XAI meets Natural Language Processing
As people tend to be more aware of AI systems and their impact, AI ethics and transparency become more and more relevant. Explainable AI (XAI) is a not-so-new term to collect methods and techniques to make predictions of AI systems more understandable. Which data points build the basis for model fitting? How is the model trained, based on which premises and assumptions? Which decisions, which parameters lead to the optimized outcome? And, most important, which model weights and decision paths result in which predictions?
Today, there are various methods to apply XAI to AI systems. But when working with text data, it is not that easy to apply known methods out of the box to NLP systems. During pre-processing and while transforming text into numbers and vectors, we often lose the human-understandable parts. During modeling, we deal with so many data points, that single weights and words lose their meaning and their importance to the human eye. Thus, we cannot simply take well-known XAI techniques and apply them without a second thought. We need to be aware of the challenges, the specialties of text data, and the possible workarounds for the NLP area.
In this talk, you will learn about local and global explanations, difficulties for modeling options and setups, useful libraries, such as SHAP or ELI5, and the importance of visual approaches. I will also show a real-world use case from my current work, with lessons learned and valuable outcomes. You will need to know the basic terms of Machine Learning and Natural Language Processing to follow the talk, but not the basics of XAI.
My goal is to encourage you to think about XAI directly from the beginning of each NLP project, as it will be central to transparency and acceptance. Also, you will learn about the right questions to ask to be sure about applications and expectations, so you can plan in terms of pre-processing, modeling, and explainability techniques.
This session took place in track Natural Language Processing and was classified suitable for some domain / none python by the speaker.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:03]
So thank you for you all being here for my talk. Welcome to my talk about XAI meets natural language processing. So this talk will be about explainable AI, about different options to use explainable AI techniques to natural language processing problems. Just a quick remark beforehand, I'm in the middle of my work about this topic so I cannot guarantee you any completeness. I'm still like trying things out and doing research so if you have any additions or remarks for me any tips I would be also be grateful and we can maybe have a chat after the talk so maybe before I start some words about me I'm Larissa I'm from Heidelberg I studied politics and data science in Mannheim before work starting to work as a data scientist in Heidelberg at Sovanta AG. In my spare time I like to do sports, play Ultimate Frisbee, play Roundnet and read a lot of science fiction, so if you want to chat about that, that's also pretty fine. And I'm working a lot with business processes and the automation of business processes, especially like in the NLP, in the natural language processing area. So recently I was involved in a customer project and that made me think about explainability in the terms of NLP, so how to explain predictions made with NLP models and systems. So maybe you have heard about that term explainable AI maybe you haven't I first want to talk briefly about what it is so explainable AI are processes and methods to understand your AI systems better so the goal is to increase the accuracy of your model because when you understand your AI systems better you can for example example, discover biases. I don't know if you were at the talk yesterday by Sonam. She really explained really good how you can see the biases in your data and in your AI system by looking into the black box. Another goal is to increase the fairness of your systems. So it also is connected to the biases and to have better outcomes of your predictions. Also, when looking into the future, it might be a legal requirement for your AI systems to be explainable. I think it's not official yet, but there are tendencies for, for example, users that they can request why, for example, they were put in a certain category. So, yeah, you might want to think about this also in the legislation way. Right now, I'm just talking about AI systems. And why is that? AI system involves the whole process that you see here. So, from the data selection, the data collection, and the setup, which model you're using, which parameters you use, over feature engineering, structure, the model that you use and the decisions it makes. This is all connected to the AI system, but I think the first four parts, they are just quite visible. I mean, you can present your data, you can visualise it, you can explain the features that you engineered, maybe it's a little bit more complex, but it's a tangible topic and And you can explain it, while models and decisions, they are more a black box. You cannot easily plot a model and explain why it comes to certain predictions or not. So in the next couple of minutes, I want to focus on the last two steps, so I won't talk about how to visualize your data, but I want to keep you in mind that this is also important and it's also part of explainability when you start thinking about that term. Why even bother about explainability? I already gave some ideas about that. There are a lot of reasons why explainability is important. It's not only for the end users who want to know why certain predictions are made. It's also for us as technical staff to know how our models behave to maybe make them better to improve the outcomes but also depending on your role for for example if you're a manager and you have to um yeah you have to to speak about why you're using ai and why you're using a certain model and a certain system it's always good to know what's happening under the hood and what yeah how the decisions are made This is especially important to think about explainability in NLP because the models or the AI systems, they solve tasks with text and text is closely connected to human beings. When you have NLP models, most of the time they work closely together with human co-workers and they need to trust and they need to understand the predictions of the model. so especially in NLP it's important to think about explainability and one last remark explainability is also closely connected to AI ethics there are a lot of great videos a lot of great talks about that topic it's also important to see those two topics together but when I start talking about that we will probably be a little bit over time so I want to present you some approaches some options how to deal with explainability in NLP projects you can roughly divide the the approaches into four sections so on the one hand you have model agnostic and model specific approaches model agnostic approaches are independent of of the model that you are using. So, it is a really good point. If you have a setup, you have already introduced explainability, you can just easily switch the model behind it and leave your setup as it is because it is model agnostic. You can just keep working with your explainable setup and maybe improve the model. Do not think about, oh, I need to keep with the decision tree because I already built my whole system on the values of the decision tree. Then, on the other hand, you have the decision between global and local explainability, where global explains the impact for the whole dataset, and local explains the impact only for one prediction. You have here, for example, to the image of the decision tree. When you have global explainability, you take the whole decision tree into account, so you can explain each prediction that is made from that tree. On the right-hand side, you have only one path of the decision tree, so you can use this explainability only to explain this single prediction that was made. This is like the distinction between global and local explainability. So, based on the experience I made in this customer project with explainability and the different approaches that we tried, I want to just show you those approaches and maybe give some feeling if this approach could fit also for your kind of problems. So the first idea, I don't know if you ever thought about it, but you can use human annotations. If you have a dataset and it gets labeled by humans, you can just ask them and include reasons why they added the label. I think it's really cool if you have the chance to do that. You can use then the reason as another label and let it be predicted by your model. But I mean in the end it's not explainability how the model came to the conclusion that this label is right, but what the model thinks that the human co-worker thought why this label was right. But I just think it's a cool idea, it's a cool way to think about explainability and If you know that option, it also makes you think about which kind of explainability do I want in my process, in my AI system. Another option is to use feature importances and model structures that you just have from your model. For example, there is BERT. I do not know if you are familiar with BERT. There are also a lot of great talks about that topic. It is a very common NLP model type, and it is trained in different layers, and you can use the layers, send data through it, and observe the attention of those layers to certain words. So for example, this is the third layer of a bird model, and the color of the words indicate how much you can change the word without changing the sentence. So a very dark color means there is only a little space you can move the word, a lighter color is you can just change the word and the sentence would keep its meaning. For example, if you would change bird, it would change the meaning of the sentence quite a lot, but for example, to or has, like those auxiliary verbs or auxiliary words, they are not that important. You can just use the structure that is already there to get a glimpse on how your model comes to conclusions. But it depends. If you're not using BERT or if you're not using a model that allows you to do this, this is not the right option for you. Then third, you could always use understandable models. This is quite difficult, I think, for NLP topics, but I don't know if you've ever seen NLP data. You have text. often you transform text into numbers, into word embeddings, into vectors, so it's really difficult. You can train decision trees on NLP data, on embeddings, but in the end, even though you can understand the model and the tree structure, it wouldn't help you. and then see, okay, the third value in my vector has a high impact, so okay, but what does that mean? But I don't know. It could be that you have NLP data that it's really good for decision trees. It's an option you can try, or also for regression or other interpretable or understandable models. It really depends on the data if this will work. Another option are LIME, SHAP and LI5. I just heard that yesterday there was a great talk about LIME also, so I won't go into too much detail here, so maybe just then ask the expert about that. But in short, LIME and SHAP, they work with correlations. So you have your model, which is a black box, and it does not depend on which model it is used, so it is a model agnostic approach. You use this black box, and you put data through it. You take your data, you permutate your data points, and feed it into your model, observe the prediction, and then correlate how the permutation and the change prediction correlates. So I think it's a really cool approach. On the top you see an example how this works with images. So you have an image, you have the prediction this is a frog and then you mask or you permutate single parts of the picture and you observe how the prediction changes. So in this case the first picture it still has as like 85% to be a frog, and then you just take those results, put it into an understandable model, into an explainable model, and observe how the weights behave, and we can now see in this frog example that the face is a really important indicator for this model that the picture shows a frog. I think that's really cool. The problem is that LIME, in this case, only works for local explainability, so you can just explain for this single example, for this picture, the importance of the face to be important. Then there are SHAP values. SHAP values, they can be local, but also used for global explainability. You can see here an example with text, so Shep values show how much single elements, for example, in a sentence, contribute to the overall prediction. You can imagine it just like a football team. Each player has his or her own impact on the game, and the outcome is like the team performance. The differences between Lime and Shep, I have a little example I want to explain now. So imagine you have a classification if a YouTube comment is spam or no spam. In this case, you have the data point for a Christmas song visit my channel. I think it's spam. Then to try out or to test your model, you take this sentence, you just separate it into words, into tokens, and then you mask the tokens. For example, when there is a one, the word is there. When there is a zero, the word is masked, so taken out. have the probability that the model says okay this is still spam or this is no spam so for example the third and the fourth option were just for visit my channel is still with a high probability it's still spam and then you have the differences between lime and Shep and this is in the weighting of this probability in their calculations. So lime is just weighting the probability with how close is my permutation to the original sentence and for a shab it is how many coalitions do I have? Oh I see I have a typo there. So how many coalitions do I find? Do I have a high amount of ones or a high amount of zeros because the theory is you can single out single impacts so if I have for example everything masked except for channel I still have a high probability and this is just quite cool because I see that the word channel has a high probability for my model to classify something as spam and this is also very high weighted by Shep and this is also why you can use Shep for global explainability and not just only for local. Then there is LI5, I don't know if you knew, but it stands for explain like I'm five, and I think it's just also a cool method to use, it kind of works in a similar way, it shows the contribution a word has to a certain prediction, and it's nice that it works out of the box for the scikit-learn pipelines, so it's not completely model agnostic, but you can use it for different scikit-learn algorithms, and it also takes the preprocessing into account, So it really works well if you have the setup that is required here. Now my last approach that I want to present are graphical representations. I think they are really good, especially for explaining what NLP is and how NLP works. You can use, for example, if you have word embeddings, word vectors into use, you can use, for example, a dimensional reduction to bring your word vectors into a two-dimensional shape. And then you can just see where the points are located on your data grid. For example, I don't know if you know, this one is VisBird. It's a Q&A website, so you can ask a bird model a question and the question gets answered and you can observe how the layers move or push the points on this grid around, so how the answer gets shaped by the different layers. I think it is really cool to visualize how this model works. But on the other hand, you have always a dimensional reduction. You come from 300, 400 dimensions, and you reduce it to two dimensions. I mean, some information gets lost there. But I really think it's a cool approach and it's really tangible to see, okay, I have words that are closely connected and I have differences between others. So these were my five approaches that I wanted to show you today. I also included a little table where you have maybe some of my personal evaluation. There are also great articles in the web that also do similar things. In case you want to keep those links, I will upload the slides in my GitHub, so you don't have to manually type them in now. to sum everything up, what are difficulties for NLP models? You have the high amount of dimensions, and that is clearly a difficulty. You have a high variability in models and representations. It is not like numerical data, which you can just show in a numerical way. You have a bag of words. You have maybe a skip-gram model. you have biograms, you have embeddings, word vectors, so it really depends on your use case what explainability you can use. What I find pretty difficult is a set of tokens is not equal to variables, so a set of tokens is more like image data because you have always to keep the context into account, and it's not just like numbers that appear together. Then you have this switch from local to global explainability which I think is also the transfer is also quite difficult and because of the size and the dimension and the complexity of word vectors it's also difficult to show this in a graphical way. I mean for numbers you can just plot them, you can have a histogram, you can have curves but for word vectors, for example, I mean, you can print them out, but then you have a dictionary-sized printout, and nobody knows what to do with it. And the last difficulty I see here, and this was especially in my customer project important, even absent words are important. So, if, for example, in a comment, you have a word that is not used, it is also just important for the prediction that some words are not used. And how do you show this on a grid? I mean, you don't have the data point there for this word. How do you show this in a table where you have the impacts? You have a lot of words that are not there, but also have an impact. So this was one point that I found pretty difficult to explain and also to show. So, just a quick wrap up about the customer project, I won't go into detail here, just this was how we explained it to the customer, we had text training data, we made some feature engineering so we did word complexity measurements, sentence length, word variability measurements, And then we had embeddings, so in this case, a custom TF-IDF embedding mechanism, and we combined it into a model. And in the end, we had a POC dashboard where the customer could see for a single prediction how this prediction was done. So we had on the top the binary prediction in class R or class B, and then we had aggregation of SHAP values depending on the location in the text. So we knew, okay, our text is structured in a certain way, we have four different paragraphs, and we can just aggregate the SHAP values within the paragraphs. This made it very tangible for the customer because they could see, okay, the paragraph four had a big impact into this prediction. And then we just had the engineered features also as SHAP values separated from the normal TF-IDF. And to wrap everything up, my lessons learned from this project, you have to really think about how to explain the explainability. So it's not enough to just calculate the values. You have to explain to the customer what these values mean. And this is, I think, a greater challenge than just calculate them. How do you present it? How do you make it understandable? Also the absence of words. How do you display them? Do you even talk about it? I mean, do Do you want to open that box to explain how these models work? You have to find a fitting representation. You have to include help texts for the user to understand it, and you really have to reduce complexity to the topic that the user really wants to know about this prediction. So, I also included some of the articles that I used for this presentation, also for research. There are really great, great articles out there. Just have a look at them. And if you want to keep in contact, you can always chat with me on Twitter or on LinkedIn, and you find the slides on GitHub. So thank you for your attention.
Speaker 2 [26:10]
you for your amazing talk that was so interesting I'm pretty new on that topic and I feel like I've just learned a lot and we have
Speaker 1 [26:17]
A lot of questions. Oh, yeah. So I guess...
Speaker 2 [26:20]
So I guess I was not alone in enjoying this. And we probably won't get through them all, but hopefully you don't mind folks to come up to you afterwards.
Speaker 1 [26:29]
Yes, yes, of course.
Speaker 2 [26:29]
Yes, of course. course. But let's try address some of the top voted ones. So Alina is asking the examples you showed are all document classification problems. Have you explored the approaches for other NLP tasks ie translation, named entity recognition?
Speaker 1 [26:49]
So for translation, I unfortunately have no experience. For NER, I think you can just use also the model agnostic approaches because you just take the model as a black box and wrap the explainer around and it doesn't really matter what's inside. So I think you can just use the approaches and try it out.
Speaker 2 [27:14]
So maybe Alina, who asked the question, can also try that out and let us know. Okay, so I didn't get the difference between local and global. Could you please explain it to me like I'm five?
Speaker 1 [27:30]
Yes, I can try. So imagine you have a lot of people, like you see here in this picture. You have a group of people, and you have a model that predicts the first letter of your name. And when you have global explainability, you can just explain the prediction from your model for each of the people in your group. When you have a local explainability, you have to pick one of them out, and you can explain why the model predicted a name with an A at the beginning, but you don't know about the others. So this is just a difference. And global explainability usually is a lot more costly, so you have to think about, do I really need data from all of the data points, Or is it enough that I can just explain single predictions and do not need information about the whole data set?
Speaker 2 [28:31]
That's a great explanation, I would say. Hopefully it also answered the person that asked it. If not, maybe they can come find you.
Speaker 1 [28:37]
Maybe five years was not the first of it.
Speaker 2 [28:41]
A smart five-year-old.
Speaker 1 [28:42]
And it.
Speaker 2 [28:43]
If you ask your human annotators for their label explanations, how can you encode that data and use it within the XAI algorithm?
Speaker 1 [28:55]
So you can for example, I don't know I come from the social sciences and when we label text or when we have have a data set we in the beginning we create a Rule set how to label data and you could do just the same so you could for example have a set of categories like 10 categories and the human annotator could select one of them as a reason for example, so you have a a fixed set of options and then you could use it as a label to be predicted. Or they can just maybe then they also can single out words that they used for their labels. So for example in the spam example you can just highlight the channel as a important word and you can just use this later on.
Speaker 2 [29:49]
Yeah, it's very interesting suggestions. Okay, so let's say we find out that channel is important for classifying something, as you mentioned the channel, as spam. To what extent is that a fundamental explanation of the model?
Speaker 1 [30:08]
I mean, it's not an explanation like how the model did this because this is just, I mean, the model is probably a mathematical construct and you could also just print some of the weights in your neural net, but this won't help you as a human to understand. And if you have the importance of the word, it's always a trade-off between what gives the model you as information and what does a human need, a human being need to understand it. So, I think most of us need some text, some context to understand values and not just can see, okay, I have here a vector of 300 values and now I see, okay, this one was the most important one. So, you always need, there's always a trade-off between accuracy of the model prediction or the explanation of the model prediction and the understandability for a human being.
Speaker 2 [31:12]
Yeah, great answer. Thank you. Okay, can you use the graphical methods on the whole texts somehow, not only word vectors?
Speaker 1 [31:26]
I mean, there are word clouds, but I'm not a great fan of word clouds. I'm not sure what this question is referring to. I mean, usually you can count words and just have a histogram. Maybe I'm not the right person to ask. Maybe we can ask a linguist. I mean, they have great ways to visualize text in their text form, not in a word vector form. But just have a look at that. I think they have great ideas there.
Speaker 2 [32:01]
Thank you again. Okay, so we're running a bit over, so I'm not going to go through the rest of the questions, but would really invite you to reach out to Larissa, and you already shared your information on how folks can do so if they can't find you in the conference today. Just one last one. It's not a question, but I wanted to read it out anyway, which was great talk, and I 100% agree.
