Why Did The Model Do That? Debugging the Ghost in the Machine
Explainable machine learning (XAI) addresses the challenge of interpreting the decision-making processes of complex models, such as Convolutional Neural Networks (CNNs), Vision Transformers, and Large Language Models (LLMs). A primary obstacle in this field is the lack of transparency in proprietary LLMs. While tools like SHAP can be applied to these models, the process is often cost-prohibitive because it requires sending a high volume of queries to the proprietary API to generate explanations.
The technical implementation of XAI often relies on calculating the importance of specific inputs, such as pixels in an image or tokens in a text string. One common approach involves using integrated gradients, which has been successfully applied to white-box models like BERT. However, the effectiveness of these methods depends heavily on the selection of a baseline. Using a black image or zero-value tokens as a baseline can introduce bias, making the definition of a neutral baseline a critical point of debate in the field.
Practical applications for these techniques include debugging prompt injections by analyzing how a model reacts to specific patterns and potentially optimizing prompt length to save tokens. While white-box approaches are computationally feasible for open-source models, the financial and technical barriers remain high for closed-source systems. The utility of XAI ultimately depends on the specific task and the level of access available to the model's internal weights and gradients.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Ethics & Privacy and was classified suitable for intermediate domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
My planned outline for the talk is as follows:
- Intro and opening hook (4 mins): A look at a clearly biased model and why "black box" decisions fail to establish trust
The XAI Decision Tree (17 mins):
- A practical overview of the landscape and walking through the tree: Selecting the right method based on your model and data
- Mapping these methods to specific Python libraries and frameworks (e.g.,
shap,lime,captum,transformers-interpret,alibi,dalex, ...)
- Mapping these methods to specific Python libraries and frameworks (e.g.,
- A practical overview of the landscape and walking through the tree: Selecting the right method based on your model and data
Closing and Take-away (4 mins)
- Q&A and Buffer (5 mins)
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:01]
That's also hard to say, to be honest, but it's a good question. That feels, yeah. Yeah. Now, the thing is, I'm just thinking of a scenario where you could use it. I mean, so the big elephant in the room is obviously proprietary LLMs. These tools here, I did a lot of research. I did a lot of, I looked into that. I always stopped doing explainable machine learning when it was about proprietary LLMs because the only way to do it that came in my mind was actually SHAP because Swiss Army Knife could use it everywhere. But this means you have to send a lot of queries to the proprietary model, which is really cost-intensive. So you wouldn't necessarily do that unless the companies actually open up the models and give you this explanation part for free or for whatever price. So just thinking of the prompt injection, you would need to understand how the model actually behaves to certain patterns of prompt injection. And to get that, you would need to have the explainability, which is really hard to get for the reasons I just mentioned. So in theory, yes. In praxis, I would love to see it.
Speaker 2 [01:19]
Okay. Does the way how you accumulate gradient from a baseline not make a difference in the importance of pixels token? Example, if you go from top to bottom or vice versa, how do you account for that?
Speaker 1 [01:34]
That's also a good one. And I know there's a lot of debate about how do you, like I'm probably derailing for that question, but that's the thing that comes to my mind. So there's a lot of debate of what a good baseline is, and I think that's why it actually goes into that question. So having a black image as a baseline could be biased. Having zeros as tokens could be biased. How do you tell that this is a neutral baseline? Who gives you the justification for that? So, potentially I would say yes, but I mean like to wrap a chart it's important to define which one is a good baseline and what you want to use and which way you want to go.
Speaker 2 [02:21]
I think we still have time for one more question. How can we generate reasonable interpretable from deep ML models?
Speaker 1 [02:31]
That's what I just... Deep ML models, I mean, you would need to define what deep ML models is. So, for instance, the image parts that I had, you can use for... I mean, I think it should be applicable to CNNs, if not also for vision transformers, which would count for me as deep ML. Okay. But it's a matter of how do you define these terms, right?
Speaker 2 [03:00]
Okay, can shop be used for optimizing a prompt length? example for token saving for language models
Speaker 1 [03:09]
And that's an interesting thought. I mean, again, for the proprietary models, we go down this cost route that I was talking about earlier. I don't know how you would actually do that, but it would be nice to try it out.
Speaker 2 [03:28]
Okay.
Speaker 1 [03:29]
I mean, you would better understand the model probably, what it's doing, where the kind of part goes, but, and it's also a big of a question, what's the task at hand, what do you want actually to do there?
Speaker 2 [03:43]
the questions are quite a lot I think you guys can just meet her after
Speaker 1 [03:48]
That's cool. I mean, this kind of means I sparked the interest for explainable machine learning and it also shows how many people are there at this late afternoon. So very happy to have you.
Speaker 2 [03:58]
Yeah. Let's take one last one. Sorry.
Speaker 1 [04:02]
I'm on is that fun
Speaker 2 [04:05]
Okay, can sharp be used for optimizing front? I think that I did this right. Yeah. Yeah. Okay. This one is a longer one even white box approaches with MLL and seems Prohibitive expensive any built-in example in the model Xal approaches in LL amps that really works
Speaker 1 [04:31]
I'm not sure that I fully get the question.
Speaker 2 [04:33]
Even white box approaches with yellow seems prohibitively expensive. Any building example, I'm not sure if this is a question.
Speaker 1 [04:46]
Yeah, that's what I was saying. So yes, I mean, I've used integrated gradients for BERT, for instance. It was disturbed, I've used it. It's not prohibitively expensive. It's just more a matter of what is the current model at hand that you can use? In a way, how likely is it that you will have a white box, for instance, BERT sitting around or there are all the other fancy models out there. That's kind of, I think, also the big question. And the cost part definitely goes for the proprietary models.
Speaker 2 [05:22]
Thank you so much. Can we please give her a round of applause?
