Simulating the World using SimPy: A practical Example
Explainable AI (XAI) addresses the "black box" problem in machine learning, where models may rely on spurious correlations—such as identifying a wolf based on snow in the background rather than the animal's features—to make predictions. To mitigate this, a structured framework distinguishes between interpretability (direct reading of model logic) and explainability (approximations of model behavior). The framework further categorizes analysis by model access (white box vs. black box), scope (global vs. local), and data type (tabular, image, and text).
For tabular data, global explainability focuses on feature ranking, effects, and interactions. SHAP (Shapley Additive Explanations) uses game theory to rank feature importance and visualize how specific values push predictions higher or lower via beeswarm plots. Feature effects are analyzed using Partial Dependence Plots (PDP) for average effects or Accumulated Local Effects (ALE) plots to handle correlated features. Local explainability identifies why a specific prediction occurred using SHAP waterfall plots or LIME (Local Interpretable Model-agnostic Explanations), which fits a local meta-model to the input. Counterfactual analysis, implemented via the DiCE package, functions as a "GPS navigator" to determine the minimum input changes required to achieve a different target outcome.
For image data, global analysis employs Testing with Concept Activation Vectors (TCAV) to determine if human-defined concepts, such as stripes, influence predictions using directional derivatives. Additionally, feature visualization reveals what a model learns at different layers, progressing from simple patterns to complex structures and specific objects.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track PyData & Scientific Libraries Stack.
Submission
The proposal as submitted by the speaker before the conference.
Real-world systems are often too complex to test reliably, in the same environment and under the same conditions. Changes are hard to measure, edge cases are difficult to reproduce, and external influences can hide the real behavior of a system. Simulation offers a way to abstract from reality while staying close enough to produce meaningful results. It allows full control over system components, timing, and disruptions, and makes it possible to test many scenarios in a repeatable way.
The practical example of this talk focuses on simulating load-balancing algorithms. Load-balancers are a good example of systems that are hard to evaluate in real environments. Some tested algorithms have no existing implementation, others differ across platforms, and cloud environments introduce many uncontrollable factors such as network latency, cloud noise, and reoccurring background workloads. These factors make fair and consistent testing almost impossible.
The problem is addressed by building an event-based simulation using SimPy. The session explains how SimPy works by using Generators to create the events, and how time and processes interact inside a simulation. An architecture for a practical example for a load-balancer simulation is presented, showing how different components interact and how algorithms can be swapped and compared.
The talk also covers improvements made to the simulation, including a command-line interface for easier execution and a YAML configuration file for flexible setup. It concludes with practical tips and lessons learned when working with SimPy, helping to avoid common pitfalls and improve simulation design.
Overall, the session provides an introduction to simulation as a testing tool, a hands-on example using SimPy, and a realistic architecture for building and evolving simulations in Python.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [10:57]
Good afternoon, everyone, and welcome to this session. Just a few housekeeping rules. Please put your phone on silence. Do not change rooms during the session. And don't forget to post all your talks, I mean, all your questions on talks. Our speaker is Kasima. Yes, that's amazing. Okay, please give her a warm welcome, please. Thank you. Thank you so much also from my side for joining the last session of today. We'll talk a bit about explainable machine learning and try to see whether we can debug the ghost in the machine. Before we get started, a few words about me. I started my career studying civil wars, trying to understand why conflicts break out again. And currently I'm working as a data scientist, and what I'm doing right now in my professional career is not so much different from my academic times. I still want to understand the factors that are driving the model outcome and whether we can trust what the model is actually telling us. And my hope for you to get away from this talk today are two things. First, to understand why it's important and B, to also get the tools to do this. And without any further ado, let's dive in and tackle the first thing. So why is it essential to actually understand what the model is doing? So looking at this image, and imagine you trained a classifier, and the classifier was trained on predicting whether it's a husky or a wolf. You give the classifier this image, and what does it say? What would you say? Husky, right? but it actually said wolf. And it kind of sounds reasonable in a way because they aren't so far away from each other. They look pretty similar. It's kind of this wolfish shape and everything and if you weren't an expert probably this could still count. But it only looks reasonable until you look at why the model actually made the prediction. And that's what we're seeing here. It's the gray part is what the model wasn't looking at when making the prediction. The white part. So the snow in the foreground or background is actually what drove the prediction of being a wolf instead of a husky. And it's kind of funny in this context or funny in a research paper where you figure this out, but it's not so funny when you look at it that this was actually a shortcut that the model was learning. It wasn't really learning the concept of the animal. And it's not funny when you think about decisions being driven for loans, for instance, or medical diagnoses, or hiring processes, or these things. And now the big kind of elephant in the room, how do you figure that out? How do you get from the classification to understanding what the model actually did? And if we look at it, most processes will go like that. You train a model, it looks good at first, maybe you have a hunch that something is a bit off, maybe not, you just want to double-check, which is always good. Then you go on Google, explainable machine learning or something similar, you find tons of libraries, frameworks, approaches, and obviously you're a bit lost. And I want to give you as part of this talk a reusable framework that helps you navigate the decision at hand and quick sneak peek there will be also something that you can really use click and drop at the end. So that's the plan for today. What do I mean when I think of a reusable framework? For me it's a structured decision tree that comes in naturally and we'll walk through that tree step by step, branch by branch until we reach each leaf, and in the leaf you will get something like a concept card where you have the concept on top, a short description, and then also potential Python libraries. And I also highlighted those that are still actively maintained and those that are maybe a bit outdated but still worth looking at. So you'll get a whole bunch of information today, be prepared. before we dive in, a few key distinctions here. First, when we talk about explainability, there is also interpretability, which is actually a tongue twister. Interpretability is about reading the model's logic directly, whereas explainability is a proxy, it's an approximation, a separate method that tries to understand what the model was doing. Then we obviously have black box versus white box models. Black box models is something that you can only query, you don't have access to its internals, to its gradients or weights, whereas with the white box model you have this kind of full access. And then last but not least there's also global versus local and with global we are looking at the overall model behavior and with local obviously at single predictions. So why did this model in this specific case go this and that way? And now let's dive in and build that tree together. Once we get started, you obviously ask yourself, what do you want to explain? And I naturally start with the data type. And for the data type, we have three different data types that we're talking about today. The first one is tabular, then we have image and also text. And we'll start with tabular first. With the tabular we have a really nice thing. Before thinking about any explainable AI tool that you could use, ask yourself whether the model is interpretable. So can you actually use the model itself or its internals itself to understand what was going on? And if you're using linear regression or some simple decision trees or GUM or Explainable boosting machine chances are high that you can stop here. You're all good and you can leave the room in case not I Have something for you and we go on with the not interpretable model and we go for the scope here With the scope as I said before you have global versus local. We'll start with global first We'll always move from left to right. Start with global first and look at what are the insides. I'm always asking myself three different questions. The first one is about ranking. It's about which feature matters most. The second one is about how does a feature affect predictions. So it's about feature effects. And the last one is about interactions. So how do features work together? and we'll start again left first with the ranking with the ranking there are different methods i'm presenting you for all of them kind of the what i think is the main or the most important method because otherwise we would be sitting here for hours so for the ranking which feature matters most we're meeting someone that will meet over and over again throughout the talk it's shop and And I'm sure many of you have heard of it. It's based on a game theoretical concept, the Shapley values. And what it does, it calculates the contribution of a feature by testing how the model's prediction changes when that feature is added to every possible combination of other features. Super technical, I know. But what it actually does and what you get out of it is it ranks features by importance. And it tells you how each feature pushes predictions higher or lower. And you can see that here it's a summary plot in the form of a B-sform plot. And I'll use a bit of time to explain that because we will see similar concepts throughout the talk. So what you have on the x-axis is actually the SHAP value, which goes from negative to positive, and it's kind of the direction that the feature is pushing the model's prediction. On the y-axis, you have the single features, and remember, for SHAP values, those features are ranked, so the one on top is the most important feature, and then you also have the color, which is the feature value, where you see whether the feature is higher or lower. And for this specific example here, we're trying to make the prediction whether someone earns more or less than 50k, and what we see here is age is driving the model apparently the most, or it has the highest importance, and there it is that lower age values, so people being younger, tend to earn less than 50k, which kind of makes sense, but I think it's really nice to see it in such a simple plot if you want, and then obviously you could go on and also look at the other features which we're not doing for time's sake. When you are in the global, we're still in the global world, when you're here and you want to look at the feature effects, there are two different approaches to do that. So feature effects is all about how does a feature affect predictions. The first one that often comes to mind is the PDP, the partial dependency, dependence plot, and it's all about understanding how a feature affects the model prediction on average across the data set. So it averages predictions across all observed values of all other features. And what we see here is you can pick some singular features. We see here temperature and we then see the nonlinear relationship of temperatures and bikes. Bikes being out there, bikes being rent, something like that. This is really neat and it's really nice but it's only nice as long as you stay in an uncorrelated world because it averages across the entire data set correlation is an issue and this is when people typically rely on ale plot it's accumulated local effects and here the the cool part is that it only looks at the data points where the feature value is actually close to your feature or to the X, and then accumulates the local difference in predictions. And this is what an AL plot could look like. They tend to look similar. It's often the case, but it's a different underlying logic underneath how they are calculated. So that's about feature effects. Now we're moving on to interactions. When we're talking about interactions, we're going back to SHAP because SHAP has something called SHAP interaction values. And these SHAP interaction values decompose the predictions into the main effects and the pairwise feature interactions, and what you can get out of it is something like that. And what you see here is again the making more or less than 50k, I believe. And what you have here is you see that this time on the y-axis it's the SHAP value, on the x-axis it's the age, and you see that there is a sex-based gap between being female yes-no that varies by age. Just in simple example, I'm sure you have tons of other interaction examples in your head that you could visualize in a similar way. And this is again, it's a global effect. It's for the entire model behavior. And with that, we're leaving the world of tabular global explanation and move on to the local part. And for the local part, we have two fundamentally different questions. The first one is, why this? So why this prediction? And this is something that will follow us also in the unstructured world. We also have something that's called what if. and the what is, if, is what would need to change and it's more like a philosophical question if you want a hypothetical but we'll be there in a second. So why this? Starting with that one, why this prediction? We have two options again and as I promised, SHAP will be our loyal companion throughout this deck here. We can use the SHAP waterfall plot or sometimes also called breakdown plot. And it shows how each feature pushes a single prediction up or down. So it's about a single prediction. It's not about the entire model behavior. And what you have here, that's what a waterfall plot looks like. When I want to interpret it, I start at the very bottom, which is the value that you see here. That's the average of the entire data set. And then I slowly go up and see how each feature pushes this single prediction until we reach the outcome of this single prediction, which is up here. And again, for SHAP, this value here is ranked. So met income, median income, has the highest impact on this prediction. And it's a really clean and neat way of interpreting a model, I think. And then you also have something that comes with LIME. And it looks pretty similar, but the underlying logic is a bit different. Where the SHAP is really high computational intense because you do a lot of different calculations and combine different things, What you have here is you basically change, slightly change parts of your features, slightly change parts of the input and then fit a local meta model and see how it differs. And with that you get to a pretty similar plot as the one on the left, but it's a different underlying math. And with that, we're now going to the what-if world. With the what-if, it's all about what would change. That's the fundamental question. It's a lot about counterfactuals, and that's why I say it's more like a philosophical hypothetical world here. So counterfactuals, they don't tell you what needs to change in the input. They don't tell you the why, but they tell you what needs to change in the input to achieve a different outcome. And here I'm taking an example from the DICE package because I think it nicely summarizes what it does. So when I'm thinking of these counterfactuals, I'm thinking of having a GPS navigation system. So I'm starting with my current location. This is here. This is my original input. And I want to know how do I get to my target. And my target is the change in the target, which is actually earning more than 50K. And then DICE calculates counterfactuals that try to optimize the way to get there. And this, try to optimize the way to get there. And this is why I'm thinking of it as a GPS navigator, right? It brings me from A to B in the most optimal way. And this is actually what DICE does. it does, it gives you based on your data, based on how your meta, like the meta information of your original data set, how they are distributed, it gives you meaningful changes, and that's what you see in the yellow highlighted parts there. And with that, we're leaving the tabular world, which has been the largest, and move on to image and text. There we follow similar patterns. We first look at scope and then at model access for each of them. Starting off with image, we look at global first. For global explanations, we have two powerful approaches. The first one is called testing with concept activation vectors, or TCAV, and it tests whether a human-defined concept, like stripes of a zebra, for instance, influences a model's prediction using directional derivatives. And what you actually do is you have your model, or a pre-trained model, or something like that, and you teach it these human-made or human-defined concepts of stripes and non-stripes, and then see what's the actual activation effect once you show the model these zebra images. That's the basic logic behind that. Again, we're on the global world, so we're looking at the global per se and not single predictions, right? Another concept that is also quite powerful is about future visualization. And what you do there is you basically see what the model has learned to detect at each and every layer. and I'll quickly flip through it because I think the images tell more than thousand words and you will see images of different layers and you will see that the model at specific layers learned more like patterns, at others it's more like, I don't know, like structures, and at others it's more like hand, foot, eye, whatever. Well, you see that in the second more patterns, more structures, and here hand, foot, and eye. Something like that. And that's really like you can see what the model learned at those layers. And with that, the global world of the image is done, and we can move to the local world, where we see which kind of...
