Inpsect and try to interpret your scikit-learn machine-learning models

Speaker 1 [00:04]

Thanks, so I'm a bit rusty, it's a long time I didn't do a training with many people so like after three years so maybe we can, you have to be gentle with me, but you have the right to ask as many questions as you want, I mean the purpose is that it is kind of so interactive and that you get outside that actually you learn something. So the way that I will do will be that I will give you a bit of information where basically to get the materials if you want to run those experiments on your laptop. And I will just make then a 10 to 15-minute presentation just to see the concepts that I will present afterwards, and then I will just directly go inside Gitlun to say, okay, what are we going to see concretely, okay? I put on Discord a link in the room A1, no, A05, A06, yes, to a GitHub repository. So you can go to my account and then you will find that the latest updated repository will be this one. And there I have two things, so I have in two formats, I have some notebooks, so there four notebooks for topics that we will see and I also have them in format of a Python script directly because actually I'm using VS code is really with GP text where you have markers and that you have the interactive windows so like this you can choose one of the overs you are free to do so here I will use the notebook because it will be easier to see and and to manage with the font and And I put a file with the requirement that library that we'll need. So just for infos, we will need scikit-learn, pandas, seaborn, jupyter for running the notebook if you need them, and then shapp for the last notebooks. So no more than that. Okay. That would be enough. And I don't know if that would work for the last because I just checked now. I created a binder normally that would install those things. I would advise you to do this on your own laptop instead of binder because it's a bit slow but in case that you have some trouble somehow with your machine, you might launch a binder there. You might have some difficulty with SHAP because I just saw today that the version of NumPy that is installed is 122, and for Numba, you will need 121. So you will need to downgrade at the moment that you play with the number of SHAP if you use Binder. OK. So before to go in the first notebook, I will take a couple of slides, which so don't worry, this is not that many. They are all the same things that I repeat. So they come from, like, I gave a talk last week in Paris. I didn't do any tutorial, but basically I thought that that could be interesting to get like the couple of slides that I have at the beginning so that we speak about the same topics here. So I'm not in France, so probably you never read the book which is on the right, which is called Les Limites à la Croissance in French. But in English, it's a famous book which is called Limits to Growth. Anybody heard about that? One person? Two? You should be more, because it's a very, very interesting book. So maybe for those persons, do you know what's that here, the spaghetti plot on the right? No idea? So this is actually the world. So this is a model, of course. This is a statistical model that represents how, like for those people that wrote the books, they say, OK, what we need, we need to have a statistical model to actually understand some of the dynamics of the system. And instead of, because the world is very complicated, what we should do is simplify it in some way, with some assumption whatsoever. So we take some hypotheses. We create a model. So they created that type of model. And then from that, they say, OK, I have my model f. I have some input x that I can define. Then I want to understand what will be the trend in the next 20 years, 60 years, et cetera. So I created those people, what they did. If you want, you can play. You have a Python library that does that. But the book was written in the 70s. And they were really interesting about two major things. Understand the dynamic of different inputs and how they will interact between each other. And then actually create several scenarios to know, for instance, until when can we pollute the planet, until that this is growing up, or until it is sustainable, how much the population can grow, until that it will not affect another variable, et cetera. So they are using it in a very different way than what I think we will do afterwards. But it's still a statistical model, they don't use it for making predictive modelling like we usually do in machine learning. Instead, what they do is just create different scenarios that are not interesting about being the most reliable possible the next day, to predict what will be the pollution tomorrow. What they want is to have a resolution of 10, 20 years, and see the different scenario and see what can happen. And for instance, this book, when it came out, was quite famous in the 70s because it's something that said, if we continue on the way that we are, so business as usual, because we have limited resources and because the planet is not like infinite resources, we are going to be in trouble in 2020. So here we go. So now, since the 70s, we know something like this with that modelization. We have maybe to change something. So this is just to show that actually what we call a statistical model could be considered as a black box, but actually there's kind of some parameters inside. And usually what we're interested in is to understand what those things are doing. We are not only interested in predicting the output, but also what we have inside this model and how it's behaving. So what we'll see today is more a problem of predictive modelling and not like trying to make experiments and scenarios, et cetera. Our task is a bit different in that we give us at hand some data, and our goal in that case is if I have a new data point, I want a model that predicts with the least error possible what is the output of this new data point. So if I would be in the regression on the left, so this is some data about some penguins, So if I give a penguin that I measure the flipper length, I would be interested to basically give his body mass, but as close as possible of what it is in reality, OK? If I have in classifications, the penguin could be a species, and then I could have a different model that will, given like human length and human depth, that will give me basically the species of the penguin, OK? However, when you have these things, it comes back as well to the model part. So the modeling part is you can choose a complex model or simple models. It's up to you when you do your design. The thing is, when you do a simple model, usually this is easier to understand how it behaves. If you have a difficult model, like a very complex model, it's much more complicated to understand what happens inside. And all the talk of today will be to show the couple of tools that we have in scikit-learn that, given predictive models, you can try to understand it. And more importantly, what I will just present is all the traps that you can have, depending on the type of models, the thing that you should be aware to not lie to yourself when you try to inspect it. So because of the classification of where I will come, I will just present three, let's say, classification of the method that we use to actually explain the models. So first, you have two types of methods. One that is model specific. So if you have linear models, you can only apply that method to this specific type of model. So three models. You have model agnostic. So it doesn't matter what you have. You will just take the prediction function of your models and try to explain, for instance, What is the prediction? How did it come to this answer? So you have model-specific versus model-agnostic. The second thing is that you can have some explanations which you try to explain a single data point, and why did you take a decision for that data point. So for instance, you have a client that came to ask you a loan. You didn't give him the loan. Basically, he has the right to ask you why you give me this loan, and then you need to come with an explanation for that particular customer. Otherwise, you have some global explanation, which is for my full data set, for my pool of people, what is more the attributes that have an influence, basically, on why did I took that decision for this bunch of people? So this is global versus local. And finally, it's something that will go a bit more into details. So usually, we never look at these programs in details. But it is also, what does your explainability method is actually evaluating? Because it can evaluate several things. It can evaluate, actually, the function f. So your model itself, given a data point, and then it will make an expression with this. But actually, you never use the true targets, the true predictions. So that would be, for instance, SHAP does take a model F, take an example X, and tell you it's changing compared to the average predictions. It's changing in these manners is how I decompose the things. But never at any point you say, hey, the true prediction of that point is actually this one. Is it a good prediction or not? Actually, we never look at it. So that's why you have the over part here, where you don't give a single data point. So you don't give only the x, but you provide as well the true y. And for instance, you will measure things like the decrease in accuracy that you have when you make some changes in your models, and is your model behaving properly or not. So you take this dimension into account. So usually it's like, do you explain only the model, or do you explain the model plus the data together. And here, just on this picture, on the right, Sage is a model that does a global explanation. On the left is Sharp that does a local explanation. That's why here you have several X and several Y, and here you have only a single X. That's just for why I have this image here. However, you should be careful about two aspects. And that's my two concluding slides. But they are super important. So first is you can explain whatever the type of model that you fitted or that you are using. If you have a very bad model in production, for instance, you can still explain what it gives you. Because you can inspect it. You can check what are the parameters inside, trying to understand. But there's nothing that stops you to say, actually, my model was bullshit from the beginning because it was just randomly giving me predictions. And the only way that you need to sometimes ensure is you need to evaluate your model before to actually try to make explainability. Because what you have to understand is that you explain the model and maybe a bit of the data, but most of the time, you have to make the evaluations beforehand. So this family here on the right, it really might help you. But you have only very few methods that actually does this use through Y. And the second thing is that an association is not a causality. So that's maybe the most important thing. Whatever will come today, I will not tell you. I mean, really, we'll get some weight to some features, saying my colon here is very important in some way to explain my y. And as a human, we just always come back to say, oh, so it means that x just causes y. And then this is completely wrong. So here I give a simple example that comes from some slides from a tutorial from somebody from Twitter, actually, which is really interesting. So if we look at the distribution of the data of those three different Python code that generated those data, the distribution is actually the same. So if I look at the joint distribution, this is exactly the same thing. However, if I look at the code here, I created x. And then from x, I created y. So x causes y. Here I did it in another way. I created y. And y just created x. And then here, in another way, I just created a variable z that has a causes on x and on y. So in terms of causal graph, you have something like this, so x have a cause on y, y have a cause on x, and z have a cause on x and y. But at the end, all of those have the same distributions. And you have to think that here, if y is a variable that I want to regress given x, so if I write a bit of cyclical encode, I will get this, okay? Linear regression that fits x, y, then predict x. I will get this fit, okay? I will get a regression line that will be the same for the three data and I don't have anything that told me what is causing X or what is causing Y. Okay? I don't have any causality there. So it's just to give you a sense that always be careful in the interpretations because you don't...apart if you do a real causal inference model and interpretation then yes you can do that. But whereas here we will never speak about causations. Okay? Is there any Any questions? Because now we'll play with the slide, with notebooks. There was one question from earlier, how's the performance of the world-free model? The performance. So as I said, on the world-free models, they don't try to, they don't measure the accuracy of the model. They do some scenarios of what would be with the different value, what do you get. What is shown is that the model in the 70s that they updated in 2000, with the variables that we could observe updated now, where it will go, at the 10 years, let's say, like 10 years visibility, they were very, very close to what actually happened. But here we are not speaking about if the population was exactly at the same levels, could have been with a couple of thousand people away. So we still have a big difference. But the trend was right. So they have already, like, during 30 years, they look at all the different trades and confirming what we were envisaging 30 years earlier. So in that regard, the trends are correct. But then we are not at the close. We don't try to make the right prediction at the given year, OK? So if it's fine, then we can start and play with the notebooks, OK. So either if you are a Git expert, Git users, you can clone the repository, otherwise be aware that in GitHub you always have a download zip. So you can just download the zip and zip on your laptop and then you will be fine. Then for me, I will need a terminal. I have locally the course on my desktop, I think. And I will launch basically the notebooks here just to have the right on your single screen development things. But be aware that you can use VS Code in any case as well. So if you go in notebooks, so today I will present, first, I will present three notebooks that will use scikit-learn. Because so in scikit-learn, if I retake is the different classification that we show. In scikit-learn, we do only global explanation. We don't do any local explanations. So I will just present the different thing that does global explanation. And I will show, first, something that is model-specific, that has a linear model. Usually we, I mean, if you have a program that you can start with logistic regressions, why not using a logistic regression? This is a simple model. And then we tell you, usually, just look at the coefficients, and then you will understand which feature just have an impact on UI. Bigger it is, better it is. So here, I will as well present some pitfalls that you need to be aware with those coefficients, because there's some corner cases that you have to be careful. And then I will present two different methods, which are more about model agnostic, so you can use them with any type of models. One that will do what we call partial dependence. And we'll show you one feature with your target and how they are linked. And another one with permutation importance. And this permutation importance is the same as to basically any model you can check the original variable that you how important they are by doing permutations of the feature. So I will come into details. And I will contrast it with something that we have from a long time ago in Scikit-learn, that are usually why people were using random forest at that time. So then we have like an attribute that is called feature importances, that I will just explain why you can use it, but under some circumstances, and what are the pitfalls as well of using it, okay? That you should be aware. Don't worry, I got a PhD because of that, so you can still... And finally, I will get into SHAP, so depending on how much time, but basically I have a basic... We start to look at this library in Scikit-learn. Again, to comment what are the pitfalls. So I have a couple of them, and basically we can discuss about that, like, around. So this is the first time that I do the notebook, so there is probably, like, some other things, but if you have any question that would be interesting to see if you are using it as well like in your work What you think about it? But we try to get some documentation out of this So let's first start with the linear coefficients Adrian, what time did we finish already? What time did we start? Okay. Cool. Okay. So the first thing that we'll check now is how linear models, because you read they are like simple models. So maybe before that, I start how many people, just to know how much people are using scikit-learn. How many people use scikit-learn first? Yeah, Yeah, great. OK, you didn't mess up the room. Then is, in scikit-learn, how many people does know what is a pipeline? OK, I'm impressed. OK, how many people already used a column transformer? A bit less. OK. So how the? OK, I will just do a quick recap of what is a pipeline in a single line. I will just do a cross-validation, just that everybody is up to speed, OK? On toy data sets, and then I will come in the notebook. So I will use a data set that is an Irish data set. And usually, we have two variables, x, which is the data that we require, the y, which is we want to predict. So I will just put that. And the first thing that would be interesting, most probably, is to make your models and learn something so that after that it's predicting. But usually what we want is what we do is that we use cross-validations. So we do a trend test split. We learn on the training, we just predict, and then we check what are the preferences on that testing. And the cross-validation is that we repeat this. We have a specific strategy that define how we are splitting the data into training and testing. So here, I will use these cross-validations that is provided in the cross-validate. That is called cross-validate. And I will use a simple model that we'll just see afterwards. that will be logistic regressions. So my model could be logistic regressions. And to evaluate that model, I can cross-validate passing the model, the full x and the full y. And for instance, here I will use a CAF fold. It will be a stratified CAF fold where I will do five splits. And every time, I will then repeat five times the same experiment. But I will take one fold for testing, the other one for fitting my model. And I will just change. Every time, I will just change the testing fold. So I will have five different scores. So if I do that and store the results, I get something like this with a nice message that you should never ignore. And I will come back to that. So then, here, I can use Pandas to make a data frame such that at least you will observe something that looks nice. pd.dataFrame of the CV results. So here, I have one, two, three, four, five, five results that correspond to my five-fold every time I took a portion of the data to test the model. So here, why I say you should take care is that here I have something that says, oh, my model didn't converge, which is not normal. Does anybody know why? Any clue? Yeah? So yes, and why does it don't find the solution here? Do you know? Okay, it could be, but here is not the case, because basically this model is a convex model, so at some point you need to know a bit, but this is a log loss, so you will find a unique solution at some point. But the problem is that this is using LVFGS, which is called a kind of gradient descent algorithm. So the problem is that those algorithms are thinking that every feature should have kind of the same scale. If you don't have the same scale, then the feature with a large scale will be attracting basically the gradient. So actually what I'm doing is like if you see a curve like this, I'm just going around like this because basically there is one gradient that just attracts everything. And then it's just much slower to get to the optimal solutions. So one potential solution here is written what is written afterwards, either you just let your algorithm more time to convert to the minimum and then you will get you will not get the warning anymore or you can try to scale the data and that's why I want to introduce that because then we need a pipeline okay so what is the purpose of scaling the data with your pipelines to say I take my training and the idea will be to make all the feature on the same scale so I will just divide by the standard deviations and I will as well in the same time Center them so like this they will have zero mean and the standard deviation of one for each of the column in X okay so I will have to learn the mean and the standard deviation on the training data and then reuse the statistic to standardize the testing set or the training set okay so you see that But basically you will always have this learning those statistics during fit, applying them at some point before to actually give them to the logistic regressions. And the pipeline will allow to not have to bother manipulating the data, but you just have to say you will apply this standard scaling and then you apply me the logistic regressions. So let's import first this magical scaler that we call standard scalers. And now I will declare that I don't have a single logistic regression, but a sequence. sequence, and this sequence will be first a standard scalar followed by a logistic regression. And if I do that, the warning disappears. I cannot say if actually it has a huge impact on the output or not. Maybe we just didn't converge, but we are very close to the solution, and at the end you don't see anything. That's a possibility. So here I just introduced what we call in CycleLens a pipeline. And here I use a cross-validation, and you see this is handy because then this model is still behaving like a predictor. You have a fit and a predict, and it does things in the middle that transform the data for you. And if I would fit on, so here you should never do that, but if I fit on some data, you might get the following thing. So a pipeline, visually, look at that. So if the standard scalars, once the data pass here, then they are provided to the logistic regressions that make you what you expect. And this model in Scikit-Learn is behaving like a list. So if I say 0, it means that I will get my standard scalars. If I say minus 1, it will just correspond to my logistic regulations. OK? Yeah? . ROMAIN GUYETT- OK, so I think I'm using the master branch of Scikit-Learn. That's why. But otherwise, in between, you can. So now it's by default in the next version, it will be by default that you will get that. In the new version will be, in the current version, you need to type the following. So in the other notebooks I put it, but here. So you need to import s-calon, and then you specifically say that you want to use to get the diagram, or you get the string, OK? Okay, so here it was just like already to know a bit about like the pipeline, so now I will go in the exercise and at some point I will just use what we call a column transformer, and I will explain what this thing is doing at that point, okay? So I will not complexify the example here. So if you understood this, you are ready to get the other part. So do you have any questions? No? Clear? Yeah? Is there a way to use interpretation and evaluation of model to spot data leakage problems? That's a good question. I mean, for sure we don't have, but basically what will happen is really when you leak some data is that you get a much better score than what you are supposed to do. Oh, okay. So that allocate is exactly what I did down there. When I did that fits and Y. And for instance, if then I ask the score of X and Y. So usually what we always do is we keep a part of the data that we suppose that we never saw during the training to basically find the hyperparameters of the model. So if at some point we use some of the testing data in the training process, then it's what we call a data leakage. If you do that, it means that you will make the life easier to your model. Because imagine that you have a model that learns by Hertz data points. If he never sees the testing, he will never be able to predict them. If you leaked a couple of data inside your training, then seeing that he's learning by Hertz, he's capable of predicting those points. So then you will get, for instance, a very good score while your model just learns by Hertz, basically, the testing set. So that's why when you do cross-validation here, we do cross-validations to ensure two things. So first is that we have a training and testing that are separated. So this, we could do it with a single split. But a single split will give you one score. So we say, like, you have one point estimate. And that's not super good, because it doesn't give you any information on what happened if I just shuffle a bit. I mean, was I lucky or not? And actually, you cannot answer to that question. So to answer to that question is just repeat your experiment. OK? So by experimenting again and again, so with a cross-validation, we mix things over. Then we get as well an uncertainty score. Okay. So here I have five score so I could repeat again and again the same thing and then I could get hundred score and could have a distribution of the score and See if we send like the standard deviation of this to be how certain Am I that this type of model can feed the data? In your questions because of the data leakage, or is it the same thing? So, yeah, is there, so I will answer like this, so is there, the overfitting is induced by the data leakage or not? Yeah, for me it's like it's a consequence, or you can, or it's like the same concept? No, no, it's two different concepts, Because you can make a model overfit without leaking data. In that case, it's just, for instance, a model that learned by Earth the data at training is a model that overfit because he's learning specific rules at training and is not capable of replicating this on testing. So the data leakage usually will just tell you that actually your testing is good as well. So you will just lie to yourself. then a model that overfits is a model that have a very high score on the training set and a very low score on the testing set, while the data leakage is like the score is good on both of them. So it is difficult to detect, but usually it's just like you should ask yourself, is it normal that I get that very good score? But this is, let's say, using a pipeline like this, plus some cross-validations. Basically, in CycleLearn, we use this pipeline to avoid any type of data leakage. If you did some preprocessing before, we cannot ensure that that part, you didn't do any leakage. But at least this, we can avoid some data leakage that are easy to mess up when you transform your data and you keep track of your data. OK. So now let's go in the params. So for the moment, I just show how you can, for instance, fit logistic regression. But now I will go into details how you can interpret that model. So let's import some data. So here I have my things to show the diagram. And the data that I will use is the data from OpenML, which are data. So we can look a bit more at x. So there are data that are coming from a survey where we have different information. So we have two types of data, one that we call numerical because there are numbers and continuous numbers, and some others which are here, categorical. And in that case, the category are encoded with strings for the moment. So we have those two types of data. So here we have a couple of statistics that we can already describe and look at what it looks like. And if we look at the first line, we'll observe the following. And basically, every line will be a person's. And for that person, we have an associated values that we try to target, that we try to predict. And here, this is the wages, the wage, which are in dollars per hours. And those data are from the 80s or 70s or the 75s. I don't remember. It's old. So basically, the goal here is not so before we did a classification frame. Here, this is a regression frame. We try to predict the wage in dollars per hours, which is something continuous. So, for the moment, I will try to explain a single model, so I will not try to do any cross-validations. It's to simplify a bit what we will manipulate, but we can always think about how should we do inside the cross-validation settings, but here I will just make a single train-tay split. The first thing as a human that we usually do and is to contrast, I want to contrast basically these analyses with what you obtain with the model. As a human, usually what we are thinking, we have difficulties to think about with more than two dimensions first. And usually the way that we are thinking is what we call, we try to find a marginal representation, so we try to look at one feature compared to the target. So I will take, for instance, the education level of people. And then on the bottom axis here, this is the wage. And then, for instance, I will say, oh, so apparently, to get a higher wage, it seems that you still have a trend that you need to get higher educations. But here, what we are thinking is really one feature, the target, and we just map those two dimensions. We discarded all the other dimensions. And we'll see later on that a linear model, for instance, doesn't do that. It does what we call conditionals. So by observing this, we could get a bit of information. And it's always interesting to look at your data. So the first thing is, if we look at the targets, you can see that the wedge here, so that's the histogram of the wedge. Here, there is a kind of long tail. And sometimes, we prefer to have more of a Gaussian target instead of a long tail thing. So we might just retransform the distribution so that we apply a log, basically, on that thing at the time that we fit it. And then we just denormalize after that. So we'll see when we do the pipeline, basically will do a transformation on the target. And probably the second thing which is really interesting is like these things here, okay? This bunch of points. So if I look at this point here, so it means that on this y-axis, I have the experience of people, and on the bottom axis, I have the age of people, and they are all aligned on the diagonal. So when you do something like this, this means that your data are correlated. So older you are, usually more experience you get. Or we can put it in the other way. More experience you get is usually older you are. You cannot be a PhD and be two years old normally. So normally, I didn't give like 15 because it can happen. So here, that means that, OK, I have this correlated data. And we'll see that this is actually something interesting that you have to consider when you explain your model, because it will have an impact on the coefficients. So just by looking at already a subset of the data, we could have already figured out before to have just looking at our coefficient without looking at the data. And this is probably a good thing sometimes just to look at a couple of descriptive statistics of your data sets. and then like if we look at the edge so we can see okay we have a kind of slightly positive slope and but I mean it's difficult to to find anything Exactly. That's why we do a pipeline. So if you use yourself, the standard scalars, and you manage yourself the data, then you can leak. The pipeline in Scikit-Learn where you put those elements are leakage-free, so you cannot do any mistake, or we do the mistake for you, but I don't think we have a bug. I'm sure that there is no bug. And that's why we always I'm starting by that things in the course because sometimes people doesn't know about the pipeline and this is actually the most important feature of scikit-learn okay so this is use it even in the cause validation settings because that would avoid you some trouble okay so now we just have a broad view of what are the numeric things and we have these details about agent experience that I will come back later on because we were a discussion about that. So I told you that we have two types of data, and I could have fine with pandas, because if I use the survey.data.info and I look at the dtypes, I have some floating numbers, which here are linked to numerical quantities. And then I have those categories. So we already know in advance, somehow, OpenML offers some information to say, those columns are already categories. So pandas have a specific dtype, which is called category, that say that, OK, here you have a finite number of categories. So this is pretty handy. And we hope that we can leverage that dtype in the future in CycleGene. So here, we can already kind of use it. And the thing is that we want, with a linear model, we'll have to treat differently the numerical values and the categorical values. So why is the categorical values, if you use them, or if you take like, let's say, there is one specific way that you could just simply transform the string into a number, is to say every category is mapped to a different integer. In linear models, this is detrimental if actually the mapping, like the difference between categories doesn't have an order or doesn't mean anything. So for instance, let's give an example with cards. If you have a blue, red, and green cards, it doesn't matter which category. I mean, and that you will replace by a number. So you will have blue that will be 0, red that will be 1, and green that will be 2. But it doesn't mean, so because of the linear models, Normally, when you go from 0 to 1, you have an increment or decrement of the coefficient on the predictive values. And here, it doesn't mean anything to say, because I'm going from blue to green, that I have, for instance, an increase of the price of the car. So it's not like you don't have an order. Something that have an order, for instance, is the size of the t-shirt. For instance, we have a t-shirt which is M, L, XL, XXL. So those are an order, and maybe if I change of one size, it means that the person is 10 centimeters taller. So every time if I change, like if I pass from s to l, so it means I have an increment of 40 centimeters. So here, there is a meaning of orders and of increments, a regular increment of the value of the coefficient between the change. So seeing that usually this is not the case, we don't want to use what we call ordinary encoders. So instead what we do is that we will just make a 1-0 encoder. So it means that every category will be mapped to a single column. And when the category appears, we will put a 1, otherwise this is 0. So it means that actually we'll have one coefficient category and it means that being that category will be receiving a coefficient and will mean that for instance when you have a red car because we have to paint it because it's not white then you will have an increment of price in your cars of 300 euros and if you take your car and that is a red but metallic then it will be 600 euros but that will be another column so we can affect a beta at each of the columns okay so you can always as well i mean it's just that this is is less efficient because you have much more columns, but here you have a different beta. So we could always think, I mean, in general, in the linear model, you can always use a one-hot encoders to do this type of managing the categories. This is a good thing to do. In the tree, you can, I mean, just because I'm speaking about the linear model, in the trees, you can use the ordinary encoders. You will not get into trouble. So now I have, so then, to do something for the categorical data, I want to create this matrix with some 0 and 1, that is a 1 hot. And then on the 1, I want to apply my standard scalars, because we saw that they can speed up actually the convergence. That's why we have different columns that will have different preprocessing. So we don't have a simple pipeline, we have really something that where we want to split the data that will go in the 1.0 encoder, and the other part that will go in the standard scalars and group everything back and send it to the logistic regressions. So this is what we call a column transformer in scikit-learn. So here we use make column transformer. What we do, we define the column which are numerical and the one that are categorical. And then here we can create a tuple in Python that will say these one-hot encoders will be matched to the categorical columns. And first, because I want to demonstrate a couple of things, I will not standardize the data because I want to see what happens on the coefficient when we don't standardize the data. And then I will tell you, OK, you need to standardize them for specific reasons. And we already saw one. There is another one that we will see just afterwards. So here, instead of using numerical columns, what we say is that whatever we didn't took in the categorical columns, we just let pass how they are. So we don't touch them. They will be aggregated with the new categorical columns. And this is to avoid to to have like too many details afterwards In the column names. I mean you can pass this this part So this will be our pre-processing and now we have to plug our logistic Yeah, so that's what this parameter is doing here so here we decided to say if it, ah yes, so the question is, do you, because when you encode, for instance, if you have a binary value, so male, female, and you encode with 0 and 1, you will get a column with male and a column with female, and actually one will be the opposite of the other one, they are just anti-correlated, so then this is kind of useless, so you might want to drop one column, and in that case you have the keyword drop that allows you to say the strategy and here we drop only in the case that we have binary. We could extend that to more columns because it's true that when you have all the columns, imagine that you have like five categories, you will have five columns, and actually one of the columns can be found from the four overs. So you can as well mention it here. So yeah, I'm not doing it. But if you don't want to have a problem with correlation, then you do that. Yeah? So that's exactly to eliminate this dummy variable drop, right? It's the same concept, right? Yes. So that the drop is, yeah. So this drop is really to kick out one of the dummy variables that can be found from the overs. And then here, so we have a pipeline that is our preprocessor, followed by a linear model which is a ridge here. So the ridge is just a linear regression where we don't allow the weight to be big for no reason. So we have a bit of L2 regularization, what we call. And we put this ridge here inside something that will transform the target before the fit. the fit so we will apply a log as a like forward functions so it's to transform I told you like this distribution with a long tail so it will just transform it a bit more like like to normal distributions and then we provide the inverse functions to just like get when we predict to denormalize to get back to the original scale okay so this is a bit of details here but it's is just to have a proper, like, to have a Gaussian y. OK. So now we just fit our data. And the first thing that we should do is probably look at what are the performance of my models. So here, we use the training and the testing to compute the mean absolute error. So here, the mean absolute error will be interesting, because it will give me numbers that say that, in average, you make that amount of difference between the salary of a person and the true salary. So let's see what it gives us. We'll plot it. So the average means that, in average, I predict $2.14 above or under the true values. So is it a problem or not? Somehow it might be, because if we look at our y here, the range of value were probably between like 5, 10, and maybe I should make an histogram, actually. So let's make just two. So v.target.east. So that gives me a distribution of the histogram. So here we do a $2 error for value that goes from 0 to 15, and then there's a couple of less. So it means that we still have a model that is not really good. But we can still try to inspect it and see what we will obtain from that. But from the first thing that you should look at is this dimension. Is my model good or not? And one way that we could do that is to get this information. And then if we have not so many data, we can just plot the true data versus the true predictions. And here we can, so if everything would be perfect, it would be on the diagonal, okay? Whatever is like further away mean that is where we commit the errors, okay? So here we make a lot of errors apparently for the high salary, okay? So it's not because, so there is like some hot layers there and probably this is kind of difficult to, like to predict them with the information that we have. So let's start to look at our coefficient. So seeing that we have a model, so the fit of x-train and y-train, we can plot, we can access the regression coefficient. So it will be the value of the input data. I thought that I corrected that yesterday. Okay. Is this working for everyone here? Okay. Cool. Okay. I'm surprised. But, yeah. I thought it was only in the last release, but maybe because we use a one-off encoder, so it's fine. It's an encoder that wouldn't work. But I executed yesterday, and I didn't get a bug, but okay. So, yeah. What is the workaround with the bugs? The workaround, we'll see in the next notebook. Because basically, up to this version of scikit-learn, we don't manage very well the name of the columns, and we lose them somewhere. So from next version, basically, you will just have to use that diskette feature names out, and you get it from anywhere in the pipeline, okay? Before, you needed to know in which order your column transformer was working to create the names. So, yeah? I have a story for you to wrap this. Just coming back to one concept you presented before, is that you said that if we have a Gaussian distribution with a small standard deviation, I understand that you can predict it better. Like if all the values are like this small range. And here, we have a positively skewed distribution. And you said that you want to like to, yeah, you did this operation of the logarithm and then It's more, so I don't remember really about the theory, but you might assume that basically the target might be like Gaussian distributed, I mean, to get the right estimator. So here the distribution is not Gaussian, I mean, it's this one, so there's the tail. So basically when you put the log, you will make it more Gaussian, then you fit on that data that is better suited for the model. So then you will predict what you will predict in the chain scale, in the log scale, and then you can just reverse this transformation and you get in the original space. So like this you didn't get into trouble with the long tail and you just put a transformation that you can make forward and backwards. So here I can go and, yes? We will go at the end of that notebook about lasso. And I will explain why there is some stuff to know. But yeah, we'll come to that. So here, we have all the different categories that have been one-hot encoded, and then we have the coefficient that is associated with them. So if we do a bar plot of those values, we'll get something like that. So usually what we tell you is that a larger thing, that basically we have an input x that will be multiplied by that coefficient that will contribute to y. So, negatively or positively, larger is this bar, more impact you will have on the target, okay, is what this thing means. But there is like one trick here, because if we look at like what we have here, we see that age and experience doesn't have a larger weight than all the other categories. That might be surprising because we could expect that that should be like the variable that should really have an impact on the salary of people. So, it's a bit of a confirmation bias because I know that they should. But here's the reason is that, like, here's the value. Remember that we didn't scale the data. So if I take only the edge and I look at this value here, or the edges here, the value have a meaning. It means that if I add one years to a person, then the salary will decrease of 0.003 dollars per hours, OK? So that's what this value means. And the second one, the experience is as well in edge, education is in another unit. So it means that all those values here are in their original units. So you can interpret one variable at the time. It has a meaning. But then, if I try to say, is this variable less important than, for instance, the union not member, are not on the same scale so it doesn't mean anything so i cannot compare them between them i mean between them okay so i need that's why when we scale the data everybody every variable is between zero and one or have a like standard deviation of of one so then it means that we can compare the the coefficient between them we cannot come back to the meaning of the coefficient, because we need to descale them if we want to understand what is the meaning of the coefficient, but at least we can compare them between them. So one thing that I could do here just to rectify the problem would be I can look at the training data and look at the standard deviations of a feature, so percolumn look at the standard deviation, and now you see that for the edge I have a lot of fluctuations compared to all the other variables, and this is normal because the other ones are just one-hot encoded. So they have a very small standard deviation, and age, experience, and education, that were numerical values, have a dynamic range which is much larger. So if I use this to normalize the coefficient, so by multiplying by the standard deviations, I would get something like that. And now I see that age and experience have really an impact. if I really compare, not in terms of their original units, but in terms of the impacts, agent experience will have a much higher impact than on the other one. However, here, it's like trying to correct something that we should have to do from the beginning. It would be much better to just use the standard scalars, and then we can directly compare the values. So here is just to show you, OK, I could look at the feature standard deviation and just have a plot like this where we see the correct coefficients. And there is a second thing which is a bit surprising when we were thinking, when we look at that plot. So here we saw a positive trend. So what we could have expected somehow is more education you have, better salary you will get, The more experience you have, the better salary you will get, and the older you are, the better salary you should get. Somehow you have a kind of a trend positive. And now if I look back at those coefficients, the edge tells you that it's negative, so somehow the older you get, you get a lower salary, and the experience is on the opposite side is saying that better experience you get, you get a bigger series. And here it's because a model like this doesn't make, you don't interpret it as a marginal contribution. It's not a marginal link between a feature and the target, but a conditional one. So it means that, so older I am, but knowing that all of the features have specific values. So I can change the edge, but all the other features have already given values for that sample. So my model already learned on a condition of all the other features. So what you observe here is not a marginal link, but a conditional link, because it's the value condition with all the other features. And here, we observe that thing because it's because of the co- we'll see in a minute, but this is because of the correlations. OK? So why do I say I am seeing about the correlation? One of the bad things that we didn't do here is that how lucky, I mean, was it normal? Or can it change if I change a bit the data? OK, so we don't have anything that can help us here with the study that we have regarding the stability of the coefficient. Is the edge always negative? Or sometimes is it positive? Or like, and the same for the experience. We don't know any of those. So that's where we could run a cross-validations. And the cross-validate that we use at the beginning, we can pass him an argument that is called return estimators, that will return us every estimator of all the different models. And then we can inspect the coefficient of all the different models. Yeah? Regarding the regression, is it fair to actually say that this is a marginal effect because there's like this regularization parameter and . So the regular, yeah, so you, yes. So you mentioned that it's fair to say that this is only conditioned by the data. I mean, that reach, because we are using the reach, actually is not only a problem of conditional between x and y, because we are forcing the parameters on the beta to have like the smallest ways possible. So this is true that the strength alpha will have an impact on the value of the parameter. So it's not only like with all the feature constants, when you have an alpha for the regularizations, things become much more complicated. And this is difficult to, it's something to have in mind. You are completely right. OK, so here we'll perform the same experiment than before, but just make it in the cross-validation. So get five estimators. And then here, we get the coefficient of the regressor. And we multiply by the standard deviation of all. So here, there's a bit of mechanics, because I wanted to get the training set of every fold and compute the standard deviation on that specific training fold, because this estimator was trained on that training fold. So I have to go and pick up the right tuning fold and not using the entire data set. So that's why it's a bit more cumbersome. And if I do that, I mean, much easier would have been to use a standard scalar, but it's coming afterwards. So if I do that, I look at the coefficient. Now I can look at a distribution. I mean, I have several coefficients. I can look at the distribution of the coefficients. And I will get something like that. And now I have information about the variability of the coefficients. So for all the one-hot encoded coefficients, they are very stable. And then we get two huge bars for edge and experience. So what this thing is meaning? It means that most of the time, the edge is negative, and the experience is easier to compensate, basically, the edge coefficient. So they compensate between each other because they are correlated. But sometimes, my model just switches. You see that actually, now my experience will be negative. And in that case, to compensate, I need to have a positive edge. So the bar will be in the opposite directions. So this plot here informs me that, oh, to have this thing that just goes and is varying a lot is probably because I have something in my data. And something just that shuffling with the data makes that we have a different model and informs us that I create here, we have two variables that are correlated. So I could convince myself that, OK, that's probably some correlation, so what I can do is just drop one of them. If they are correlated, it means that I don't need the two variables to be able to predict. I can use only one, and then my model will not get confused. So let's remove. OK, so here, this is a plot to show that it was a former plot, but basically that the age and experience are correlated. And now I will drop the age from my model, and I will just redo the same experiment. So no age anymore, but now you see that the experience become always positive. So actually, before, we really had these things that the experience was compensated for the age, And that's the problem of correlated variables here. So somebody was speaking about LASSO. And so for the one that doesn't know, LASSO is a linear model. So Ritz imposed a L2 regularization. So he tried to make all the weight going to 0. So bigger is alpha, more the weight will be put to 0. But you can put as well a constraint which is a L1 norm, which means that some of the weight should be exactly zero. So you should discard weight. So I think there is an example at the end. And in that case, you could make a kind of feature selections. And saying, for instance, it will drop age of the experience because you don't need both to predict very well. Yes? Wait, I have to go because I have to. Can you please submit them on Slido? I will ask them. Yeah, if you can go on slido.com. Could you please submit your questions on Slido? Because then when I ask, removed people also will hear them. Then the speaker doesn't have to repeat the question. OK. What's the Slido you have in mind? Slido.do, and then you go hi there, and then you choose the room when you ask the question. So slide.do, and then you do piper. It's there. You see the . Maybe you can come to the front. It's kind of complicated, you know, to iterate on questions, because I want that to write. It's good. You can come here. Or you can enter the microphone, and I can answer. Because it's hybrid. It's not there for the hybrid model. OK. Sorry? What is it? Slide of what? Slide.do. And then you can ask questions on the website. And then Adrienne will repeat them. The instructions are also on the room of the conference on Discord. We have a question, in the example that you use standard scaler, do we have, like, are there recommendations for, like, all the preprocessings that we have, or, like, when they make sense? Yes, so regarding the type of preprocessing that you can use instead of standard scalars will depend on the type of data that you have, so you need somehow to have some knowledge about it. So if you have, for instance, some outliers in your data that are away from the main distribution, so usually you don't want to take into account, and if you scale your data, you will take into account those. I mean they will have a bias on new mean and standard deviations, so you want to exclude them and to exclude them You you will just use something like Robust Scaler that will just neglect like data that are far away from these distributions And otherwise sometimes you are interesting of not centering the data. So you will use something like mean max scalar to let Not max app scalar. Sorry to let the zero being zero. Okay, so like this you have sparse data So it means this is an efficient format to have a lot of zero in your matrices So you want this impose you to not break this sparsity so you want to keep so you will use max apps Scaler and why there is some time that you are interesting about having data between zero and one for some reason So for instance like I know Images that you have pixel that are between 0 and 256 and you want to bring everybody between 0 and 1 and in that scale So then you will do a min-max scalar That I have no idea Probably not, because the why is that, I mean, this is exactly what we have here. So we have a Newton, I mean, it's between a Newton and a gradient descent, I think. So it suffers from that. And the read should somehow reduce the problems, but the algorithm itself doesn't solve the problem. Yeah? How do you mitigate the frequency for categories when a category is uncommon but very discriminant, but it's normalized code will be very small. So uncommon. So the thing is you will encode those category anyway, so they will have a weight. I mean if there's discriminants, I mean at the time that they appear like one, they will have a huge weight. So I mean you will associate a huge weight to those. The only problem is that you need to at least have some of the sample doing training to be sure that they are up here in that case this is more of a problem of selecting your right training feature i give an example it's my question i give an example so my point is if you have a category especially categories and you wonder how much you should actually split them So if you saw here we don't normalize yeah so here you said yeah if we standardize very unfrequent variables so basically there are we'll have a very small standard deviation so they will become very small. But here we don't standardize the categories. What is important is that you don't have a huge difference feature range between the numerical and the categories, but we don't reapply like a scaling on the categories. So we only want to encode them. So we don't apply a new scaling. Again, I think it's something that's complicated. I cannot tell you that for sure you should always do that. . OK, so here we saw that if we drop one of the variables, then we don't have the other variable to compensate. So then, like the experience, and if we would do with the a, that would be the same, OK? a would be always positive, or experience would be always positive, and we don't have this problem. So now what happened? Yeah, what I could have do instead is to put the numerical standard scalars instead of just letting the data pass through. And it would have directly solved my problems of trying to correct the feature, and we would get the same results in terms of performance. And it's just that by default here, I don't need to re-multiply by the scaling. And in that case, those features which were numeric are at one standard deviation. And all the coefficients are in the scale of the encoding, basically. So that would be the usual thing that you do with linear models. So you apply a 1-0 encoders on the categorical data and the standard scalar on the numerical data, and then you go on. About ridge, what happened just before is that we were using your ridge with a very small alpha, which mean that we didn't apply any weights. I mean, a very small regularization. So we let the weights being as big as they wanted. So you have to know that when you apply a reach, so usually you want to use cross-validations. You can use reach CV to actually find what is the best value of alpha in your predictive settings. And you will use the internal cross-validation to find this alpha. And this will have an impact on the weight, because more regularization you are putting on alpha, the weight will just shrink. All the weight will shrink in that case. So then you cannot, if you have an alpha and you try to express it like in the original scale of the original units, I mean, if you take the coefficient and you want to express them in the original unit, that doesn't have any sense because the alpha is as well shrinking the weight because this is its job. So you cannot interpret them in this way. But here, if we do a, this always go, 10 minutes? OK. I should have to do a three-hour tutorial. So here, we do, basically, we are seeking for the alpha. And this model, we just find the alpha directly for you. And if we look at, for one model, at the alpha, so for instance here, you pick up an alpha of 10, because from the range of value that were from 10 to e minus 10 to log of minus 10 and log of 10, this was the best value that you found. And this might improve somehow the score. Here, this is not the case, but somehow you should always tune your parameters. And here, we can see that by just applying some regularizations, if I look at my coefficients, it has an impact direct on the coefficient. So for instance, now the experience and age are both positive. So this is difficult to, I mean, the alpha fight for the correlation somehow. And then I put both positive. So this is like, normally the job of the regularization as well is to make as well, like, your model more stable. Okay. And now... Yes, and of course, if one that you apply, you alpha here, so those, seeing that the model now is kind of constrained, and you just have, like, positive values, What happens after that is that if you look at the coefficient and you do cross-validations, you will see that the regularization solves the problem of the correlations, and the standard deviation as well. All these things together just manage to reduce the problem that we had before. So as I said, we could have used a LassoCV, and here this is an example where I'm using the same way but something that will select some specific feature and if i use that and look at the the coefficient so i redo the same thing this is performing as good but what is interesting is now looking at the a coefficient you see that the experience have been set to zero okay and some overs have been set to zero so here the model found that there is no way of using both variable and you just put one to zero because of this regularization part okay so one important things with uh with lasso to know is that here we picked up experience but there is no reason we could have picked up age okay so the algorithm in cyclone just make a cyclic way of of picking up things so you might have more chance that it will be always the same variable that is drop but this is not a given, so to see that we can just run a cross-validation and reproduce the same things, and here we observe these things. So here we can see that most of the time edge is kind of like positive, most of the time experience is zero, and sometimes this is not the case, okay? So one of the things is that LASU is not a super estimator because the algorithm is as well not stable. I mean, it's not always stable. So just by changing a bit the data, there's some cases here that edge become negative, and we have the same frame of correlated things. So maybe I could have increased the value of the alpha, but then it's already automatically set by cross-validation. So I would need to do it by hand. So this is not usually the way that we do. So you want to use LassoCV to optimize these parameters. Okay. So it stay? Four minutes. Okay. Did you learn something? Maybe? Yeah, I mean... So one of the questions here is that, like, this is awesome, but the other parts, is there recording in other places that you've given the talk that people could watch? I have to check. I think the part about permutation importance and partial dependence there is because we have in the so I will just give we have a channel in scikit-learn that should be here where these talks and I assume that in those ones is one of Olivier Grisel that that presents the feature importance and the permutation importance and the partial dependence. Sorry for the SHAP because it was the first time that I was trying to present it. But we can discuss it as well for the people that are here, we can discuss it later. Does Scikit-Learn support Bayesian approaches to model uncertainty of coefficients? Yes and no. We are very bad at that. we should use PyMC3 for that. We have two models which are called Bayesian read and RD regressor, which are like proper Bayesian approach where you give a priors, et cetera. The problem is that we don't give you the distribution of the coefficient. We just give you the uncertainty of the predictions. So we should be able to do that, but our API is not good. So probably something that we should fix because we should be able to do that. So, but I advise you to use like real Bayesian framework like if you want to do that. Thank you. Okay, sorry for.