Automated Feature Engineering and Selection in Python

Speaker 1 [00:02]

A nice introduction. And thank you all for being here. I didn't expect such a huge crowd. Okay, I'm going to dive right in. So in machine learning, we're dealing with complex prediction problems. And often the first thing that we do if we have a complex problem is that we want to use a complex model. So for example, a neural network, or a kernel method like a support vector machine. but they also have some drawbacks while they give great predictions most of the times it can be kind of annoying to train them, especially neural networks I don't know how many of you have trained a neural network before but usually the first one you train doesn't do much and you have to fiddle around with a lot of parameters to get something to work and they're also hard to explain And usually when we want to move our machine learning research into production or influence business decisions with that, we need to explain what we're doing to someone who probably only did some high school mathematics. And you need to convince them of why your model does something great and why they should trust the predictions. and you can't really do that with kernel methods and also neural networks have some problems of explaining what they actually do. And if you really look closer at those methods, you also see that what they're really doing is using a linear model just with better features. So the kernel methods are designed as linear models in the kernel feature space. And the trick there is that you project your data into this kernel feature space, but then actually it works out that everything is just scalar products in this kernel feature space. And so you never really need to compute those really complex features because you can compute everything with the kernel functions. But in the end, you just work with linear methods, like linear regression in this kernel feature space. And similarly, for neural networks, If you think about the neural network as having two parts, one part that maps from the input through several layers until the layer just before the output layer. And then the second part, which maps from this last layer to the output. Then this last part, that's also just a linear model mapping from some kind of pretty good feature representation to the output that you want to predict. So, yeah, the same as a linear logistic regression. and the cool thing about neural networks is that they learn those better features by going from the input layer through all those layers until they learn some good representations of your data so what if we already had better features then we could just use linear models and I'm writing my PhD thesis about neural networks but I'm actually a huge fan of linear models mainly inspired by my freelance work because if you work with non-scientists They usually want something that just works and that's easy to explain. So a quick recap. Linear models predict something as a linear combination of the input features. So you just have this weighted sum of your input features to get your output. With one feature, the whole thing looks like this. So it's a line. That's why it's linear. They're great because they're really easy and fast to train. and they work with big and small data sets so you kind of have this requirement that you should have more data points than features but in general you can use them with pretty much any kind of data set that you have and they also give transparent predictions so this equation that's something that yeah anyone who has taken math in high school should be able to understand okay but of course linear models are linear so if you have a more complex prediction model then you need some feature engineering um yeah to get that to work um yeah creating those additional informative features uh usually requires domain knowledge so you need to know where your data comes from and maybe then you have some intuition about what might be useful features that you could construct from your original input data and most of the time it's also just like trying a lot of stuff out until you find something that's somewhat useful and yeah anything that's annoying we would like to automate so that's the same thing with feature engineering usually feature engineering frameworks have two components the first one is of course the feature engineering component where you generate a lot of additional, complex, non-linear features. And then since this usually happens that you just generate away and create many, many features, you also need a second component where you select the features that are actually useful for the thing that you want to predict. There are two types of strategies going at the automated feature construction. construction um so one thing is that you can just uh yeah first uh start with your original input features from those create a lot of um yeah non-linear complex features and then once you have your big feature pool uh you can um apply feature selection and then select the features that you actually want to use uh moving forward and this of course uh requires a lot of memory because you have to create this huge feature pool, which needs to be stored somewhere. The alternative is an iterative approach where you stepwise create some features, then you apply feature selection to throw out those features that don't seem useful to you, and then you create some more features until you're satisfied with the feature pool that you have. Here is the problem that you might miss important features for two reasons. so first you might throw out some features that might be useful later on so if you have some kind of non-linear feature that might not be that helpful by itself but if you combine it with another feature then you might get something useful if you throw that feature out before you can use it to construct a more complex feature then you're going to lose that complex informative feature and it depends on your feature selection strategy so i'm going to talk about that later as well. But if you use some very simple heuristics to select your features, you might also miss some features that might be helpful in conjunction with some other features. And if you have a more complex feature selection strategy, like training a model in each iteration, then that's going to take a lot of time to construct your final feature pool. So yeah, I would always suggest to use the first approach um because usually you don't really have a problem with memory and i mean if you if it doesn't work on your laptop you can just rent an rbs cluster or something so there's always enough ram somewhere yeah okay and that's of course also what auto feeds and the library that i was writing does okay um so first we're going to talk about feature engineering uh there it highly depends on what kind of data you have. So if you have relational data, so stuff in different tables, for example, you have one table with all your clients, and then you have another table with all your products, and you know which client bought which product, then you can create something like the sum of all the products that they bought as an additional feature for each client. So that's going from one table to another table and aggregating stuff. So if you have relational data, the library that you want to use is feature tools. Then if you have time series data, there's a nice Python library called TS Fresh. They do stuff like detect how many peaks you have in your time series, what's the maximum peak, minimum, stuff like that. And those two libraries are also going to be covered in the next talk. So if you're interested in them, stick around. I'm going to mostly talk about myLibrary autofeed, which is kind of for your normal data. So you have something in a single table, your single Excel sheet. And it's also mostly targeted towards scientific data. So for example, different sensor measurements in different units. You can also supply the units that you measure stuff in to create more informative and physically relevant features by respecting if something is like a temperature or pressure or something like that. So how does feature engineering with AutoFeed work? So we start with the original input feature vector. So in this case, that has three feature dimensions. Then in the first step, we apply some nonlinear transformations. So logarithm, square, square root. And yeah, there are a bunch of different things implemented. And that gives us, yeah, so that's one step of feature engineering. Then the next step is combining those features. So we have different operations to create some kind of combination of the nonlinear or regular features that we had in the first step to create more complex features. Then the next step is, again, to apply transformations on those features. so apply more non-linear transformations and you can repeat the process for as many steps as you want more combinations, more transformations but of course the feature space grows really quickly, really fast so in practice it's also fine if you just stop after two or three steps you already get pretty informative features additionally we also provide a wrapper for the one-hot encoder from scikit-learn to deal with categorical features. And as I already mentioned, you can pass the physical units of your features and they're respected in the feature transformation process. So for example, you won't take, you don't subtract the temperature from a volume of something. You only compute something that's physically meaningful as features. And that's done with the pint library, which deals with units, also messy units. And we also use that library to compute the Buckingham Pi theorem, which gets us some additional features of dimensionless quantities. It's a physics thing. Probably won't need it. Okay, so that's for feature engineering. Then we're here. We have this huge vector of complex features. So the next part is to reduce that to some manageable, informative amount for which we're going to use feature selection. There are two approaches to feature selection. The one is univariate feature selection, where you look at one feature at a time. And the other one is multivariate feature selection, where you always consider multiple features at a time. Don't use univariate feature selection. so if you have two features here um for the classification case you would say okay the first one um that seems pretty informative i'm going to use that the second one yeah it doesn't look that great so we can ignore that um that's fine if you look at the individual feature but um since you usually don't have a problem with just one or two features but many many features um this strategy gets you into trouble because on the one hand, you're going to select a lot of features that might be correlated. So those are two pretty informative features, but they don't really add anything to the problem. So with univariate feature selection, you would take both of these features, but one of them is not really helpful for your problem. The second problem, besides adding redundant features, is also missing out on important features. So those will be two features that by themselves are not very informative. But if you take them together, they actually help you solve your problem. So yeah, if you would only look at one feature at a time, you would kick out both of those features. But in the end, they would actually help you for your problem. So yeah, what should you do instead of using the univariate feature selection? Always look at a lot of features together. You can do that with your regular machine learning models. So you can train a regularized model on all your features and then check the influence of the individual features. So for example, if you're just using a plain decision tree, a random forest from Scikit-learn, after you fitted it, it has the attribute feature importances where you can see how important each of the features was for the decision so that depends on at which position in the tree each feature occurs the feature that's at the root of the tree has an influence of all the data points that go through to make the decision so that's an important feature whereas some feature that occurs once like somewhere at the bottom that's probably not very informative and the second approach that you can take is when you're using a linear model for example for linear regression and the one that works really well is the Lasso-Las cross-validated model. That's a sparse linear regression model, so it drives a lot of the weights to zero, and only those features that are actually helpful get a higher weight. And then you can look at the coefficients, which is also an attribute after fitting the model, and they tell you how important each feature is. And you just need to be careful. If you didn't standardize your data before training the model, then you need to consider the standard deviations of the individual features because you might have one feature that has like really large values and one feature that's really small values and deviations but even though you have the smaller deviations that might actually be an important feature so you yeah need to scale that and that works pretty great if you have yeah a manageable amount of features so like about the size of your data set and if your features are fairly independent but especially after the feature generation not only do we have oftentimes many more features than we have data points a lot of the features are also correlated with one another and that kind of messes the whole thing up because for example in the decision tree when you have a random forest and you have two features that are highly correlated maybe in one of the decision trees you have the one feature at the top and in the other you have the other So the feature importance is not really going to reflect how important each of the features is because you have those correlations between features. So for autofeed, what's implemented right now is something that's based on the LassoLas model because we want to use linear models later anyways. But it has a little more steps because we have this case of those many features. um so what we do is that we have this set of candidate features that we generated and um the residual that needs to be explained so the part of our target variable that's not yet covered um by the features that we selected um and then to select the set of features that we want to keep um we start by computing the correlation um of this residual with all our features. So this gives us some features that might be helpful to explain a big part of our target variable. Then we use these features, so the most correlated features, to extend our set of good features. And we take as many as the linear model can handle, so like half of the number of data points that we have. Then we train a lasso regression model with those features. On the one hand, we're going to use that to update the residuals so that we know the features that we selected right now, explain some part of the target variable we have, and then if we subtract that, we get the part of the target variable that we didn't explain with the selected features. And we also use the coefficients to filter the features that we have. So if the lasso regression model said, okay, those features have a really low weight, then we just throw them away and continue with computing the next correlated features. And we do this until we have a stable set of features. And what's really important is that in each iteration, we take a subsample of our whole data set so that we don't overfit on all our data, but we know how well the set of features that we have generalizes. Once we convert to a set of features, we do a final noise filtering where we train another lesser regression model on all the features that we have selected and then also some randomly generated features and we only take those features that have a higher coefficient than the randomly generated features. Yeah, so that we don't just select some features by choice. Okay, so that leaves us with the set of features that we want to keep and then we train a final prediction model that you can use later on when we use the final model. And how do you use the model and autofeed? and so it's pip installable first of all and then you can yeah just import the auto feature regression model right now it only works for regression but i'm also working on extending it for classification and it just works like a scikit-learn model so you instantiate the model and then you can fit the model and fitting the model if you call fit transform that also gives you a data frame with all the new features that you generated. Then you can call predict on your data as usual. And you can also, so if you want to say, okay, I don't want to use the simple regression model that's implemented with AutoFeed, but I want to only generate the features and then use them with my own model, you can just call transform. And that gives you the new features that you can use with your model of choice. Okay, yeah, how well does it work? So first a toy example where I have some, yeah, target variable that's a pretty complex formula of three original input features and a little bit of noise. So if you look at it, yeah, an individual feature doesn't really give you much information about the target variable. but yeah so if you train a regression model on the original features you get a pretty crappy result after one step of feature engineering yeah you already get some useful features but it's not perfect yet with two steps of feature engineering you get a pretty good r square but you're still using a lot of additional yeah you generated a lot of features even though you yeah originally just had three terms. And with three feature engineering steps, you basically have a perfect prediction and AutoFeed found the two features that you wanted to find. Of course, that's just a toy example. So testing it on some real data, you want to see if the predictions improve and also what kind of features were selected. I tested that on five data sets from the UCI machine learning repository, they all have fairly small sample sizes. So first of all, what we can see is that the prediction, it's not as good as a random forest or something. So we're not state of the art, but we're definitely better than a simple regression model. And we still have this added benefit of having an explainable model and we see which kind of features it uses so we can derive some insights from that. If you look at the selected features versus the features that were engineered, so with one, two, or three feature engineering steps, you see the number of features grows a lot that were engineered, but there's actually a fairly stable set of features that's selected in the end. So we're not, yeah, it's effectively selecting just a few informative features from all the features that were generated. Those are the features that were selected most often across all these data sets. So what I noticed there is mostly that there's a ratio of two features. So if you normalize, you kind of normalize one feature by the other, or you set one feature in perspective to the other, that's often what seems to be an important feature. So if you don't want to use a library, but go on with feature engineering by hand, that's something that I would recommend just taking the ratio of two features. And then we're coming to a close already. So I hope what you take away is that linear models are great because they give us some interpretable, meaningful formulas. And if you have the right features, then also the performance can be pretty okay. For feature engineering, there are different libraries. So you have feature tools, TS Fresh, and of course, Autofeed. and for feature selection don't use the univariate strategies and if you use some models you really have to be careful if you have more features and correlated features than data points so yeah there you have to be a bit careful but Autofit takes care of that and as I said right now the library only works for regression so the feature generation like feature engineering part works for everything everything. It just generates this general set of features, but the feature selection strategy that's currently implemented relies on the LASA regression model, which is why it only works for regression. But yeah, in a few weeks, that's probably going to be updated. And that's it. So thank you.

Speaker 2 [23:22]

Thank you, Franziska, for this very insightful talk. We have six minutes for Q&A, and I'd like to remind you that questions are preferred over comments. So there's one over here. So first of all, thanks for your great presentation. So I have a case, actually, that's multiple classification with really unbalanced label, but I have a huge amount of features. all of the features are binary actually zero or one do you have any recommendation actually for me

Speaker 1 [23:56]

So you're mostly looking for a feature selection strategy?

Speaker 2 [23:58]

for feature extraction actually and also feature selection after feature extraction

Speaker 1 [24:04]

Yeah, I think if you already have a lot of features, you should probably first, yeah, either you can use something like PCA or something to reduce the amount of features that you have, or, I mean, I would first go and select a few features, and then from those promising features, you can try to construct some better ones. But, I mean, if you have more than 100 features, it's really going to be tricky with the amount of RAM that you have to construct more features.

Speaker 2 [24:32]

more features. I tried this PCA. So PCA is good, but for normal data, actually, when we have numbers or something like that. But with zero and one, actually, I was not that satisfied. I wanted to ask you, so do you have any? So, okay.

Speaker 1 [24:46]

Yeah, you should probably first look if those features are highly correlated and then throw

Speaker 2 [24:51]

throw away I did all of them so maybe

Speaker 1 [24:53]

So maybe, yeah.

Speaker 2 [24:53]

yeah Well, thanks anyway.

Speaker 1 [24:56]

Anyway. Thanks.

Speaker 2 [25:02]

Thank you for the talk. Just a very simple question. When you were putting the results on real data, you said that the output of this library was not as good as Random Forest, I believe, or or something. But did you try to take the features that were found by your library and put them into these kind of models to see how they perform?

Speaker 1 [25:22]

No, I did not. I mean, yeah, that's obviously one thing that you can do. My main... I mean, it definitely shouldn't be worse than the random forest on the original features. I mean, the original features are always kept. You just get the additional engineered features. So if the random forest was already happy with the original features, it could only get better with the new features if you regularize enough. But yeah, I mean, what's interesting is how much better you get than a normal linear regression because it's the same model just with the added features. But yeah, I mean, as I said, you can also just use autofeed and generate the features and then use them with whatever model you want to use them with.

Speaker 2 [26:09]

You said your main motivation for, or one of the main motivations you said was that you wanted to stick to a comprehensible model, to the linear model. Don't we lose this benefit if we have such complicated auto-detected features? Had you any problems explaining then your model to your non-scientific co-workers?

Speaker 1 [26:36]

Yeah, so I mean, if someone doesn't understand what a logarithm is, they're probably going to have a problem with the formula. In general, yes, I've been working a lot with other scientists who don't understand the machine learning models, but who know general math, and they're usually really happy. For example, like chemists, they work with complex equations, they just don't always know how to get it from the data that they have. So if you give them some equation where they see, okay like those two features have yeah you need to set them in relation to one another or something like that and that really helps them so it's yeah

Speaker 2 [27:18]

Hi, great talk.

Speaker 1 [27:19]

How is it different from the scikit-learn's polynomial features interaction? And if that already exists, because I already give you a good explanatory model using, for example, plotting trees, for example.

Speaker 2 [27:38]

And if you couple that,

Speaker 1 [27:38]

And if a couple... if you couple that with the polynomial feature interactions, how is that different from your autofit? Yeah, so first of all, when I showed the features that were selected, most of the features were ratios of one another, and you don't get those with the polynomial features. So they just give the interactions of one feature multiplied by another, but you don't get the ratios. So yeah, that's definitely something that's missing from the scikit-learn implementation. And yeah, for the trees, I mean, a random forest consists of like 100 trees. They can be pretty different from one another. And I don't know if you plotted a tree with like, I don't know, depth 10 even. They get so complex, you don't really want to look at them. So I mean, you can look at the future importances to kind of see the direction of why the decisions were made. But decision trees are in principle, they can be well explained. But in practice, if you actually look at the tree, it gets pretty complex. And it's kind of hard to just extract the simple rules that were followed from the tree if you plot it. So I still believe that if you have just the actual formula, that can be more informative.

Speaker 2 [28:51]

Thanks for your awesome talk. I was wondering, how did you select which operations you should apply on the features? You said there's a multiplication, some division, but is there any scientific meaning why these are selected?

Speaker 1 [29:06]

No, that was mostly random. I mean, I kind of looked at formulas that exist and I saw which operations are in there and I thought, okay, I'm just going to use those. Yeah, so I mean, the combinations, like, yeah, you don't really have more than, yeah, times plus minus and the divide is covered because you have a transformation that's one over the feature. So that's, yeah, a complete set. But for the transformations, yeah, I implemented some. There are also more, you can select which one of those you want to take depending on your data. And it's really straightforward to implement some more. I mean, it's relying on the SymPy library, which covers mathematical operations. And everything that's in there can, in principle, be extended to autofeed. It's just like two lines of code. Yeah.

Speaker 2 [29:52]

Okay, our time is up. In five minutes, we'll have another talk on automated feature engineering. And please give another round of applause to Franziska.