Unlocking the Predictive Power of Relational Data with Automated Feature Engineering

Speaker 1 [00:07]

yes welcome to the tutorial here my name is alexander ulich i'm the co-founder and ceo of code 17 gmba and today i want to talk about one of those big challenges that remain in classical machine learning in today's data science world and particularly about predictive analytics on relational data and the difficulties and the challenge that comes with manual feature engineering on it. Later on in the code section or in the hands-on code tutorial, we will quickly go through a solution how you can alleviate that problem using GetML FastProp. So let me quickly run down through the agenda introduction. This is the theoretical party of the presentation. feature engineering a little motivational speech of why this stuff really matters and then setup of get ML if you if you want to run at least one of those notebooks parallel on your machine but I advise you bring either a lot of memory or clouds machine relbench is the benchmark that I promised that I will quickly introduce and then the two tasks followed by the conclusion and the results okay but let's put everything a bit in perspective why is relational data important why is predictive analytics on relational data that that that important to us at the end relational data is fundamental to pretty much all the interesting business applications like machine learning in business applications that like many of you probably also have heard of sales forecasting customer churn prevention algo trading one of my like favorite topics fraud detection and then the good old loan default challenges or prevention strategies that then the business department has to come up with and all of those applications of classical machine learning rely on a relational data set of some sort and some extent and if you're now a data scientist working on one of those data science challenges or applications you will like probably turn rather quickly to one of those traditional or those workhorses in industry like gradient boosting models that are out there so they are they are kind of the hidden hidden champions they are still in the the algorithms that perform best in in even the most recent kegger competitions and tabular data but like also turning to the title of the slide those gradient boosting models require features and what does this mean they can only work on a single tabular data set and kind of learn the dependency between the target and whatever you as a data scientist supply as a feature column to these machine learning models and i mean them being the most of the widely used machine learning application models out there this is kind of funny because the data that we work with the data that the use cases that i on the second slide presented to you they are not based on tabulated data but they are kind of derived from raw data so what does it take to make the connection between raw data that resides in i don't know your data warehouse of choice and And then the tabular data set that these machine learning algorithms can work well with. Is it just like picking out a random table from our, I don't know, Oracle database and slapping labels onto them? Is it a weird kind of concatenation of different tables? Does this solve the problem? By far not. Some of you might already know the answer to that problem. manual feature engineering and it's difficult it's actually that difficult that some people consider it as the art part of data science which is interesting because for me art or at least i would define art as this as something different as something creative and where i'm kind of happy to express my wishes and thoughts in a different manner and not writing very concise SQL code. So you might guess the answer. For me, the art part felt more like the hard part of predictive analytics. Like given my background, I studied physics. I learned to work with Lagrangian and Hamiltonian equations. and kind of picked up some programming along my studies, were kind of pretty down when for the first time my task was to, I don't know, build a customer churn predictor in automotive. It didn't have anything to do with the kind of statistical fundament that I had. Instead, it was this big hole of like oracle tables and a database that I had to work through to kind of uncover the important factors, the important features that might have an impact on churn. And because it was that fun, I quickly want to take you through this whole experiment of building these hypothetical churn predictors yourself so welcome to the requirement from the business someone defined churn he gives you a labeled data set with your customer names and the target that you now should predict for other customers this is the statistical population it's defined on the right hand side and now you as a data scientist would typically start with feature discovery so how should the feature columns that the tabular machine learning models need actually look at and should be defined like so you look at the data that is available to you in this particular machine learning scenario and the first table that you pull out of the database is the orders table and the orders table i mean it's quite straightforward contains all the orders that this hypothetical customer trio has placed in an e-commerce store and it lists the number of items that used to be in our shopping cart and the total order volume and the problem with that is it's in a one-to-end relationship with the population table and this is something what now you as a data scientist requires you to become creative so how can we simply get it into a format that our machine learning algorithm then can understand and the trivial solution to it and you would be kind of surprised how often people come up with something like that would be a full outer join i don't know whether you have an idea what this would mean to the model but it actually will distort the population inflate it first of all and will bias it to an extent that it's just not reliable or able to transfer it to your data set and unless you're here in this case i mean the full outer join we can quickly go through the details i mean bob of course is one customer in your in your store and if we do a full outer join against the order table bob now appears three times and what does this mean this like would only work if you clone your customer bob three times before you then start the prediction and this is not what machine learning is about and then charlie's gets probably in my prediction he doesn't churn, he isn't loyal, he is just null, this also doesn't work. So given that the full outer join like the trivial or not trivial but the stupid non-working solution doesn't work, we need to get into feature engineering and these are the typical ideas that at least should immediately to my mind when I have to think about this one-to-n relationship like how to actually reduce this order table to a sensible number and to sensible features that then we can predict on one example would be the sum of orders like the the total order volume of each customer this is like a first very valid feature another very valid one would be like the average order value within the last 30 days by a certain customer or another feature candidate if you then look also into different tables and look into the item table what was actually inside the shopping cart and then figure out whether probably a voucher was applied onto an item inside that shopping cart and this could be another valid feature like the count of orders within the last 55 45 days but only if a voucher was used on them and i mean the story continues and it's getting more complicated the longer you basically deal with this or think about that problem and so it's no surprise that in real world machine learning scenarios and predictors these features can get pretty pretty long like 10 lines i probably have never seen a feature that is only 10 lines of sql i guess it would work but at least my features like easily reached two three four hundred lines of sql to like deal with all the constraints that the database places on you like you have to join the different tables multiple joints everything is nested so it gets pretty complicated and you also have to deal with things like time stamps in the example that i showed you earlier i didn't mention any time stamps so it was just an all-time churn but this typically is defined differently so it's a churn of a customer is defined as someone not placing any orders in the next three weeks something like that and this is relevant if you want to do supervised learning on it because you need to create a population that contains all kinds of historical timestamps that allow them to train for this churn classifier on different prediction dates so you need to factor in this whole timestamp component into your features as well and yes if this is not motivation enough let me like quickly go through the elements of an like typical data science project like you start with relational data you start with the feature engineering and in in in my case of course i was able to access the database but I always had to have some business domain expert at my site that actually explained the underlying process that I had to model so lengthy meetings and the result of lengthy meetings was hopefully one or two hours where when I could get back to code and write these features that I then had to compute put in a feature table in a feature store or in any other infrastructure component, then trained the predictor on it to get the output. And this is just a single run in practice. And if you didn't run in any budget or time constraints that are usually placed on top of your projects, you had to redo the whole cycle because you saw, well, you hypothesized the new feature, added it to the table, but it didn't bring any boost in performance. So it's a continuous circle of trial and error and again, if this is not motivational enough to think about automating the problem of feature engineering, there is also this topic that we don't know the features that we don't know. So we can always ask for business expertise, put it into our models, put that into our features. But if we, the data scientists, don't know what signals to supply to the predictor, like as these statistic relevant information, the machine learning algorithm will never be able to pick up on that. So this is also an interesting problem that an automation could solve. And then lastly, it's about features drift. i mean drift can be fun if you're in the driver's seat and like that kind of hobby but if you're the data scientist that watches his own features drift this always meet means that your your mobile model performance probably degrades which then or in return means that your i don't know ad spend will go up your customer churn will or your customers can't be kept and the loyalty program is probably triggered for the wrong kind of customer so it's it's it's it's bad for the business then so then you as a data scientist have to redo with the result of it so you need to redo feature engineering and this at the end of the day like not a car crash but doing it manually triggered the kind of wish in us as a data scientist and software engineers to figure out means and ways to automate feature engineering using supervised learning. And if you think about it, it's kind of an immediate idea. I mean, when we speak about predicting customer churn, we know the label, we have the outcome, we know what to train on. Also, the relational data schema, it doesn't change. It's like given, you get an export of the table and that's all set in place. And an error metric is also easy to define. I mean, it's the same error metric like, I don't know, square loss or something like that, that you would normally use to train your predictor. And our mission as GetML or with the product GetML was to automate this laborious work of feature engineering on relational and time series data. And what does GetML look like? So, I mean, it's basically filling out the one single blank that we see in every machine learning pipeline that still requires some manual work, like some work on the predictor itself. And this is this feature engineering part. And on the predictor side, we use what already works well. So it's basically a solution for finding the right features quickly together with classical gradient boosting, XGBoost, that we have implemented in a single GetML prediction pipeline so that we can really build an end-to-end model where we specify the relational schema on the one-hand side, the target on the other-hand side, and just can train it in one go. We have developed different algorithms for this automation of feature engineering. The first one, which I want to showcase you later on, the one that is part of our GetML Community Edition, is FastProp. FastProp is short for Fast Propositionalization. At the end, it's the kindly said just-try-out-everything approach. So we take all sorts of predefined aggregation functions and apply them on each valid or each joint relationship that we defined in our relational search space or data model, which I show you later on. But this is FastProp. It's comparable to what you would know from feature tools or these typical time series libraries like TS Flex or TS Fresh. And this is maybe also an important side mark. Every time series can also be considered as a relational learning problem because even a univariate time series, if you join the time series onto itself, gets relational. And this is why we have some benchmarks where we compare the FastProp algorithm to what you probably know from TS Flash X or TS Fresh. And because we implemented everything very, very efficient, I will get to that in a minute, in benchmarks it was quite easy to outperform them in terms of time needed per feature by a significant amount, like 114 to 412 times faster than these libraries, which for us as data scientists like living with that algorithm allows you a whole another level of flexibility so you you have model runtimes at the I don't know 5 10 20 minutes mark and then you can be pretty sure that you more or less are at like 95 or 99% mark of what is at all possible in terms of the overall predictive model accuracy. And then there are more advanced feature learning algorithms. They are kind of the really interesting ones. These are not the ones that I will later show you, but they basically go one step further and really generalize the concept of gradient boosting to like this feature learning approach. So enough said on that. Like with these typical data, like infrastructure requirements, it was very important to us to design the GetML software, the framework in a way that is that it's very flexible and compatible what to the the tools that are we are either required to use or that are just available to us and also external requirements so i mean automating all of feature engineering so just to start with these two can can be approached but again sometimes you have these business requirements that just want to have their kpi appear in the model and a certain important score that we should return to them so at the end we can easily build an ensemble that doesn't spend just over the automatically generated ones but also include other features that you supply from feature stores or feature scripts that you run alongside this pipeline and then at the end if we just choose get ml to generate these features it is also possible to just take these feature transforms and put them into your existing machine learning pipeline so it's more or less a transform call and then we get back a feature table that i don't know you can use in any any other pickup in any other prediction workflow and at the end observability was quite important, so we built an out-of-the-box MLflow integration, where we autolog some model artifacts and something that I would like you to show at the end of the presentation. It's also possible to get through the API to all sorts of interesting model insights that can be used and visualized later on. Because the feature learning itself is possible, like the learned features that these feature engineering algorithms discovered can be transpiled to SQL. So there's a whole another dimension of interpretability waiting for us later on all right but first let's go through the first notebook one like side mark i'm always asked like what is get email is it just a wrapper around numpy or pandas i mean at the at the surface of it we have our python api it's what you would it's a modern info interface and kind of behaves what you are used to from scikit-learn. We have this MLflow integration, and given that GetML can be used as a Python package, it's compatible to any batch prediction pipeline on any platform and infrastructure. But what is underneath the surface and what makes this whole problem of feature engineering in this high dimensionality space possible is our GetML engine. It's the core of it where all the feature learning happens, where we spent four years of engineering effort and wrote around 800,000 lines of optimized C++ code to get these six unique feature learning algorithms to work. Another important note, it's just CPU bound And because we had to basically implement all the data structures from scratch, we decided to implement native memory mapping, so we are not limited by the RAM that is available to your environment as long as the SSD is fast and large enough. So the only really limiting factor is the number of cores or the number of CPUs that are available to you but from my personal experience i haven't had the honor to work on a data set where we where it wasn't possible to like crash it with the typical machine size that the hyperscale of your choice is providing you all right another point and always and ambiguity in this domain what is feature engineering I mean this this kind of confusion especially at the beginning came up quite often there are these two terms of under two terms but feature engineering can be seen two different ways like what I just defined the feature aggregation way like really reducing and relational data set to a flat feature table this is what you can do with SQL or pandas or what else and then there are these the definition where people consider feature engineering as these classical feature transformations so we just take some interaction features so we take feature column one the already existing feature column one multiply it with existing feature column two or I don't know some polynomial feature where we square it or something At the end what would I do if I had both choices is probably as you can guess starting with the feature aggregation part because this at the end of the day is the limiting factor. If the information isn't contained in your feature set, like multiplying two different features for some interaction miracle doesn't really get you much further than, I don't know, the last percentage in your model accuracy. So, then if you want to get started yourself using GetML, I guess the easiest is if you go to the GitHub repository that is already provided in the description of the talk. There is a setup environment where like all the dependencies are readily available to rerun those notebooks. And as right now we only support like natively Linux. So if you are on Mac or Windows, then you need Docker for this. All right. So talking about the benchmark. So this was now the whole motivational part. Why do we spend time on feature engineering? Now going over to the benchmark part, we discovered RelBench. It's really difficult, really hard to find reasonable-sized data sets to really benchmark these relational learning approaches onto them. And luckily here, maintained from the folks over at Stanford and PyG, We really went through all of the internet basically and created a nice collection of data sets with predefined prediction targets and also given splits so that the typical potholes that you would stumble into are really alleviated. So here we, for the purpose of the demonstration, we will turn to the H&M database. It's a data set that originally was part, I think, of a category recommender challenge, but luckily the data set itself is able to, or it can be used for predicting customer churn or the item sales forecast as well and this is what we are now going to do so and besides the h m data set they have all sorts of other data sets that are like highly relational and interesting to work with and also in parts solved so let's start with the h m database The first task, the first notebook that we will go through is about predicting customer churn within the next week. So a customer that doesn't have any follow-up interaction in the next seven days. And the second task that I will go through second is the prediction of the total sales of an article in the next week. So one is a classification problem and the other a regression task. So how does the data set look like that we train the predictor on? This is what we basically created out of what is provided for this classification challenge. So on the right-hand side, we have our statistical population. Inside the statistical population, we have a churn column. That's the target. It's this churn identifier. Then we have a timestamp. I mentioned earlier that, like given a whole database, we can define multiple timestamps where we predict churn for, like 1st of January 2020 and then the 7th of January 2020 can be used as training data based on this data set. And then we have the relationships like the tables that we define as the relational search space. This is the customer table and the transaction table. And if we define customer churn or if you want to predict customer churn and have customer attributes you might already think well is it really necessary to aggregate over the customer table because a customer in the churn table or in the training table can only appear also one time in the customer table so this is more like an attribute dimension that we can left join against the population without the need of calculating aggregations over it because there There's simply an end-to-one and not a one-to-end or one-to-many relationship. And then the table where feature learning really is necessary, that's a transaction table. There are customers that didn't do any kind of transaction in a certain time interval. And then there are customers that had many transactions. And this is the table that we actually want to apply these aggregations on. And then there's also another table, which is the article table, which contains more information like attributes of the products that were actually bought by the customer. So like product group name, department name, the article is listed under the product code, the section number, color group code, like all sorts of categorical columns that can be used to enhance the output of the model accuracy. So, let's go into the code part, or let's start with that notebook. It's a lot of text, I keep it brief, but it's very easy to follow up and go through all the concepts on your own, self-paced. So, let's start with importing everything. I said getml it's like a python package here in this case you can install it from pip right now we are at 1.51 I think and also install our getml ml flow wrapper and then afterwards we like import everything here we load the package that is provided from the relbench guys where the data sets and the prediction tasks can be gathered from. So this is what we do here. I mean I spare executing every cell now on its own, so let's move on. Here we define the MLflow autolog setup and then we can already start with actually loading the data into the GetML engine. So as it's a relational database that we are working with, it's not a single table that we need to load. So these are multiple tables that are stored in the directory structure from that relbench package and the first table that we are going to load is the customer table. They supply it as a parquet file so we simply can use our parquet loader and directly load the data into the getml memory. And this is what we are doing here by default and this is also like a very important concept to understand, no roles are set. I mean, if you are working with a predictor, you need to specify whether it's a categorical variable, whether it's a numerical one, or in this case, it's a join key. And these are informations, what we call data annotation, that right now you as a data scientist need to provide on top of those tables that you load into the memory so that the GetML engine knows how to join these tables to each other and what are the kind of aggregations that are allowed on certain columns in that data set. So there are different aggregations for categorical columns than of course for numerical ones. So this is what we are going to do in the next cell here we use our set role function to supply the information that the customer ID is the join key in this scenario so again we weren't able to pull it in from a database no constraints were set we had to load everything from parkey where this information of course is missing and the next role that we are going to set is the age role on the age column which should be treated as a numerical column and then there are all sorts of categorical columns like FN but I don't know what it is actually and then active club member status fashion use frequency postal code these are all columns that are available and that we can use in this predictor this is the customer the attribute table next we turn to the article table it's basically the same here we are going through data annotation for the for the articles that are bought from the customer and after loading again the article table in memory and here you can see we have an article count of more than 100,000 so it's not really a small data set problem that we are dealing here we can do the data annotation in this case and as you saw in the slide and the accompanying slide there is no numerical column just categorical ones and this is what we kind of supply here in this cell and afterwards the only table that we need aggregate over it's the transaction table here we have two joint keys like the article joint key that relates the transaction table to the article table and then the joint key customer ID which relates the transaction table to our statistical population and this is the second joint relationship okay data annotation done we now can go over and load the statistical population in our memory I mean it's like an proper data science setup so we have three different sets that we are working with train validate and test these are all supplied as separate parquet files these parquet files we load into the memory of the instance and then again annotate it. In this case we used our roles class that we supply during loading so we didn't need to do the annotation afterwards but right at the moment where we import the data into the memory of the engine. Has some performance implications but let's not worry about about that right now too much. Here you can see the train table contains 3.8 million rows. So at 3.8 million unique timestamp customer ID combinations we are going to predict whether that customer will be loyal or can be classified as churn in the time period ahead so data loaded let's continue and next is the definition of the data model like we have specified all the join keys inside the tables but now we need to supply the information of how many times a specific table shall be joined to the population. It's important, I mean you may ask yourself, but why do I need to join the transaction table twice against the customer table? When working with an propositionalization algorithm like FastProp it can be useful to join those two tables over two different memory periods and this is what I will like a concept that I will introduce in the next notebook but right now it's just a single join of the so basically what we do is we define placeholders so we tell well there's a table it's called train we will add it to our data model we add all relational table store data model and then we specify that the customer table shall be joined on the population table on the join key customer ID and the relationship as I said earlier can be one to many or many to one in this case it's many to one as we don't need to aggregate over that joint relationship the next one the transaction table here we again join it on the population also on the join key customer ID but in this case we supply another very important information and this information is about the timestamps that should be respected I mean as I said earlier if you want to predict customer churn at the 1st of August 2020 it doesn't make sense if you don't restrain or constrain your features to only aggregate or consider data that was available up until then and this is what the timestamp concept is about so it puts hard constraints on these features so that there should never appear a data leak in the features that are going to be learned and i mean if there is a data leak you can easily spot it by and feature importance that is that goes through the roof so let's continue like one last table a as i said earlier there is now this attribute table on the or for the articles that are being bought and this is again in a many-to-one relationship with the transaction table And with the tables annotated, the data model defined, we then can turn over to the containerization. So as we like to contain or have our data at a single place and available, we introduce the concept of a container where we supply the actual data frames that we want to train on and that we want to predict on. So here in this case, we supply the earlier loaded train validation and test table together with all the peripheral tables. And with that being defined, we continue with defining our GetML pipeline. so here we now strap everything together we have our data model I mean if you work with neural nets I mean a data model can kind of be considered as the as the input layer of your of your of your network has to be referenced and then of course the feature learners that we want to use as well as the predictor that should be used to predict on the features that the feature learners generated so with that done we can just call a pipeline dot check i mean there are certain sanity checks that we've we found to be useful so if like join keys only exist in one table and not the other we put out a warning if we think the cardinality will cause a very long run time of this whole pipeline, we will send you some warnings and you can take the respective steps to mitigate the problem. So, for example, if the cardinality is really too large, it's questionable whether it will contain any significant or any prediction-relevant information in the first place. So, we just can leave the column out. And this is what we do here. So on the previous specified data tables, we now call the set role function again, where we said that the postcode is probably not relevant. So we give it the role unused string so that it's not considered in the training and the longer. And then after a simple pipe.fit, everything is done. Here in this case the runtime turns out to be 3 minutes 45 and like afterwards we can call the score feature where we this time supply the validation and the test set. And at the end of everything, we have this little overview here where we tell you what kind of accuracy or AUC in this case we achieved on those respective test sets. And yeah, that's it for the training part. But as I said earlier, these feature learners that we implemented can learn a logic that we can transpile into SQL. And this is very interesting when it's about model interpretability, about feature interpretation. And this is what I will now show you next. So, first of all, and pretty straight on, everyone who worked with gradient-boosted decision tree models knows the concept of feature importance. So in this case we can of course calculate the feature importance and say, well, normalize to 1 of course, that 58% of the importance basically falls onto one feature. It's called feature 139, one of those many features that we learned. And what is feature 139? This is the following function. Again, it's just a transpilation. We didn't like re-implement a SQL database in the GetML engine that runs under the hood. It's really just a transpilation of the logic that we learned. And in this case, we see, OK, it's account distinct of the timestamps. And if you now take a closer look at the column T underscore dot and this count distinct logic that is applied on, we can immediately conclude that this feature is kind of a proxy for the customer or for the overall activity of a customer, which absolutely makes sense. I mean, you are less likely to churn if you're shopping often with a certain company. And this is now exactly what this feature uncovered. So it's about separate shopping days a certain customer had. And then if we take a look at the second most important feature, here we are already down to an importance of just 11%. It's a trend aggregation. And this is a feature that can be an indication of the momentum or the recency in a customer shopping behavior. So if the shopping event is rather long ago, this might have a different impact on the churn than if it's a shopping event in the recent past. So this is kind of the first insight that we can provide and the second interesting concept is that we can generalize the concept of feature importance to columns that we used in our data set. So we can propagate the information, the feature importance back to the columns that are part of the feature. And this then will give you a good overall understanding of the importance of the different columns in your data set, which then helps you to, I don't know, further tune your data pipeline, your extraction processes or scripts. So that if time is of the essence in those models, we can really just extract and train on those columns that matter most. And this is what we call column selector. We also implemented a logic for that. So that's it for the first example. So it's about customer churn. and let's continue with the second one. It's the, this is a quick rundown of the pipeline. Again, we just loaded a parquet file and defined the GetML pipeline to classify chown node churn. The second notebook that I'm going to demonstrate is a bit more complicated. I told you earlier about the integratability of GetML and that you can use GetML to transfer or to transform features and then use it in your own prediction setup. This is what we are going to do in this item sales forecast, this regression problem. So we first just build a base pipeline where we start with a minimum data model and an XGBoost predictor to get to a result. Then we do some tuning, and then I will show you how to extract the feature columns that FastProp learns and uses these feature columns inside an Optuna light GBM pipeline if you really want to go for an optimal model accuracy. So the data set at this time kind of looks the same. I mean, the raw tables look the same. of course the population now are defined differently so let's quickly head over to this notebook hmn item first of all i mean in this notebook we we already made use of some some shortcuts it's like the more advanced notebooks of both so if you work through them please start with an hmn churn notebook before you turn to this one here in this case we assume that we already know the columns that are that exist inside these parquet files so we don't do an like full discovery here we just specify the roles for all the different tables that we are going to load like population transaction customer and article and then just go over and use this like shorthand to quickly load all the train test and validate test set in a dictionary and then also load the customer transaction and article table in getml with the already or predefined rules dictionary so in this case data loading wrapped let's continue with the modeling aspect so here we define the base model so just the very like the easier setup of both. So here we join the transaction table on the population table. Again we want to predict the item sales of like these 100,000 articles in the next seven days and as we expect some strong autoregressive component it's of course really important to join the sales table onto itself like transaction table onto population in this case the join key is obvious it's article id not custom id any longer and the timestamp our timestamp constraints are also important here as we don't want to train on data that we shouldn't know at that point in time or historic point in time then there's this other parameter that we use it's called memory and memory truncates the maximum information that is available from that peripheral table. So you can think of it as an upper timestamp. So whenever we join or when this join relationship is activated or used, we just join over the most recent six weeks before the date where we want to do the prediction. And then, again, we have these many-to-one relationships, so just attributes that we here join from customer on transaction and then from article on transaction. This is what the data model looks like. So here's our population table. This is the attribute table, so just, again, the article attributes that we can just left join against the training table. and then this one aggregation joint where we have set a memory of the last six weeks or 42 days. So this is our base pipeline. Then we just continue as you already saw in the last notebook. We specify our pipeline. We supply our data model, define the feature learner and the predictor. In this case, of course, we need to use a different loss function. Then we do the fit and the score part, and we get to these results. When I looked at these results for the first time, I saw this quite drastic drop, especially in the mean average error, what we want to optimize for here in this challenge, from 0.47 to 0.7. So mean average error, again, this should be as low as possible, and a drastic increase like that suggests that we didn't capture enough of the information and that the model wasn't able to generalize well over the unseen test data. So this is what we want to fix next. And the solution to that is we just add another join, like a second join, to our data model where we not use a memory parameter or memory of six weeks, but just instead of one week. So in this case, this join will deliver plenty of features that help then the predictor to get a better understanding of these recency effects towards the item sales prediction. And with that applied, we also have some timestamp preprocessors. So right now, if we set up the timestamp role on a column, it's just a numerical value that we use as limiting boundaries in this join. But something that you normally also would do is extracting like week or day of week components out of the time step that you have at hand. And this is what the seasonal preprocessor does out of the box. So we just add a seasonal preprocessor to our whole data model. Then, and again because of memory and things that you discover when you run this notebook yourself, we exclude the seasonal preprocessor on a column where it doesn't make sense. Then we need to sync the container again so that we don't get a warning that data was changed after the creation. And then at the end, and also a very important element, is adding more aggregations that FastProp is allowed to try out. So different aggregations have different computational complexity constraints, and the default aggregation set is limited to the faster ones. And if we know that it's a strong autoregressive component in the time series that we want to fit or predict, then we add additional aggregations to it. The n most frequent parameter is also very interesting. In this case, it's like the kind of aggregations that you can apply to categorical columns and numerical columns are, of course, different. Yeah, there's a question. So let me repeat. So why is there a relation between the memory and the aggregations? I mean, these are the aggregations that are applied on this time window. So if we put a memory of one week, we basically specify that we have a sliding one week long window. And on that window we apply all the aggregations that you see down here. So this doesn't make sense for the one week window, it only makes sense, and this is what I skipped, if there is this six weeks memory window in that data model as well. Good catch. And yeah, as such, let me continue. So here we just pull up more aggregations, predefined aggregations. The n-most frequent parameter, yes, right, very important. We have some numerical and categorical columns, and of course on numerical columns different aggregations make sense than on categorical ones. I mean, categorical columns count distinct as a good candidate for it, But if you want to try to gather more information from certain categorical columns, we create additional dimensions for that. So it's like we apply an aggregation to all columns of a table, but only if a certain categorical column has a certain categorical value. And this blows up your data set drastically and plenty more features will be generated because we create these extra feature dimensions for, here in this case, the two most frequent categories in this whole data set. And then another parameter which we don't tune or which we don't change here in this setting is the numFeatures parameter and this just tells FastProp that it should generate or just should return those 200 features that have the highest correlation with the target. And I mean if you look at this as an ensemble learning problem it absolutely makes sense. I mean we can't of course be sure that there aren't any cross-correlations but it gives you a good first estimate and you can always do more advanced tuning later so you can return like uh 2000 features and do your feature selection using traditional methods like the the earlier mentioned feature importance this is something what we also do and what is also part of the getml pipeline but something that we don't use here all right so um enough on that pipeline right now so then we just call again a fit on the pipe refined this time we can already see that the mean average error dropped from 0.07 to 0.04 which is already a good improvement so this additional join help the predictor to or help to generate a feature set that is better able to help in predictor generalize over the different train and validation sets that we use all right and now begins the exporting part and this is where we do it in a very lean and memory efficient way but at the end it's just one column that matters and this is the pipe.transform column line and pipe.transform basically chooses requires the split the data set that should be transformed so here in this case we iterate over the train val and test set and generate feature transforms for all of them so So at the end we are left with those transform exports, I mean we can look again at the size of the population, I think it was 3.8 million or something, so 3.8 million times 200 feature columns, this is what the last line there will generate. And then begins the external tuning, this is what I would like to skip at that point, I mean, the script is also supplied, so we have this, one second, the H&M item tuning script. So here we basically just go through the normal Optuna and light GBM setup, we load in all the data that we earlier exported also set the right like put it into the right data frame and then we here in this case also interesting what what I said earlier we can always use external supplied features in the predictor and this is what we did here because XGBoost is now known to work less good on categorical features than LightGBM because XGBoost needs to want to encode all the categories of attributes in your population. LightGBM instead has a better way of handling this and this is the reason why we didn't supply all the attribute, the categorical attribute features to the GetML XGBoost predictor in the first place, but instead just load those categorical attributes of these items here in this final HyperOp setup. So we loaded everything back into memory, now we have our X and Y train, val and test data frames and And then it's just an Optuna optimization routine with 50 trials and a hyperparameter range that is supplied from the benchmark. So now getting back to the benchmark and results, we basically compared the scores of those pipelines that we generated with GetML with the scores of the other known solutions to feature engineering come up with. This whole relbench set again stems from Stanford and one of the backing parties behind this is PyG. Of course you can solve these relational learning problems also with graph neural networks. Some of you might have already heard of that. so this is the setup and this is now the benchmark that we quickly will discuss and just to finish up on the hyper optimization this is because it's similar to what a human would do those hyper parameters that the human here in this case used for his scores of the predictions and so at the end I think everything is quite comparable to each other and what we what we see here is that our propositionalization approach together with an traditional machine learning with an XGBoost predictor are working comparably well, like if not best, compared to like a machine learning pipeline that a human would train or something that even a graph neural network would come up with. So at the end, and also leading over to the conclusion here, manual feature engineering still works well. I mean, we didn't reinvent the wheel. If you have enough time and no budget constraints and are happy to dig through relational and dig through processes to understand where the relational data come from, then of course this works. Graph Neural Networks are there to solve the same problem, but entirely different. I mean, it's a completely different setup. They always build on top of PyTorch, often needs some heavy GPU power to be trained on. And then lastly, not compatible or comparable at all with what you would normally would be used from if you had data scientists or if you would go the manual feature engineering way. So this boosted relational learning approach that I presented and that we implement is at the end a very strong solution. and without compromising interpretability and compatibility. And the interpretability aspect is like the last slide that I want to show. It's not a slide. It's actually something interactive. I told you that these... One second. It's possible to transform features, like transform the learned features into SQL code. So with all this text-to-SQL success out there, like using LLMs, we thought, well, let's try to go the other way around. We have the prediction problem. We know what the setup looks like. We have summary statistics of all the columns. So let's try to have an LLM explain me as maybe a business or even as a data scientist, explain back to me what the feature set that was learned actually picked up. And this also works very well. This is like just one of those apps that I showed you earlier where we just have a streamlit instance running that gets all the features, the learned features from the engine and puts it through an own interpretation cycle. So this is not written by a human, but instead just generated based on the features that we learned. And this at the same time is already like a hint to the more advanced feature learning algorithms that I haven't shown at the moment. It's called multi-rel. Multi-rel features can be much, much more complex. But still, the SQL code that they generate can be easily translated to quite self-explaining feature names. So here in this case, it was also customer churn set up from the famous AdventureWorks demo data set that Microsoft put out there. And they also have a hypothetical bike store that sells bikes and services and predicting customer churn is one of the other tasks that we did there. You will find the same notebook in a different repository from us. And putting these features through the interpretation agent, I mean, let's start with the summary page. First of all, it gives you a good understanding of the feature importance, like the min days order to special offer is the most important feature. And here we have our correlations. Sorry, it looks a bit cramped on this screen. usually the bars are a bit nicer and if we dig deeper and just not look at the ensemble overview we can also visualize what each of these features learned or came up with so the min days order to special again explained from an LLM is the minimum number of days between the order date and the associated special of a start date apparently in days considering specific conditions on due dates difference mapping and shipping method so I think it's maybe opening up and really another interesting dimension to make like end-to-end machine learning on a relational data way more transparent and explainable by just basically transferring everything that we know to works well on tabulated data like gradient boosting first of all to relational data and make it learn aggregations as you can see here and in the second feature again we can also plot density distributions where we see the frequency of a specific feature value gives you an understanding if you dig deeper into the into the sequel code here or want to know what the target average at a specific feature value is. And that's it for my presentation. Thanks for your attention. Thank you so much, Mr. Alexander. A lot of you submitted a lot of questions on our Slido forum. So, if you still haven't submitted your question, please submit it. We will start now with the Q&A session. So, the first question is, which is the most popular one. How does GetML handle temporal relationships between features? I have to think about that for a minute. so temporal relationships between features I mean at the end they need to work well as an ensemble so if we go the propositionalization way we don't really have an influence on that so we just apply all the aggregations that we have on all the columns and look what basically sticks these are the important features that we then put to the predictor and this kind of answers I think already the first half of the question if you think about the more advanced feature learners that we use and I mean there there it's an the question about an iterative improvement of the error function so at the end we have a loss loss that we can calculate and we can try to iteratively grew through these boosting and begging approaches that everyone knows these feature sets that capture most or try to capture most of the effect so in this case we try to avoid any kind of correlations by design inside the feature sets that we learn but at the end it's it's left to be seen yeah maybe I need to get a more a better understanding of the often of the actual problem yeah thank you second question is how does get ml scale when working with big data 1 billion plus rows it is often necessary to directly compute features in the database? Good question. I mean if we try to alleviate a lot of that with this memory mapping approach so that it's just CPU bound and if you have like one billion rows in your population, I mean my first question is does your organization have enough budget to run the query to get that data out of the So we would probably simply start by sampling. But this is like normal data science practice. This isn't like directly... Has a direct implication to what we are doing. Understood. Next question is... Is there a built-in feature selection functionality in getml? Yes, there is. Good question. So if you just go through our... API documentation, we have our pipeline class and inside the pipeline we have a so-called feature selector. sorry yeah features a lecture so there we go we simply use the same XGBoost classifier that we also use as a predictor usually with a different set of hyper parameters and first train it calculate the feature importance and then take the top 10 20 50 percent share of these features according to the importance calculated by the first feature selector and then we retrain it on the whole data set like on the on the remaining features thank you next question is could you please comment on over overfitting especially when a large number of features are generated or learned I mean overfitting occurs it can be easily spotted if the the performance on your train data set is much better than on your test on validation data set in this case I would just go down the normal hyper parameter optimization route and just find the parameters that regularize the models in a way so that they don't overfit next question is is Is distributed compute automated feature engineering on the roadmap for big data scale data set use cases? I mean, it will be on the roadmap as soon as we get to the first data set where we need a scale-out solution for this. So, you probably should email me. Can you also apply this to hierarchical data? Hierarchical data? Good question. I mean, starting with the Postleitzahl code that we have here in Germany, there is this hierarchical structure to it. And we have other preprocessors that just apply basic string operations to it. So just take the first three digits, two digits, one digits, and then create additional columns in our peripheral tables that represent these hierarchical grouping. ICD codes is another famous one with very, very high or complex hierarchy. And again, just splitting on certain parts of that string does the trick. Next question is, does GetML work if the column names of my data frame are non-speaking? Absolutely. And the beauty is, it works even if not the column name, but also the column content is unspeaking. I mean, this is the beauty of this automation, that we as a data scientist, as the one being in charge of finding these features, doesn't necessarily need a translation of whatever cryptic attributes people tend to give certain things in their database. our next question is how granular can be can the time series be and does it support time zone aware timestamps in case of summer time basically our timestamp parsing is quite popular but again you can break it so it depends but this is not a Typical feature learning problem here. This is more in data cleaning and Efforts that, of course, you need to put into your data first. Next question is, are the sql queries generated using an llm? When knowing the database structure, it would be possible To parse queries without usage of llm, making them more reliable? It's important to understand that we just use the LLM to try to translate existing features. So we learn them, we create them. It's a very deterministic approach that we use to get to the logic. And this then is interpreted. And of course, there all the hallucinations can happen, all that happens. Next question is, what is the limitation of GetML, if any? You can break any machine learning pipeline, I don't know. I mean, I would really love to learn more about its limitations, but given the limited test data or relational data that is out there makes it quite hard so maybe also appeal in the community more relational data sets are highly appreciated and maybe we discover some new inspiration for for features that are that we haven't thought of yet next question is is or will there be shep integration Actually, wait one second, it's experimental again, it was more or less a hackathon project that we did a while ago. We already have something like that. So if you translated your feature logic into more or less speakable feature names, then combining this with these popular SHAP frameworks helps to get a better grasp of what you're doing. So, again, orthogonal to what we are doing and fully, yeah, compatible to the approach we are solving feature engineering and learning. Next question is, can column roles be detected automatically? We wrote all sorts of scripts for that, but it's not yet part of our API. And then we still have two more questions. Where do you see the future of feature engineering heading? More automation or more customization? Like define customization. I mean, these traditional feature concepts, those KPIs that businesses care about and that we already have plenty of them in the current feature store landscape out there, they have their absolute relevance. But I think if we are in a scenario where predictive performance is key or where the time that can be spent on an end-to-end machine learning pipeline is very limited, I think we provide a good framework that gives you a good compromise of this. That was our last question, actually. So thank you so much, Mr. Alexander, for this amazing presentation and for these insightful information. Please, everyone, a round of applause for Mr. Alexander, and we hope you like the talk.