The foundation model revolution for tabular data

Speaker 1 [10:02]

Hello again, it's me again, session chair, my last session today, and today I wanted to present our next talk, Foundation Model Revolution for Tabular Data.

Speaker 2 [10:13]

Our...

Speaker 1 [10:15]

The person who's going to present it today is Frank Hutta. He's the CEO and co-founder of Prior Labs. He's also a Hector Endowed Fellow and PI at ELIS. He's also at ELIS, and he's also at the Institut Thuringen. He's a full professor of machine learning at the University of Freiburg. He holds a PhD at the University of British Columbia, and he has three ERC grants. Frank is best known for shaping the field of AutoML, and he has a long long list of papers that I would recommend to read and his focus has been on tabular foundation models and you probably heard of TAPI-FN that's one of them so for you now, Frank Hutter

Speaker 2 [11:06]

Yeah, thanks a lot for the introduction and for the opportunity to speak here. Very excited to speak to you. And if you take one thing away from this talk, I would like it to be that the tabular space, tabular prediction is really undergoing a revolution. Deep learning claimed to work on tabular data for the last five years. And it was always a failure. It worked on one data set and then was the worst on all other data sets. And this has stopped to be the case. And in the last year, there is all these new methods here that actually outperform sort of your standard CAD boost, XGBoost, et cetera. And this used to be the case for 10,000 data points. Now it's the case for 100,000 data points. At the end of the month, it's for a million data points. And so it's really very quickly advancing. And the good news is that these benchmark results actually translate to practice. We published WFN2 in Nature last January. It has 929 citations by now, and out of these 207 use cases actually, yeah, there are 207 published use cases where we had nothing to do with that, and people showed it's state of the art for domain X, Y, Z, et cetera. A lot of these, like 94 in the medical space, that's where people picked it up first because medical data is typically small and the first version was limited to 10,000 data points. Finance it's only nine because people in finance don't publish. We have a lot of traction with banks and insurance and so on. Energy utilities, manufacturing and really others, it's completely horizontal just like LLMs. LLMs are amazing for unstructured data, no matter the domain. Just like that, tabular foundation models are amazing no matter what the domain is for structured data predictions on databases, C3 spreadsheets, et cetera. In contrast to LLMs that are really good at language and knowledge, they're good at numbers and statistics and patterns, and they directly plug into the ML value chain that people have have been setting up for the last 50 years. They're tackling this boring binary classification problem, multi-class classification regression, et cetera, that sits at the heart of so much value. So for example, credit risk assessment, binary classification, designing clinical trials, seeing which patients will actually respond to this drug, churn prediction. I could go 270 use cases. I will not do that. Rather, I will show how this actually can change machine learning. So the typical design cycle, data science cycle, you know, you take a data set, you clean, do model selection, feature engineering, model tuning, and then there's the model deployment, maintenance, governance, etc. And next data set, the same thing happens all over again. And when we speak to our customers, very often it is the case that people take three to six months and a dedicated team for one of these tasks. Maybe the people in the audience will do this faster, but there is typically zero transfer in terms of the methods, and a lot of the engineering time actually goes into the maintenance. You have these models flying around that an intern built three years ago, and you don't know the data anymore. You don't know how it was tuned, and then the data starts drifting, and the model is not good anymore, and what do you do? And that can be replaced by just one foundation model. We have one company that has replaced 108 different models with TAPI-FN. There is no more models flying around. They don't know where it comes from anymore. But all you need is the data. You put that in context into your model, and that makes predictions. And it's the same story as for LLMs. there were all these bespoke computer vision architectures before, all these bespoke NLP systems and all of them are gone it's just foundation models now and just the same in the data science space and I really do hope that this will enable data scientists to become sort of the 10x data scientists that you don't need to do all this stupid stuff anymore that very few people like doing, like tuning hyperparameters and so on And, you know, I have the background in AutoML, and this is sort of just a natural convergence of AutoML. We built PriorLabs, that's my only slide about PriorLabs, to win in tabular ML. We have an amazing team on the science board. We have Yann LeCun, our founding advisor, Bernard Shulkov, number one in the world in causal ML, which really we need here because in the end we don't predict for prediction's sake, but actually to act in the real world, and that's where we need causality. We have Philip Singer, who is a legend on Kaggle, who's number one there. Nick Erickson, who's a legend in tabular AI, built Autogluon at AWS, and a lot of other really great people. But I hope to have motivated you to listen to the rest of the talk. What I want to do next is talk about TPFN, sort of set the basics, and then also focus a bit on trustworthiness, because this is revolutionizing the space, and trustworthiness is of course super important in data science in order to really revolutionize also the market and the value chain. And also agents are super important, Sebastian talked about that earlier, and I want to show how TapiaFN can interact with agents and show some demos, time permitting. So TapiaFN. V2, that was last January. As I mentioned, published in Nature. It's actually Nature's most cited AI paper in 25, and 24 it was AlphaFold, and widely regarded as sort of this ImageNet moment for tabular data. In terms of results, yeah, here's LightGBM, XGBoost, CutBoost, in the default and in the tuned version, tuned for four hours, and still like TPFN in a forward pass here, on average 2.8 seconds is better than these four-hour tuned boosting methods. How does this work? So this is one slide on how does it work, if you're interested in that. It's all in context. So you have your training data, X train, Y train, and then a test data, X test, Y test. Y test you take out of a data set and feed all the rest into the WFN, which is a neural network, it's parameterized by theta, that makes predictions for the y-test in one forward pass, and then you just look at how likely is a y-test under that predictive distribution. And that is what you back-prop on and actually optimize the parameter theta in order to get better predictions. Then you throw away this data set, and you take another data set and do this again and again and again, and you do this like hundreds of millions of times. So one data set gives us one back-prop. It's not that one data point gives us one mini-batch, we actually do mini-batches of data sets. So this is important, it's all in context, you push the entire data set in, and that's actually exactly the same that on the right you do when you have a real data set. So on the right, that's the first time you see a real data set, on the left everything is synthetically generated, and synthetically generated to create data sets that look like the ones that we actually are likely to face in practice, so we base this on actually a lot of principles from causality, structural causal models. We also put in missing values, we put in outliers, we put in all the data complexities, imbalance and so on that we expect to have to deal with in the end of the day. And then this one model actually works across all these thousands of different applications. The architecture, it's not a standard transformer, but it's similar to a transformer. There's a tension pattern, so where you do a tension across the features, and then you do a tension across the samples, and then, yeah, an MLP, and you do this in the case of TPFNv2 12 times, and then you read off the predictions for the white test. You can get a multimodal output distribution, you get the full predictive distribution, And that's also very well calibrated. Qualitatively, for some scikit-learn toy data sets, this is the types of predictions you get. So away from the data, you are uncertain. And the predictive surfaces are very smooth, in contrast to, for example, cut-boost and so on, where you're very axis-aligned and you have these jumps. It is super simple. I love AutoML. Still scratched out here, the AutoSQL Learn picture. You don't need the complexity of Bayesian optimization, of ensembles, of meta-learning, et cetera. At test time, at meta-train time, you have the meta-learning, but at test time, when you actually get a new data set, you just do a forward pass, and that's all. And you can take this type of event, you could put it in ONNX, put it on a sensor, and then on the sensor, you actually have a machine learning algorithm. It's not just a classifier, but it's a new algorithm that you can feed entirely different data sets, a medical data set, a finance data set, anything, push that through the sensor and get predictions. And, of course, there's a lot of tooling for machine learning, neural networks. All of that applies here. It's a foundation model, so you can fine-tune it to your own data, just like you download a LAMA and fine-tune it to your language data. You can fine-tune this to your company's data sets. Here we have an example of sort of there's a sine curve. that is not known, it's just the orange points are known, this is from the Nature paper, and the blue is a prediction of the TAPI event, it's not perfect, but then after fine-tuning on a couple of sinusoidal curves that are not this sinusoidal curves, it knows, oh, sinusoidal patterns is something that I should know, and then actually predicts much better. You can also fine-tune on some real data sets and get sort of this real TAPI event that is yet a bit better. All right, there's further foundation model capabilities like density estimation, data generation, and the embeddings are meaningful. And, well, that was TPFN 2, and we've been building. And TPFN 2.5 came out in November. That clearly outperforms V2 again and also scales, in this case, to 100,000 data points. So, yeah, TPFN V2 was here, and then, yeah, TPFN 2.5 here. and real TPFN a bit better. And what we also did in that version is to show, to introduce distillation because some people in regulated businesses, for example, need to serve trees, and then we can actually distill into trees and also serve trees. And actually, yeah, distill this TPFN model into a tree ensemble, and the tree ensemble is still much better than individual trees fit on this new data set. And why can we do this? It's because, well, the TAPI-FN has learned not to overfit, so it's a better predictor, and then we basically can have an infinite number of data points that we generate in order to say, well, for this data point, I should predict what TAPI-FN predicts, and so I can train this tree with infinite amounts of data and can actually get better than training on the original data set. And the same with MLPs. There you can actually also put this in an output layer and get this in a forward pass. We also introduced scaling mode, which is sort of, you can think of basically as a rack-based method and also scaled this up to 10 million data points. And at the end of this month, we are releasing TapioFN3. This will natively actually handle a million data points. And then with scaling mode, we basically can handle more or less infinite number of data points because you don't have a billion data points set as IID, and then you need a lot of different mechanism anyways. So while we started with a WFN Nature paper with 10,000 data points, that was clearly what we heard in the market is what people wanted fixed the most. And so we've really focused on that. And here's a sneak preview of our current results in the API. We're getting much, much better. And this is the deep learning revolution. It's just exponentially growing. Which brings us to trustworthiness. This is going to, like TAPFN and other tabular foundation models, will be the future of tabular data, and we need to make sure that we can actually trust them. And actually, I also want to argue that we can already trust them quite a bit. Because for example, we have smooth and well-calibrated predictions, and I would say what makes us think that this tree ensemble is particularly interpretable. There's all this, you know, uncertainty estimates are not super calibrated, and it's very jaggedy, which also leads to SHAP being sort of all over the place with cutboost, and for TPFN it's smoother, and then you actually get much, much clearer patterns in SHAP, so you get actually clearer post-soc explanations. Again, if you really want trees, we can give you trees, but we can also give you GAMS. We have a paper with Andreas Müller that outputs generalized additive models in a forward pass. We have a paper on symbolic regression that is also trained sort of in this PFN style. There is a lot of things you can do with this. And I do want to take the time and contrast this to the previous deep learning that did not work, which is sort of, yeah, there was a bit of a fight between the data science community and deep learners because deep learning has claimed victory and they didn't have it. And so deep learning has a bad rep, and I'm trying to say why this was. It was slow. Data scientists don't want a slow method. They want a fast method. They want to iterate and tinker with the data, doing feature engineering and all that stuff. You need to be able to do that. Deep learning used to be brittle. You would have divergent runs and so on. You just can't have that in a production system. You needed to do hyperparameter optimization, learning rate tunes, and so on. It would overfit a lot on small data, and also generalize poorly, and all of that is now gone with in-context learning, where it just works out of the box with a forward pass. There is no need for hyperparameter tuning, and it actually works particularly strongly for small data. In the Nature paper, we showed that it only requires half the data to get the same performance as the next best model. And that actually, in sparse data regimes, like in medical data, for example, if you have a patient cohort where you only get away with half the patient cohort, the societal value that comes from that is huge. You get medicine to the market faster. there's also a lot of money saved in the medical system that you can actually use and put to good purpose. You could use LLMs for tabulate data, but it just doesn't work very well. Also, they hallucinate and they're non-deterministic. So those are criticisms that don't apply to TPFN. And deep learning used to have issues with interpretability and fairness, etc. And with TPFN, you can actually do feature importance in the forward-backward path. You could do data point importance. How important is this data point here for the classification in a forward-backward path? In order to do that with gradient boosting, you would have to retrain n times when you have n data points. And it's also state-of-the-art for robustness to data drift. We have a NeurIPS paper in 24 for counterfactual fairness, ICML paper last year, and for causal reasoning. And for causal reasoning, I want to give a few details. So we often do make predictions in order to act in the real world. And this acting, that's intervention. For example, for pricing, we set a price and then demand actually reacts to that. And there is correlation between the features. We can't just use a machine learning model that doesn't know that this is an intervention. And so likewise, yeah, if I want to judge the survival chance of a patient, I can't just say, well, what is the probability of survival if I observe that this patient gets a surgery that is different than if I decide that they're going to get a surgery. One is an observational data point, one is an interventional, and you get systematic biases for any type of standard machine learning method, including TAP-PFN. And with doPFN, we get around that. We do this by actually doing, in the pre-training, we generate the data once in the observational way, and then this T here is intervention, we actually cut the incoming edges to the T and regenerate the data, to regenerate interventional data, and then we learn to take observational data and predict interventional outcome distributions. And that actually allows you to do the same thing you can do with randomized control trials, with A-B tests, et cetera. I believe all of these fields will be entirely revolutionized in the future based on these tabular foundation models combined with causality. Yeah, this yields data of the art. I can go quickly here. You can also combine this with knowledge where the data scientists might know A causes B, and then you can actually put this as an input to the model. I know that A causes B. Please give me the basal and posterior overall structural causal models where A actually causes B and not the other way around, and get better predictions that way. And yeah, again, this is all state of the art. I do want to show some demos, because we all know and love LLMs, and we don't want to say, oh, LLMs are bad. Data scientists love data, love this, because it can do all the feature engineering or the data cleaning and so on. There's so much you can do with LLMs and the genetic systems. But at the end of the day, when you actually do the predictions, you shouldn't use the LLM. And so here we have a demo from our product person, Diana, who is here at the booth also, where we have a demand forecasting CSV on the left. Claude is doing this natively on the right with TAPEF and MCP. And the LLM is great at formatting the data, making a training set, making a test set, et cetera. Then on the right here, we have formatted it. Now it's calling TAPEFN. Now that's done, yep, no, actually now it's uploaded, now it's calling tapiafn. So the tapiafn part was very fast, gives us a good R squared, and on the left you see this tabular blindness of LLMs where it's actually great training performance, but the generalization just isn't there. Philip Singer built a great demo for submitting to a Kaggle competition, so he just says Playground series S5E3, get the data from a Kaggle competition, train TPFN on rainfall target, on full train data, predict test, and make a submission with probabilities, submit this to Kaggle. This demo usually runs two minutes, it's sped up here, but that's all it requires. And of course, Cloud is good at doing all the orchestration, but then the actual predictions, it can use TPFN. Yeah, we integrated this in Databricks, so in Databricks, of course, you have all these different dashboards, and Databricks has this Agent Bricks that is really cool. You can do all kinds of things with it. For example, here you have this sales analytics assistant where you can actually chat with it. tell me which products actually this customer is most likely to buy next, and that then actually builds a training, a test set, runs TapFN, makes predictions, and yeah, you can iterate like that. And there's a lot more to this demo, but we can show that in detail at the booth. You can try it yourself, 15 seconds until first prediction if you go to ux.priorlapse.ai, You can sign up in with your Google account and just directly go to this playground and try any of our data sets that are in there or also upload your own data sets. And give it a shot yourself and see if it works for you. And if it doesn't work, please let me know because I would be surprised. And we should know and actually improve it. All right. With that, we are hiring. We're an exponentially growing startup. We're 30 people. to be 80 people at the end of the year. We hire data scientists, dev rels, AI scientists, engineers, product people, everything. If you refer someone, we pay you when we hire that person. If that person that we hire is female, we pay you twice this. We're moving to the same office as Black Forest Labs. And those are the takeaways. I want to have some time for questions. Thank you.

Speaker 1 [33:16]

Thank you very much Questions lots of questions, but I'm gonna take the one that are highest ranked and for the rest You can approach the team in the booth. Yep

Speaker 2 [33:25]

Yep. I'll also be at the booth for a bit.

Speaker 1 [33:27]

So the first question is, from what we understood now, TAPI-FN is trained on dataset, and that dataset can sometimes include cliers, missing data, dirty data, unclean data. So TAPI-FN supposedly doesn't need to get clean data, and can you explain how TAPI-FN handles such datasets that would cause other models usually to fail?

Speaker 2 [33:50]

Yes, so if there are missing values, then you know top FN one would just be really bad But then because it didn't know that missing value is what what is this but in top FN? to already in the data that we used for training we just said well missing value that's going to be encoded as none and then just That had a particular embedding that was learned for a nun and and it's just deep learning magic It learns what to do with this in order to, in the end, get the best predictions for my test. And that is, in the end, the training signal over hundreds of millions of data sets where there are missing values. The objective function is optimize my cross-entropy loss. So you are penalized for bad predictions where you're confident about. And it's not autoregressive. It just makes direct predictions for the entire data set. And as such, it's just, you know, we do use the deep learning magic during pre-training in order to actually automatically, so it's really the epiphany of AutoML, come up with the right method that can take these complex inputs.

Speaker 1 [34:55]

Thank you very much. I hope it was clarified. Next point. Does it make sense to pass only a subset of data to TPFN? I think what the person means here is usually if you have millions, then it doesn't really matter if it's 100,000 or 50,000. And the question is, how do you pick X train and Y train on very large data sets?

Speaker 2 [35:15]

Yeah, I mean, back especially when we had the Nature paper with 10,000 data points, people are like, oh, I have 50,000 data points, so I will use 50,000 versus CatBoost and 10,000 versus WFN and then compare them. And while we showed that we only need half the data, well, that would be 10,000 versus 20,000 and 50,000 would actually be better with CatBoost, right? So just taking subsets is actually not a good idea. That's actually this plot here. So tap EF and 2.5 Here is with sub sampling and so that's 1% of the data 10% of the data It still improves but then a hundred percent of the data you just sub sample and you get the same and that is bad But with this rag was actually on that that is proprietary It runs on our end, but it picks the right data the right context for on the particular points You do see that it just continues to improve So we do need the entire data set in order to pick with Rack what to put into the context.

Speaker 1 [36:19]

Thank you. Next question is how does TAP-PFN compare to time series foundation models? So I think the main point here is like, yes, for classification problems, it works and we're seeing it, but how does it work for regression problems?

Speaker 2 [36:31]

question problems. Yeah, for the interest of time, I didn't have the slides here, but we also benchmarked this on time series that we had in last January. We were number one on Giftywell by just, Giftywell was then the most prominent time series forecasting benchmark by Salesforce, and it was better than Kronos and Times FM. Those were the most recent ones of that time on Giftywell. And this was simply taking the index and sort of then some simple features like time of day, time of day of the week and so on, putting sine and cosine features and casting it as a tabular prediction problem. And it was the model from the Nature paper that had never seen a time series data set before and that's never seen an actual data set before, only synthetic data. And nevertheless, it generalized to that. We haven't focused on time series yet, but it's very much on the roadmap for this year.

Speaker 1 [37:28]

Very exciting. Next question. How important are well-described columns and names and text in the, like, in the name of the columns? Is this something that the TypeEF intakes in itself?

Speaker 2 [37:39]

And so far, only in the API, and then the API, yeah, it is very helpful. It's for the open weight model, it actually has a scikit-learn interface that is the same interface as XGBoost and just does not even look at the column header. So all of the results I showed today, except the API result, does not even look at the column header. We can also handle text within the table, and actually also have an ICML submission on that, too. And there, using LLMs makes a lot of sense in order to get the right embeddings, and that does help a lot.

Speaker 1 [38:22]

Thank you. Last question. What are the main limitations of tabular model in general? If I understood correctly, it supports regressions and prediction of known values, but if it is predicting a color column, it will never predict yellow if only green and blue are in the training data. So can it predict, if I understood correctly, something that doesn't exist in the table or maybe emerge?

Speaker 2 [38:47]

Um, yeah, I mean, it, if you don't, if you've never seen something, for example, during training that is in your X train, so if you have seen green, yellow, et cetera, strings, it can generalize to new ones, but for the test, it cannot. not. I would probably also put that under hallucination. But you could train with unknown or a new class. And actually we can also do outlier classification and so on. And one thing that we could do is say this data point really seems like an outlier. I'm going to say, even though you're binary, I'm going to say, I think it might be neither of these two classes. We don't do that yet, but actually that could be cool.

Speaker 1 [39:45]

Cool, thank you very much. Thank you all for being here. And yeah, that was Frank

Speaker 2 [39:51]

All right, thank you.