The Foundation Model Revolution for Tabular Data

Speaker 1 [00:07]

So today we will talk about the foundation model revolution for tabular data. And the TLDR is that after years of false promises, where deep learning claimed to be doing great on tabular data, now it finally does. It finally gives a state-of-the-art performance for small tabular data sets, better than XGBoost, etc. And, like LLMs, it's actually trivial to use. There's no training stage anymore, it's just a forward pass through a neural network. work. In the end, we can go through a demo, if this works with a technical setup, and I would invite you to install dependencies in the background now. You can either just use this QR code or also simply Google tapvfn, the top hit is our GitHub from prior labs. All right. So to motivate this talk, why tabular data? Well, there's a lot of tabular data prediction problems in the world, and they take the following form. There's some training set, some test set, we have some features, and we want to predict something. So, for example, in this case, which patients have early-stage Alzheimer's based on some omics blood markers, some numbers that I don't know anything about, but you can measure these omics blood markers from your blood, and then predict whether, well, based on historical data, people with SEAS blood markers had early stage Alzheimer's or did not have it. And then you have new patients where you can wait three years and you know whether they had early stage Alzheimer's, but then it's too late to treat them. You want to know now whether they're having it right now so you can treat it. So that's a typical tabular prediction problem. There's a lot of, for the example, thousands of possible examples. It's really machine learning 101 where you would use random forest, linear SVMs, whatnot. Will this customer churn? Is this payment fraudulent? Will this credit be paid back? Which product will this customer buy? Will this treatment work for this patient, et cetera, et cetera? So this is the general tabular data prediction problem, and there's also time series forecasting which can be cast into tabular prediction. So there you're wondering how a certain quantity develops over time, and the quantity could be the supply or the demand or a stock price or an energy price, income and expenses of your company, machine health for predictive maintenance, et cetera, et cetera. So there's a lot of these problems in the world, and deep learning hasn't really had that much to say for those problems in the last 30 years. But we actually now want to have foundation models for tabular data. Why? traditional machine learning, you do training for every new data set anew, and that leads to individual siloed models with a sometimes lengthy task-specific training, whereas in foundation models, you actually do a lot of pre-training, so you do a lot of work on many data sets, but then you package this up into a box, and you never need to train again. So it's off-the-shelf use without retraining, and if you want to and you have different data sets from your domain, then you can also, we'll open this box again a little bit and fine-tune it to your particular data sets once, and then it does even better for your type of data sets. So foundation models have revolutionized text and images for images actually now for a decade. Text, well, since ChatGPT or slightly before, a couple of years before. And there's lots and lots of companies with immense impact. And for tables and time series and relational databases, et cetera, foundation models just haven't really cracked that, and this is what we're trying to do now with our company PriorLabs. And our community is now on its way to revolutionize this tabular data. So we published a paper in Nature magazine on TapFNv2 that is more accurate than previous machine learning methods like XGBoost, CatBoost, et cetera, on 96% of the use cases. It's more data efficient than previous methods, only needing 50% of the data to get to the same performance that previous methods got with 100% of the data. And we open sources, there's already over a million downloads on PyPy. Our open source has over 3,000 stars on GitHub, and is used by a lot of different companies, and many companies have nice things to say about it. So that was the motivation for tabular foundation models. Let's get into it. I will give some technical insights on tabPFN, how does this work, talk about tabPFN for time series, and then demonstrate some foundation model capabilities, then I'll talk briefly about practical applications, some limitations, et cetera, and time permitting, we'll look at a CoLab demo. All right. Just for nomenclature, the tabular data is extremely widely used, so each row here is going to be a data point, each column is a feature, and there is a special column to predict, that's a column Y, and I'm going to abstract this as X train, Y train, so we have this historic data and then X test, and we actually want to predict this Y values for the test. So what topEFN does is actually it treats a data set as a data point as an input to the network. So you actually sample synthetic data sets, split them into X-train, Y-train, X-test, and Y-test, take the Y-test out, and feed this part here into a neural network. So it's a set-valued input, X-train, Y-train, X-test, go into the neural network, and the neural network outputs a probability distribution for each Y-test data point. And then you take this Y-test that you took out here and look at how likely is it under this prediction of the network. You compute the cross entropy and optimize this neural network here in order to output probability distributions that actually better reflect your white test. So that was one back prop for this entire data set. And then you sample a new data set, and you do this again, and a new data set, and you do this again. And we do this hundreds of millions of times. So what we need is really a mechanism in order to synthetically generate tabular data sets because there aren't a whole lot of tabular data sets out in the world. There's OpenML that collects data sets, a really great effort really supporting it. But there's maybe 50,000 data sets on there that are sort of high quality. And out of that, we've sort of collected maybe 200 high-quality data sets. So there's really a dearth of high-quality tabular data sets in the world. And this is different than, for example, for language or for images, you know, where everybody puts their content online. If you have a high-value tabular data set, that's not going to be online because that actually gives you company value, and you're not going to share it online. And so that actually leads to this problem that in contrast to text on Wikipedia or anywhere on the Internet or images, people share them. And tabular data, that is not the case. And so, really, we would like to have trillions of tokens, trillions of data sets. We're here on a pretty small scale compared to neural nets for language, so it's enough to have hundreds of millions of tokens, but one token for us is basically a data set. And how do we generate these data sets? Well, so that's based on a highly parametric structural causal model that goes a bit beyond this more practical focus of this talk. But really, you have all kinds of possible causes of your data. You could have, like, the features could cause the Ys, or one feature could cause a Y that could cause another feature, or just the Y could cause all the other features. You could have latent causes and whatnot, all kinds of different possibilities that are being sampled in the synthetic data. And then at test time, after we've trained this type of N on hundreds of millions of of synthetic data sets, we actually take it, and then we have some real data sets. So this is the first time that we see a real data set. There we actually don't know the Y test, and it has the same interfaces over here. You just feed it in and get predictions for the Y test. And it's just one forward pass. There's no training anymore at test time. So in contrast to previous deep neural networks that were really slow to train and so on, nobody likes us in data science. You want to be really fast. That's why people really like tree-based methods. This here, it just gets the data, does a forward pass, gets a prediction. That's it. And surprisingly, this now works. All right, so I just said that. Simplicitly, it's just a forward pass. I sort of was one of the people starting the AutoML community, organized eight years of AutoML workshops, was the general chair of the AutoML conference for the first two years. I love AutoML, but it is very complex, and it never really made it to the point where data scientists love it, because it just took a long time, and it was super complex. You know, we have this Bayesian optimization procedures, hyperparameter optimization of the data pipeline, like, for example, Scikit-learn, we built AutoSQL learn, wrapping around Scikit-learn, doing meta-learning over different data sets, doing ensemble building at the end. It was this complex beast. And actually now, with TapEF-NG, you just do a single forward pass, and it outperforms the decade of work we've done in AutoML. Qualitatively, it gives smooth and well-calibrated predictions. So here are some toys I could learn data sets, this half-moon data set. You see just very smooth predictions compared to, for example, CatBoost gives you very axis-aligned predictions, as trees would. But yeah, away from the data, you get more uncertain, et cetera. So that's sort of what you would expect, because we train this with cross entropy, which penalizes being wrong but sure. If you're wrong, you at least want to be unsure. That's what cross entropy says. And that's why we get sort of well-calibrated predictions. Quantitatively, we had a paper at iClear 2023. We're on fairly small data sets. We showed that this sort of was 36,000 times faster than AutoSQL Learn, which was a state of the art at the time. If you have a GPU, if you don't have a GPU, it's only 3,600 times faster, so like one second versus 3,600 seconds. But it had a ton of limitations. So this really just worked for continuous data, only up to 1,000 data points, 100 features, 10 classes. It wasn't designed yet for categorical features, missing values, uninformative features, only Only classification, not regression. So really anyone in data science was like, eh, can't use this. But it was sort of the core of the method already worked then, and since then we've been building and just resolved most of these limitations. Now it works for up to 10,000 data points and 500 features. Now it also works for categorical features, missing values, uninformative features, and also for regression. And the inference time we also reduced. And it continues to be better than previous methods. So here now in five seconds it's better than other methods in four hours. And so really basically it becomes a new default for small tabular prediction tasks. And that's sort of what nature was looking into. Is it that? And they said, yeah, this will be used a lot. So they accepted the paper. One big thing in the paper is, or in the new type EFN, is a new architecture. So this is a new architecture that really knows about tables. In contrast to, for example, an LLM, that would just serialize a table, we actually know something about rows and columns, and we do one axial attention over the rows and one axial attention over the columns. And then, yeah, we do, like, 12 layers of that, and, of course, in between have a standard MLP. So it's a standard transformer, really, with this axial attention, so you have two attentions per layer, not just one. And in the end, there is this histogram distribution. All right, yeah, for classification, so there's, of course, a lot of different classical machine learning methods, like linear regression, MLPs, SVMs, random forest, and then all kinds So boosting methods like light GBM, cut boost, XGBoost, they're all fairly similar. And then top EFN is sort of just really quite different, quite distinct in performance. From these in particular, the dark blue is a default performance. XGBoost is actually pretty poor in the default, but quite tunable. That's why people like it. You can fiddle with it and it gets better. But people would also like it if they don't need to fiddle with it, and it's already better. Disclaimer, it's not better on every single data set. So here is, for example, a data set where cutboost in the default is better, and on these 28, tapfn is better. And after tuning, this is sort of still the case. For regression, results are similar but not quite as good. So still the default is better than the tuned version. But there's more data sets where CatBoost is already better in the default, and we also compared to the now leading method in AutoML, AutoGluon, so AutoSQL Learn, we sort of stopped developing a couple of years ago, and Nick Erickson at Amazon is really, he's a gifted engineer, he's really been pushing on AutoGluon, and it's great. But even though TAPEFN in five seconds actually outperformed this ensemble of all kinds of different methods in autogluon trained for four hours. And autogluon is well-based on AutoML and has this really nice post hoc assembling, and we can take that and put it into a TAPEFN and actually assemble different architectures and hyperparameters in TAPEFN, and that makes it yet better, getting to a 70 to 30 win rate against autogluon. And that also helps a lot in regression. So in regression, actually, the default is not quite as good. It's actually worse than autogluon. But if you tune it, it gets as good. But if you add PHE, then you also get to this 70 to 30 outperformance of autogluon. Yeah, some ablation, so type BFN is now robust against uninformative features. This is something that really plagued previous MLPs and so on, like deep learning, when you take a feature and you just add a bunch of uninformative noise features, deep learning was just very poor at figuring this out, which you see at the MLP here that just massively drops in performance, and type BFN. We just told it in our prior for the sample data sets, there might be some uninformative features in there that just do not affect the Y, and it learns that this might be the case and is not affected much by these features. Likewise, it is now also robust against missing values. In the prior, we said, well, we just punched random holes into the data, and now it also does not care about missing values anymore. And also works for categorical features by us really putting into the prior a lot more work in what could categorical features be looking like. There's trees in our prior of the generating process, et cetera. And it works with less samples, so 50% of the data give you sort of slightly better performance than CatBoost with the full data. Explainability is something that people, of course, ask a lot in the context of deep learning. Because Gboost is not super explainable, to be honest, if you have, like, thousands of trees that are doing random things with these really weird decision, yeah, sort of surfaces. But it's sort of simple to understand a tree and then sort of to interpolate, okay, well, we do 1,000 trees and so on. And that is a little bit harder for neural networks, I fully understand that. But you can do post-hoc analyses just like you can with any method, and actually the smoothness of TAPI-FN's predictions really helps in order to get clearer patterns in this post-hoc analysis. So, for example, if you look at the SHAP results of CatBoost, they're sort of all over the place, whereas for linear regression they're just very clean, and for TAPI-FN they're actually similarly clean as for linear regression, but the predictive performance is better than the one of CatBooth. So that was technical about tabPFN. Now how to apply this to time series and foundation model capabilities, these are very short, practical applications too, maybe we'll get to the colab. So time series. We can actually cast time series forecasting as tabular regression by featurizing our data. So let's say we have a univariate time series, all we have is a time stamp and a value. For example, demand. And then we can say for each data point here, well, what is the running index? What is the year? What is the month of the year? What is the week of the month? What is the week of the year? What is the day of the week? What is the hour of the day? Things like that. Put a cosine and a sine features of those. And then, basically, we have featurized every data point here, and we can treat them as IID data. So we can then also do the same with the data in the future. So 17 days in the future, I can just say, well, what is the day of the week? What is the hour of the day, et cetera? And so I can actually create this X train, Y train, X test, and question mark is, well, prediction for the future, and calls up EFM, and actually this really simple extension achieves state of the art on the most broadly used time series benchmark, outperforming Amazon's Kronos large, with a much larger neural network, also Times FM from Google, and this is the same network that we used for the Nature paper. There was no retraining. This network has never seen a time series. It has never seen this featurization of the data, whereas, of course, the other time series foundation models have actually seen time series, synthetic or real, and so this just goes to show that there is so much to be gained in time series foundation models, and I'm super excited about pushing down on this. All right. Foundation model capabilities. It's a foundation model. fine-tune it to your own type of data, just like you would download a llama and fine-tune it to your text data. You can download this and fine-tune it to your tabular data sets, and that can make it dramatically better. So here is a default TPFN predictions on some data points that follow a sinusoidal that we're not telling it that it's a sinusoidal, and it's making not-so-great predictions, and after a bit of fine-tuning on just a few sinusoidals, it actually makes more sinusoidal predictions. You can also do prompt tuning, data generation, density estimation, outlier detection. We have a drift-resilient version. We have a counterfactual fairness version. And you can even go further and output a hypernetwork, so output a neural network that is particularly good for this one data set so that you get faster inference. Or you could output a GAM for your neural network so that you can see this as a different way of training a GAM that's actually faster and slightly better than previous GAMs. As a foundation model, it also works out well with other foundation models, so you can embed textual features and feed them into top BFN just like as normal features, and that actually then works really nicely. Yeah, and it has been used in a lot of practical applications already, so for example, Boston Gene reported that it's this machine learning powered blood test and analysis platform. They're identifying immune profiles and try to enable better treatment prediction. And they report that it reduced their errors by 53%. So ROC went from 0.81 to 0.91 and saved them 90% of the time, so making them 10 times more efficient and let the data scientists actually focus on stuff that they wanted to focus on enhanced efficiency, allowing them to concentrate on meaningful clinical analysis, rather than the more boring ML part. TechTel is using it in a risk decision automation platform to enhance fraud checks and risk anomaly detection. And yeah, it's open source, so the open source community is contributing. Actually there's this really nice plug-in built by Benjamin Lalanne that, yeah, puts TabPFN through the API into Google Sheets, and you just get predictions directly by calling the API. Of course, I will make this faster, but super nice to already see this. All right. Some known limitations. So when to use TabPFN, I mentioned it. When you have tabular prediction tasks, classification or regression, small data sets up to 10,000 data points and 500 features, and when inference time is not crucial. If you try it and the results are not so great, maybe it's because you tried CPU, CPU is terrible. You need to use a GPU, even like we used RTX 2080, it's a five-year-old gaming card, it's like 100 times faster than a CPU. The defaults can be suboptimal for regression, as we saw in many of the examples. Then we have this AutoTabPFN, which makes things a lot better. And we also have a Discord with over 500 people already. Please reach out there if you have issues. When to not use it, when you shouldn't expect that it will work. If you have millions of data points, it's not going to work as well because attention is quadratic, and we're using standard attention so far. And if you need instantaneous inference, then in-context learning is just too slow. We do some KV caching, which makes it like 300 times faster, but it's still not quite as fast as just trickling down a random forest. We're also overcoming our current limitations. If you have more than 10,000 data points, again, AutoTapEFN has something that has trees with TapEFN in their leaves. I would rather use that than downsample the data. And we have some extensions in progress, scaling up, time series forecasting, doubling down on that, and actually really focusing on Causality, where it's sort of the best person in the world in Causality, Bernard Schilkopf. All right, Colab demo, I don't think I have time for it. Maybe in the, yeah, maybe go through it afterwards, or I'm happy to go through it after the talk. But let's have some questions. Thank you. And I should have mentioned, we're hiring, we have all kinds of positions, we just got like 9 million of pre-seed funding. And we're looking for all positions. We even give you $2,000 if we hire your referral. So send us your friends. We're trying to get the best of the best, a diverse team. Yeah, please send them our way.

Speaker 2 [25:12]

Thank you so much for your talk, frank. The slider has been exploding. In case you have noticed or Still have a question, you can still ask it on slider. Let's just start with the first one. I assume one can still use Chap for explainability. Are there internal features to Explain the predictions?

Speaker 1 [25:31]

There are no internal features yet to explain the predictions. We are definitely looking into it, but there's this whole field of mechanistic interpretability and so on, and that's what I definitely want to look at more with my university hat on.

Speaker 2 [25:48]

Thank you. Then next question. How does tab pfn work for tables that have free text fields in addition to categorical and numerical values? Do you recommend an ensemble approach in this case?

Speaker 1 [26:01]

No, actually, there, I would recommend to just take the textual features, push them through an LLM. That's what we do in the API, only because we don't want you to have to annoy you with doing that yourself. But then just feed in the embedding into TapioFN, and it just directly works. And TapioFN, again, wasn't told that the embedding is going to come from a neural network. So it's very surprising, in a sense, that this directly works.

Speaker 2 [26:26]

Thank you, and then the next question does it handle a highly unbalanced target in classification tasks

Speaker 1 [26:32]

Yes, but we have tried sort of if you have like 0.01% then you know and you have only 10,000 data points and there's sort of one in the one class that's not going to work but if you're like 100 to 1 should kind of work.

Speaker 2 [26:47]

Thank you and the next question isn't tab pfn essentially a shadow model and you are running an attribute inference attack given the fact that synthetic data will to some extent represent reality

Speaker 1 [26:59]

No, the synthetic data, so it's not the case. Many people ask, oh, it's trained on some data that's basically probably gotten from some real data, and so it's remembering the real data, and then you're sort of like elephants never forget. You kind of know your test data. No. The data is entirely synthetically generated from structural causal models. There's no real data that enters this process at all, so it can't possibly remember any real data. It has not seen any.

Speaker 2 [27:35]

Thank you can tap pfn predict multivariate outputs

Speaker 1 [27:39]

Not yet. Great feature request. We want to have it.

Speaker 2 [27:44]

Okay. Why the limitation of 10,000 samples? Any plans to overcome this?

Speaker 1 [27:48]

Yes, it's quadratic attention. We're using a simple quadratic attention, like simple standard attention. And yes, we're looking into linear attention and have promised that by the end of this year, we definitely want to have something out that natively works on 100,000 and gets state of the art with a million.

Speaker 2 [28:06]

Thank you. How much you need to pre-process normalized data before making an inference?

Speaker 1 [28:11]

You don't need to normalize. We do that in the model, so we just standard normalize, but, of course, if you, like, we actually want to enable data scientists to do their work better, right? Like, all the feature engineering, this ingenious work that people do manually to get the right features to, like, do it through the right type of feature wrangling and so on, it will still work a lot, and we actually have this paper, CAFE, that shows that feature engineering on top of TapiaFN actually really helps it. Helps it more than it helps sort of like the contextual knowledge actually helps it more than it would help random forests.

Speaker 2 [28:50]

Thank you, and then the very last question does tap pfn also work with panel time series or hierarchical time series

Speaker 1 [28:57]

It's just standard time series so far in the TUPEF NTS.

Speaker 2 [29:03]

Thank you. Let me ask you one thing. There are so many unanswered questions. Would you be available to answer them on discord?

Speaker 1 [29:08]

Discord. Yes, absolutely.

Speaker 2 [29:10]

Okay. Thank you. Then thank you very much for your talk, Frank, and you for the questions.