Foundation Models in Forecasting: Are We There Yet? Lessons from the Trenches

Speaker 1 [00:00]

So, hello and welcome all to the afternoon session of the final day of this conference. We have a very exciting topic coming along our way. The title for the talk is Foundation Models in Forecasting. Are we there yet? Lessons from the Trenches. And this talk will be given by our speaker, who is Dr. Irina Boyarovska. She's an applied scientist at Zalando, and she works on time-series forecasting and demand prediction. Outside of her work, she leads a math circle for children and enjoys spending time with her family. Now, as far as questions are concerned, so please post your questions on talks.pycon.de. If we have time, we'll also take questions in person. We have an additional mic, so yeah, just raise your hands after the talk, and yeah, we'll be happy to take your questions. So without any further ado, let's give a very warm welcome to our speaker, Irina.

Speaker 2 [01:24]

I'm going to start with a question. How to cook without cooking? Do you have some tip for me? I think any working parent can probably relate and would dream of having a solution to this problem. And we are here in Germany, and I was thinking that probably one answer I will get is buy a Thermomix. so what i want to speak today about is something about cooking without a cooking when trying to forecast time series is it possible that we are able to predict some data without actually training this is the so-called zero shot forecasting problem at the same time we still have to use the covariates ideally we have to be i don't not have to but we could be in a multivariate setting and a global fashion which means that if a normal thermomix can just cook one meal at a time are we able to deliver a five course menu with just one go and even without preprogram such a menu. So it sounds like the dream and this is what I want to take you through today, a journey on foundational models for time series forecasting and what we have tried about at Zalando. So this is the title, the content of my talk. I'm gonna go with you a little bit about a short but very deep history of foundation models, so think about LLMs but specialized for time series data. Then we're going to check what is the interesting thing about having a zero-shot forecast, is it really working, and what it can bring us when we think about forecasting at the scale of Zalando or any other big e-commerce or other type of company that works at scale. Finally, what should we look at? Should we be happy looking at the accuracy only and close the project, so to say, and go rest? Or is there something more that we should care about? So first of all, I want to share with you why is it important for us to look beyond the classical time series forecasting. Zalando is a leading multi-brand fashion and lifestyle destination in Europe. It's a very big assortment, first of all, then very big scale in terms of we are operating in 25 plus markets and we are generating more than 17 billion GMV in one year, and we need to know when is this GMV coming and where is it coming from. Moreover, we are not only interested in forecasting GMV, but a set of other variables or KPIs, like, for example, the number of items that we are going to get, for example, the cancellation rate, our partner program share, and so on. So we have in total nine KPIs and then we have a horizon up to 112 days and we need both daily and weekly granularity. So as you can imagine, we really love to forecast, but we also really struggle keeping this whole layer of forecast alive, accurate and usable. So is foundation model, any foundation model able to help us in this task? how it all started so here this timeline is probably not so correct in the years because you see we have a few years in a row we all know how time series forecasting was solved in the past with statistical models then transformers were very popular at some point trees and so on and And then in 2023, a series of time series foundation models appeared, like TimeGPT and like LAMA, then 24, more of them appeared, and we were reading the news, reading the papers, and thinking, okay, we really have to try this out. And we had the chance to have a three-day hackathon in May last year, where we explored everything that we could find. And I will share the results with you in a moment. Afterwards, in October, a new version of Chrono's foundation model appeared that was very promising. And then again, we had the mindset, OK, we have to try it out. And we tried it out. We had a 10-day research sprint where we focused particularly on this model, investigated a lot, researched, got some results, got excited, And finally, this quarter, we had an end-to-end initiative to see how much of this we can actually even put to production. So it was quite a journey. And let us now see how it worked in the two hackathons that I mentioned. In May 2025, we were following our mindset, think big, act fast. That means, what is everything that we can find? what are the usual baselines that we use in forecasting and how can we compare those different models if we focus on market level GMV and just forecasting the next seven days. Mostly we did zero shot, but then we also tried fine tuning and we tried covariate integration, but this was a problem to most extent because it was just not possible with the given models. Then in the other hackathon in December, our focus was on finding a solution. So we gathered the forces and we decided to go all in trying out Kronos 2 because it was promising in terms of being able to take future covariates and also being able to even forecast globally, let's say, more markets at once and even more KPIs at once. Now, I think I'm sure you are curious to see the results. So in the first hackathon, we played to win, but we also dared to fail. And we kind of also failed, because our models were matching to some extent the baseline, but this was not enough to go live into a business context. Being able to have a model which is sensitive to the covariates is very important, and this was just not given. And the accuracy was also not mind-blowing. And we were thinking, okay, it probably just means that we need to fine-tune more. And we didn't do it, and we kind of blame it on that. Then we had the next hackathon in December, and there, funnily enough, just taking the model as it is, trying zero shot, was already showing great results with the right context, Like we used past and known future covariates, and it just worked pretty well. So on some KPI, like for example, GMV, which is our most important one, the weighted average percentage error was on average improved by 4.6 percentage points. And we were like, okay, in May we were thinking we have to fine-tune. And now we were like, wait, we don't have to fine-tune. And this was really cool. And now I want to try to help you understand how come. So Kronos is a very cool model with a lot of great ideas, which I think are valuable on their own, not even in this particular setup. So the architecture is very interesting, and I think probably would take another talk like this or even longer to go through this. But I would like to pay your attention to the things that are different from the previous foundation models in forecasting and which make it more successful, in my opinion. That is these two layers here, that is time attention and group attention. While we all know about time attention and how usually the model is learning from the time series itself and the things were happening throughout the history, group attention is focusing on a given time point but then checking what was going on not at one time series but at a series of time series so you can think okay how is Germany acting upon let's say a Black Friday and how is Austria or Poland or another market performing at that same period Moreover, not only markets per market, but even if I think, okay, if my GMV is changing, how is the number of items changing, or the average price? So all of these things are captured in the group attention. And another, I also think, secret power, which is not secret, the source is open source, and there are very nice introductory videos that you can watch. The model was trained on synthetic data. And the way this synthetic data was built is very elegant and very powerful because it tries to capture a relationship between time series that we can find in reality and not only normal and usual correlation but even causal effects. So this is really exciting and this is I think what changed our perspective but also gave us a different win, which we did not expect, but we were happy to get. So yeah, fine-tuning was a hope for us. I was thinking about building a foundation model for fashion data or something like this. We have seen it for other types of data. But it turned out that if we have the right context, then actually we don't need it. And I think here the hypothesis is that this synthetic retraining is already capturing the relationships, and this is then much more effective than retraining the model. So the analogy is here, if your ThermoFix doesn't know how to make a tiramisu, maybe you don't even need to teach it, but it's very important that you buy the right ingredient. And now I'm going to go back to this original big problem that we are facing, right? So having an accurate and good and stable forecast without training, I mean, it's great. It's nice to have, but we were also kind of thinking, okay, wait a second. Here we are getting something even more. And that more comes from the fact that the Kronos model that we were trying out is able to forecast all markets at once, all KPIs at once, and all time horizon at once. The only thing is that daily or weekly, one has to decide on the frequency. So you can't have both. And this is something that you still have to take care either afterwards or decide only for the lower granularity, for example, and then do some kind of reconciliation afterwards. And this is really amazing, right? Because our models that we have in the past and also very recently we had great success with, for example, global light GBM when we were forecasting all markets at once. We still had the task on figuring out this relationship between the variables that I mentioned, like number of items times the price is equal to my GMV. If I forecast this in three different forecasts, the numbers will not match. So we had to reconcile. And the relationship is more complex than what I just said. And Kronos, thanks to this in-context learning, is managing to capture this relationship and to give us consistent data. And this is really amazing. So what do we do? The thing is that I don't want to stop here. And there is one thing that was in my mind already before, let's say, playing and actually using foundation models. That is, we have to listen to the people who are using our forecast. Our forecast is not just there because it's cool and because we are wizards who can predict the future. But the forecast is there because some decisions need to be made upon that forecast. And if the forecast is not, for example, stable enough, if the forecast for the future is changing a lot as we come closer to this date, then our planners cannot rely on it. If the forecast is not sensitive to the covariance that we are giving to it, then again, this is not that useful because we want to know how our actions impact our GMV that we are expecting in the future. And here, there was actually, I'm very happy, there was a talk on forecastability, just I think yesterday or before yesterday by Ilya, you should check it out. It's very nice that there are ways to measure this. And there are ways to measure this and there are ways to even figure out how to adapt if this stability, for example, is not given. So that's why I want to say that we have to look at the big picture when we are choosing what should we go with at the end. And I have something here that I call the production readiness scorecard and it has five pillars. Each of them, I think, is very important. Of course, the accuracy is very important And here, again, a connection to another talk from my colleagues at Zalando that was just before the break on the accuracy. I say here, it is very important to have an accurate forecast, but do we even know how to properly measure this accuracy? And when we are comparing, do we know how to properly compare? It's very easy to go into this trap of comparing apples with oranges, and then it's a problem. So carefully choosing the accuracy metrics, comparing, and then being able to, and also the period, right? See, here I wrote on the holdout set, we had this negative experience in the past when we were training a certain amount on certain amount of data and doing an optimization on it, and then testing in a test period, but not leaving a holdout period on which we did not perform the HPO. And this is very difficult in a way that it can really lead to overfitting and it can lead to promising a number that you are not able to deliver when it will come to putting this forecast live and use it. Then the second measure that I already mentioned, this is stability. And here, there are two types of stability, vertical and horizontal stability. And both of them, I think, equally important. One is measuring, OK, if I'm going to forecast some number in the future, as I'm coming closer to it, if my forecast is jumping around, I cannot really use it. Because I need to plan how many people are going to work in the logistics warehouse one month in advance. And if you're going to change that number two days before they are about to come, I can't so fast or it's going to cost me much more to now all of a sudden hire 100 more people. But here there is again an issue that is sometimes these numbers change not because the forecast was wrong, but because the reality changed. because what if the marketing department decided all of a sudden to play a big voucher campaign in some country and then that is what is raising the forecast and it's right to be different from the original one. It's a very difficult topic. Then we come to the consistency. When we are producing different KPIs, we need to make sure that they are consistent with each other because our forecast is used by different pillars in the company and different decisions are made. If you think again about the warehouse, probably you can imagine that they are not that much interested in how much GMV do we expect. Because if we tell them, okay, expect 100,000, but this is maybe 10 super expensive jackets. Okay, this number doesn't make sense, right? but you can match like 10 expensive jackets versus packing 300 socks. You need quite a different amount of people in the warehouse to pack this or the other, although the GMV number was the same. So that's why, for example, number of items is another very important KPI that we forecast, but it has to match to the average price that we are expecting. And finally, coming to the exogenous sensitivity, Here, all the future covariates are super important, and I think this is what made Kronos 2 being so good, is because we could include these covariates. So a model that doesn't know about our plan, per default, somehow cannot be that good. And I'm really curious to see, because since October, I'm starting to get a bit worried, have to say, because since October, I have not seen any big, let's say, new models coming out there that will capture this idea and try to implement it and integrate it and that we can test it out. And finally, scalability. This is for me a little bit the sad point. I just moved to article level forecasting team at Zalando and I cannot bring this with me because the research sprint that we did, at least the first iteration, so that at this scale, at the moment, the model is not ready to give us the solution that we are looking at. And I think this is not only related to the scalability and to the performance, but also to the architecture. And here, if someone has some successful stories, I would be very happy to listen to them. So now we come to the final question are we there yet and my answer is zero shot is real but zero effort is not and the future is hybrid so it is true we can focus without pre-training we can cook without having a recipe and without needing to sit in front of a stove, but we still need to have some effort made. And this effort is going to the right shop, buying organic ingredients. I mean, to be honest, one of the biggest problems that we have during the years in working in forecasting when we try to figure out what was wrong with the model, most of the time that was something wrong with the data. If we give wrong data, inaccurate data, wrongly prepared data, anything of this can lead to some results that were not as expected. So this is one thing that we have to continue to do. And the other thing that we need to take care of is then again, how is this forecast used, right? So the five pillars that I mentioned, can we tick all those boxes? And this requires quite some amount of work that goes beyond that zero-shot forecasting. So with this, I still hope that you get some glimpse about how we try the foundation models at Zalando. And I am now happy to answer any open questions and get your feedback. Thank you.

Speaker 1 [23:02]

Thank you, Irina, for a wonderful talk. We have a couple of questions on the app, so I'll just take one by one. First question. How different are different foundational models in performance? Who is systematically winning the race? Kronos?

Speaker 2 [23:23]

Good, thanks a lot. So here I would first ask, sadly I don't know if it can be followed up, when we think about performance, do we think about the more accurate or in terms of speed? Because if it's in terms of speed, then here when we are working on an aggregate level, as you can imagine, we don't have any issues because the data is relatively and absolutely in general very small, right? When it comes to performance, at the moment, Kronos 2 on a set of KPIs and markets, Kronos 2 was for us the winning model, but also on par with Global Light GBM. But when we compare to other foundation models, then definitely that one. We had a long collaboration and still active with Nixler and TimeGPT, But unfortunately, the topic that I mentioned on the future covariates looks like has not been completely solved yet. So that's why there is still an open question about the performance.

Speaker 1 [24:29]

Okay, thanks. We have questions of similar flavor related to Kronos 2, so I'll just club them together. So is group in Kronos 2 the same as hierarchies? Does Kronos forecast market and detail articles at the same time?

Speaker 2 [24:50]

No. So this model is not hierarchical in the structure. So there is a group attention, so there is a knowledge about the different things that we put in the model, but it's not per se hierarchical. It's not something that will take care of predicting both article level and category level, let's say, together and make sure that they are reconciled. And I would be happy if some extended version comes out that can do this. It would be super nice.

Speaker 1 [25:25]

Okay, thank you. One more question on Kronos 2. In your experience, what number of covariates can Kronos 2 handle reasonably while still improving the metric?

Speaker 2 [25:38]

Super good question. I kind of hide that part in the presentation, but choosing the right covariates, I think is one of the most important and crucial aspects. And it's not so easy to decide. And here, I think the size of the set, like how much, I guess, it also depends how much you have. So as compared to, let's say, light GBM here, We did not use covariates like taking legs of some KPIs and so on. We just used actual covariates about things that we know related to Zalando, like the commercial calendar, average coupon rate, discount rate, maybe the weather, personalized marketing costs, and so on. But even here, even if I take just the commercial calendar, it's so complex that there are some hierarchies, And there I have a variety of choices how to go from this commercial calendar to some informative data that I put in my model. So I don't think that there is a limit on the model itself. And also it's very cool that the model can forecast a lot in the future and it forecasts it all at once. So it makes it also super fast. But at the same time, the covariate question is still left to the experts. And there I have one idea that I still didn't manage to try it out. Maybe many of you have heard about auto-search. So this is like if you give some opus or some other LLM the task to iterate over different configurations and try to come up to an optimal configuration, kind of like an HPO, but done by an LLM instead of Ray or Optuna and so on, which is difficult to do if you just have a set of covariates where the number of subsets that you can take is arbitrary. So I was thinking that maybe another LLM can decide which are my optimal covariates, but I haven't tried it yet.

Speaker 1 [27:47]

Okay, thank you. We have next bunch of questions related to explainability. So I'll group all of them together. So what is the state of explainability in time series foundation models? And do you use explainability methods to guard against forecasting mistakes?

Speaker 2 [28:06]

Yes, it's a good question, and I think it's another difficult one. I don't think at the moment the architectures as they are are giving the opportunity to explain. We used to use transformers, temporal fusion transformers, for example, in the past, and they offer, as per the paper, let's say, that these models are explainable. We never managed to make this work. other simpler methods like profit maybe there we did some demand driver sensitivity where we try to show which are the covariates which are more important but i find this topic super hard and most importantly even if i know and i'm able to say that there is a correlation it doesn't mean that there is causality so we try to convince our stakeholders that they should better focus that it's accurate because explainability is hard.

Speaker 1 [29:08]

Yeah, follow up question, is there a way to plot some kind of feature importance then?

Speaker 2 [29:15]

Not that I know about, good question, but I don't think so.

Speaker 1 [29:19]

Cool. Next question. So, did you try to do top-down reconciliation of more aggregated predictions based on the foundation models?

Speaker 2 [29:31]

Good question. We have not. We have tried forecasting categorical levels, so something between top-down, so not a market and not an article, but a category, like male jeans or female shoes and something like this. And actually, this turned out to be pretty good, and also good for cold start, like the start of the season, where our current trees are struggling. thing, but otherwise not. We have tried different types of reconciliations, and here I really recommend, for example, an external library and other libraries have so-called mint reconciliations, so optimal reconciliations where you don't go bottom-r or top-down, but just find the optimal by taking both outputs, but we were using this for the task that I mentioned about forecasting different KPIs. So if I have my GMV, my items, and average item price, how do I reconcile them without throwing away any of these forecasts? Because in top-down you end up throwing up if you had it in the first place. And like this you can remain everything and still have optimally reconciled KPIs.

Speaker 1 [30:51]

All right, final question of the session of where did foundational models succeed and where did they fail are there some phase? Space where they outperformed Tree significantly for example new product introduction

Speaker 2 [31:07]

Yeah, very good question. So I think, interestingly, they do perform well both on established, so I did not focus that much on articles, but more on the markets, but markets themselves, they have the same property, right? You have mature markets like Germany and Switzerland, and then you have lower performing markets or new markets. And here we do see a better accuracy compared to what we have at the moment in the newer markets but then it's a question is it because the model is so good or is it because our baseline is so bad so it's hard to say but yeah I think reasonably well in both scenarios

Speaker 1 [31:53]

All right. That's it from this session then. Thank you all for your questions and being a wonderful audience. And let's end the session by giving Irina a big round of applause for a great talk.

Speaker 2 [32:06]

Yeah, thank you.