Accuracy Is Overrated: Ship Stable Forecasts (Without Lying to Yourself)

Speaker 1 [00:00]

Hi, and welcome everyone. My name is Irena. I'm going to moderate the session today, and it's my pleasure and honor to welcome today Ilya Babomnikov with us. He is an accomplished data scientist at VOIDS, and the topic of his talk is Accuracy is Overrated, Ship Stable forecast without lying to yourself so let's just wait one second to everyone to have a seat and remember that you can post your questions in the talk spike on de website and yeah enjoy the talk and please help me welcome ilia Thank you.

Speaker 2 [00:52]

Thank you. Thanks for coming. And let me start. I'm Ilyan, and it is my already fifth year at PyCon, so I hope you enjoy PyCon as much as I enjoy it. And I'm already more than 10 years in data, and I think it all started when I put my head in accelerator at CERN. That's how I started with the data at Large Hadron Collider, and proceeded to e-commerce and supply chain. And currently I'm working at Voids where we run AI power, procurement software and blah blah blah. So all this marketing stuff. But what is important is we do a lot of forecasting. A lot of forecasting. We work with a lot of segments from pet food to stylish hoodies. And you see quite a lot of revenue we are forecasting. It means we need really stable and accurate forecasts. So it's one of our prior responsibilities. And imagine we ship some forecasts. It may be even a bit better. WMAP was improved, reduced by 4%. Nice. Now we need to put it into production. You put it in production, but unfortunately, you need to sometimes retrain your model and re-forecast. For some reason, you cannot just ship the forecast and forget. You need to operate it. maybe every week, every month, and that's pretty annoying because your forecast actually can jump up and down significantly. Here's an example. You see this super nice time series from one that we work with. First, you deliver some forecast in November, this blue line, okay, then one month later you retrain your model and that is your new forecast. 30% down. Is somebody happy? No. One more month later, even 50% down. Is it acceptable? No. Planning team need to redo decisions. Trust in the forecast is lost. Doesn't matter what was the accuracy number. If somebody see like this, it could be quite unacceptable. And it looks like your model is insecure. And with every revision, meaning like retraining and re-forecasting, it changes its long-term forecast dramatically and what that means? It means forecast is unstable. They need to replan, redo decisions. Complex. And to be precise, when we speak about instabilities, there are multiple levels of instabilities. It could be that we take the same origin, the same target and it That means our model is just not replicable. What we are more interested in the top right corner, called vertical stability, where we do forecast from different revisions, for example, retraining last week and this week, and forecast, let's say, for the next month. And then we see that, okay, our model shifted up and down by, let's say, 10-20%. And that's the one that we are interested in and we would like to study in this talk. You can define multiple metrics, here I selected one of the simplest ones, mean absolute relative difference that we will be looking at. It is, you sum up the forecast of an overlapping period for the future between different revisions, for example from previous week and this week, and you calculate the difference. So formally it is bias how the new forecast this week is biased according to the previous forecast from the previous week. And just you take relative and see how it looks. Let me explain a little bit what is our experimental setup. We take a real online sales data sets, anonymize them of course, train model and produce one year forecast ahead of us. And every new week or every new month we create new revision with new retraining and forecast again. And at the end what we like to measure, we measure stability metrics and we measure also accuracy because we would like to see how they interfere with each other. We measure accuracy over out-of-sample period and stability for the full one-year horizon into the future. And we would like to put on our stress test a lot of models to see how this looks like. We start with the baseline system, the classical statistical models, profit, trees, XGBoost, LightGBAM, GAM model, even neural and hits. Also hyping now foundational models. Let's see how they perform. We're really curious about your expectations, and we will test Khronos and TimeGPT and AutoML. And a few words about datasets. Datasets that we have are quite interesting because this is not nice clean M5 dataset whatever you can find in Kaggle. No. This is a real dataset from real life. We have spikes, short data, and super big spikes, and super sparse products, some time series that I don't know what we can forecast here. Fine. And we put all the stress and let's have a look how the end result looks like. So this is types of plot I will be showing a lot. Let me explain in a few words what is here. Here on the x-axis we have WMAP, one of the most popular, not the best metric, but one of the most popular metric. Means lower is better, more accurate the forecast is. And on the y-axis we have MRD. This is our stability metrics. Again, lower is better, means the forecast is more stable. Here, blue dots you can see simple time series and statistical forecast. While it is rather on stable side between stability around 0 and 0.2, you can see that error vary a lot. It could be quite our lucky shot that statistical model worked out, it could be quite unlucky that statistical model was not the fit for this particular dataset. Then we look at the trees, and you You can see the trees are mostly grouped in the middle of our accuracy, and also kind of rather on the stable side. But still you can see that a lot of red dots, our tree models, jump really high in instability, meaning that our focus with trees can explode from revision to revision, sometimes depending on datasets, anomalies, and etc. Then we try a simpler model, let's say GAM-generalized additive model, and you can see while we're losing a bit in accuracy, the stability is much better. We are below 10% of stability, meaning that our new revision doesn't jump more than 10% from one to another. Then you can see N-heats, these triangles. While we have some nice accurate forecast, stability quite suffering. Most of the N-heats and stability metric between 0.2, 0.3, meaning we have from revision to revision quite big jumps from training to entry training. Chronos, similar story, could be stable, but, sorry, could be accurate, but quite unstable. And auto-gluon, you can see those here in this corner. I don't know. Maybe we can invest a bit more time. Most of the setups are quite vanilla setups models with best parameters we can find online and with some upturned tuning, but you can see results. They speak for themselves. Let's look a bit deeper in another foundational model at Nixlam because we invested some time and talked to them. And here you can see similar experiment we performed with Nixlam. Here I'm just showing light GBM, GAM model and time GPT. Time GPT with the stars. And you can see again, you can have quite accurate here with time GPT model and an accuracy level is similar to light GBM, a bit better than GAM model on average, but again, stability here is suffering most of the points between 0.2 and 0.4, meaning that from monthly revisions, it could be a jump up to 20, 40, 30%. Yes, it outperforms in the case of short data. Here you can see time GPT model produced nice to focus where light GBM model actually failed. But it also could be vice versa where it underperforms for long and sparse data where we have a lot of data and with a lot of anomalies and events. So now we learned, okay, different models can bring different stabilities, but what we can do in post-processing, maybe we can do after this, something like stabilize a model without losing too much of accuracy. And the different approaches, I will mention three. The first one will be reconciliation stabilizer. The thinking is quite simple. We have much simpler model that does some forecast on high level, let's say on the full shop, on the full category level, and we used reconciliation with top-down approach where we split high level forecast to the smaller forecast on product level based on the product level forecast. And the expectation is that model on average become more stable because it always scale up back to the simpler shop-level model. And here is the type of plot that I will be showing in the next few slides. On the y-axis now we have delta in accuracy, data in the metric, and left means that we improved our accuracy. So here, left corner, or left side, means we're improving the accuracy with this method. And on the y-axis, we have our stability metrics, and lower means that we improve stability of our model. And here you can see also, of course, ideal case is left bottom corner, where we improved both stability and accuracy. And you can see that does it really help? Not always. What we observe is that nervousness or instability of the model can be perfectly learned actually on the shop level as well, on the high level, and then you have a mixture of two instabilities on the shop level, on the granular level, and you mix them together. Quite often it doesn't work. But it helps against really big fluctuations. You see a number of examples, a number of datasets, this has really helped. both stabilize and improve accuracy, but this approach is debatable. Then another approach is how to stabilize a model is assembling a stabilizer. You might have seen like M5 competition or other Kaggle competitions where the winners or top five just drop tons of the models, assemble all of them, and they produce even better results. Let's give it a try. We try the simplest assembling, averaging. You just mix a lot of models, all the models give equal weights, forecast. And surprisingly, it works. So you can see that a lot of dots here are in the bottom left corner, so it really helped a lot of models at least to become more stable or a bit more accurate, especially versus mean within this group of accuracy or stability. But also you can see that the number of really bad results that we lost actually in both stability and accuracy. And when it happens, it happens when inside your ensemble there is one, two models that are really unstable, really can escalate, especially long-term future, up, down, and it spoils the full ensemble. So in my conclusion would be it works, but you need to, you cannot just blindly drop and assemble tons of models, you need to preselect them, find really on a rather reasonable scale stable and then ensemble can even make the full collection of the models better. And the last approach that was quite interesting and you can find more details in this paper, this is origin assembling or full interpolation where we take the model from previous week and mix it with a model from this week's retraining, and then we can go like n weeks in the past. We mix it with a particular weight, here you can see the formula, for example 0.2, meaning we take 20% of previous week's model, mix it with 80% of this week's model, and forecast in the future with this approach. And here you can see quite interesting results because when we're using this origin assembling or full interpolation a lot of our experiments end up in win-win region that we improve both accuracy and stability of our model of course majority that's kind of a bit expected end up in trade-off we trade off between stability and accuracy so model become more stable but a little bit less accurate on rather short evaluation period So it's interesting, this approach is kind of, you almost cannot lose when using this approach if using this approach in a rather smart way and worth experimenting. So this was three types of stabilization one can use. And for the conclusion, I would say it's like if you care about stability, it means you're building a real product that can deliver value to the users. Because otherwise they lose trust, otherwise they don't know how focused would you perform, because accuracy always is calculated in some short period, you don't know if this period is the same, you can have concept drift, you can have a lot of things, and usually much more stable forecast on the long term performs better in business processes. Unfortunately, I really hoped there will be some more magic pill where you just okay use foundational model it give you super great results or use best pick model and assemble they will give you really great result you are done. But But unfortunately, there is no this kind of magic pill and you need to unfortunately learn your data, spend time on tuning your model and what I can recommend, try simpler model. Not every use case of forecasting deserves the most complicated model you can find. The more complicated model, the more unstable it is. If you can use simpler model, linear, statistical model and deliver the same accuracy, the end result in most of the cases will be much more stable. If you are using assembling, preselect high quality models and then in this case, assembling will deliver much better results. So that's it from my side. Thanks a lot for your attention, and I think it's time for questions.

Speaker 1 [17:59]

Thanks a lot, Ilya. This was very insightful and a deep dive into a very important topic for everyone doing forecasting. Let's go now through the questions, and I'm going to start with the first one. Why not to consider the forecast uncertainty for the model? This is valuable information that is not used in this experiment.

Speaker 2 [18:24]

Yes, that's right. We were only discussing point forecasts, not probabilistic forecasts in all the studies. So I think there's something that we can expand this research and see how to perform with uncertainty and how the uncertainty will be correlated actually with real instability. Because, yeah, retrainings, yeah, I would say it would be interesting. It would be interesting to consider this and see if uncertainty of the forecast will be a good primer of potential instability of the forecast.

Speaker 1 [19:02]

Thank you. Another question is on stacking. What is your opinion on stacking, feeding the predictions of all models to a final simple model like linear polynomial regression or a fully connected neural network with linear activation?

Speaker 2 [19:23]

So, stacking is a bit debatable, so we haven't tested it, it's something that we would like also to test. But it goes a bit into the direction where you learn your stacking in some historic previous evaluation period, and you believe that the stacking will make sense in the future. What we did try, we did try like best peak forecast where we're assembling and doing the stacking, but the winning forecast, the winning model for particular product takes us all, so formally we do the stacking with activation zero and one for each forecast. And we saw that it's very unstable because as soon as you learn the stacking, your model can fluctuate up and down quite significantly, yes. With linear activation, it's something to test it out. I think it will be smoother, but I think the general problem is there when you're on the stacking, it accidentally can flip some models or give totally different ways to different models than in previous revision that would lead in events disappearing, some seasonality disappearing and some significant issues.

Speaker 1 [20:45]

I see. Thanks a lot. Another question is the following. Did you utilize short-term lags and predict it with continuously available covariates, meaning lags from actual, or did you utilize lags from the predictions, i.e. iterative approach for longer forecast periods?

Speaker 2 [21:07]

So we tried both. We tried both. I think here in trees models we use lags from predictions as well for long-term forecast. But for example for linear model we use a frozen lag, so it means that we use only lag that is frozen after the training period is finished.

Speaker 1 [21:33]

I think, okay, very good. Then a question about sparse time series. Do you have any advice on creating robust forecasts for sparse time series, i.e. you are missing observations?

Speaker 2 [21:48]

Yes, the advice will be similar, use simpler models. Simpler models, especially for sparse or very sparse time series, would do the job, and as well, reconciliation quite helps in learning. Because the problem with sparse days, you cannot learn much on time series level. You need some reasonable, not over-trained model on lower level, But when you reconcile it on some shop level, category level, whatever, you can extract a lot of good features like events, seasonality behavior, prices, and etc.

Speaker 1 [22:28]

Very good. And I think this is related then to the next question, which is exactly on the hierarchical forecasting. Could you elaborate a little bit more on hierarchical forecasts, shops category? How do you bring them together? Have models like TimeGPT automatic handling of hierarchies?

Speaker 2 [22:49]

Okay, let me start from the end. About time GPT, you need to ask Nixler. I have no idea if they have automatic handling of hierarchy. This is something that I would also be curious to ask them. How we handle it at the moment, we do it with reconciliation, where you have multiple levels at which you are forecasting, and then you can reconcile it either top-down or middle-out, or actually you can also learn the reconciliation metrics on ELSO Historical.

Speaker 1 [23:29]

Thank you. Did you utilize a short term? Oh, sorry. No, that one is answered. Sorry. Now we have another one. Instead of assembling, what about making the predictions of model X1, X2, Xn new features of model Y? Could that could be that model Y learns from signals and then learns also which signals, which model captures the best?

Speaker 2 [24:00]

This could be the approach. I know a few examples where it worked out when you have really a lot of data with really a lot of clear signals and etc. On our data sets, I would not go so much into this behavior because you just don't have enough data to capture these signals without overtraining. I would say, because what you're describing, because we described, okay, you can have best peak, then we described, okay, you have linear activation, and this is one more next step where you train model of, another model of input of the, and use a lot of simpler models as input tool. Second model is something to experiment with, but I don't see it will perform better than, for example, the similar linear activations and et cetera. I am quite often conquest against overcomplicating the models if simpler models can do the job.

Speaker 1 [25:15]

Yeah, I think that's a good mantra to have, it helps. Did you try to combine different methods? Maybe this was partially answered, but the question is, how about assembling and convex combinations of models from different weeks?

Speaker 2 [25:33]

No, we didn't try. I mean, face space for experimenting is huge, so we needed to limit some face space for experimenting and take a more basic approach in every direction and see how it works. So maybe next year, with the next talk, we'll find out much more complex experimental setups that will work.

Speaker 1 [25:59]

Sounds good. Another question is going again to the direction of ensembling. Do you think that similar stability or accuracy metrics can be achieved by defining one unique right model architecture or feature selection for your specific use case instead of ensembling different models?

Speaker 2 [26:23]

That's a good question. There's something that we are attempting to find, this magic pill, maybe really good architecture, but the big problem is that we have a lot of different use cases and a lot of different clients. And some models that could be quite stable and accurate for one client, one data set could be a total disaster for another client, a total data set. And that's exactly, if you look at these plots, you can see boosting decision trees are jumping really quite significantly in accuracy. and on this plot maybe it's not too much visible but then when we have other tables where the winning model per data set and usually the winning model per data set is quite different for different data sets because they are on average winning models that win let's say three data sets out of ten but not more

Speaker 1 [27:34]

I agree. Okay, good. We have a few more questions.

Speaker 2 [27:38]

I should have made my talk a little longer.

Speaker 1 [27:42]

Yeah, but I think it's very valuable for all of us. So I am curious about this one. Might it improve the results when using the median instead of the mean for all models to improve stability? I think this refers to like taking the previous predictions, but instead of the mean, take the median. Or on the models. I think we saw two types, right? we ensemble models or we ensemble previous forecasts. So I don't know where there was a mean, taking the mean and would taking the median improve.

Speaker 2 [28:18]

I mean, for some reason, it sounds a bit like that we are from probabilistic forecast trying to make point estimator. But if somebody from the room asks a question, so can they comment? Do you take the mean in order to improve it? You mean assembling a stabilizer here? Yes, exactly. Okay. So, this variance. But what do you mean, taking median, yeah, then we will have... Because the outliers would be neglected more or less, so it might improve. So if you have outliers, they might be more neglected, but it's not so important. No, I mean, that's a good point, that's interesting to experiment with. The problem was that here we're taking mean of the forecast and what you call outlier could be a signal, like event, promotion, et cetera. So on average, it would, of course, improve stability because you're getting rid of stronger but if you want your model forecast some spikes or some big movements, then you actually could reduce your accuracy in this way. But again, that's an interesting idea to experiment.

Speaker 1 [29:56]

Okay, so I'm going to read the last question. There are a few more, but maybe you can check them, and everyone is welcome to also approach Ilya after the talk to continue the discussion. Wouldn't Bayesian methods ship both model uncertainty estimation and stabilization by assembling or by sampling from the posterior naturally?

Speaker 2 [30:24]

In Bayesian estimator, you are using the same model, the same family, so they could be biased in the same direction or exploding in the same direction. One good thing about assembling and why it works, because you sample over different models that if one is exploding or is not robust anomalous, another will be robust anomalous and and bring better performance. I'm not sure how this would work in Bayesian and something.

Speaker 1 [30:57]

Thanks a lot. Yeah, maybe it can be a topic for the next research. So thanks a lot for the very insightful questions. Thanks a lot, Ilya, for your great talk.