From stockouts to happy customers: Proven solutions for time series forecasting in retail

Speaker 1 [00:07]

So first of all, welcome also from my side to my talk from Stockouts to Happy Customers, Proven Solutions for Time Series Forecasting in Retail. First and foremost, I'm super happy to be here, super happy to be at this venue to see all those bright minds among us. And if they would have told me up front that it put me on the big stage, I probably wouldn't have ended in this talk. But anyway, here we are. Proven Solutions for Time Series Forecasting in Retail. You might be wondering, why is this now a specific or dedicated talk? I mean, in the end, it's more or less like any other forecasting case. There is some sequential data coming in. There is some time-serious model. It could be a stats model. It could be a time machine learning model that just produces some forecast. And there we are. And four years ago, I probably would have said the exact same thing, and I would have seen not any hurdles or any complex complexities in there. But now, four years after, I can guarantee you it's not. It's a very complex field, and especially forecasting and retail has very complex problems that are overlaying on top of each other, and therefore I'm very happy to give this talk today. Before I start, can I just get a hands up for all the people that already worked on a time-series project and brought a time-series product into forecasting production? Cool. Who of you already worked on a forecast in retail so far? Okay. Wow. Okay. I would say roughly 15% of the audience. At the end of this talk, I basically want to share my experience from the past four years, want to present all the pitfalls that waited for me and also the company Pareto's I'm working with, and to just share a wide understanding, a diverse understanding, what's basically waiting for you. So whenever you will start a retail forecasting case, you definitely know what's ahead of you. Towards me personally, as I said, I'm Robert. I'm a physicist as a background, changed my profession from physics to computational physics, and then which brought me also to make me one of the early adopters of transformer models in 2019-2020, which eventually brought me to sequential data modeling and then also to time series forecasting. In the area of time series forecasting, I'm working now for the last four years, and today I want to share my experience on that. towards the company i am working i'm working in at the moment is paretos paretos is kind of a company that helps other companies manage their complexities in their business decisions especially in areas close to the supply chain so whenever there's like a retailer who needs to fill his warehouse with products they need a forecast which we produce with our products but on the other hand also we have downstream business applications where we for example automate the replenishment and automate the capacity planning. So all those decisions where companies struggle a lot with, we do help with our solutions and retail forecasting is a crucial part of it. So let's start. Why is retail forecasting so important? So basically what you see here is kind of a typical picture of a supply chain. So basically in the beginning there are some raw materials that are being ordered to the manufacturer plant. The manufacturing plant produces some goods, which they then ship to their primary distribution centers. Those primary distribution centers, they distribute it into secondary distribution centers or the so-called retail warehouses, which then eventually ship it to the end consumer. So basically, what we are talking about today is kind of the very, very last step of the supply chain. Why is this so important? Because if you get this step right, and you kind of can predict the demand of your customers up front very accurately, you can, in a perfect world, just plan the supply chain backwards. So by knowing the actual demand, you know what to store in your warehouses, you know when to ship your goods to your warehouses, the manufacturing plant knows when to produce what. So basically, it's a backward propagation from the very end of the supply chain. And therefore, the retail forecast is so important. Why is retail forecasting so hard? In the end, it's about customer demand. And what really makes it so hard is that there are a lot of separate business problems connected to one retail forecast. So we have those typical external demand drivers where weather, holidays, economic factors play a huge role. We have a very diverse product portfolio. So there might be very low runners that are being sold twice a year, and there might be high runners where we see very large volumes and peaks. And we both need to comprise it into one good forecast. We have small and long planning processes that range from seven days to 12 to 15 months. And on the other hand side, super complex product hierarchies and promotions, they do play a huge role and they also come up for a lot of non-linear singularities. and there are also other things like order books which we also talk later on about new product launches, substitutions from old products with new products and also what you see in basically every retail case is that there will be constrained demand and constrained demand is I think the one key factor that I want to highlight today because it's very important and it really has the potential to destroy a lot of business impact that you would create and for example if a product is out of stock Then the demand will be constrained And therefore you need to find strategies to come up with that So these are the four areas I want to dive deeper today And for the other ones I will give an outline at the end So you also have something in mind there What is very important for each of those problems If you want to solve them You need to have the dedicated data from your customer available So understanding the business problems At the very beginning of your retail forecasting case, it's crucial to get the right data to actually be able to address all of those problems. Otherwise, you're lost. So it's really important to understand business problems at the very beginning. So let's start with the first one. The first one, as I just said, is for me personally the most crucial one. If you haven't solved that in your case, your case might not likely find daylight and not make it to production. It's about the sensor demand. So this is like a typical example where there are some normal time series data that can be tracked. And in the meantime, for example, the warehouse had some shortages. So there was not a possibility that products could have been sold. And this is how the time series typically then looks like. If you now do a normal forecast, not really taking this into account, your forecast basically learns these kind of zero-inflated periods. periods, and then also will forecast these zero inflated periods. And this is what is shown here on the right side, this is kind of like a devil's cycle, because if there is a shortage, there will be no sales tracked in the sales system. Your future, your machine learning model might just predict zeros, also because you think, okay, it's not sold, then I also predict zero sales in the future, and then the supply plane will also replenish zero stuff. And then, of course, this cycle just starts from the beginning, there will be shortages. So this is a very, very crucial point to get that right, otherwise you destroy a lot of business potential. How to address that? There are multiple ways of addressing it. I will now share two with you. You can also dive deeper into that afterwards, but just to have this on your mind, there is like the possibility of just imputing those shortages. There are a million ways of doing it. Here I showed you one strategy where I used some seasonal imputation step to impute specifically the shortages with the unconstrained demand periods. And therefore I get like a time series that is not zero inflated, where I, for example, then can use typical machine learning models, you name it, could be neural networks or machine learning models that basically can handle that and get rid of this negative bias that is induced by the shortage periods. So this is one strategy, how to handle it. The other strategy, for example, is to really invest into some summary statistics based on the data you've got. And also for that, the stock data is very crucial. Otherwise, you have very hard times coming up with the summary statistics. One example that we often do, and it also works quite well in production, is to really count the steps in a given week how much you have been out of stock. Because not using this week at all for your model training is kind of a pity because there is some signal included in a given week, right? But you need to account for the day's time step whatsoever where you had some constraint demand. And if you just put some summary statistics on top of it and just count, for example, the time steps where you have been out of stock and pass it as, depending on the model, pass it as a future covariate or pass it as some constraining factor, the model has the ability to come up and to unconstrained the data which which which is which is which is good and which again mitigates the negative bias that is induced by those shortages one thing that is super interesting when when you're running all your pipelines you're testing multiple models against each other is how you evaluate what is in the end the better forecast so maybe in the first rows what would you say would you consider as the better a forecast. Forecast one or forecast two? It's not that hard, come on. Forecast one, yeah, why? Exactly, right? In the end, we want to supply, we want to plan to the full potential of what the demand, the customer demand actually is like and not plan accordingly to a forecast that is completely zero inflated. And now here's the tricky part. If you do just a normal metric calculation on top of that, actually your metric calculation would tell you that forecast two is the better forecast because we have zero inflated periods. So if your forecast predicts zero, your metrics say, yep, that's good. Let's take this forecast to production. But it's a massive threat to the quality of your forecast product. And what you really need to do, it's just one strategy, right? There are others. But one strategy is to really don't trust those metrics and really only calculate the metrics on the periods where you actually know you had unconstrained demand. So, for example, now I highlighted this here. So, here we knew that there was no shortage. And calculating now the metrics here shows us, as you said correctly, that forecast one is the better forecast. So, really let's make sure to not getting fooled by constrained demand cycles in order to train or evaluate your forecast. And I think this is a very, very crucial point whenever you are faced with a project like that. Cool. Let's go to a different section of what we see very, very often when dealing with especially e-commerce retailers. So these are normally companies where they do launch a lot of products very regularly. So every week, every month, there are new products. And the difficult thing about it is that those products, often they have no time series history, right? So you need to come up with some reasonable forecast without having the actual history in place. And this is what we define as cold start problems. So as a cold start, so for every time series where there is no actual history available. Let's have a look into that. So from our research, we found out that for those cold start items, the static metadata is something you cannot miss out. So we need to know up front if, for example, if there's a new shirt being launched in a shop, whenever the shirt is red, the color, the size, the shape. Maybe colleagues of mine already showed that posting, like embedding pictures with some LLM into some vector space and passing these vectors to the model actually does help. So any kind of static metadata really does benefit your model quality. On the other hand, obviously, we can also look for similar items in the past. So if you embed, for example, metadata, your forecasting methodology automatically embeds the metadata in a way that it can look for similarities and then can learn from the one product whenever there's a future product being launched that there might be some similarities. So this is something also which proved to be very successful. And lastly, it is recommended to use global models there. So in time series forecasting, there's always these two type of models, so to say. The first model is kind of the local model, where each model only sees one specific time series. The other model is kind of the global model sees the whole time series data set as a whole, and therefore can learn some cross dependencies across the whole data set. So we found out that for especially for those code for those code set cases global models are recommended that did To give you an example all the booster trees the extra boost the lgbms cat boosts and also the neural networks They are all global models, right? They all simulate at the same time So therefore using this as one model in your model zoo or model portfolio is definitely recommended Yep, this is already talked about that One more thing that I found pretty interesting, the model choice for the cold stats really makes a crucial difference. So there are, again, two types of models. There are models that require some input data and they cannot accept any kind of missing values, right? And there are models that can do that. So for the models that need actually input data and they cannot accept non-values, for example neural nets or statistic models, there is some literature how you could make use of similarity-based imputation. So that means you impute the missing data with some similarities, and the similarities from different products you get, for example, by a k-nearest mean approach. on the other hand also for example transformer networks they do have the possibility of masking certain time steps so therefore they don't really take them into account later on in training and you can pass some dummy values on top of it and they won't really trigger or impact the gradient descent algorithm on the other hand as i said for me if i would if i would recommend you a model class to handle those cold start items i would definitely recommend the booster trees because Because the boosted trees, they do have this great behavior of accepting the nonce. So they really can naturally handle missing information. And especially for cold starts where you would like to focus on the static metadata, that's a big plus. The next point is about the diverse product portfolio. So you need to imagine those retailers, they do have 20,000, 30,000 different SKUs as products. This is not like something special. And within those products, there are probably 1% of the products, let's call it 5% of the products that make up 80% of the revenue. And the rest, 95, maybe 20% of the revenue. So you can imagine how diverse and how heterogeneous these data set distributions are. For example, there are never out of stock products. There are new product launches, what we just talked about. There are promotion special days. We talk about this in a second. And there are also unpopular styles, yes. So in Germany, yes, I would say this style is very, very popular. But for the rest of the world, it's not. So yeah. And we need to kind of handle those different patterns, right? And I just brought you kind of the respective time series that I see every time there's a new use case coming up for us. There are those regular demand patterns, which are not very hard to predict. But nevertheless, there are some trends, seasonality, sometimes some peaks, overlaying, it's not that easy. There are new product launches, there are promotion days with very high non-linear spikes. So this is something which is always hard to grasp, but we'll talk about this in a second. And there's this intermittent time series, which is kind of the sandals with socks. Yes, how to deal with that? You can imagine it's very hard to have this one best-fitting model that does all of it. So showing you once more about this data distribution we see in production most of the times that there's a lot of, there's a small fraction of products that make up already quite some good amount of volume and of the revenue. And then this is like a normal distribution of what we see day to day. And as I said, especially with those, with this imbalanced data distributions and those completely different patterns, one model wins it all. Probably not going to happen or not so often at least. We also saw in the Frank Kutter talk earlier that also his foundational model performs good, but it performs even better whenever it's put into an ensemble, right? And I found this insight very valuable because this is also what I experience a lot of times in production and training those models. And just to give you a feeling, so what is this notion of assembling? So whoever already used like an XGBoost algorithm just for testing purposes, probably almost everyone. Yes, exactly. And what does XGBoost, what is so special about this model class is kind of his ensembling architecture, right? So it uses some boosting ensembling in a way that it fits, tries to do a prediction, tries to do a fit and the next tree kind of fits on the residuals of the first tree and therefore every new tree kind of sees the residuals of the first one and so on and so on and so on and tries to fit those residuals and or something is a very very powerful method and I show you now basically how we do it basically internally apparatus in order to make a difference so as you can see here for example just like a super simple example when there is like a statistical model on the first hand that sees only the time series itself and it has a good fit of the seasonality but kind of misses the peak and on the other hand we have this machine learning model that fits SMP quite well even over fits bit and also have a slightly less quality on the seasonal stuff And if you combine them together, it's kind of fits. This is kind of the idea, right? So now making it a bit more explicit. I'm not looking into How we deal with with internally within our product So we do have like a model zoo model portfolio of 100 plus models That we train not not of course not every time that would be computational insane But we train a lot of we have certain presets that do it's that includes some of those model classes and The interesting thing is then in the end we have like Different methods how we could combine or how we do combine those different model classes And I think especially this this one here at the very end proved to be very successful for us So basically we train a bunch of models we say we save those predictions and we then we leave an optimizer It could be a black box optimization or could be like a stack rank optimizer all both as possible and in the end let the model let the optimizer combine the models for final prediction and that proved to be very resilient so whenever you're confronted with a data set with a lot of diverse data time series as an input or sampling is definitely one way how you could mitigate it just one comment there are also in the literature examples of better learners so this is kind of a machine learning model that is being trained on top of the other time series models. We didn't make that positive experiences with that because it's just another layer of complexity if you train another machine learning model and optimization so far is well more understood and well more explainable so therefore getting rid of some complexity definitely proved to be very powerful and by the way also AutoGluon, you probably also you use that if you're into times with forecasting it also has some very prominent stacking or stumbling approach in place which is also open source and one can have a read and that cool let's go with the last section that I wanted to show you because promotions are still especially for our b2c customers a very crucial area of data so to say a lot a lot of this these customers really do have a lot of promotion data and this is one of the data source that is more or less always the dirtiest one since they just throw everything in there that whatever made promotions and then let let let the models decide kind of what to do with it and just on a very very briefly what what we do with it so first of all we experienced that data cleaning cutting through the noise is especially important when you work with those noisy promotion data so what we do kind of is We analyze up front automatically which promotion types do have an uplift that we see in the past data. And this is the promotion data that we keep and throw away kind of the other promotion data that is just pure noise. And that really made a difference for our forecasting quality. On the other hand, and I think I come to that on the next slide, But decomposing is one strategy that might work, and to decompose into trend seasonality and some, like a noise component, like the residual components. And how you really make use of that is that boosting mechanism, similar to XGBooster I just explained earlier. So what you could do is, for example, I'll roll it down here. This is like a typical example of a time series, very well behaved. So there's an upward trend, there's some weekly seasonality, and then there's one spike being produced by a promotion. And now there's a tricky part. On the one hand side, you do want to capture those trends and seasonalities going upwards. The tabular models from this world, they do not really are strong in capturing those upward trends. Since the tree-based models, they do have some strange decision barriers. And in theory, they are capable of predicting out of sample, but in practice, this is not their, I would say, their biggest strength. So they fail sometimes to capture those trends if you don't do any preprocessing on top of it. On the other hand side, we have those stats models that are really good in fitting those trends and seasonalities that I really would recommend. always using statistical models plus your machine learning models to really have combined the best of both worlds. So basically what you could do is having a sequential multi-stage approach where you first fit like a seasonal model that captures that basic trends and seasonalities and then on top of the residuals you get like a forecast from a very powerful machine learning model which is shown here. So on the first two plots we do have some very simple forecasts, but they are very powerful, to be honest, if they are fitted like that and if they are capturing those trends. And then at the last plot, we see kind of the typical nonlinear spike that is where those tree-boosted models are very well suited for. And if you then combine all of those predictions into one prediction, you get kind of the final output. And this proved to be also very, very strong for a lot of diverse problems. So seeing one minute, I speed up a bit. The other challenge I still would like to share and also to present is on the one hand side if whenever there is like super complex product hierarchies and small and long planning cycles comprised in one use case problem, then hierarchical forecasting or temporal hierarchical forecasting is quite like a novel literature that I really would recommend looking into, because I think every good forecasting model should have those hierarchical approaches, which is quite interesting. And also, on the other hand, some order book data is also some special problem, because it has time-varying information for the same date. So whenever you do a forecast or you do a train step, you need to have that exact data in place, whatever was known at this point in time. So it just varies over time, and therefore these order book data is also like a super interesting problem that you will probably face whenever you dive into this. So finishing up with the learnings, I think most importantly, before starting a retail forecasting case, make sure you understand all the business problems and ask for the right data. Otherwise, you're lost. Secondly, there are multiple sources for demand constraints. For example, stock outs. Identify them and handle them appropriately. that really accounts for modeling as well for metric calculations to really avoid those pitfalls. And lastly, the retail forecast cases I saw so far in the last four years, they always comprise multiple problems at the same time, so therefore flexible modeling approaches like ensembling approaches will definitely help out and boost your model quality. That's it basically from my side, thank you so much for listening. On the right hand side you see like a QR code, we are actually hiring, we have three open Positions for back-end engineers and also ai solution engineers that basically they're really focusing on solving those cases making customers Happy and basically doing all of the stuff that i described to today hands-on So looking forward to connect with all of you and yeah, thanks for listening

Speaker 2 [26:37]

Thanks a lot. Let's dive into the questions because there are 17 of them. We will not be able to do this at all. Okay, so the first one that has the most likes is, what approaches do you undertake to balance the business impact of overstock versus understock in the model? For instance, one mis-sale is as bad as 10 items in overstock.

Speaker 1 [26:56]

Yeah, that's a that's a very very good question to be honest because this is also what I've been Wrapping my head around the past month So basically that's correct. Whenever you have a negative bias on your prediction It's much more costly to the production and a positive bias because having one item on stock Costs less than having a item not on stock because this is just a completely a lost sale How I would approach it Make sure you differentiate between the forecast and the decision itself The forecast should always be as unbiased as possible It should always reflect reality as good as possible. So do not try to Induce some bias onto your forecast in order to make it positively or negatively biased Try to keep it as unbiased as possible and then afterwards whenever you produced your whole unbiased forecast Then it's the time to put some decision logic on top of it to really make sure the customer makes a good decision and not mistakenly swaps it or changes it with the forecast itself. So really forecast unbiased and then decision logic could be, for example, towards positive bias. And this is very important because you want to make multiple decisions on top of one forecast normally.

Speaker 2 [28:14]

Thank you. Next question is this extreme gradient boost versus a light GBM. So which one do you prefer and why?

Speaker 1 [28:22]

Hey, GBM.

Speaker 2 [28:24]

and why.

Speaker 1 [28:26]

So, lgbm and xgboost, normally the behavior comes out relative similar to each other, but I just prefer the lgbm because it's way faster than the xgboost and therefore gives me more iteration speeds, is my answer.

Speaker 2 [28:42]

Thank you. Next question is also technical. If you have tried foundation models with zero shot.

Speaker 1 [28:48]

We have tried it, but I don't want to step on anyone's toes, but I don't think we need it yet at the moment. So I also had a discussion with Frank Hutter before, and we just discussed, and especially those shortages and those constrained demand, the foundational models nowadays don't really can take this into account very properly. So therefore, it's not appropriate to use it in our situation. Nevertheless, we had some discussions. There are possibilities to also enable models make being more specific for certain problems and industries. And then I think it might be a valid add-on to our ensemble strategy.

Speaker 2 [29:33]

I think we have time for one more question. The question is, is the model result always accepted by the end user, and how do bad model predictions impact the trust in future results?

Speaker 1 [29:44]

Yeah, it's also a very good question. So I would lie if I say results are always sufficient. I think there's two things that are very important. Mostly what I experience is that most of the times the predictions are not sufficient because the data input is not sufficient. So you can get trapped quite easily by seeing the outputs of your model and then say, hey, model is shit or your model is not what I want and it's super easy to say that it's super hard to translate it back to your input data and say hey here please customer this data set is missing or it's it's it has a bad quality or here we have no stock data these are kind of the explanation factors and then kind of it's a shared journey to improve the data situation which then automatically improves the quality of the performance so whenever there's bad You really need to understand the explanation or the root cause behind and i think by that Customers normally accept it because they know that they also need to improve their data situation

Speaker 2 [30:48]

Thank you very much with this we don't have time for more questions, but I will send these to you later Maybe you can comment on them on this course or some such. Let's thank the speaker again