Tackling the Cold Start Challenge in Demand Forecasting

Speaker 1 [00:06]

Yes, thank you very much for the introduction. It's actually very cool to be here, and we are really looking forward to sharing a little bit of our experiences in the everyday work with demand forecasting, and especially for this topic of cold star problem, which also we think is kind of underrepresented. It's often addressed, but not in a very concise way, we feel. And this is our contribution to say to give a little bit of structure and condensed content to this topic. us that's my colleague daria and me we both work for a company called paretos it's a heidelberg based decision intelligence startup just a little bit of advertisement bear with me our mission statement is no more bad decisions and um yeah so what we often observe is that if you look into businesses and decision processes how they are taken Often you find that a lot of critical things are, even in 2024, in times of AI, machine learning is still done in Microsoft Excel, right? And a lot of manual work, a lot of error-prone things that don't scale well. And this is something that we tend to address because we're convinced we can do better. And demand forecasting is one part of that. And for today's talk, we're going to stick to a specific example, namely demand forecasting for, let's say, some fashion retailer. Like here we have our small, our nicely AI-generated fashion retailer, COA. Yeah. As a typical fashion retailer, what you do, you sell fashion items, you have like a supply chain behind it, so you have some suppliers. And the question is, especially getting interesting, once you want to launch a new collection, like maybe summer is coming up, you have new items that you want to sell. The question is, how many do I have to order for each of those items? On the one hand, I want to order enough in order to meet the demand that my clients or my customers will have. On the other hand, I don't want to just order too much because then I will overstock and this will induce all sorts of bad consequences like warehouse costs and so on and so forth. And there is somewhere in between, there is a sweet spot. And this is actually a classical use case for decision intelligence, where you have some optimization problem, and what optimality here means actually depends on you, right? It depends on your KPIs, it depends on your risk affinity, it depends on what's important to your business. But the point being here that having a good demand forecast, even for that scenario where you have a new collection, is a very critical part of that process to look into the future. Now the examples we are looking here is actually drawn from a real data set, we'll also give you the reference later, so this is also something that you can play with and try it out, it's called a Visuelle, it's a data set consisting of around 5,000 items from a real fashion company, an Italian one, contains data of around four years, so all sorts of, also those Those product images are actually taken from the data set. They have descriptions and so on, and most notably, they contain the time series data of the historic sales that you want to learn from. Speaking of time series, do we have anyone here that has any practical experience with time series? Maybe raise your hands. Nice! That's quite a few. So okay, then I guess we can, we still, for those of you who didn't raise their hand, We will start with a little recap, but I guess we can keep this one a bit shorter.

Speaker 2 [03:53]

So what is time series? What is time series? You probably know. So you can see two examples in this image. So it is date on x-axis, and then for each date, we have some number of sales. So in this case, you see target on y-axis, and this is number of pieces sold. And so time series means that there is time dimension, there is a point, which we can think of as prediction time, that divides past and the future. And in this task, we stand in this prediction time and our task is to say what is going to happen, how many pieces we sell for each item. And in this case, we have two different items. Maybe it is a short and shorts or something. So what else do we have in terms of data? We have static covariates. So what is covariate? You can think of it as a feature that you feed to model that learns from it. And static means that it doesn't change in time. So for instance it can be an image of a product and it can have textual descriptions like size which category it is like short color fabric and so on besides that you can also have time wearing covariates these are features that change in time for instance it can be calendar features and they are known both for the past and for the future because you probably know which day of week will be in 15 weeks and so on. And you also have information about prices and weather. And now let's think about travelling back in time, three months, and now we stand in this point, this prediction time, and what is notable in this example is that one item doesn't have any past target, but still we need to make some forecast. This is what we recall cold situation, cold start problem, and it means we don't have enough target data. What are the challenges that appear here? So first of all, standard solutions don't exist, and if you refer to a book, which probably everyone who has experience with time series know about, from Rob Heidman, judgmental forecasting is usually the only available method for new product forecasting. What is meant here with judgmental forecasting? It actually means that we have a human expert. This is an AI-generated picture of an expert. So we have a domain expert that uses their own judgment and tells us how many pieces we sell. This is a good approach, probably, but then it's hard to scale. What if we have 10,000 pieces for which you want to make a forecast every week? That is not probably doable. And also, it's hard because usually in fashion, how far in the future you want to forecast, it is 12 to 20 weeks, which is really hard. And the second challenge is performance evaluation. Okay, what if we have a point, we make a forecast, how do we guarantee that it's good? What if it's random numbers? We need to make sure that we really tackle cold start forecast without seeing any historical sales. And for that, we can suggest some basic ideas on which we can base our methods to tackle cold start. So we don't have enough past target, and we see it as a problem. But let's not forget that we have other available data. We have images, we have time-varying features or covariates, and we can also be encouraged to be creative with features and think what else can help us. Even if you don't have it in our data set, if it's not directly related, we can think what else can we use? For instance, Google trends, why not use that? And of course, arguably the most important piece of information in our data set is historic target of other items. So if you don't know anything about sales of this shirt because it's not launched yet, We look at our data set and we see how many times and which seasonality periods did similar short have in the past. So we have this to learn from it. To summarise it, we want to search for available data, we want to be creative with features. We want to make sure that we have similar items with the past in the data, because if you sell vegetables and fruits and then suddenly you want to start selling some merchandise with shirts it's probably not going to help you're probably not going to forecast it with your data and finally we also want to think of it in terms of global models global meaning that we have one model that has access to all items to all time series and it is able to learn from other items to forecast for new launches and we have a reference that tells that actually this is in general better approach to have global model unlike local one that is like one model per one time series and having all this data how can we proceed so in general you see that it has different modalities we have images we have textual information and we have numerical information so with numerical one is very simple you just feed it to your model and it learns then with textual we can do one hot encoding and to reduce it to vectors and for images we can apply vector embedding and this has been done in two references that we add here for you so it is also from demand forecasting they take pictures of products and they do vector embedding and essentially what you end up with from all these data sets from all these different modalities are vectors and having vectors as you know from math you can always measure distance. So now you can know what is the most similar product to the short for which you want to make a forecast or maybe how many of them you have that are similar and how similar they are. Now let's think about models that require past target. The most obvious thing you can probably do is to simply take the most similar short to the short for which you want to make forecast but hasn't been launched yet. You can take the most similar one, take its history and pretend that it's actually history of this new one. You can train on it and then do per usual, just train on it and make a forecast. You can also be a bit more creative here, you can take mean sales of all the similar t-shirts, you can also apply nearest mirror imputation algorithm for that which we also reference here and essentially impute the past. Alternatively you can do some dummy padding and masking. What it means? Doesn't matter what was there you tell your model don't learn in it. It's not like it's not real data. You may add a feature that informs the model that this is imputed part or you can actually delve deeper into the model to the very core and say that this data should not be used in loss functions for instance. We add a reference here for a paper presented by Zalando. What they did there is that they applied this dummy padding and masking in encoder decoder architecture so the model didn't learn on padded zeros. And also something practical, there is a library Nixla so it's written in Python and you can also use it in Python. With Nixla you can actually very easily in a couple of lines use neural forecast models and in only a couple of lines you can add this masking part which will do everything else for you. And then we can also think about models that don't even require pass target. So for them this missing part, it doesn't have any consequences. For instance, we can think of some tree-based models like XGBoost, it can easily support nonce, and then we can also have models that inside of their architecture don't even require to have pass target. To illustrate it better, let's look at this example with tabular representation of the data. So we have a target which we want to forecast, we have time dimension, this is the number of weeks, we have some features like day of week, what was the weather, which colour the product has, it doesn't change because it's a static overhead, and then we have these sales yesterday. It can also be sales two days ago, sales one week ago, and so on, this is also what we call lag. So in call start case, this sales information of the past has limited availability. What we can do, first of all, again, to recap, maybe some model can already support this column with nonce, maybe we can fill it with sales of a similar item and pretend like it's history of this one, this certain item. We can do dummy padding and masking, and finally, let's just imagine this column doesn't exist, we don't use it, we don't learn on it, and it should work. So now we talked a lot about different methods, how we can do forecasting, and then there is still evaluation part. Even if we use one method, how can we know how good it is? How do we have guarantees that we can handle a cold start problem? And how can we say which one is better than the other? And for that, I think Aleksandra can help us with evaluation.

Speaker 1 [13:42]

Yes, exactly. So in the time series context, the first thing that may come into your mind when we talk about evaluating the performance of a model is backtesting. If you're not familiar, think of backtesting as some kind of cross-validation scheme that accounts for the time dimension and makes sure that you don't leak any things you are not supposed to know from the future into your testing data. And yeah, I mean, if you don't have any data, then you're going to have a hard time do backtesting, right? And it's per definition like a tricky one. So classical backtesting is difficult for cold starts. But on the other hand, whenever you do something, whenever you model something and you deploy something, you want and you have to make sure how good does this actually work. And the good news is there is something that we can do. And we want to have a look at two general strategies of tackling these things. two backtesting strategies that you can use in the CodeStart context. The first one is what we call pseudo-cold backtesting. The basic idea is that you can artificially make your data cold. So what does that mean? Let's just say, for the sake of simplicity, we have some time series. It actually has quite a long history, so it's not cold, as indicated by this flame. But we can just artificially chop away the history and just pretend it's a cold start time series now. It's indicated by this pseudocode. And now I have a testing sample to work with, right? I can, like, do my cutoffs here. I have, like, I don't have really much data in the past, and I have backtest samples that look like cold start data, artificially generated. um so that's something that you sorry that's obviously something that you can do rather easily given your data just chop some things away so this is an approach that's generally very flexible and straightforward but then on the other hand i mean just discarding the history is not something that's the true start right of an item so if you think about in the fashion example if you think about a product launch a product launch often is accompanied by I don't know certain advertisement and maybe it's a new season there is maybe special market interest and what not right and just cutting away the history and doing this technical approach in general will not reflect that it's just something where you can use for the technical part of your model but arguably it doesn't reflect the true starts of your items and also don't want to go into detail here but also you have to stratify this a little bit if you do it, like especially if you have product hierarchies you have to be a little bit careful, you can't just randomly pick any items or you should not randomly pick any items and do that also it's a little bit unclear or at least difficult to actually draw these subsamples but it's possible next thing would be what we call true code backtesting and the idea here is pretty simple that no matter, let's say you have again this item here with a long history the idea is every item no matter how old it is it has been code at some point in the past right we just have to travel back long enough to get to that point um and that's what the true code backtesting is doing so you um you go back um as far as you can afford with your data to capture the actual beginnings of the time series now yes this now solves the problem this is something that reflects the true dynamics of your item start because per definition you go back to the item start and that's what you test on on the other hand this is and depending on your data situation depending on how many cold starts you have observed in your historic data this may be expensive in the sense that you have only limited amount of testing data available to do that but it seems a bit more like the cleaner approach okay what now so um as per usual it depends right what what am i supposed to do general rule of thumb is if use the true cold if possible and if possible means if you can afford it with your data so if you for example a fashion retailer and you have every i don't know every couple of months a new um new products launched and the code start is probably something you're very familiar with, you have a lot of samples in your data, that is something you can jump on this true code approach. If you don't have that, if code starts are more like the exception, then A, you're probably going to have a hard time fitting that anyway, and B, no chance of doing this true code approach because you just don't have the data to do that. One point we would generally like to stress out in this discussion is not on a quantitative level but qualitatively is whatever you do in backtesting and I think this goes beyond cold start like whatever you do it should reflect it should reflect your real world scenario that you expect as close as possible. What do we mean by that? I mean, Daria, you already mentioned it if we think about this counter example of a guy who's selling food and vegetables and now all of a sudden he starts dreaming about well I could as well start my fashion brand and sell t-shirts and sweaters and what not then yes he may have the data to do cold back tests with his fruits but this doesn't give you any indication about how plausible that will be for fashion products this is an extreme example of course but this is a general thing to think about okay so to the very end we have seen methods like how can we deal with the code start we have seen a few evaluation schemes how can we judge those methods how good or bad they are and we want to jump into a quick example that we did and this slide is actually also a good opportunity to give a quick shout out to our colleague Simon who assisted us in designing all these slides. That's the reason why they look so fancy. This wasn't our work. Because it's obvious, because in this slide we did the last minute thing without Simon, as you can see. This slide illustrates the data set that we are considering, the Visuela data set. We want to do some evaluation. We are going to consider the true code backtesting scheme and the testing here is indeed interesting because the testing data consists of cold start only it's like of an extreme case we are really only it's like a couple of hundred products that we are considering in a testing period we have a few thousand products to train on and only cold start so whatever the model has to predict you always have no history at all and due to time i will keep it a bit short and i like don't want to go into the details of the models but I want to give you the high level idea of which models did we consider and what was the outcome so we have three concrete examples three fashion items in shaded gray you can see the the true time series respectively like in time steps after their respective launch and now we are we considered different methods and one thing that we think is always a good idea is to start as simple as you can, like with very simple statistical or naive baselines that you understand. And so we did here. The most naive thing we could think about is just take the historic averages of everything, and this is that line, and it's the first baseline we are considering. The next sophisticated step is take the average, but per product category. And then a little bit sophisticated, still take the average but per product category and per time step after launch. So these are the simple baselines and then we compared it against a actually very architecturally rather sophisticated deep learning model with image embeddings, text embeddings, enrichment from Google Trends. So very nice architecture. Also, we included some rather old-school-ish machine learning approaches, so CatBoost and XGBoost only relying on the tabular data. And we got some results. The results, I think there are only, aside from the numbers, I don't want to go into details, two messages here I think are important. First thing is, yes, it was worth it putting the effort because the deep learning model was the one that wins. But if you look at the numbers, and the margin is rather small, right? So it's not a clear winner. It's rather a case where even with very simple statistical baselines, you can achieve results that even like an XGBoost cannot beat, right? And I think this is an important message overall. Solving the code start problem is something that is possible. You can apply different things. You have to evaluate them properly. and don't forget about the simple baselines. And the cool thing for you is you will be able to, if you want, you can have a deep dive and replicate these results rather soon, right?

Speaker 2 [23:37]

Yeah, as Alexander said you can also give it a try very soon we're going to publish the code that we used to have this experiment results and We also want to collect there all the literature review that we went through in the last years So we couldn't find a place where we can really store and find all this Cold start literature information. So we're going to also update it as we go and besides that you will also find all the references That we use in this presentation

Speaker 3 [24:17]

Thanks a lot for that nice presentation. A lot of questions, so time series and forecasting. People are really interested in that stuff here. Okay, let's start. Which model types approaches do you use for time series with a lot of zeros, intermittent time series?

Speaker 1 [24:39]

Yeah, that's actually a very good question. That's a problem you face often, and I would say there are two dimensions to that. One is the data dimension, because the first question you should ask, is this really, for example, the right level you want to forecast if you have very sparse time series, because often these low-level kind of things may contain too much noise. But that aside, there's a whole model zoo for that, right? Like intermittent time series is an active branch of research. There are classical baseline models, which we think you should always use and always include, like adida and so on and so forth but also you can think about i don't know prosol regression and you can plug that in any kind of model so that's but usually it depends i would say

Speaker 3 [25:20]

Thank you. Is there research on whether vector embeddings of images of clothing items and their similarity corresponds well to the perceived similarity by humans? I think it's fair to say I don't know. Okay. What advantages provides Nixler versus other Python libraries for forecasting, such as Darts or SK Forecast?

Speaker 1 [25:55]

Generally, a good question, I would say. The Nixla, what we observed, like the community aspect is really nice. Over the last couple of years, it has rapidly grown, especially also in the deep learning community. A lot of models that you find where, I don't know, papers have been published. There is already somewhere a branch for neural forecast where this model has been implemented, so this is quite cool. It's a unified interface, kind of. Yeah, as for differentiation to darts or other libraries, I'm not so sure to be honest.

Speaker 3 [26:27]

I would add it's really fast compared to the others. So how, no, it keeps changing. So voting up, I try to keep that in mind. Should my key takeaway of this session be for cold starts, just be naive?

Speaker 1 [26:48]

I don't think so I think the key takeaway that we wanted to transport is more like you should start naive always like compare it against something that you understand right there were I mean a couple of weeks or months ago there were these blog posts going viral about some time series transformer model and then two weeks later it was shown that well actually some season expansion smoothing is better that's that's unfortunate because I think this is really something that you include the simple statistical stuff in your analysis and if you notice that you are not really better than that, then either you can continue and ask the question why is this about? That's something that we in this case didn't do, also due to time constraints we just stopped at some point or maybe also raise the honest question, maybe it's not possible to get better than that because the data situation is not good. So that's definitely not the takeaway, the takeaway ways more like include it, do it as well.

Speaker 3 [27:45]

Definitely. So now a question to the latest developments like time GPT, etc. Have you tried transfer learning in the time series domain? And if yes, what's your experience with it?

Speaker 1 [28:00]

We didn't. We want to, but we didn't. We're also interested in that field. We read a lot about it, and it's also something that we want to dive more into. I think what we've seen is that for the code start, it may be a little bit tricky because a lot of the models depend on also the history as the most informative thing for us for predicting, including covariates, I think, is really even more challenging in this case because how do you unify different semantic covariates across your data, right? I think that's really, yeah, that's obviously a challenge. So I'm not sure if there are what's to gain for the cold start problem, but definitely something we are also interested in going deeper.

Speaker 3 [28:45]

maybe talk for next year how is training on the data from similar products different to just using the forecasting models of those products and optionally apply transfer learning

Speaker 2 [29:00]

I would say that the suggestion is also a valid approach and can also be thought of another method and can be evaluated. But how different it is, it's hard to say. This is the most naive way to transfer learning, right? If you just use something that already exists and impute it instead of the missing time series.

Speaker 3 [29:17]

Okay, now last question. There's still a lot of open questions. How can your models predict fashion trends, items that are popular only one season for a single year?

Speaker 1 [29:36]

So items that are only suitable for one, I'm not sure if I got the question right, like if we,

Speaker 3 [29:43]

If there's a new trend that now yellow is the color to wear, will you get that?

Speaker 1 [29:48]

that? Ah, right. I would say no, because this goes really more into the direction of marketing research, and this is a different use case, I would argue, than like the replenishment forecasting things that we have shown. Still people are doing that, and it's very interesting, but then I think you need to think a lot more about external data sources, because if there is any signal you want to pick it up, then it has to be somewhere, right? So the question is where, I don't know.

Speaker 3 [30:15]

Thanks a lot for your talk. Thanks a lot for your answer.