How to compare apples with oranges: Proper evaluation of article-level demand forecasts
Article-level demand forecasting at scale involves predicting sales for hundreds of thousands of items across multiple markets, resulting in approximately 10 billion records daily. The primary challenge is the disparity in sales velocity; absolute metrics like Mean Absolute Error (MAE) or Root Mean Square Error (RMSE) overweight high-selling articles, while naive relative metrics, such as Mean Absolute Percentage Error (MAPE), fall into a scaling trap. In this trap, fluctuations in the aggregate error metric often reflect changes in the assortment mixture—the ratio of high-sellers to low-sellers—rather than actual changes in model performance.
To resolve this, a benchmarking approach using Poisson-based simulations is employed. By generating a perfect probabilistic forecast—where the mean of the Poisson distribution is the actual observed value—a theoretical lower bound for the error is established. This is paired with a naive upper bound, such as repeating the previous day's sales. By anchoring the actual model error between these two bounds, it becomes possible to determine if a model is truly improving or if the error is simply shifting due to seasonal changes in the product mix.
Key takeaways include the danger of overfitting to a single aggregate metric, which can introduce bias toward high-selling items and degrade overall accuracy. To mitigate this, a multi-metric evaluation strategy is used, combining demand error with bias measures and Gross Merchandise Volume (GMV) error. These calibrated metrics are then translated into a traffic-light system for stakeholders, allowing for market-specific diagnostics where a high error in a low-volume market may be acceptable, while a lower error in a high-volume market triggers an investigation.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Machine Learning & Deep Learning & Statistics and was classified suitable for intermediate domain / novice python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
At the pricing department in Zalando, we are predicting future demand for millions of articles on a daily basis by large-scale machine-learning models. These forecasts are key for discount decisions taken downstream. As evaluating every forecast on its own becomes infeasible at this scale and frequency we created a set of aggregated metrics that help us make informed statements about the performance of our models. On the one hand these metrics are being used by us to further improve our forecasting models, on the other hand they are used by our stakeholders to make informed decisions.
To handle this volume, we use PySpark for data processing and scaling our evaluations across the entire assortment. Furthermore, evaluating forecast performance in this context is crucial in two different scenarios, namely when analysing past forecast performance and when creating and comparing alternative models. In both cases we look at different time ranges and possible different subsets of the forecasted articles and calculate aggregated performance measures to compare them. We want to answer questions like
- “Is this forecast performing better in low-discount periods than during sales events?”
- “Did we make a higher error on highly discounted articles during last week?”
- “Is this model well-suited to predict high (or low) selling articles?”
- “Did our model perform well for sneakers during the last voucher event?”
Evaluating aggregated metrics like a relative mean squared error (MSE) or an mean absolute percentage error (MAPE) over different sets of articles has lots of pitfalls. Comparing different parts of the assortments leads to an "Apples vs. Oranges" problem that we want to elaborate on based on examples we experienced in our daily work.
To answer the questions above we developed a set of aggregated metrics that we monitor on a daily basis using plotly and streamlit for clear, interactive visualization. We want to present these metrics and explain how they are useful for the questions and tasks mentioned above. We will highlight the techniques and best practices to draw meaningful insights from evaluating forecast performance and how we are able to compare apples with oranges using meaningful lower bounds for our aggregated metrics.
We also want to share how observations from our monitoring influenced the evolution of our LightGBM and PyTorch models and how it shaped important parts like feature engineering, hyperparameter tuning and the choice of our loss functions. Lastly we will touch on how to communicate these sometimes very technical numbers with stakeholders so that they can make informed decisions without being overwhelmed by details.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:20]
Good afternoon to you all. This afternoon we are going to be looking at how to compare apples with oranges, a proper evaluation of article-level demand forecast by Stefan, who is an applied scientist at Zalando, currently working on developing large-scale forecasting systems. He holds a PhD in mathematics, where his research focused on analyzing dynamic dependencies in time series. And prior to his three years at Zalando, He worked for five years at EON as a data scientist, creating algorithms for smart meta-analytics and forecasting. Then we also have Mons, who is a senior applied scientist at Zalando, where he builds large-scale predictive models for pricing, which requires thinking about forecast evaluation from a business perspective. He also holds a PhD from TU Berlin on foundational properties of neural networks. Can you please give them a round of applause as they take their stage?
Speaker 2 [01:22]
Yeah, thanks a lot for the kind introduction. We are very happy to be here and represent Zalandov at this great conference. So, let's dive right into our topic, how to compare apples with oranges. This is a topic that hopefully is useful for many of you who are working with, let's say, regression problems, where you train models and so on. So it's not specifically tied to only forecasting, we would say, but this is our use case, so we'll present that. To give you a bit of corporate context, Zalando, as most of you might know, is one of Europe's largest e-commerce retailers, right? So we operate in 25 countries. We have a revenue last year, I think, of roughly 18 billion euros with over 60 million active customers. So it's a large-scale company. And this also means that if you go to the website, this is how it would look like. I guess this is familiar to most of you. And what this means, like here you see three articles, but we have hundreds of thousands of those articles, right? And this is now where our use case comes in, namely demand forecasting, essentially. So this is a fundamental question that we are trying to tackle here, and this is prevalent throughout many, many different business domains, right? So you need forecasting for pricing. This is where Stefan and I are working in, but you also need it for supply, like how many articles do we need to order in order to basically neither be overstocked nor understocked, right? So this is kind of a difficult problem that we are trying to answer because it also obviously has this time dimension in it, right? So I want to know, okay, how many articles are going to be bought tomorrow, in a week, in two months, and so on. So these are questions that we are trying to tackle. Breaking this down a bit, we have essentially time series on article level, right? So for this shoe, or for this, I don't know, sports set, whatever, or this bag, we basically have time series that we need to model. And these are very noisy because we have, as I said, like on any given day, 500,000 articles. These articles also change from time to time because the assortment is highly seasonal. We have 25 markets that we need to predict for, right? So Germany, UK, you name it. And we need to forecast up to 26 weeks into the future for different price levels and so on. And so if you take all of these dimensions together, we are essentially in the need of providing a forecast every day with roughly 10 billion records. So this is essentially the scale of the problem we are talking about. Meaning also we have a large pipeline behind all of this with a very diverse set of tooling. This is not the focus of this talk, just to give you basically a bit of an intuition about what we are dealing with. So we start with our data preprocessing part, which is heavily focused on Spark, Databricks, and so on. Then we do model training and experimentation offline with tools like LightGBM, PyTorch. We use experimentation tracking with MLflow, hyperparameter tuning. We do an Optuna, and so on. Then we deploy our models. Inference happens online, as I said, on a daily basis. That means we create these forecasts every day. and then we need to evaluate them using tools like Streamlit, Plotly, whatever. Taking a step back, what happens here essentially is, of course, so first of all, we process all of the data that we have. So we know the histories of our articles, right? So what were the past sales? What was our stock? What were the prices and so on? We have article metadata. We have calendar data like holidays, sales events. like think about Black Friday, and so on. In the next step, we then do modeling, and we go basically along the two of the more typical directions that you would use in time series, namely gradient-boosted trees or transformer models. On the transformer model that we currently have in production, we also have a paper, so a bit of shameless self-advertisement. Then we do model inference online. That means basically we have the history of an article, and then we try to predict the future. So how many items are going to be sold tomorrow or in two weeks or whatever. And the topic of this talk is forecast evaluation. And it's, as I said, not necessarily only tied to forecast, but we will frame it in this setup. So where we need to be careful and so on. All right. So why do we need to be careful here? We create forecasts on an article basis. and sometimes to get a bit more insights we also look at single articles in order to see how these time series develop or to see some patterns and so on but as I've said we have 10 billion records that we produce or 10 billion forecasts that we produce every day so it's impossible for us to obviously look at single time series on the scale So meaning we need to somehow restore to aggregate metrics, so somehow aggregate the performance over different articles. And then for each market and each day, we basically calculate these aggregated metrics over all articles. How these metrics look like, what our apples and oranges are, and so on. For this, I will now hand over to Stefan.
Speaker 3 [07:15]
Thank you. Alright, so this is about forecast evaluation and the metrics that a lot of you know is like for example the MAE, where you like look at your forecasted value and look at the ground truth value and you sum them up and take the mean, or you can do an RMSE if you are more into squared distances. The problem that you get here, there's like a small example where we have one example apple and an example orange here, we have this basic t-shirt where we like sell 500 off on a given day we forecast 480 and like this is pretty good right we are just four percent off and mae gives now an error of like 20 right then we have this fancy blazer which is also cool article those we don't sell that often we sell one on a given day we forecast at five that's pretty terrible right we are like far far away from the ground truth. But if we look at the MAE, like
Speaker 2 [08:13]
like it's
Speaker 3 [08:14]
it's just four points of error, right? So this is like a problem that a lot of people know if you have like a lot of, in this case, time series, but in general if your prediction problem has like different scales, you get into trouble if you just use these naive absolute metrics. Normally like the solution for that is to use like in relative forecast metrics, so we are trying to to normalize our errors by the amount of ground truth we see. And typical things is a mapper we know, and there's something called an RMSE. Basically, we sum them up, and we divide by some scaling factor in general, like the size of the ground truth. This is the error we are using in our production setting, more or less. This is like just a form of the RMSE that I had before. Like in normal words, we take the prediction, we subtract demand, square, and then we normalize this by the ground truth. One thing that always helps me to calibrate this a little bit, if the prediction is just 0 for all articles, then this error will be 1. So 1 is very bad. Everything below is better. Okay, what we then get is like this is our error over some subset of articles over a couple of months. So the error itself is a time series, right? And if you look at this, you can see some time series characteristics. Like there's like a trend where the error is lower during November, December, and it's higher in September and February. So if you look at this, you could say, right, this is the point of lowest error. model was performing really good. Over here our forecaster was struggling a bit and it doesn't look so good. And then naturally you can ask, why is the forecast in November and December better? Or why is it worse than the other times of the year? And what I'm trying to show in the next couple of slides is that this idea is just wrong. The forecast is not worse during November, December, it's not better in February. Actually it's the different way, the direct other way around. So what's the problem we have here? There is a pretty nice paper from Maltitihi where he pointed out a lot of problems with these relative forecast metrics and scaling. If you have the time and nerve for some mathematics, have a look at the paper. It's really nice. We are trying, like, we are picking out the part which is relevant for us, and I'm trying to explain what's going on there. So the first point, this velocity disparity is the first thing I explained in the apple and orange case, right? If you have an absolute metric, you're going to overweight the fast-moving articles, the one that have a lot of sales. And the other point is the scaling trap. If we have a naive relative metric, like the one we are using in production, for example, a lot of the variations you see in this metric actually has nothing to do with your forecaster. They just come from how your assortment is structured and put together. And one solution they are providing in this paper is this Poisson-based benchmarking where you do like a simulation run with...
Speaker 2 [11:31]
Uh, uh,
Speaker 3 [11:32]
Poisson distribution and how this is going to work I'll show in the next couple of slides. So quick refresher Poisson distribution, this is this distribution which is doing count data so what can come out of this are like positive integers it has just one parameter which is both the mean and the variance and so if you see here lambda equals five this thing we see there has mean five, variance five and it's a pretty pretty nice way to start if you're trying to model count data. For example, sales. Sales as count data. Poisson distribution is a good measure for that. Let's do some simulations. What I'm going to do now is I run a simulation where I put just 10,000 articles. We will already see the effects we're looking for at 10,000 articles, so you can imagine how this looks in production for us. And we will mix two different kind of articles. One we will call low sellers and one we will call high sellers. And the low sellers will have a mean which we draw from a uniform distribution of 1.5 to 3. And the high sellers will have a mean that we draw from a uniform distribution that goes from 5 to 30. Not unusual values in this article demand forecasting. And then we're going to do the following. We are going to draw from Poisson distribution with the lambdas we just chosen as means, and this is what we're going to call our ground truth, so what we will observe later. And then we create an oracle forecaster, so a forecaster that's cheating a bit, because it knows the realization of the ground truth. And we'll just do a Poinçon draw again. And we will later call this thing perfect probabilistic forecast. Then I'll go ahead and just calculate the demand error between the ground truth and this perfect probabilistic forecast. Mind you, this forecast knows much more than a usual forecast can know. And we are kind of just adding a bit of randomness around that forecast. And this is what's happening if we change the fraction of high sellers and low sellers. So on the very left of this graph, where there's 0%, the whole assortment is just low-selling articles. 100% means it's just high-selling articles, and the rest is the mixture between. So what we see now, our demand error, although this is the same forecasting model, varies quite heavily, just depending on how the mixture of our assortment is. Right, so, and the values go from 0.2 to 0.6. That's quite a bit of a range. And for our use case, for example, we are usually somewhere in this region here. Like we have between 5% and 30% of articles, because we have articles that sell like three times per week. We have a daily forecast. There's a lot of zeros inside and ones. What's also interesting is I did the simulation just 100 times and also putting the 10% and 90% intervals of the simulation there. You can see the variation is not very high, which makes sense because we have 10k articles and then averaged out over them gives very, very stable values. All right, so what we have just seen is that also this relative forecast, they still depend on the mixture of articles heavily. So we cannot compare two different sets of articles. We can't compare two different days. And we can also not compare two different sets. For example, if we calculate the error in our case in UK and Germany, you can't compare the two values if the assortment mixture is different. And what we have done now to go around this problem, and this is partly adapted from the paper that I showed a couple of slides ago. Whenever we look at this demand error, we are adding two additional lines. We are adding a lower bound and an upper bound. The lower bound is this perfect probabilistic forecast that I showed how to calculate it. And the upper bound is just a naive error. We are just repeating yesterday's observed value, which is like, if we're not better than that, then there's a problem, right? And this is what we are actually looking at. So this is how our error plots look like when we do experiments or if we do monitoring. We have the blue line in the middle, exactly the same plot as I showed at the beginning. And then we have the red line, which is like our lower bound, this perfect holistic forecast. And you can see there's actually quite a bit of variation over time in this plot. And if we now look at this part here, where we thought our forecast is really good, we can actually see that we are barely beating the naive forecast in this instance. And we are pretty far away from the perfect forecast. And on the other hand, if we look at this part where we thought we might be struggling with forecast quality, actually we are doing pretty good, right, we are pretty close to the perfect forecast and pretty far away from the naive one, right? So it really matters if you compare your forecasts on different part of the assortment, this can be like either different times or different parts, like, I don't know, sports and kids articles in our case, or different countries, you really need to make sure that if you have this relative matrix, that you have your baselines in place. Otherwise, you are, like, doing some really blind stuff. All right. A couple of more plots. This one here, I'm putting in some other relative metrics. Right? The plots before, this was always our demand error. Now this has, like, MAPE, WMAPE, and, like, some other ones. And all I want to show is like this error is like the problem is the same no matter what your relative metric is Right. This is not specific to our case. This is like whenever you have this relative metric that does this normalization you are in trouble Yeah, and this is like the punchline right no matter what metric you take you always have to You have to have to watch out and just like it just Changes which part seems to be hard to forecast All right, then one thing we thought about is, okay, why do we take Poisson as a baseline? We could also add a bit more of the dispersion, so we took negative binomial as the next count distribution that comes to mind, and then try different rates of dispersion, which makes the perfect probabilistic forecast less perfect, more or less. We add more variants, and it's exactly happening what one would think. It's that the perfect probabilistic error goes higher. Poisson is the blue one on the bottom, and then there's the other ones. In production, we are using Poisson, because as soon as you start to do something else, where you have to parameter to wiggle around, it becomes really, really hard to justify how to choose your R. And we just stay on the safe side and take for song, which is like lowest variance that makes a lot of sense on account distribution. All right, and then we have the question, are we happy now, Moniz?
Speaker 2 [18:45]
And that depends, right? So you have basically, we can go now and say, okay, well, have we saved our scaling problem? All is great. We can now fit our error metric perfectly and then think, okay, well, cool, I have actually something to anchor it against and then can decide, okay, well, looks good or looks bad and then try to improve our models on this metric. Okay, so this, of course, is usually, in a larger scale setting, not the picture that you only look at. So usually you also look basically at other metrics, and this is something I would like to motivate now, particularly with regards to some sort of bias metric. So what we have seen is if we now want to improve on the demand error, then somehow, according also to what Stefan said, it makes sense to exclude low-selling articles from the training because it makes, from the business perspective, also sense to just concentrate on some high articles. We know that these errors are somehow driven by high-selling articles. It makes training easier, faster, and so on. And if I do that and actually train our models only on high-selling articles, then you can actually see that our demand error gets better, right? So on the bottom, you have, again, this perfect forecast. On the top, the naive forecast. And in the middle, we have the blue line, which is the model only trained on 50% of the low-selling assortment and the rest is high-selling assortment, and the green line where we just take the full assortment. So you can see, okay, great, I optimize for these articles, and my error metrics drops, even if I evaluate on the full assortment. Problem was, and this is something which basically happened back then, model went partially live and then you plot essentially how does your forecast actually look like. So we have the total sales, right, and then we can also check, okay, how do these sales in the future behave like for these two models. So the black curve are the actual sales and the blue curve is the model where we just saw that the error metric, the demand error is really good, but it seems to be that we have some sort of bias variance trade-off here so here we over predict we are biased towards these high selling articles and so we over predict and then questions get asked whereas on the um bottom we are the the green curve this is the one that was trained on the full assortment actually helped so just the the punchline essentially is just looking on one metric and overfitting on this one metric could be uh an issue if you also care about other stuff like if you want on the aggregate level to be more less correct and so on so this is like another another instance of that you should basically know your metric right so and what our conclusion was we don't only look at this one metric which which aggregates everything nicely together but we also look at something like a bias measure case and we look at something like a gmv error which looks at market level performance of our forecast right so we don't look at article on article level anymore but we actually uh look okay how biased am i essentially so you can also scrap these absolute values if you are interested in the sign of this whole um of this whole um yeah quantity questions now so what so what do we do with this right so we we have shown you instances about how to calibrate our errors and how to train models or evaluate models with regards to different metrics and how is this actually used in our case. So we use them both for internal model evaluation and also to fine-tune our models. But what we also do is, of course, we somehow communicate with stakeholders. So these forecasts are used downstream for pricing decisions, and this means also stakeholders look at the forecast accuracy on a day-to-day basis, and we somehow need to distill everything together here. And this is essentially what we show them. So we have a set of metrics. So here you can see the demand error that we have talked at length about. We have some sort of bias measure. It doesn't really matter how they look like. We have other metrics, the GMV error, and so on. And this is something that took a bit of time also to get calibrated to that, that we can show pricing managers in a traffic light system, and they are happy then with that, basically. So they can say, hey, our pricing was completely off yesterday. And then we can say, hey, it wasn't our forecast. So it was as bad or as good as usual. So this is something that we can answer here. And here I would like to highlight something that we have talked about and where you can basically see this calibration action. So in terms of the demand error, we have this traffic light system. So we calculate the actual metrics. times 100 here, okay, it doesn't matter, but what you can see is, for instance, a demand error in Lithuania of 91 is dark green, so it's a good metric because we don't sell that much in Lithuania. We have a lot of high slow-moving assortment, meaning that the expected error is high, whereas in a country like Germany, an error of 66 or 0.66, as we've talked before, is already concerning so something is happening and this means we can actually use this traffic light system here okay this one error metric is really off so this is like we need to investigate what happened and this means essentially we can use this in order also for model diagnostics so to conclude you should know the pitfalls of our metrics scaling trap are we biased does the bias capture actually the patterns in the data that we use and so on. We use reference forecast to anchor these metrics properly. And then we can also use them for outlier detection. For instance, if we have this high demand error that somehow speaks for an outlier, so we can go look, okay, is there an outlier that's in the data somewhere? We can do data quality or forecast quality checks and so on. Of course, then also depending on the business context, one should consider other metrics and avoid overfitting on just one metric as we have seen in the example before. But in the overall end, keep reporting simple for stakeholder communication. With that, we would like to conclude. Thank you very much. And we have attached our LinkedIn profiles and Zalando is also hiring if you're interested.
Speaker 1 [25:38]
Okay, thank you so much. We have a couple of questions for you. Do you assemble the gradient-boosted tree and transformer, or do you use one of those models per product?
Speaker 2 [25:49]
Currently, we use the Gradient Booster Tree separately. We have tried out assembling, but it didn't give us big performance boosts so far.
Speaker 1 [25:59]
Okay. Could you explain how you computed or driven the perfect forecast in easy terms?
Speaker 3 [26:08]
Yeah, so you're kind of cheating. You're just taking the actual observations and then you draw from Poisson distribution with the mean of this actual observation. So what you're doing is you take the observation and you add a bit of randomness around it. So that's the perfect probabilistic forecast and then you do this like for all articles.
Speaker 2 [26:32]
So it's essentially you say, I expect three items sold on average, but I mean if I actually forecast two, that's not too far off, right? So you account for that a bit instead of saying I need to be 100% right all the time.
Speaker 1 [26:47]
Okay. Isn't your naive tool naive border? It looks like struggling a lot with seasonality and trend, a lot of spikes. Yep.
Speaker 3 [26:59]
Yeah, probably the naive one should be naive seasonal. So maybe we don't repeat yesterday, but we should be repeating like on Tuesday, repeat Tuesday and so on. That would be a bit better, I think.
Speaker 1 [27:12]
Okay. Would scaling the demand by the cost of the product make a difference?
Speaker 2 [27:17]
Yeah, so this was a simplification that we did here. So this is actually what we do. So usually the formula takes the price of the article into account. So we just did it for the purpose of this talk that we didn't didn't show that.
Speaker 1 [27:30]
Okay. How far in the future did you or could you forecast for?
Speaker 3 [27:36]
It depends. We have one forecast that goes like for the next 10 days basically and we have another one that goes 26 weeks, but then on a weekly scale and not on a daily one.
Speaker 2 [27:48]
So different granularities here.
Speaker 3 [27:49]
Varieties here, different forecasts, different horizons.
Speaker 2 [27:52]
also different use cases like one we need to be directionally correct for like what happens in half a year the other one needs to be pretty on spot for the next days so
Speaker 1 [28:01]
Okay, do you decide pricing based on demand forecasting?
Speaker 2 [28:05]
It's an input to an optimizer and the optimizer then chooses the best prices. So it's the most important component
Speaker 1 [28:13]
Have you looked at metrics that consider temporal shifts and prediction before or after the actual demand ages peak?
Speaker 3 [28:21]
No, we haven't that sounds cool
Speaker 1 [28:25]
All right, what's the business impact of these forecasts?
Speaker 2 [28:28]
Yeah, that's always a good question. The problem is it's a bit like we we usually measure the forecast end-to-end because it goes into an optimizer So we need to check the impact also in conjunction with the optimizer How do we distill this into one number
Speaker 3 [28:45]
Yeah, this is super hard like if this is kind of like the holy grail of our department if like we ever get to the point where we can say like X point of Demand forecast accuracy gives us this increase in business metrics. We are like super happy our usual approach is like if we have something that Shows accuracy improvement we go through an AB test And then the AB test gives us the the business metrics that we can report and make a decision on if we want to roll this out
Speaker 2 [29:11]
So every improvement that we have basically can be measured in something like in the range of millions of euros basically, so single or double digit.
Speaker 1 [29:19]
Okay, how do you measure elasticity?
Speaker 2 [29:24]
Good point question. So we what we do is we all of these metrics are on observed data only right so we ignore here that we actually provide a sort of we call this a demand grid so we provide forecast for different price levels and then this gives us a curve and this curve we can fit to an elasticity model and this is how we would usually measure basically the quality of this curve. But this is highly counterfactual or non-observable, let's say, so a difficult problem.
Speaker 1 [29:55]
Okay, thank you so much. There are a lot more questions, but I think the people...
