The Forecast Whisperer: Secrets of Model Tuning Revealed

Speaker 1 [00:07]

Thanks a lot for coming. So, last data talk before going home and having final weekends. I'm already dreaming to sleep normally enough in my bed at home, so not sure about you. But let's do it. So, I'm Ilya. I'm joining PyCon since 2022. So, on the right you can see my first photo from my first PyCon for LinkedIn. And I started working with data at Large Hadron Collider, and of course everybody knows that quantum physics is fun, but actually I wanted to do something with real life connected. And for the last five years I'm delivering forecasts and optimization models to various businesses at Blue Yonder. By the way, who is working with forecasting, please raise your hands. Oh, nice, nice, nice, nice. So I will be judged by experts, cool. Today I would like to show some less common forecast tuning aspects that address real-world problem that we are struggling with different businesses. We will start with business objective, discuss a new data centric paradigm and talk a little bit about interpretability. And let's start. So imagine your boss comes to you with super idea or a stakeholder or new customer and as to create or improve the forecast model. And the target, obviously, improve accuracy, just decrease error, easy. But actually each metric could be leading to different forecast and actually each evaluation set up, you're forecasting for next week, next month, horizon one year, will lead to actually different optimal forecast. And then accuracy is not business objective. But you're smart and say, yes, it's easy. For me it's easy. Just tell me what exactly metrics, I will do everything else. But you know the background that it's not so easy. And this is a bit slight about statistics. Imagine we have two sales examples. One is posted on distributed with a rate around 9.5. And another posted on distributed with 0.6. So it's typical sales. So it's count numbers and we would like to optimize some forecast for this particular metric and let's see what numbers we get. On the left side, if we optimize for mean square error, we will achieve the best mean square error when we will get actually to the mean of our Poisson distribution. It means it will give us around 9.5 to the left and actually 0.6 to the right. But if you take mean absolute error, we will achieve the minimal mean absolute error with actually median of our distribution, of our sales distribution. It means that when we will try to optimize for something, some real world sales that kind of follow Poisson distribution, we will end up with 9 for the left, so kind of high selling units where you sell around nine points per time bucket and to zero actually as a right and now you see like okay we have 0.6 and 0 and the kind of it sounds like quite a lot especially for some things that distributed as a right example but actually it's quite quite popular real was example to work with slow sellers. Let's jump to actually what we like to see is time series. I think in previous sessions you've seen a lot of nice smooth patterns with weekly patterns, monthly patterns, nice trends and so on and so on. But the reality looks like this, you know, it's count values, daily granularity, it's typical example for example for manufacturing shipments. And huge, huge, look at the y-axis is like going up to 600 and a lot of days is just zero so it's real shipments they don't ship every day and you need to forecast actually there is some seasonality some trend hidden and what should we do okay and even before doing this you come to the stakeholder and ask like for what will you use the forecast and planner or some stakeholders say like okay just Optimize WMAP. I mean, to me things like they use this WMAP from prehistoric period and will continue in further. Okay, but actually business process will care about profits and if you assume some simple profit formula like revenue minus cost minus lost sales minus overstock, we can also estimate a profit as it comes from our forecast. And in this case of course the profit will be punished differently under forecasting and over forecasting. So again it will be already some bias metric. We can do some wipe coding, take some super complex model, whatever is on the hype. I took some temporal fusion transformer and the architecture is not super critical for us. We know there's a lot of parameters, a lot of things to optimize and there is some loss function behind it. And we go and take simple mean absolute error. We import mean absolute error and actually we are in PyCon, so there's Python code above. And we take a mean absolute error and try to optimize it and here how you see how we predict sales. So orange line here is actually our forecast that optimized mean absolute error. And you see that on daily granularity mean absolute error is kind of okay-ish. It gives something, taking that the sales are quite high. But what planners or stakeholders might care, they might care actually about monthly WMAP. And then it's kind of not really nice. And actually the forecast looks quite awful, we just forecast close to zero. And the final prove it we calculate and it is minus 16,000 so maybe not the best idea for metric selection we think a bit more and select something for more this sparse data and remember that MSAE was optimized by the mean and maybe we it's good to forecast more as a mean of the distributions and median in this sparse use case. So let's try MSE as our loss function. What do we get? We get worse, so now we have 47 instead of around 30 MAE, but much better WMAP. It was 100%, now it's 37. And the forecast, well, it doesn't show any really good nice patterns, but at least it's moving in the right direction. And profit as well, 3,000 okay at least we are positive we will not run bankrupt tomorrow but can we go better less what we usually do in our modeling let's do hyperparameter tuning yes we take for example for Bayesian tuning we take some open source library I took Optuna and I tune based on mean absolute error and go to profit increase the profit, nice. Okay, but I remember that planners actually want monthly WMAP. Let's go, let's define monthly WMAP, put it as an objective in our hyperammeter tuning, tune the profit and get, wow, now it's double, so it's much better profit, we have much better WMAP, remember in previous iterations we 30 something, but worse MAU. So you see how actually on different granularities metrics start diverging based on whatever your loss function selects, whatever your objections, whatever your tuning and etc etc. But then you remember, okay but what about there was some profit stuff, maybe we can directly try to tune our model for profit. Maybe it will be not in part of the loss function but maybe at least we can use it as objective for our hyperparameter tuning we take we define our total business profit calculations put it as part of objection objective and voila we have even worse mean absolute square we actually learned some looks like some pattern maybe there was some day of the weak pattern behind it something but look at the double map it's almost the same even a little better and profit perfect yeah and surprise surprise unexpectedly the optimized model for profit and the guess the highest profit so nobody expected it and but in reality it's not something that often happens Often data scientists sit out in the bubble, optimize some particular metric and think that business users will tweak forecast or use the forecast somewhere separately to optimize the business objective, whatever. But it's not what quite often happens. So quite often they just take the forecast. So let's summarize. Business objective is a loss. It's possible. yeah there are some challenges like to make it differentiable to make it smooth and nice and etc you can surrogate create some surrogate differential loss and there's nice paper summarizing this when they taste different loss and also define task oriented prediction network you can do it but since in the business cases quite often your forecast is not used for one business case but for multiple business cases might be it's not the best way. You might quite overtune and produce a biased forecast. What could be easier way is to select reasonable metric for loss function that will be rather unbiased for your use case or perform needed data transformation and then optimize based on the, optimize your for example hyper parameters, optimize your additional feature engineering or introduction of new features based on business objective. In this case you can really boost your business objective or whatever planner KPI will have for you. So now let's talk a bit, be done with a bit with the modeling, with this vision we defined, we discussed with the business objectives and let's a bit zoom out and think about the full modeling activity and actually role of the data in the picture. Traditional way is to think about IML models like kind of machine where we put garbage in, we get garbage out. What does it mean? What does it imply? It means we don't want bad data. We would like to get good data, clean data. So we do the cleaning, we do pre-processing, feature engineering and start with the modeling. We freeze the data, we start with the tons of modeling activities but Andrew had a little bit different opinion what he said like okay actually we have such advanced model so you you seen from the foundation models there is out ML there's so much good stuff maybe we can focus a bit more on data and what he proposed he proposed let's try data centric actual approach for modeling where you freeze your model that a little tuned before and you iterate on data and try to iterate on data to see how you actually you can improve your metric or objective at the end. Even they introduced the corresponding computation, it was about handwritten Roman numerals recognition, not really about forecasting, but we can actually learn something from this. What they did, they did label correction, so they identified and rectified mislabeled data points, as well as they did a lot of data augmentation like the domain agnostic pipeline, additional data validations, cleansing and additional augmentation like rotations of the data. Yeah, maybe rotating time series data is not an option for us, but let's think what we can actually do in this domain, in this let's switch, we freeze our model, we try to improve now data. We can check label consistency. It is forecasting use case, we have product attributes, for example, if we're shipping products, we have location attributes, hierarchy, a lot of this stuff. Maybe we can do check label consistency, improve our labeling, improve our tableau information, and since everybody loves Gen-I, I include this even some Gen-I library here that actually can help so I have tested this actually can help improve your tableau data a little bit and extract some additional for help instructing additional information from description then we can detect more errors and think about what story coming from the errors and argument our data with additional features now in this data centric approach let's tackle two problems to real life problem first problem we have some again time series data this time we have sales this is weekly granularity and you see that while average is somewhere around 175 there are some clear weeks with clear zeros some ways one week some ways many weeks what is it like supply shortage, evergreen blocked service canal, office is closed because they have some sudden holidays or lockdown because of the COVID. Maybe we know, maybe we don't know. Maybe we don't have data actually from somebody who is providing data and we have no way how to collect this data. We need to work with what we have, we have time series. So and even we don't know if a product will be available. What's the impact of the forecast we run some naive model and obviously we learn some drops from the history and start forecasting these drops from the history that we might don't want to forecast and what we can do in data centric approach we can identify the change points for example okay from Bayesian let's say I'm identifying the change points where I have clear sequence of zeros I can check if what is the probability that this clear sequence of zeros happens based on all my historic sample and we are now in data centric we don't want to introduce new features on twigs a model what we can do we can actually filter them out from our whatever model you use either as a table or from your time series, you are cleaned out, and voila, your focus is actually much more reasonable in this case. So yeah, there's also some random drop happened, quite unfortunate for us in the same time period, but you see that the focus stays on the level in comparison with what we had before. So this is one way how we can approach. Let's talk a bit about explainability and another use case. Please raise your hands if you know what the Shapley values means. Okay, I see some people don't know, so you can Google. For our use case, what is important to understand, so we give input features, like for our forecasting We give some product attributes, seasonality, and etc. And as output, as Shapley values, we get a marginal contribution of each feature, not only the forecast, but also what actually impacted our forecast. For example, here, age impacted our forecast positively, gender negatively, and etc. And here you can see the example, if you see it close enough. You see some now rather smooth seasonal data. And orange, our forecast shows quite some drop. And then if you use big models, and the big question, like, where does it come from? Maybe there are, like, 100 features in your model. You run Shapley values. Let's say you introduce three Shapley values, one seasonality. You can see what is the seasonal effects, up and down, one prices. You see there was no really much price effect. And then promotion. And you see there was clear promotion effect that actually lead to the decrease of our forecast. And okay, if we would be a bit in model domain, we could say, yeah, let's work in promotion implementation. Let's maybe cap the effect of the promotion or introduce additional loss contributions that will penalize negative promotion effects. But we are trying to be data-centric. We look at the history and see that these promotions actually doesn't have any real impact. It's some just ongoing promotions for a long time that customers don't really care. But one occurrence of these promotions happens during COVID. When there was overall decrease of demand and model learned this correlation and we remember ourselves that correlation is actually not equal to causality. So the model learned this correlation and introduced this correlation in the future forecast. So in this interoperability we could catch this wrong correlation that we introduced in our forecast, go look at the data where these correlations were learned and fix actually the data. and that's what we're doing and in this case we just remove the promotion like because nobody really cares about this promotion it doesn't have any impact even if the customer believes there was some impact no not really people really don't quite often don't care about promotions so we clean the history and voila the forecast looked much better so but i will not show how much better It will be a secret. So coming to my summary, what we have discussed, first of all, important, even if you're working with data science, with models, and et cetera, you think it's everything what you need, it's sometimes important to understand what is the business objective and where your focus will be used and how it will be used. a key for modeling and actually key for your model to success to succeed in to deliver some value and you can implement it directly in the laws or in your optimization or in your feature engineering but you need to implement this objective somewhere and what I would recommend is to mix model centric and data centric approaches so it means that okay you need to observe what What story data is telling you? What actually correlations are in your data and is it consistent with reality, like with our promotions that actually didn't have any impact? Or does it tell the wrong story because of multiple data errors, anomalies, mislabeling and et cetera, where we can improve our labeling? And second question, we ask ourselves, what actually our model learns from data? correlations will learn by our model on based on our feature engineering based on our this one our data problems and this is where a lot of explainability enter the game and I would quite recommend them give it a try and base your modeling on explainability as well so that's it from from my side thanks Thanks a lot for your attention, and if you will have any questions, you can come to our booth. I think it will exist in the next 10, 20, maybe 30 minutes, so we still have some time. Okay. Any questions? Very quick. Here. Here, I'm coming.

Speaker 2 [21:59]

First of all, thanks for the talk What I was wondering is if you remove promotions that don't have an uplift The model will never be able to understand or explain which promotions will be silent So if you predict all the promotions will see an uplift or an higher uplift Doesn't it this include a bias for higher uplift for promotions?

Speaker 1 [22:22]

Yeah, maybe I was not 100% correct usually there's tons of promotion types. Yeah, and What we quite often observe it doesn't make sense to implement all of them. Yeah, so it's better to identify what real promotions What promotions really make an impact and implement only those promotions. It's really make an impact And this is one was one of the promotion types is just ongoing like like a typical bonus for some new signing customers and etc. So this actually at the end doesn't play a significant role.

Speaker 2 [23:02]

Okay, perfect. Thanks Hey also from my side, thanks for the talk I was wondering about this data centric tuning from an MLOps perspective because obviously changing the data also changes The the model you you get in the end, but if you leave out some data clean some data how can you track this with an experiment tracker and And do you have some tooling and MROps experience there?

Speaker 1 [23:32]

Yes, I mean, first of all, of course, you need good data versioning, yeah?

Speaker 2 [23:33]

Yes.

Speaker 1 [23:37]

So it's not only about model versioning. So you really have good data versioning with clearly monitoring of data drifts and what is happening. So I would say it's similar way with model tuning when you tune your data, with, yeah, Similarly, you save your model parameters and model metrics at the end and historize them to observe the same you do with data. Okay. Any other question? Well, if not, please let's thank again our speaker.