Hierarchical Models in MMM: Can Structure beat data size?
Marketing Mix Modeling (MMM) often faces the challenge of insufficient data for specific regions, where the number of available data points is equal to or less than the number of coefficients required for prediction. This data scarcity leads to high uncertainty and unreliable coefficients, particularly in smaller markets. To address this, hierarchical Bayesian modeling is used to implement partial pooling, which allows models for data-poor regions to borrow statistical strength from data-rich regions.
The approach utilizes PyMC for probabilistic programming and the Hypothesis library for property-based testing of transformation functions. To reflect real-world consumer behavior, the model incorporates ad stock functions to account for the delayed effect of advertising and saturation functions (such as the Hill function) to model the plateauing of returns as spend increases. Testing ensures these functions remain bounded between zero and one to prevent unrealistic simulations.
Three modeling strategies are compared: pooled (all regions combined), unpooled (separate models per region), and hierarchical (partial pooling). While pooled models ignore regional variance and unpooled models fail in data-sparse regions, the hierarchical model uses a group mean and a deviation parameter to balance these extremes. To improve sampler efficiency and avoid restrictive distributions, a non-centered parameterization is applied.
Key takeaways include the importance of calibration and uncertainty intervals over simple metrics like RMSE or R-squared. Hierarchical models are most effective when regions share domain similarities, such as shared culture or audience demographics. However, they fail if regional behaviors are too divergent or if data is uniformly sparse across all groups.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Machine Learning & Deep Learning & Statistics and was classified suitable for intermediate domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
What we are going to show
- Country-specific marketing Data that is, unfortunately, never good.
- Function and Python transform like Adstock and saturation (with the tests so that you can see it in action).
- Differentiation between pooled, unpooled, and partial pooling.
- Meaningful diagnostics.
- Wins and losses of hierarchical modeling.
Why this is interesting and relevant
How do you model marketing effectiveness when you only have 12 months of data per country, some channels are interrupted for weeks, and your manager wants reliable ROAS estimates yesterday? Most teams think: "We need more data." But getting more data takes time, costs money, and sometimes isn't even possible (or the quality is bad).
What if you could get better estimates by changing how you model the problem? This is where hierarchical modeling and partial pooling come in. Instead of treating each market as separate (unpooled) or pretending they're all identical (pooled), we let markets share information through partial pooling. Countries with thin data borrow strength from the group, while markets with strong signals pull away from the mean. You get stability where you need it and flexibility where the data supports it. We show this end-to-end in Python: from building testable transform functions (Adstock, saturation curves, lag effects) to assembling three different model architectures in PyMC, to evaluating which one gives you calibrated intervals and stable ROAS estimates. You'll see the good, the bad, and the ugly.
Main challenges
- Making transforms reusable and testable_ Marketing transformations like adstock and saturation are usually hidden in modeling code. It is generally very difficult to imagine how they look, how they change the data. We pull them out as pure Python functions with clear signatures, unit tests (pytest), and property-based checks (hypothesis). This makes them composable, debuggable, and easy to understand and even improve.
- Building fair model comparisons: We construct pooled, unpooled, and hierarchical models with identical priors where appropriate so the comparison isolates the effect of structure, not prior choice. We walk through the PyMC code, show how partial pooling works mathematically, and run short MCMC chains that still demonstrate the key differences.
We go beyond "we reached 90% R2" to actual decision metrics:
- Posterior predictive checks: Does the model generate realistic data?
- ROAS stability: how much do channel estimates vary across groups?
We use ArviZ throughout to visualize traces, compare models, and compute these metrics. You'll see exactly when hierarchical structure pays off and when it doesn't.
Practical lessons and the repo
We share what we learned building this:
- Data checks and control using Pydantic, so you catch errors before MCMC runs for hours
- Test your transforms independently: Yes, for unit tests!
- Use synthetic data with known ground truth to validate the whole pipeline
- Calibration metrics matter more than posterior predictive RMSE alone
The repo will include:
- Typed transform functions (Adstock, saturation, lag) with unit tests
- Three PyMC models with matching priors
- ArviZ evaluation scripts (calibration, PPC)
- A Typer CLI to run everything on a predefined CSV
When hierarchical lose (and what to do about it): Partial pooling isn't magic. If your groups are genuinely wildly different and you have almost no data per group, hierarchical models can still produce overconfident nonsense. We show a scenario where this happens and discuss alternatives: stronger priors, splitting the hierarchy, or just admitting you don't have enough signal. The takeaway: structure beats volume in the right conditions. We help you recognize those conditions and build models that respect them.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:00]
Okay, hello. Welcome, everyone. Welcome on a bright, sunny evening. Yeah, today's talk, before introducing to today's talk, please remember one important instruction that whatever questions you have for the talk, please post it on talks.pycon.de. That is important. No questions will be entertained during the talk okay so make sure you use that talks app yeah so today's topic is hierarchical models in marketing mix and match can structure beat data science and this talk will be given by our speaker who is Muhammad Amin Jabari just a short intro about him that he is a lead data scientist based in Berlin who specializes in large-scale machine learning systems marketing mixed modeling and applied NLP so let's get right into it and I would encourage every one of you to welcome our speaker with a big round of applause
Speaker 2 [01:20]
Thank you everyone. So yeah, today we're going to talk about the eternal struggle between data and models. Which one would win? Before we go deep into the topic, a small introduction. That's me in the picture. I'm from Tunisia and I've been in Germany for nine years. And yeah, I'm a data science lead at TD Reply. TD Reply is a consultancy firm. We work for a lot of clients, which takes me to my segue that these clients expect a lot. They might tell you that your data is good. They might tell you that they have everything ready. But most of the time, it is not. So that's why I tend to say that data is expensive. Of course, you can put all your data in S3 for almost no money, but the processes to get that data and to ask them to go get the data usually take weeks, and in tight time frames of consulting, it is usually not possible. So what can we do about it? I'm going to talk about something called partial pooling. Maybe it's known for some of you. I will go into it a little bit. I'll talk about when it works, but also when it doesn't work. A bit of Bayesian modeling, just to set the scene, and I'm going to talk about testing in Bayesian modeling and marketing models, because my boss told me to. And yeah, the two packages we're going to use is going to be PyMC, which is a package for probabilistic programming. If you were at 11 today in the talk with Juan, he talked about it extensively, and that little animal there is a library that's called Hypothesis. It's a Python library that allows you to test, actually. It's called property-based testing. When you have a property of a curve that you expect and make sure that that test will kind of challenge that property to make sure that your function doesn't explode. Meet Volta. Volta is a music distribution company. I chose music because I like music and I dream that one day I can make music to be distributed to the world. Volta is specialised in putting music on Spotify for different musicians. In this case, we're going to take one musician, we call him Amin, and we're going to try to analyse things like Meta, Spotify, TikTok, YouTube, radio and playlist pitching. For this one, it's when you are a musician, you want your playlist to go into Discover Weekly for Spotify, or a chill, focused playlist when you want to work, and that song would appear there. So these are what we call marketing channels. This is how you would get exposure, so people would listen to you. What we're trying to explain is the revenue you get from the streams. So, a lot of charts. Let's go slowly. So, this is our time series of the revenue that that artist so far has made. As you see, Germany and UK, we have quite some data. It's quite complete. You could make quite a good model out of it. But if you look at Sweden or Poland, you have only six data points. and that is horrible. But the client or the musician or the distribution company, they still want to know how can they create an uplift in these specific countries. That is the first problem. Second point is that somehow the data collection of TikTok in Poland did not work. So we have no data there. Second problem. Third one is the data to parameter ratio problem. So when I started studying data science, they told me for each 10 rows, you can get one coefficient to predict. I don't know if it's a myth or true. I think there was something related to like a normal distribution about it. But nevertheless, now we are in a worse position when we have six data points and we have six coefficients we need to predict, which is also, or sounds kind of infeasible for Poland and Sweden. Last point is our spends are quite volatile. So there's a lot of outliers. You have eight countries you want to predict and you have one month to do it so how to as I showed all these countries are quite different but they're also a little bit similar and this is when we start thinking a bit more about the problem first all the eight countries I showed you they're all in Europe that is great because Europe has quite this at least the countries that I chose have a quite of similar weather has similar holidays to some extent, has similar culture. We are also, in this case, trying to explain the streams of the same artist. I mean, and the audience. Usually, people who like rock are quite a clear audience, or like metal in this case. So, there's a lot of similarities. And the most important assumption, if you spend money, you're going to get revenue to some extent. we'll see why later. So, three models we want to try. First one is what probably the most pragmatic person will tell you. Sum everything together, consider them as one data point, run the model, whatever you're going to get, it's good enough. Second one, eight countries make one model per country. That will probably work for Germany and UK because we have 104 data points, but for Poland, making a model with six data points, not sure it would work. Third one is what I'm going to show you, which is kind of a middle ground solution. Use the knowledge that you have from Germany and these countries, try to give parts of it to Poland and Sweden and all the missing countries, and try to end up with a nice middle ground. So this is the equation for just a regression model. So what we're trying to predict, Y, which is the revenue per country at a certain time. You have alpha, which is how much revenue the streams would have made if we don't advertise at all. And then the sum of the different coefficients. In this case, we have six. We say this meta, TikTok, Spotify, et cetera. How much we spend on them and some error rate. And this is where the caveat starts. Yes, if you spend, you get revenue, but at some point, it's going to plateau at some point. Think about it. I want to be famous in Sweden. There's 50 other people who want to be famous in Sweden. How do we share that piece of cake? Am I going to reach everyone? But at some point, I would have reached all my audience, and that would have stopped. Second point is that the ads in themselves, they don't impact you directly. And to be honest, this is my favorite part of my job and what we do. It feels like a bit of sociology, psychology. If I would listen to a radio today, I might hum the song. One week later, I will know the lyrics. By a month, it will probably be my favorite song. And that's what this function kind of shows you. I'll go in it very deep, but if you want more, very fast. But if you want to go deeper, talk to me later because I have only 20 minutes left. So ad stocks, as I said, 100% of your spends will not affect me 100% today. They will affect me over time. I buy radio for three months. So once that spot of, hey, listen to me on the radio for the next three months is going to be spent once. But the effect is going to go slowly, slowly until it sticks in people's mind. This is what we define as ad stock in this case. and we need to include it in our model if we need to really explain reality, not explain like a point in time. Second thing are the saturations. So saturations are, like I said, doubling your spend doesn't mean you're going to get double the streams just because you would have reached already everyone at some point and any money you're going to spend more, if someone doesn't like metal today, they're not going to like it tomorrow, even if you spend money about that song. and the testing part I'm trying to finish it fast so that I can tell my boss I did it testing so actually it's not true I love software engineering and testing okay so testing this is exactly what I showed you here but 50 times so hypothesis is kind of a library where you give it the state giving that amount k so k in this case is when the spins would reach 50% of their effect Giving that K is between 0.05 and 5. S is the steepness of the curve, how fast you want it to go up or to flatten at the end. Giving K and S, now do N examples. In this case, I chose, I think, 50 examples. And the deadline is none. Deadline is none means if you fail or if you find an error, when you go past the boundaries of 0 and 1, I will explain why it's dangerous, then stop. But in our case, we said deadline is none, So even if you find errors, you have to keep going. And then we call our hill saturation function. So why do we need to test saturation specifically? Because most of the time at the end of the projects, clients come and tell us, it's a good model. Now can you simulate what would happen if I change my spans to some extent? If at some point the total spans or their effect go past one, it's quite unrealistic because the whole effect and information should always stay between zero and one because this is the maximum spends that the person did. So if it goes past one, it's kind of unrealistic. So this is a function we're going to use when we're going to simulate. So when we do a simulation and that function is wrong, we're going to tell to the client, yes, you're going to increase your streams by 50%, while in reality, they could have only increased their streams by 20%. They would have lost quite some money for no reason. We do this usually for a lot of the other transformation functions like add stock and other functions. Now back to the modeling. So we saw the transformations. We saw the different channels we want to predict or understand. And we do our first model. So Bayesian modeling is, I don't know if I would call it counterintuitive, but we're not used to it because we're used to the usual frequentist approach. You can think about it as instead of letting it find the betas, we tell to the model, the betas are most likely in a normal distribution between 0 and 1. And this is my data. Now, 4,000 times, choose one point in the distribution. Compare it and see if it works with the data. If it's good, then stay in that area and keep discovering. If it's not good, go to another area and try to discover. But you have to stay into that zero and one. This is basically what Bayesian, or what we're going to see here in a nutshell, is. The first model, if you look at it like this, it's not too bad. And green is the real coefficient, because this is synthetic data, so I know what the beta is supposed to be. So we actually caught five out of the six. This one was a complete fail. But personally, this one and this one and this one also failed, because what you see, the two blue bars, are what we call the uncertainty. The longer these bars, the worse it is. The shorter the bars, the better it is. The worst thing that you can have is something like this. The playlist effect is most likely at 0.2, but it can be minus 1, or it can be 1. If you go and tell someone this, they will tell you, I can do nothing about this. Is it negatively impacting me or positively impacting me? And that's quite a problem for us. So we can do nothing, basically, with this model. The main reason is that that coefficient worked well because all these values here are quite close to each other. So even when you summed everything together, we didn't really lose so much information. It was quite okay. Nevertheless, something like this one, the space between all the countries is quite different, or the real value of all the countries is really different, and that's why the model doesn't know what to do. So we go to the next model. Next model, we try to explain every country by itself. Germany was, I think, okay. For example, if we take Spotify, what is the effect of Spotify? It's between 0.5 and 3. Still too wide for me, if you want to be sure of my opinion. But Sweden and Poland and Italy and all these that have so less weeks of data, it was, again, a complete failure. And if you look at Poland, who has six data points, I would not expect anything, to be honest. The width of the certainty is very large. That means that the model doesn't know shit. So, what we want to do now, all the points that I showed you, they exist in the space of, okay, how effective is the channel? These are the countries. We're going to try to pull and pull all the countries partially together and get them closer to the average. Why? Because, like I said in the start, they're all European countries, there's a lot of similarities, So we assume that the effect might be similar. Whatever you're going to do on Spotify in Germany, yes, it probably might work in Poland to a certain difference, so we cannot take what Germany did. So let's try to assume that and try to include it in our calculation. So what we're going to end up extracting at the end is the mu of k, which is a group mean. So the group itself has an average. We want to find that. That would be interesting to know what is the average of all these countries together. than how much the countries deviate from themselves. The bigger this, the worse your model is going to be, and this will prove to you that you shouldn't use this method. So this would be usually quite low. Z is for each country how much it deviates for the center of the channels, and beta is the coefficient of that specific country that you want to get. It's kind of a compromise to some extent. So we start like this. The width, I want to focus on the numbers up there, they're quite interesting. So Poland, we start with a width of 1.8. It was not bad, but we were failing to catch meta here. Some of them were negative. Then we did the next model when the width increased a lot. So a lot of uncertainty is now in the model, unusable. And then we end up with a third model. Still some problems, but as you see, we're getting very close to that green dot. Optimally, this should be next to the green dot. Now, some of them are still the negative. the width has decreased on average so we're back to 1.8 and this is due to the fact that I created a terrible data set where there's like a lot of multicollinearity I want to really to keep it as close to reality as possible usually you cannot do a good model with only six features you usually need more you need the competitor you need seasonality I didn't include any of these things so that's why we are not optimal and it will not be optimal but it's a showcase so to give you more detail. Here, as you see, Germany, we're very close to reality. France, not so much. Sorry. Poland, we're actually quite close to reality. So I think the models improved a lot compared to unpulled. So unpulled, as a reminder, each model is per country, one model per country, versus this one, the blue is like the concession we did. I had to put code because it's PyCon. So this is a pooled model summarizing everything together in one equation. So like I told you, we are assuming that we want to find beta, whatever coefficient it is, and we assume that it should be in a normal distribution between 0 and 1, and try to find it for all the channels. This is what this syntax is. Usually the models are way bigger, but I wanted to just focus on what I change. pulled here. Beta should be in a normal distribution. It should be centered in zero and can deviate up to one time. And please do it per channel and per country. So they're completely independent. Then the partial pooling. When we say we have a beta of everyone this time, we start with a beta for all the countries together. And it should be in this zone. We try to keep it small because we want to actually find the difference. We put it also in 0.5 to let it deviate and stay quite positive. That's also another discussion that there is an assumption of positivity about the spend. Sigma is how much the countries can deviate from each other. And then the last beta, which is of the specific country, should be in a normal distribution, but it should be close to the centre or close to the average of all the countries and should deviate this much. This works, but creates huge problems for our sample, which is what is going to go find the solutions, because it's very restricted. Let's say the model decided that beta is 0.5 and that beta sigma is 0.03. The freedom of the sample to try and explore the coefficients of the count will be very small, because I have to stay in 0.5 and move very little at this 0.03. So there is a solution for that. It's called a non-centered parameterization. We start the same. This doesn't change. There are still the two things. So we have normal, half normal. But we add this offset. So the offset is another feature we add that gives more space for the sampler to go discover further. Small tweak, that fixes your model. Now, this chart is actually pretty cool because it tells you how much each country took from the average. And I think, I'm not going to lie to you, I didn't do this at work before, but I think I'm going to start using it every time. Because it shows you that Germany, or the countries that have a lot of data points, they did not really take a lot from the average because they're confident about their numbers, I know what I am. I am 104 points. You don't need to help me. I mean, in some cases, I need to help. Again, some of these numbers are messed up because of the multicollinearity, but if you go to Poland and Sweden, on average, these two rows are like 50%, because they actually don't have so much information. They have just six points, and they need help from everyone else. How did we improve these models? So, pooled, reminder, everyone together, one model. It had a high RMSE, but it was quite calibrated. And this is due to the fact that Germany did the most of the lifting and did the work, like the countries that had a lot of data. Unpooled, we improved, again, because the big countries did most of the work. So if you average the model of all the countries, it's going to be fine. But the uncertainty gets very high, and our solution is kind of a nice middle ground. We're still very close to the unpooled model, and the certainty improved a lot. That is hierarchical modelling. When it doesn't work. So, first, imagine we have eight countries, eight data points per country, for all the countries. Then there is nothing you can do. The priors that you chose, that normal distribution we talked about, are going to take over the model, and it's not really going to be a representative model for reality. Second thing, Amin wants to go on tour in Poland as soon as possible. So there will be a weird peak in that country, or maybe it did already a big tour in Germany. So that's going to create a very different effects regarding the time, and the model will be worse. So you have to add what we call time-varying coefficients so every country will have its specificity in the events in time that happen. And probably the worst one is choosing the pooling in a wrong way. An example here is like, I assumed I recreated my data, but I assumed that TikTok is very used in Sweden, Italy, and Spain, and not used at all in Germany, France, and UK. In this case, it's Europe, you assume, so you put them together. But there is a very big difference, because TikTok has almost zero effect here. And here it's almost one. You cannot put them together in one model. The model is not going to be able to reconcile this. So instead, you will then create one model for these three countries because they're simulating their behavior and a model for the three countries. For me, that just says that a lot of what we do, especially in Bayesian modeling, really has to do with understanding a lot the domain, what is happening in that country, spending a lot of time doing analysis before modeling. Learnings, test your transforms, very important because if you simulate wrong data, it's going to cost you a lot of money. Second, calibration and certainty matter, not only RMSE, because as you saw, my RMSE was not great, but my calibration was improving over time. If you add four other features for seasonality, for competitor price of Spotify maybe, some kind of things, then your RMSE will probably improve a lot, but always look at the calibration and the certainty. And synthetic data helps. It helped me really tweak, change. I discovered the shrinkage chart by creating the synthetic data and seeing how good my model was improving. And that's it. Thank you very much.
Speaker 1 [23:22]
Yeah, thank you, Mohammed, for your talk. Yeah. One question is, how did you perform validation using hierarchical modeling?
Speaker 2 [23:34]
Test-train-split. Is this the question? How do we validate?
Speaker 1 [23:39]
Like the validation set?
Speaker 2 [23:41]
Yeah, the validation set, so yeah, you have to take, if you have, in this case, probably Poland and Sweden, you probably cannot validate it, that is a good point, but for things like you can at least validate Germany and validate the countries where you have more data, that could probably allow you to get a signal if your model is doing great or not.
Speaker 1 [23:42]
Yeah. or what all metrics you use for your evaluation.
Speaker 2 [24:04]
So in this case, I showed RMSE. To be honest, this is a very interesting topic because we internally try to avoid to use R-square. It's quite good to know about it, but it's just too good to be true most of the time. And clients love it, so we tell them that. But we also try to tell them, but we also try to include your domain knowledge and the model is well calibrated. And so at least in my experience, When I use Bayesian, I usually tend to have three or four different ways to validate what we are showing to the client instead of just R-squared is great, good luck.
Speaker 1 [24:44]
Will this approach also work for data outside of Europe, like can this be extended to all the countries?
Speaker 2 [24:52]
Yeah, the example of Europe was just because I thought of which place has some similarity. Yesterday we were talking about the topic, I was talking about we could do it also for North Africa. You could also do it for people who like pizza. You can put them together and make something. You just need to, before writing your model, can I have enough similarities? And that's why I spent so much time showing you all the charts and slides, because you have to really see the similarities and do extensive data analysis and what we call artisanal work code and all data science work to really make sure that what you're doing is actually good or not before doing the model. Probably the model for us, it's 20% of the project and 40 to 50% is a back and forth data analysis, assumption collection, talking to the client.
Speaker 1 [25:40]
All right. That's it. Thank you, speaker, for a wonderful talk, and let's end this with a big round of applause.
