Using Causal thinking to make Media Mix Modeling

Speaker 1 [00:07]

So my name is Carlos Trujillo, I'm lead data scientist at Wise. I don't know if you ever used the app or have ever transferred money abroad. This is one app of a finance app and at the same time I'm principal data science at PIMC labs. If you ever use PIMC as a framework, PIMC also have like consultancy side where all the writers of the packages that contribute in open source are able to as well you know provide consulting and yeah I'm there as well so today I want to talk a bit of a statistical media mix modeling but before of that I also want to have a sense of the room so I want to know how many of you are familiarized with regressions raise your hand nice how many of you are familiarized with Bayesian modeling great okay so I will not need so much introductions and the final two questions how many of you are familiarized with marketing okay and what about causality or causal reasoning you the apparel theory yes nice great so what we are but I just want to quickly define right now is media mix models and to put it in simple media mix models are what I like to call glorified regression model. So we are doing a more sophisticated regression model to understand what is the impact of certain drivers. Those drivers, as you can imagine, the media in part of the name, it's media advertising. So we do marketing media advertising, we spend money in certain channels and we want to understand how much money are we getting back from what we spend. So that's the whole logic and And in order to do this, we apply these sophisticated regression models. Now this is, let's say, the first layer. And this is interesting. We can do the math. But what I have been more interested in the recent years is about causal media mix models. And what the causal work brings right now is just a bit of additional structure. Because when you are building these models, you're actually interested in understanding what is the impact of your intervention. So you are not only interested in understanding what is the, I don't know, probability of Y, given the data, given some X, you actually want to understand the probability of Y given do X something. Once you intervene X, what will happen? That's the question. What is the causal impact of our spend? In the same time, the good thing of these models is that they are very interpretable, so usually every time that we do these models, or these causal models, we can also drive down this causal DAX or direct acyclic graph where we can basically also observe how our variables or our different nodes are related between themselves and against the target variable that we want to model. So I usually face a scenario during these days where we have you know this causal models that we spend a lot of time structuring and defining and running and somehow they say something let's say oh you know channel X worth two million dollars but then you do a simple A B test or you do any type of experimentation and all of the sudden you see that should be actually like I don't know seven million or five million or whatever so again models can or are usually wrong and sometimes you do experimentation and you know that you actually do something in the world and you see that it's different from what the model interpreted at the beginning and for that reason we invent or not invent but we came up with an idea of calibration and calibration it's a very interesting idea more in Bayesian models in simple we are just taking the likelihood from our model and we are appending a new observation that should be fitted in the process so we are not already only fitting against our observations we are adding an additional observations that you feed the contribution of an specific parameter so we have again the parameters that estimates the value of x2 and we have an observation where those parameters should fit to make x2 have certain amount of contribution against the target on top of the regular feeding where we are trying to reduce the you know the square errors against the target against the total target givens the rest of the parameters that's more or less in simple what you can see here and we can do that because every time that we intervene in the world we know our actions we know how much spend did we spend extra like 20 30 percent we know this Delta usually if the experiment is successful you also know the Delta of the outcome so you increase 20% and then you see 5% extra of new sales or users or whatever and using these things again you can append these to the likelihood so we can get a calibrator posterior distribution so these zones very nice and this it's a very easy way to sell to your stakeholders or managers because you go you say like oh I will do some model we We will get some results, they probably will be wrong, but then we have experiments, and we will append all of these experiments to the regression, and that's it. We have the perfect model that will always say the right answer. But what I want to say here, oh, and by the way, usually the information from the experiments, my bad, came from instruments like difference on difference, synthetic control groups, regression discontinuity, or regular randomizing control trials like A-B tests. So you can run any of those type of experiments, either observational or control it, and then you can use the information from them to fit your regression. But sadly, and this is the point, and hopefully we go to it by the end of this presentation, calibration is not magic, it's just math. And that means that we need to treat it in that way. We can just append experiments indiscreetly and just expect to get some output. So I will just show you this in the Bayesian way. And what I mean in the Bayesian way is, like, we will create or I create a small set of data or a simulated data set where I generate in purpose some expense channels. I generate some, you know, noise and special events and some trend and contributions. And given all of that, I will create a target variable that then we will analyze. A small shout out here, I did this using PyTensor, which, if you don't know, is basically the back end of PyMC, and allows you to do a lot of very interesting things, more if you are trying to do graphical models or data generative processes. So in this case, as you can see here, I just create four vectors, I give them some values, and then we have all the channels. I repeat the same thing for the components and then I just start to define the structure between those channels so once we have all the parent notes all the things that we see before I can start to find like how all of those variables relate and again I can define a graph in this case this graph that you see there where we spent some money that money generate some exposition that exposition correlates with other channels and we also have events like that one there that increase also the amount of exposition that we have against our users you can imagine this like I don't know Christmas so for sharing Christmas either you spend or not some more more people will see all of these you can write it again in PyTensor which is very handy because then you have the possibility to actually express these graphical model in PyTensor and print it and you know just get an understanding actually of you know if what you are running or the more abstract model that you were writing here is actually what you did so you can use I don't know functions like I don't know the py.print and then you can see how all the variables are related and if your more abstract representation is actually close to your more explicit graphical representation so I don't know if you have ever used by tensor this is just a small shout out that you should use it we are contributing a lot and it's a very nice tool if you are working with graphical models but by the end of the day if you repeat this process several times with several variables we will get the target which is this one that will be representing this graphical DAG that we have there, this causal DAG, which at the same time will have the following underlying structural causal equation which is the one that is under that. So this is our data, this is just for context. Now that we have our data we want to run a model, a media mix model that you can build in PyMC with a few lines of code, defining your priors and a few other things and then running it basically. So right now if you want to use or you want to build MediaMix models you can actually use PIMC marketing. This is just an out-of-the-box tool that we build inside of PIMC and just defining the type of saturation or add stock effects that you want to observe, you can feed your model in again few lines of code. To give you also an introduction, in MediaMix models we have these two main effects which are adstock and saturation, basically saturation is talking about like the diminution returns effect of a spend so we spend more and more and more but that spend is not returning the same amount of value as you move forward in X and on the other side you have the adstock which is the lagging effect I spend today but because I was exposed to people a lot today I will get some returns in the future even if I don't spend in the future so this will call it kind of the brand effect so those two things are will be in our model and these are things that we also want to estimate from our marketing effects we feel all of that we get our PIMC model or our Bayesian model is what we have there we can also print it again because all the library like all PIMC itself is an structural causal Bayesian modeling process we can get a graphical representation of whatever that we are computing. So it's very important to understand that every time that we write, let's say, any PyMC model, we are writing a model of the world, a model that follows a specific structure, a model that follows a specific logic. We are not just writing a simple equation. That equation will have some meaning under it. And even if you don't know it, even if it's explicit or not, it's present. So, we feed it, and as in any model, you will see, oh, my sampling is running properly, I don't see any major problems. If you do some checks, you see, well, the feed is kind of caged, I get some interpretation of the channel, so, I don't know, X4 is great, X1 is poor, X2 is also poor, and so on. And you are sort of happy. And if this model is true and their causal interpretations are true, then we can apply a lot of do calculus. So we can ask to the model, tell me what will happen if I condition certain variable to be different. And then I can start to compute the causal effect of that variable. So I can say, okay, let's try to understand the effect of X2. What is the causal impact of X2? So I can run and sample the posterior based on the parameters that I have and say if X2 is equal to zero, tell me what happened. What will happen if X2 is different to zero and the difference of that will be the effect or let's say how many users you will be losing, okay, if X2 is zero. And we can compute that again in PyMC. We get the result and we can observe the daily effect and the cumulative effect. This is what you will observe if you run, let's say, some synthetic control group and you observe a contrafactual and you compute the difference. That's more or less what, again, your contrafactual will tell you. If you have ever used, I don't know, Causal Impact from Google, maybe you're sort of familiarized with those pictures. So here we have the model interpretation of what should be the effect of X2 if we intervene, we drop the channel to zero for sure you want to be you want to validate this information in this case the model is saying like if you drop that channel to zero you lose like a user a day and that will be like 10 users if you run it for several days but what happened if we actually intervene if we drop the channel to zero and we compare what we see is that our model was totally wrong our model at the beginning was estimating that we will lost something like 10 users but when we reduce the spend to zero we actually see a drop of 600 users so the model was totally under representing the value of x2 and this is where calibration goes to the rescue and says well if this happened give me now your input from the experiment let's append it to the likelihood and let's try to now correct that error and we can do that again in PIMC marketing we have a specific function that it's called at lift test measurements we can create a test a test we can define as I mentioned how much is the Delta of a spend that we have how much is the Delta in the observational Delta in our target variable and we can now in this side we will have to the parameters of the model in this case to the saturation parameters of the model we will have a likelihood that he needs to satisfy on top of the likelihood of the regular rest of observations so again this is where Bayesian models becomes handy because we cannot we can feed them feed them again different type of observations and we can feed like basically any of the parameters that we are defining against those and the result of that when we compare is even worse so we run this new model we did the calibration in theory should be okay we have some correction like you can see that now the channel was steeper like it's even moving close to the same direction of the initial way but it's still like 10,000 X run and then this this is where you know people go like side dock here and it's like okay so calibration doesn't work like how can I get a model that is actually estimating properly my marketing spend or my marketing actions and that's where causal theory pops in because that's what can help us to identify what we need to do in order to get the right estimates so if you have ever read you the apparel theory you will know that that causal structure or causal DAGs have different type of, again, structure, sorry, which are called chain, forks, and colliders. Based on the structure of your DAG, you need to define a specific type of regression, or you need to adjust for a specific type of channels, depending on the structures that you have there. And what happened is like here in this DAG, this is how our data was generated. This is how our world works. We have a lot of different things that we need to control and need to take account for. One of those, and maybe even the most initial important one is like every time that we run a diff and diff, a B or any of the methods that I previously mentioned it, you are estimating the total effect of your node. In the meantime, if you just run a simple regression and you put, let's say X1, X3 and X4, you are trying to compute the direct effect of the channels. As you can see in the dark, we have like the purple line where we have the direct effect of X2, but at the same time, X2 goes to X3 and X4, and that's an indirect effect. So every time that you measure again, any contrafactual experiment, you're measuring the total. So not only the effect that go through the purple, also the effect that it's go through the yellowish lines. And if you just compare it, like you're not comparing apples to apples, you're comparing apples to other stuff. So two very different things. So that will never match starting from there. On the other side, if you see this, when we run a multiple linear regression, we are not only observing the relation of one variables against the target, we are actually observing the conditional pairwise relationships. And that means that if I condition on X3, I'm also generating an spurious correlation between X2 and X1. And this happens because here in the DAG, we have a collider structure. We have X1 that goes to X3 and X2 that goes to X3 as well. That's basically a collider. Again, if you have ever read about causal structures. And that will generate, again, in our model, an spurious correlation between X1 and X2, but that relationship doesn't exist, but the model will see it because when you condition on X3, they relate. So if we want to observe the effect of X2, we cannot control for the total effect of X2, we cannot control also for X3. We actually should remove it from our regression model. And at the same time, we have some other confounders like events, that if you don't account for events, then you will have a bias estimate as well of X2 over there. So all of these things are not, let's say, statistical questions, are more like causal questions. And if you just create models where you put, let's say, everything on the Blender, which even if you don't believe it, this is very common these days, you just do regression, you don't think about causal relationships, you don't think about DAX, you put everything on the Blender and click run, then your estimates will be wrong. Now, if we follow Perl theory, we can say okay causal reasoning to the rescue we can feed a model only on x2 we run this model because we know the causal structure or we discover it we will know that the effect of x2 will be isolated we will not be blocking with any extra variable like x3 and x4 we will not be blocking the path of information and by consequence we will capture the total effect in that regression and if so that regression against our experiment should be close and that's exactly what is happening so now we have a regression that is built to capture the total effect of x2 and we did an experiment that will capture the total effect of x2 and those two are very close together and we can make it closer if we apply experimentation on top of it or if we now add another likelihood to our model to fit so the insights that I hope that you get for this presentation is even if you have a lot of experimentation and you want to apply this experimentation to your model to correct for those biases like no amount of experimentation cool you know calibrate or get make a model better if you are causally miss specifying it you need to understand the different DACA structures if not then you will never able you will not be able to recover the true causal effect of each of those nodes and on the other side you need to understand that adjustment serves are like crucial you maybe don't need a model that explains everything maybe you need several models where each one of those explains a certain power of reality so one model explains x2 another model explains x3 and so on so I think that pursue that single model that explains everything it's also a very hard thing to do. If you want a deep dive I will just share the notebook where you can see basically the whole process here and if you just want to understand or read more about these topics at Wise, at Bolt and at PyMC we are doing like a lot of content just to explain all of the things so you can go there as well and I think that that's it so thank you for your time

Speaker 2 [21:50]

Thank you so much for the presentation and we have a couple of questions, please don't forget to use slider for Q&A So the first question is how did you come up with the initial model?

Speaker 1 [22:02]

how I did came up with initial model if like in this example I just build the initial model as a example of what you what do you do when you're like in your company they give you a data set with three variables and then you just need to figure like okay what is the value of each of those so you just plug everything in and run it so that's a bit of what I wanted to show with the initial model is just an example of well you just run a model with whatever you have look everything and click run so that's

Speaker 2 [22:35]

And the next question is how to deal with bad, wrong data in my model.

Speaker 1 [22:43]

that's a very interesting question so I will say if you have like any piece of data that doesn't make sense to you at least in causal terms what you usually want to do is try to check the conditional dependence of your data so let's say you you want to understand which variables are dependent on each other or and or are dependent on others if a variable is dependent on other then Then you need to start to maybe account for it depending on the structure in your regression or not. So I'm not sure if it's like wrong data, but it's definitely a problem of what data you put into your model at the time of the regressions and how did you decide which data goes there is through causal discovery. So that's something that you can also find out in my blog, how to do causal discovery with the different variables that you have.

Speaker 2 [23:39]

Could you apply causal structural learning here without including your expert knowledge?

Speaker 1 [23:46]

I think that you can do like it's you have a lot of it in a lot of algorithms out there for example if you go to do why or the guys that build the why you have something that is called a greedy search where you basically just plug your data in and they will do a search over the notice space over several different type of DAX that could be possible given your data you could get some answers so you don't need to put any expert knowledge. But I think that the expert knowledge is actually valuable because we'll help you, like, the number of combinations that you can have with four or five variables in your data set is almost, like, it's huge. So when you add a bit of expert knowledge, you basically cut the number of possibilities that are also, like, non-sensible. So maybe you have, I don't know, something that is saying that, oh, actually target, you know, your target variable is driving your spend in Google. And, you know, like, that's not true. So while you will explore that part of reality, you don't you want to discard it with your expert knowledge So add knowledge to your data. I think this is what brings meaning to it

Speaker 2 [24:56]

Do you have any book suggestions for understanding causality and the dual calculus?

Speaker 1 [25:01]

calculus yes so in that sense I will recommend the first one I really like it's Alexander Moloch and causal discovery in Python I can also give you the link I will put the link in my notebook so you can have it there you have you the apparel do the the why do why the I forgot the name but it's the book of why sorry the book of why from you the apparel and also your apparel have one that is even older that is in early 2000s I think 1999 version it's a causality for for Python or something like that so you can also take a look to that to that book that's also interesting it's more foundational and then you can actually see if you read you the apparel you can see how he went from Bayesian modeling to causal modeling because these two topics are like very related and again even if you don't know it if you have ever write a modeling by MC you write a causal model I mean maybe maybe you know you know it or don't but you did it so that's a very important thing that

Speaker 2 [26:09]

But and we have one more question Costs are dependent on the season. For example Christmas should I include costs or just go with impressions as fourth input?

Speaker 1 [26:20]

Yeah, that's a very good point. I will say Impressions in marketing is basically exposition which is closer to our target variable and by consequence the noise between Impressions and the final target is usually less than if you just take a spend again The amount of exposition that you get is not only depending on the spend is depending on a lot of other factors That will add noise to the relationship against the target So it's better to take variables that are closer to that part of the funnel than further so impressions

Speaker 2 [26:53]

Okay, thank you. So with that, we can wrap up. We don't have any more questions. So a big thank you to the speaker and to the audience for questions.