Causal Inference through the lens of probabilistic programming

Speaker 1 [00:02]

Hi, everyone, and thank you for joining the third talk for the session today. My name is Irena, and I am very happy to introduce Juan. He is an applied scientist and PI MC, and he is going to give us a talk today about causal inference through the lens of probabilistic programming. Welcome, Juan, and please join me in welcoming him today.

Speaker 2 [00:38]

All right. Thanks for coming. I'm really happy to be here. So maybe for the ones at the back, these slides are on the program. So you just can click there and you should be able to read them in case you don't see them. All right. So yeah, the objective of this talk is to kind of inspire or trigger interest in causal inference to the lens of probabilistic programming. So it's less about the details, but more about pattern and problems that we have solved and I have solved during the last years where I believe these frameworks can enable us to make better decisions, which is what we end at the end of the day. So this is a brief outline of the talk. So I'm going to give some very basic intuition about cultural inference and Bayesian modeling. This is not a cultural inference talk nor a Bayesian modeling talk, but somehow something in between. And I'm going to motivate kind of the techniques going through Bayesian A-B testing. So I think most of us have experience of her about A-B testing, so I want to just present some ideas which I believe can be very useful when kind of approaching this type of problems. A-B testing, because it's a randomized kind of setting, it's easy, quote-unquote, because you don't need to deal with confounders or other causal paths, so this is a good start, but then we move to another concrete example on this item five where we actually face kind of a non-random treatment assignment and then we need to understand how can we tackle this. This is the only example that I will try to work a bit on detail and then I'm going to talk about other use cases which follow the same pattern and the code is available on the slide so you can take a look afterwards. I don't want to more focus on the concepts. At the very end, it's about fixed effects, modeling, mediation analysis, and a little application of what I'm currently working on, and I'm adding some reference at the end. All right? So that's the plan. So again, in a nutshell, cultural inference is about understanding kind of cultural effects of interventions. So is it a price change? Is it a marketing campaign? Is it a change in our recommender system? This is the type of thing that we can intervene and we want to understand the outcome. The biggest challenge is that we don't know what the outcome would have been if we hadn't gone through this intervention, right? So let's say I get a voucher. I'm a client in a food delivery company. I get triggered. I buy a burger. That's increased revenue. But the question is, had I didn't receive the voucher, had I ordered the same burger? Maybe I just like burgers, and I just took advantage of that. And this is something that we don't know. So to reason about this, it's interesting to think about the concept of potential outcomes. So if we think about our treatment variable as T and the outcome Y, let's say, voucher revenue for my burger, Then we have y, i, it's an individual, so i equals 1. It's me, Juan. And then we have two outcomes, 1 and 0, depending whether I received the treatment. These are just kind of abstract notions, because I just see one, like the realization of whether I got the treatment or not. And we're interested in the individual treatment assignments, so what would have happened if I had received the voucher or didn't, and let's say if this is the same, then there's no effect on the voucher. But if it's non-zero, then there's this effect. It's very ambitious to actually go at the individual level. So what we are typically interested is taking the average of that over the whole population. And you can actually go a bit further and condition or certain covariates. But in general, this is the first thing that you like to understand. And again, the biggest fundamental problem of causal inference is that you see just one of the potential outcomes. So this makes this field very hard, because you don't know how to validate those. You need to just use a lot of external data points and experiments. So this is about what we're trying to solve. And the how, what I'm going to present here is about how Bayesian modeling can help us address these problems. So if you're not familiar with Bayesian modeling, we have kind of three ingredients. So if I get some data, then I can define two quantities. Let's say one is the likelihood, which means the probability that given my model parameter, let's say if I want to model my data with a linear regression, what's the probability for fixed beta coefficients to have arrived to those predictions with the data? So this is where you need to think about how you parameterize your model. Maybe you're going to do a linear regression, maybe a logistic regression. You have parameters, and you can reason about that. The other concept that we have is the prior, right? So this is a very powerful pool, but with big power comes big responsibility, and I'm going to do a lot of prior predictive modeling. And this means that without seeing the data, and you specify your model, let's say your linear model, you have an intercept and a beta coefficient, you can reason about these quantities even without seeing the data is the intercept supposed to be positive negative do i have information about that you typically have you in most cases like you're not blind to the problem so constraining the model uh parameters even before seeing the data and this is and you should be responsible here this is not cherry picking so again and you need to be kind of conscious about what this can do, but it's very, very helpful. Once we have this, there's a way, and I don't want to go into details, to compute what we call the posterior distribution, which actually is the, let's say, fitting the model, which in the Bayesian setting means getting the probability of these parameters given the data. That's what we're interested in, and using that, I'll show you how we could potentially estimate treatment effects. So, when I talk about probabilistic programming language, I'm thinking about the computational engine to allow this to actually work in practice. So you can do this by pencil and paper, but it's going to be long and tedious, and there's a lot of research in making kind of the computation of the posterior very efficient, so I wanted to just highlight kind of PyMC, non-Pyro, or STAN as different flavors of probabilistic languages. They have a lot of similarities, but a lot of difference. I don't want to go into detail I'm gonna mostly focus on pyMC and on pyro and the reason is because I've used them the most And so again a bit subjective I'm gonna have some notes on the slides that I'm going to ignore but it's for you more to reason about them later All right so what we're trying to understand is why with Bayesian modeling actually help us to solve cultural inference problem because Because a priority, they feel like a bit disconnected. So the first thing that I think is super useful is, as we will see later, when you think about a causal problem, a causal inference problem, it's very helpful to draw a DAG, like a directed taxicly graph, where you actually estimate or kind of understand and try to kind of hypothesize about the causal path between your variables. And when you write your Bayesian model, you're forced to somehow think about that. So you cannot just ignore your assumptions. You really need to think about your assumptions. So just thinking about the problem, actually think it's the most valuable thing that you can do. So even if you draw the deck, write your probabilistic model, and then use frequentist approaches to fit that, that's a win. Because again, here, what is important is not the tool, but the thinking part. The other thing is that you get uncertainty for free. So all of these probabilistic models just kind of handle posterior distributions by you. So you don't need to vectorize or think about broadcasting. This is kind of already baked into these frameworks. Prior predictive modeling, I think, is also super useful, where you can pass prior information to your problem in a way that kind of makes the results more understandable and reasonable. Again, that doesn't mean that I'm going to cherry-pick the prior to make my stakeholder happy. With power comes responsibility. You also need to acknowledge, let's say, how to do this process correctly. Something that actually makes this even, OK, so before that, you have a lot of flexibility, which means that this PPL, probabilistic programming languages, give you a lot of distribution. They're integrated with deep learning libraries. So kind of the possibilities are very, very big. So you won't be constrained by kind of flexibility. Another thing that is very important, I'm gonna talk about this later, is about the do operator. So the do operator, it's kind of a technique to simulate interventions. And this is something that you can reason about, but these probabilistic languages actually have a way of implementing these kind of graph surgeries so that you can simulate interventions even if you didn't do them. So if this sounds a bit funky, it's fine. I'm going to expand on that. And I think nowadays that we can say that we have fast inference for these type of models, which means that before, we will need to wait a long time. But now we have very good MCMC samplers available that scale well with GPUs, for example, with a JAX backend. And if we want to sacrifice certain kind of precision of the procedure, we can go with variational inference, which translate this into an optimization problem, and then we can use deep learning techniques. So yeah, and again, I think we're very lucky to have this, and this is actually going pretty fast. OK, but importantly, this does not mean that Bayesian methods are better than frequentist methods. This is something that I'm not going to say, and you shouldn't think about it. What is important is that either if you use Bayesian or frequencies, you need to be responsible of the assumptions and know your assumptions. So again, this is not like better than worse. They're just different viewpoints. And if you, at the end of the day, using frequencies methods actually make better decisions, then you won. And the other thing is that the bigger problem is about identification. So do you have enough data to actually estimate what you want to have? In many cases, you don't have the data, and there's nothing you can do rather than maybe improve on your experiment design, try to do some proxies. This is something that I'll show at the very end, but also knowing that this is a limitation. So the PPL won't kind of save you from that. So there's no silver bullet, there's no free lunch. But nevertheless, I'm going to show you that we get some interesting tools to solving the inference part. All right. So I'm going to talk about typical Bayesian A-B test. So you think about, again, food delivery. You have a new recommender system. You deploy it, and you want to estimate whether there was an uplift in your conversion rate, right? So your conversion rates are bounded to be between zero and one, and you're interested in this lift quantity, for instance, which is the treatment conversion rate divided by the control, and you just take this one just to make like a percentage and then you want to understand let's say okay so do we have the conversion rates a that are between 0 and 1 you have your lift and then you have your observations which I can sample from a binomial distribution this is not very important but this is a typical case of an of an A-B test and so before seeing the data I haven't run the A-B test but as I am a good modeler I reason about the test before getting the data. So what I can do is, okay, I can still simulate my priors. So for the conversion rate, I'm going to assume nothing. It could be between zero and one, which is a bit naive if you think about your applications. The treatment is also the same, like I have no idea what it is, and it's going to be between zero and one. And then I have my conversion lift, which is a deterministic quantity, which means this is just something to store in the trace, but it's just the same formula. So you see that this PIMC code just resembles the math very closely. And then you can reason about this. So here we see kind of a heat map or like a between conversion rate and for the treatment and the control and we don't see anything just because like there's no pattern you just chose like random stuff. And if you look into the prior predictive distribution, meaning you just sample from the specification, you see that your uplift could be between zero and whatever. Let's say, typically, your kind of conversion rate lifts are relatively small, so if you do like 1% or 2% increase, you're going to get promoted. So this is a bit too naive. You know better. So we can talk about informative priors and in this case you can set priors in a sense that they are both let's say for the treatment and the control priors there sometime somewhere in between one and four percent right pretty reasonable and what could go wrong and why they can vary I mean there's maybe some systemality on the conversion rate right maybe the conversion rate is higher in in summer or in winter. And if you see the plot, we do see that the kind of prior distribution is constrained, but still the relative uplift is big. It's in between, let's say, 50 to 100 percent. That's already quite high, despite of the fact that we have reasonable, let's say, marginal distribution for the conversion rate. And the problem is that when we do this kind of just specify this independently we are kind of ignoring the fact that in general the conversion rates of the treatment and control are correlated right so let us assume that you do nothing you do the split there's no intervention a then both conversions will oscillate in a similar way based on seasonality right so they actually are correlated they're not just randomly typically will fluctuate together so what you actually can do is reason about this and say, okay, actually I can do something different. What I can do is that I'm going to explicitly model the lift as a variable, and I'm going to use the treatment conversion rate as an outcome. So here my conversion rate of the control is the same as before. And now I'm going to set a prior on the lift. And I know this is something that I should know and say, okay, this is between 0 and maybe 10%. I'm still being generous about my new recommender system, and then you can just invert the relationship of what you had, and the outcome is that you have now this correlated prior predictive, and the lift is where you want it to be. So this is, let's say, this indicates that if you're going to do inference with data that's going to come, let's say the expected values of your lift are reasonable, right, and not super crazy, and this piece of kind of thinking process is super valuable, and again, I haven't seen the data. All right, and now I can just run the model, I run the experiment, and I just need to pass the observed data of the control group, of the treatment group, and this is my result. So the conversion rates actually are in orange, the observed one, and it's around 2 versus 3.5%. But actually the posterior distributions are much closer to zero because the model is like that cut me noise, I'm going to be more conservative, and I think it's not 2 or like 3.5, but around 3.2. So this is because there's something I think is Tremont's law that says that if something looks weird, it's probably wrong. So you really need to have evidence that these conversions rates can go really up to 3.5. So this is the type of regularization that you can get from using Bayesian models in the context of A-B testing. But when you think about sample size estimation and all of this pre-analysis that you typically do for a real A-B test that you should be doing power analysis, you can still do that in the Bayesian setting. just do it in a different point of view. So what you can reason here, and this is one of the things about thinking innovation way, is that when you think about the null hypothesis of the lift being zero, this is always false, because you'll never hit zero. The probability of getting zero is like zero, so there's no meaning on that. But you can reason about maybe a region of practical equivalence, whether if your lift is between zero and 0.001, you you call that like there's no uplift, right? And you can now reason about, OK, let us assume that my uplift have certain distribution. So I generate lift distribution for different lift sizes. I simulate data through my generative model that I showed. And then I compute the Bayesian estimates. So I'm simulating forward past, and then compute Bayesian estimates. And then I see and count how many of these posterior distributions excludes this area of practical equivalence. And then you see a lot of Bayesian models on the right, and some of them hit zero, some of them they don't, and then you just count this many times. And you don't need to use for loops, you can vectorize this very nicely, and you get this typical power curve, where you estimate, okay, based on the sample size And the power defined as kind of this share of kind of models that didn't touch the rope region. This controls how, let's say, the true positive rate. And you have the frequencies, the non-informative, which was uniform 01, the informative and the correlated priors. And the colors are, let's say, on the lift that you are expecting to test. Let's say the true lift, if the true lift was 25%. So one thing that you see is that for large sample sizes, you don't care. Like, that's kind of going to be dominated by the likelihood. But if you're interested in doing something like early stopping on things like this because experiments are costly, then you might be interested, okay, how much I can push into the sample size. And for the frequencies, let's say, even if we get very small sample size, the power goes up because for the frequencies, effectively, there's no constraint there's no regularization so it's not very conservative but in the correlated one you see that it takes some time so it's very skeptic about kind of uh moving up of course if the probability if the lift is very high that's let's say fine like the red and the blue one but if it's something in between zero and twelve percent then the correlated model is going to be very skeptic so this is a balance between kind of this false positive and kind of true positive rates, and you as a modeler need to decide where is this trade-off. So you can still do all of this, but reasoning about the problem. This is something that you might want to control or not. This is your choice. This is, again, a tool. So in A-B tests, you want to reduce also the precision of your estimate, and then kind of what you can do, there are a lot of techniques like Cupid where you take pre-experimental data and kind of use that. You can actually do it in the Bayesian sense and also do the prior predictive game to actually combine these two. So I have some reference if you want to do it. But anyway, let's go now into the concrete example where we don't have a randomized treatment, and this is a very famous data set about a training program in the 90s, and kind of this This is about kind of education, and the question is, does this program actually improve, like, the outcome of people taking it, for example, earnings? So the participation of the job program is the treatment and the outcome of the earnings in the years, in the year 78. And we have a lot of covariates, things that we know about the population, like socioeconomic and demographic data. And if I just take my Excel sheet and just compute the results, I get that the treatment individuals got on average $600 less. So educations make you earn less. So that's an interesting kind of result. But we want to understand why. And the reason is, again, what you're seeing in the right, that this treatment was not randomly assigned, let's say, depending on, you could choose to just be in the program. So this creates what is called like a backdoor path. There's some thing that you need to control because if you just compute this in a naive way just by taking the difference in mean, this is probably not going to work. So this is what is called a confounder and a spurious correlation between these things. So you need to do something in this case. So here I kind of encoded this at covariates, but you have like education, age, pre-earnings, and a lot of things, which both affect the treatment and the outcome. So how we can actually do this, well, there are two estimation strategies that I'm going to show you. is that you can run a regression model to estimate earnings, which is your target variable, as a function of your treatment, which you have here, and then you add covariates in your regression. That's going to help. And then you are going to estimate, let's say, you're going to read the treatment effect from this coefficient. Pretty straightforward. You just need to do one regression. The other thing that we're going to do is actually do operator, vector, which means we're going to model the whole graph, we're going to estimate the parameters, and I'm going to simulate interventions as if the training program was at random. And then I'm going to take the difference. And they should be the same in this case. So this is the only kind of code that I'm going to show. You don't need to go about the details, but more about the flow. So this is in PyMC model. You specify your data, so you have covariates that I'm going to vectorize, so I don't need to loop over them. I have some priors, so I didn't think about the priors in this case that much. They're doing a bit of regularization, but I'm going to skip this. And this is kind of a linear regression, right? I'm going to take the sum, like there's an intercept, the dot product between the beta coefficients and the covariates. I pass it through the sigmoid, and model this distribution. So that's the treatment component. And I also have the component of the target, and this is the same thing, you have some priors, you have your covariates, and your normal distribution at the end. And the only thing is here you have the treatment, someone here, right? This is what we want. All right, and this is how the model looks, and the only thing that is in gray is the data. The rest are distributions. So you can actually sample from these and generate posteriors of your earnings, even if, let's say, and you can do all the prior predictive modeling that you should do. And then what you do next is now that you kind of have this prior predictive process, you can condition on the data. So okay, now I have my model, now I'm going to condition on the data, and I know that treatment, and earnings are observed. So this is more like probably you can do this at once, but this is like the mental exercise that you should be thinking when you do these problems. And then I just run the machine, and I see that the procedure distribution is around 0.55, and it crosses zero, and here we have our naive, like minus 0.63 value, and the red line is like if you get OLS with stats model, the same model. So you see that you have indication that the treatment actually had a positive effect and you can compute probabilities from this distribution and it's a bit better, let's say it's regularized as compared to OLS and the orange lines are another method called matching and the reason I put these lines is because this is part of a bigger blog post where I compare various methods and results. But long story short 0.55 make a lot of sense and you have the uncertainty. All right. So then we're done. But I'm going to just give you another recipe, which actually I believe is going to this is dancing. Okay. So another way of doing this that is actually where PPLs shine. So this is about the dual operator. And the idea is that I take this computational graph and I can cut all of the covariates going to the treatment. I can literally just cut them off. And let's say reason about this treatment as being random, right? Because I'm cutting all of the confounders going in after I've done inference. And then if I have this model, then I'm going to say, okay, what about if we all had the treatment or we all didn't, and we just take the difference, let's say, individual by individual. So this, you You can do this very easily with the dual operator from PyMC, and you say, okay, just import it. And you say, okay, you know the conditional model that I fit, well, I want you to now cut and make treatment variable always 0, and the treatment variable always 1. And we see that here the treatment variable does not have any arrows coming in. So it's like data. So it's effectively like an external treatment assignment which was done at random. And then you can take the difference between the prediction of these two models and generate counterfactuals or these potential outcomes for every individual. And you see there, and you see that on average, the treatment is higher than the control. And if you take the average, you get the same thing. thing. So the blue line is on the regression coefficient, and the orange line is with the do operate. So we just did the same thing. So the question is, why butter? Right? Because now we have this baseline, and we know more about our program. So for instance, we know that earnings are positive. And I'm using a normal likelihood, so what if I use a distribution that is positive, let's say like a gamma distribution, and kind of play the same game. Here, then reading the treatment effect from the coefficients becomes really tricky because you have this what is called the link function to make everything positive. So you need to do some math, right? Which you can do. But on the other hand, you can just follow the do operator flow and do the same thing. Condition your model, get samples, do operator, do operator, take the difference, and that's it. And in this specific case, I'm not claiming that kind of this holds in general, but in this specific case, this parameterization actually improved the precision of the estimate. And this is something that we actually have seen in practice, that understanding the data generation process and trying to model that explicitly brings a lot of value, And at the very end, it reduces the precision of the estimation in some cases. So this is now where you can keep adding more kind of complexity into the mix, and I'm going to go intentionally a bit fast just because I want to just focus on the fundamentals and just show the resources at the end. So this is now where you have something similar, where you have a treatment, in this case X, you have some covariates C, which are not confounders, and you have some groups. Maybe you have treatments in different cities or for different countries, and you have data for all of them, and the treatment is based on the groups, and also the outcome is based on the groups. And this is a confounder that I see. So I have the G in my data set. So bxy is the coefficient that we're interested in, and bzy is this coefficient here. And this plot shows that if you run the naive model, and in this case it's a simulated data set where I know that bxy is zero, if you do not control for confounders, you are going to be biased, right? It's going to be very wrong. And the reason is because you have a confounder. So now, let's say you need to think about what you want because in your regression, let's say, the recipe would be, okay, you have here your target. You need to control, okay, here's the treatment variable. Here are the confounders. And how do I confound, let's say, understand the grouping? So there are various ways to do this depending on what you want. So if you just want to have the treatment effect, you can use something that is called, like, fixed effects. And what you do is essentially put a dummy on the groups. Easy peasy. The only problem is that if you do this, you won't be able to tell anything about set because there's you will have two intercepts that are redundant a if you are interested in both you can actually think about a multi-level modeling, but then Because this is actually putting some constraints on this additional intercept You are gonna ruin kind of the what you gain with the fixed effects So, there's a way of fixing this with something that is called the Moondlack trick, where you add the multilevel model, and then add the X bar, let's say the mean over the X group to control for that. And there's a better way to do it, which is use latent variables. So, depending on what you want, you have this thing at the disposal, again, there's no free You need to think about what you want, but you can run the machinery after that. So these are the results. So again, notice that we are interested in this coefficient being zero. The naive model, the blue one, is very off. The orange one gets it, but the precision for the group level, let's say the set variable is very bad. If you go multi-level model, this doesn't work either. But with this iterative approach, we can make it better by, for example, adding a hierarchical structure plus using a proxy for the group level confounder. So this is the average over the groups. And in these nodes, there's an additional way of doing that with it using a latent variable where, again, instead of having one regression, I model the whole data generation process and treat this group level confounder as a latent variable, like a random variable. And this on average is going to be much better because it's going to propagate the appropriate uncertainty. And again, I just need to specify the model and then do operator here, do operator there, take the difference, I'm going to get the results. So resources are there. So now when things get funky is what if I have a confounder but I don't see it? And this is like a question that i'm very interested in and there's a really nice exercise from this book by robert ness on casual ai and the idea the following you have some gaming a problem where you are interested in the treatment okay if you're engaged if you have side quest engagements like if you engage more are you gonna have more in-game purchases this is mediated by how many items let's say, have you won while playing, you could be part of a guild or like a group of players which actually affects the outcome, the in-game purchase, and how engaged are you with the app, but you have other things that you don't see and you need to be conscious about it. And I model this as said. So yeah, exactly. This is a confounder that exists, and you need to think about this problem in the context of the castle framework, regardless of whether you have the data are not. So we cannot adjust for the back door because we don't have in our data set. So one solution, which is not the solution, but one potential solution is, again, using again latent variable modeling, as in the previous slide, where we actually model set with maybe a deep learning model or any type of model. So this is going to be a bit funky. I actually did another small simulation replicating the paper there, so just click there, it's better. But the idea is that the model is going to be kind of the forward path, it's kind of the decoder, so I'm going to model the whole process, so I'm going to model the set that I don't know as a random variable, and I'm going to model the engagement, which is my treat variable of interest as a variable, as a function of the guild membership and set. Same for purchases. I have another model. And that's my kind of forward pass. And on the other hand, I'm going to model kind of the encoder part that is trying to learn the other direction by modeling, let's say, the approximation of set through a normal likelihood. So I know this This looks weird. It's weird. So I had to spend roughly two months going through this. And I can run variation inference, but it's the same idea of what we mentioned before. Using set as a light model, and then pushing the results. So my take on this is that it makes sense, it helps, but you have some freedom on defining the target space of the decoder, or the encoder, and this dimension that you choose actually affects the results a lot. So my advice is you can reproduce this simulation, but use this with care, because depending on the choices that you make, you might get actually pretty bad results. So I just wanted to present this as something that might work, but please do your simulations. I'm running a bit out of time, so I'm going to say that you can do mediation analysis, which means if you're interested in paths, like, say, cultural paths along the way, and this is another kind of known data set about family intervention, does it has an effect of drug disorders or substance use? But this is mediated by the fact that you can have bad friends, that it gets inside you to try drugs, or you can do experimental drugs yourself, because you are an individual. And then you can have these mediation paths that you can understand. And the game is the same. You model the whole data generation process, in this case, a lot of logistic regression. And by using different type of do operators, OK, what are you interested in? OK, I'm going to just assume that, let's say, the treatment is random, and I let the effects propagate. Or do I also want to intervene in the mediators? And what you can get is a decomposition of, let's say, the total effect, which is blue, as a sum of a direct effect and also effects mediated by the kind of in between nodes. And this is actually extremely helpful because you get much more information and a typical kind of DAX in reality look more like this and less than like the trivial one that I wrote. People have critiques about mediation analysis because you are ignoring unobserved confounders. But, I mean, you can still add them and treat them as latent variables. So at least you can reason about that. And what I'm currently using, this type of method, so I've used a lot of these methods in different applications. What's taking my time now, it's on media mix modeling, which is about understanding efficiency of marketing spend from a target variable. So it's inherently a causal inference problem in which we actually have unobservant founders, which is, for example, competitor spend, and we have latent variables, which is brand. So here is just a toy model example where I have kind of contributions or causal effects of two different channels. We have some seasonality component, and this latent variable in orange is something that I model with a Gaussian process, which becomes very handy in this type of applications. And when I have enough circumfounders, which is most of the time, we need to calibrate this model with experiments using custom likelihoods. But everything can be used in the same framework, and we recently gave a talk. I'm going to share this reference later. Yeah, so a bit of self-promotion. So thinking about causal inference and probabilistic programming, we have PIMES in marketing, which is about marketing applications. media mix model, CLV, customer choice model, partial pie, which is more general framework for kind of observational partial inference. And we recently launched Decision Lab, which is trying to automate these workflows through agents, right? Because there's a lot of boilerplate that we can know. Let's say maybe you don't need to write the whole PIMC model yourself or do the dual operator thing if it's something that we can automate and kind of really kind of ensure that we can iterate faster. Okay. The takeaways, this is going back to the first slide. Again, I hope you can, when you see these slides, really understand and kind of these ideas should resonate to what you have seen. And, again, I'm going to share the resources on this material. So there's a lot of things that are there to be seen, but I hope you get the message that there is potential to look into this type of thing. And, yeah, that's it. Thank you very much.

Speaker 1 [41:17]

Perfect. Thanks a lot, Juan. It was a great session today. We learned a lot, and there are some open questions. I think more than the ones that are posted, but let us start with the time given. The first one is, you mentioned Perl in your presentation. Is all the mathematical processing done in Perl?

Speaker 2 [41:39]

No, this is Judah Pearl as in a person.

Speaker 1 [41:43]

Okay, not the language.

Speaker 2 [41:45]

No, so he is like a very famous cultural influence guy.

Speaker 1 [41:51]

Very good. The Bayesian A-B testing example made explicit assumptions about closed-form likelihood like Gaussian, Bernoulli, etc. How can we know or test whether this assumption is valid and what to do if not?

Speaker 2 [42:09]

This is something that you need to reason. I mean at some point you need to specify the model So for example when you run a linear regression second learn you don't think about like because you just trust it But in essence that's happening, right? So if you for example think that you want to control for fat tails You can use a t distribution and kind of make the same game You also also need to find that all of these are approximations kind of of your problem Let's say probably the true model is really complex, but you don't need that for decision-making making so starting with a normal distribution for the uplift might be something very reasonable you have fat tails a t distribution you for whatever reason think you're going to have bimodal distributions and you can use a mixture but again this is about you this is a tool where you can reason about that and if you have enough information you do the prior simulation and if the uplift results are in sync with what you expect let's say fat tails then you can do that

Speaker 1 [43:02]

Thank you. So another question is the following. How can fixed effects, random effects and latent variable modeling be integrated in a Bayesian causal model to handle observed and unobserved heterogeneity?

Speaker 2 [43:20]

Yes, so this is something I went a bit fast, but you can actually find all of the materials in the blog that I shared, and I actually ported these examples to Bambi, which is this Bayesian package which allows us to specify Bayesian models as formulas. So if you go to Bambi documentation, you will see the code, which is very short, to reproduce this plot. And if you want to use unobserved latent variable, then you can, again, use PyMC or non-Pyro and just work out yourself. So I've done a lot of work for you trying to reproduce the paper code. So if you open this simulated notebooks and ask Cloud Code, I guess we're going to reach a pretty good level.

Speaker 1 [44:09]

Amazing. Clarification question. What do you mean by PPLs do not solve identification? What does identification mean in this context?

Speaker 2 [44:22]

All right, so it's in in this case, right so where we don't have like said we cannot measure Like that's a problem and PPLs are not going to solve it PPLs can help us Reason about it and maybe potentially find a solution. So this is a solution which is tried to model this interaction, but again, there's no free lunch and Kind of depending on the choice on the embedding size. This can go terribly wrong In the case of media mix models, for instance, we know that we have unobserved confounders and we know that our model estimates are going to be wrong.