Bayesian Marketing Science: Solving Marketing's 3 Biggest Problems

Speaker 1 [00:03]

Thanks for the introduction. Thank you all for coming. I had a nightmare this morning that I got here and then my laptop broke down. Then once we fixed that, I was standing here and I was wearing my pajamas. It's terrible. So I'm very excited to be here with a functioning laptop and no pajamas. And talk to you about Bayesian marketing science, solving marketing's three biggest problems. As we already established on Thomas, it's my mission to bring Bayesian modeling to its fullest potential and I started this when I was in grad school at Brown and found that Bayesian modeling is a really amazing tool to solve all kinds of applied problems and build custom models to solve these problems. And however, there are still and were back then even more things standing in the way of those wider adoptions and that started with the tooling and it's why I got involved with some other folks like Chris Fonesbeck to develop PyMC and then during my work at Quantopian which was a quant finance company I also discovered that it was not only really useful for academic research but also for solving applied business problems and then after that chapter ended I started PyMC Labs to really try and bring this to establish it in business. So, PyMC Labs is basically a bunch of the people that have developed PyMC and joined the development team over the years to help companies use PyMC more effectively or just solve any kind of problem where Bayesian might be a good fit. And I couldn't be more happy about the team that I'm working with. These are some of the most talented, amazing modelers I've ever had the pleasure of working with, which is also why I like to call them the Bayesian Avengers, because each one of them is special in their own way and hardcore. We work with clients from all over the industry and solve their problems. And so, for example, HelloFresh, you probably know them. they're also here in Berlin. We helped them with some media mix modeling, marketing attribution. Visual West is a company also here in Germany with custom lifetime value modeling. So you see, there's definitely a large exposure of marketing companies, marketing departments in e-commerce companies that we're helping. And that was a bit surprising to us because none of us really had a marketing background. And nonetheless, there was this huge demand there. So marketing does seem to be very eager to adopt these methods. And the other big branch of companies that we're working with are health care slash biotech companies, which I guess also makes sense because a lot of us have a biotech background, and also because it's often much closer associated to academic research where Bayesian modeling is a bit more established. But we're not limited to this, so we work with all kinds of companies. So I want to start this talk by introducing our protagonist, Amy. I generated her on This Person Does Not Exist. She is the VP of Data Science at an e-commerce company. And she is, of course, not an island, but embedded in the whole organization and has certain, well, requirements and people that provide input to her and certain resources. So I wanted to map out that and see what challenges she has and then what I think the market needs and what we developed to help people like Amy in the marketing space. So there's the CFO, right? And he or she sets revenue and profitability targets. And also, of course, has probably a pretty tight grip on the budget. So Amy can't just go out and buy software solutions for every little problem that she has. She will have to develop solutions herself, right? And of course, like any retail or e-commerce company, marketing will probably be the biggest expense in the business and a large effort is going to be placed on that. So really it's very important to maximize the marketing return on investment. The other team that she's working with closely is the CMO slash the marketing team. And what they really want is to maximize the efficiency of their marketing budget. And towards that goal, what they want is accurate measurements of all the different efforts. So they're asking data science to help them measure the things that they're doing. And one complexity, well, several complexities with that, but one is that they have all kinds of diverse different data sets. So they have data sets on the marketing dollars that they're spending on different channels. They are tracking the new users that are coming in, the sales that they're doing, they're launching campaigns and they want to know the impact of those. They have lift tests, so just all kinds of different data sets. And somehow out of that stuff, they need to form a coherent picture, right? And that's what they're asking data science to do. But not only a coherent picture but also insights that are transparent and actionable. And things that they often are looking for, and these are these three things, are what are my customer acquisition costs? How much do I have to spend in marketing to get a single customer? As well as, once I have acquired that customer, how much do I earn from that customer, right? The customer lifetime value. And then other things related to, for example, campaigns or any type of action that they're doing, what is the effect of that? What is the causal impact of the actions that we're taking? So these are the asks of Amy, but also there are resources that she has. So for example, she will have a data science team working for her, and probably many of you will be familiar with what that is, I assume most of you are data scientists who are working in a group like that or at a different level, and there's often a lot of pressure to produce results fast, right? So the marketing team wants those reports and insights for this quarter, this deadline, and that makes it difficult to really come up with a long-term R&D strategy to say, okay, well, now we're going to build, like, a huge, amazing statistical model that incorporates all these different data sources and attributes uncertainty in the right way. So that compounded with the fact that it's very hard to find these statistical modeling experts in the market makes that pretty challenging. And another challenge, of course, is that most data scientists are not themselves experts in marketing, right? So there's sort of a domain expert problem where the knowledge is differently located in the company. And then any solution that the data science team comes up with needs to be deployed. And so they will go to the data engineering team, and they really care about things like maintainability and being able to run these things on-premise and things like that. Okay, so this is after having worked at PyMC Labs with in many of those settings, I think we have provided some tools that will hopefully help Amy and people like her. And that is what I'd up here the PyMC Labs open source marketing stack. Consisting out of currently CausalPy and PyMC Marketing, which are the tools that I will be talking more about. So a key tenant, and because you came to this talk, you probably know that we'll talk about Bayesian modeling, and they have many benefits, but one of them is that they are inherently transparent in what they do, right? So there is a model graph, and you can explain this to your stakeholders, to the marketing people of how your inputs relate to your outputs. And that, of course, is very important when you want to build trust. But not only can you explain the model, you can also explain the results. So the parameters are not some crazy thing in a very high dimensional space, some embeddings, but rather they are parameters that relate to things that you actually want to estimate. So for example, customer lifetime value. Also this stack is open source, which I think is very important. And this is also what I think is the big difference. There's many media mix models out there that people sell for hundreds of thousands in annual licensing costs, which of course is quite a big drain on any budget. So obviously it's free, which is cool. It's transparent. You can look at the code. If you just get a SaaS solution, you don't really know. They tell you all that their model is state of the art and amazing, but how can you really verify what it's really doing? It's modifiable, so you can just use that code as a template and do your own thing or inherit from that base class and build your own custom model. And of course, it integrates with the PyData stack. So your data scientists will love it, because you can use it with Pandas and Jupyter. And the engineers, hopefully, will like it too, because you can just deploy it in-house in Airflow and use Kubernetes and all these other deployment tools that are out there. Now, one central problem that we've come across over and over again in PyMC Labs in trying to solve these problems is that every customer is very, very different. So in the beginning, we thought when we did this once, we were like, okay, cool, now we can just do the same thing for everyone, but we found that's not at all what it is. So we always needed to do custom solutions, which is what we're mainly doing, right? We're coming into the company and helping them solve their very particular problem. So we found that it was pretty much impossible to build like a one-size-fits-all solution that has like everything in one tool. So rather, I think the Unix principle will come into play or it will be helpful here where you have simple tools that you can combine simply, which makes sense if we're talking about like an ecosystem of tools that are compatible with each other, and then build your own custom workflows around that. Also, that idea from the Tidyverse, where you have all these individual tools, and they just speak the same language. So this is really the vision behind this. And of course, developed by the Bayesian Avengers, so the modeling is legit. But also, as I said, we're not marketing experts ourselves, but we teamed up with domain experts that have done this for many, many years. So PyMC marketing is something that we just released. So I'm very excited to announce this here and tell you a little bit more about it. So it is, as I already said, Bayesian, so built on PyMC, which is a probabilistic programming package for Python that allows you to build these fairly complex statistical Bayesian models. And it's open source, Apache v2 licensed. So just do whatever you want with it. use it in commercial selling, sell it, whatever. And we pride ourselves on having made PyMC as user-friendly as possible. Not to say that it's very user-friendly, but we try. So we try to extend this here to PyMC marketing and just give user-friendly high-level API that is easy to use. It comes with two models that are commonly used in marketing, media mix models, and customer lifetime value models. We plan to add more, but that's where we started. And in that, it's the success of the lifetimes library, if you're familiar with this. So this was done by Ken Davidson, and he, I think, mostly abandoned this. So people were looking for new solutions to do customer lifetime value models, and that's what we provide, and also doing it in a Bayesian framework. So this is the URL, check it out. And also a big shout out to the people who helped us develop this, domain experts. Juan did a lot of work on this also with the code. He's a data scientist as Walt and has a lot of experience with this. Colt Allen was largely responsible for the customer lifetime value stuff. Niall Olten and Luca Fiaschi, they just have decades of experience in this space. So yeah, we really tried and not only do what we as statisticians think is the right thing, but also what people told us what they need. If you're more interested in learning about this, there's a webinar today, actually, where you can sign up. Just go to Meetup, PyMC Labs, and then you can find it. But I wanted to talk a little bit more about Bayesian models and why that is so powerful in marketing, but also specifically for the problems that we're trying to solve here. So one of them is the ability to set constraints on modeling parameters using priors. So this is extremely helpful when working with other stakeholders because you can just get the input. So you can ask them, well, what do you think, in general, like a rough parameter range of the customer lifetime value is? They will already have several ideas on that, right? So they have been doing this for many, many years, and you don't have to throw away that domain expertise because you have your black box machine learning model that just learns everything from data. You can take the input and enter into a conversation with them and really get the download from them and co-develop a model with them that also really helps to, well, make people adopt your solutions. Related to this, we can incorporate existing knowledge about effects of advertising. So not only by asking people, but also from other data sources. So, for example, if we have run A-B tests or lift tests, we can incorporate that to get better estimates. And this is the idea behind this, that we have these diverse data sets and we ideally want to come up with a coherent picture that takes all of those into account. Also, because we're in a Bayesian framework and the inference algorithms by now have become very, very powerful, we can just build whatever crazy model we want and usually the sampler will have no huge issues sampling everything that you put into your model. So, for example, the Robin model has a very sophisticated fitting structure with iterative optimization and some evolutionary algorithms just because if you don't have priors and you are in this very difficult statistical modeling space fitting these items can be a challenge, but fortunately that's the strength of PyMC that it's just like code your model and then we try and figure it out for you. And finally the idea that we produce generative models. So the benefit of that is that once we fit the model we can just generate new data creating forecasts, we can use that for model validation to see does it actually replicate the data that I know that I used to fit this with, but also things like scenario analysis like what if I doubled my marketing spend. So this generative framework is very powerful for this. So what are media mix models? Well, they start with a certain media budget that you allocate and then you want to know how effective each marketing channel is. So you have a total budget, you get a certain number of activations or signups and from that you can back out the customer acquisition cost. There's a lot more nuance to this, like transfer functions, but this is the core of it. So let's look at an example that was provided by Juan and we have two marketing channels in our example and over time we spend a certain amount which is normalized here to be between zero and one and on the output side we measure how many sales we got right so we want to know how do our marketing efforts drive sales. Now with PyMC marketing this is we try to make this as simple as possible, you just call the delayedSaturateMMM class, you instantiate it, you pass in all these different things, so you see, well, we have a target output column, we have a date column, the channel columns, you can have certain control columns, you see these sine and cosine, these are the Fourier bases to model seasonality effects, we can include certain events like Christmas, and then parameters for other transfer functions in the model. So with this single line, and then to fit it, you just call .fit, you build a model that's actually fairly complex, right? So I'm not going to go through this, but we have the likelihood here, we have inputs, we have parameters for our transfer functions, and everything that goes into an MMM. So you can just imagine that if you were to write this from hand using PyMC, it would probably be quite an effort, and there's many tutorials out there, and talks, and people asking questions, how do I do this? So here we just try it and do this for you that it's just out of the box. When you fit this, the first thing you want to do is probably just see how good is it as replicating your data. So this is called a posterior predictive check and in blue we have what the model predicts and in black we have the actual data. So you can see it's actually modeling the data fairly well. Then we can also decompose the signal, right? So we have certain things like the sine-cosine pattern here for modeling seasonality. That's one component. We have these events here that are these spikes. We have the channel contribution in blue, an intercept term, and these together give you this sales data. Just including this part here because we've just gotten it from Robert Kübler, who just checked out the package and did a pull request to add this. So this is just my love note to open source, that you put stuff out there, and then people start working with it and contributing back, which is just amazing that that is already happening. You have these contribution plots, so you see that the more you spend on the x-axis, it's not that you get just more and more on the y-axis. There's a leveling off effect, a saturation effect, and of course, you get posteriors because when a Bayesian framework is not just single estimates but whole probability distributions and this helps when assessing your uncertainty in these estimates but also helps with optimization for example. There you really want to take uncertainty into account. And since this is a simulation study we can compare this to the true return on ad spend and see that we recover the true underlying parameters pretty well. The other thing that I laid out before was analyzing the effectiveness of a marketing campaign. So this is the next task that Amy has to solve. And here's another example. So we launched a new product, and in 2022, this was an example contributed by Colt Allen, and we see that sales have been pretty steadily increasing since then, which is great. And then here in February 2023, the marketing team decided to run a campaign to boost sales even further. And now the question is, did that campaign work, yes or no? And well, just visually it's kind of difficult, right? I mean, yeah, maybe the variance may be also increased, so hard to say just by gauging. So obviously that's a statistical question, but actually not a trivial statistical question. Because usually what we would hope for, if we're like good scientists, is a proper experiment where we have a control group where we didn't have the campaign and one where we did give the campaign to, but we don't have that here. So this is the framework of causal analysis and quasi-experiments, and this is the other library that I'm very excited about called CausalPy, developed by Ben Vincent, and this is doing not only causal analysis but in a Bayesian framework, which I think has many benefits. So you can see basically the logic behind this just from these plots, I hope it makes sense, is we have this path sales data and we just fit a model to this and then we predict what would have happened if there was no intervention, right? And that's why this is called a synthetic control. So there is no real control, we're creating one. And so we just assume that, well, it would just continue like this, and then we can look at the residuals, and then compare the residuals in the out-of-sample, essentially, to compare the difference between what our predictor would have happened and what actually happened with the intervention. And then we can also just look at the cumulative effect to really see what is going on, and And we see that there's a very strong effect that indeed it seems to have made a difference and increased sales even further. And also, I'm not showing the code, but this is just a single line also to do this. And there's a lot of flexibility in this, so it's not just linear models. You can add covariates. You can also use scikit-learn models to do more complex forecasts and synthetic controls. And also different type of... So this is a synthetic control example, but there's differences within differences, and there's a whole zoo of things, and we include many of those. The last thing that I wanted to show is estimating customer lifetime value. So that's the other task. And usually data, and this is actually the data underlying the plot I just showed. It's not just sales, right? Those sales come from individual customers. Some are repeat buyers. So we want to get more insight into this. And we can use a, they have somewhat obscure names, a beta geo model from the customer lifetime submodule in PIMC marketing, and put in data like this, like how many, how often did they buy, when was the last time they bought, and how long have they been a customer with us, for each customer. And then get, for example, over time, let's say they bought something here on day 31, what is the probability that this particular customer that I use as an example here is gonna buy again and as you can see over time we're losing confidence that they will come back right like customers churn all the time and the longer we haven't seen them the lower the probability that they will come back or assume now that they have actually then bought again here so now we have this confidence that well probably they're gonna they're they're definitely still alive and we're gonna just expect that they are gonna buy more. So this and many more analyses you can get insight into which customers are driving your revenue, which customers are at the highest risk of churning and of course you also get uncertainty in all of these because we're in that Bayesian framework. Now the last point I want to make is this idea of combining models so again I would like the way I laid it I was like oh task a tool a but the power comes from this Unix principle as I explained before where usually in a company the goal is not to minimize customer acquisition costs or to maximize customer lifetime value but rather to maximize revenue right that's really what we're after at the end of the day and all these other things are just sort of waypoints on that path because we do know that not all marketing channels for example provide equally valuable customers. They are different and really what we want to do then is combine the media mix and CLV estimates to get a measure of our profitability and not just these other metrics and of course we can just include customer causal impact analysis into this as well and not only estimate the the causal impact of our marketing efforts onto sales, but really profitability, right? And this type of thing. So the way that I like to view this is as just cogs in a machine, and each tool is an individual cog, but the power comes from combining them together. And also, by combining these things together, there's this idea of triangulation, right? So this is the thing that we're after, which we can't observe, but we have different data sets and models that provide insight, and if you just have a single one, right, you have much more, you can say many fewer things about this, but if we triangulate and combine many data sets and the insights, we can get an estimate that's much more accurate. The possibility is endless. So where are we going with this? Well, we're just getting started. plan to add many editions of you, follow our blog, then you have heard that we have hierarchical models across channels and location, time-varying parameters, Bayesian optimization, a dashboard, but really there's much more we can do. So asking everyone here to help us build the most amazing toolbox out there, find the code on GitHub, and here are some more links if you want to know how this can help you. We also offer a free 30-minute strategy session of how these tools can most effectively help you. So thank you all for your questions, for your attention, and now we have time for questions.

Speaker 2 [26:16]

Hey, thank you so much, Thomas. We have a bunch of questions on Slido. And I'll start with the most popular one. How is it possible to scale PyMC with growing data? And is there a distribution mode, I believe?

Speaker 1 [26:33]

Can you ask again?

Speaker 2 [26:35]

How is it possible to scale PyMC with growing data?

Speaker 1 [26:39]

So Bayesian models have the Many people think that they're slow and don't scale and that was definitely true and it's becoming less and less true Because many of those scanning problems have been solved So with PIMC you we have a JAX backend where it can run it on the CPU, but all the GPU So GPU sampling we found to be especially effective when having really large models for really large data sets. So So hundreds of thousands, millions of data points, not billions, like deep learning, for example, does. So there are limits still. So that is the first thing. And then we have another sampler called NutPy that just came out that's coded in Rust and then calling into the number backend of PyMC, and there we're seeing even better speedups on the CPU. So yeah, that's something that is being actively worked on and we already made a lot of progress on.

Speaker 2 [27:38]

Next question. Does PyMC marketing support CLTV projections for ad-based business models, for example, mobile apps?

Speaker 1 [27:48]

So, I think that is something that you can get out, but I don't think we have a particular function for it. So, there I would ask, like, to pull requests, we'd love to have that.

Speaker 2 [28:00]

Okay, next question. How do PyMC and PyStand compare in terms of sampling performance?

Speaker 1 [28:09]

So it used to be about the same But now with the things I just mentioned with the jack sampling on GPU and the not sampling Using rust and the number back end we found in some benchmarks that are not representative and not like Yeah, not done so properly and over such a wide array of models that I can I confidently say it's always going to be faster but we have seen cases where it has been significantly faster PyMC over PyStand with these new sample additions.

Speaker 2 [28:47]

How do you validate and then recalibrate your MMM models?

Speaker 1 [28:52]

So one key method of validating is the posterior predictive check and then That is the first step in what is called the Bayesian workflow where you start with simulation You look how well does it capture my data you look for any things that it doesn't capture yet And then basically refine in a loop the model iteratively and improve it until you get a good fit

Speaker 2 [29:20]

Okay. Last question. Is PyMC able to generate the models, generate graphic representations of the models such as a mermaid diagram? And is it for complex graphs really easier to to explain to stakeholders what's going on.

Speaker 1 [29:38]

So maybe the question aims at this model graph I showed with all these different boxes. So I mean, yes, so that comes from PyMC proper. So you're just like it's a single function call and uses GraphVis to create it. And so yeah, that's a good question. How useful is that really? I don't personally I don't really look at it. I know many people like it. And I yeah, it's it is difficult to explain these models to be honest. And that is one tool that could be helpful. Is it super amazing to show someone like 100 boxes that are connected in a weird way? Not really. So that's a challenge, and it's one of the better things we have, but it's not a correct thing.

Speaker 2 [30:24]

Great. So that's all we have time for the day. Give our speaker a round of applause again. Thank you so much.