Introduction to Uplift Modeling

Speaker 1 [00:03]

I'm very happy to be here and again thanks for the organizing and all the volunteers that make this conference possible. So today I'm going to talk about up-flip modeling, which is related to causality and machine learning. When I was preparing this talk and I was writing the abstract, I thought about going through a specific example, showing some code and let's say going through the concepts at the same time, but this turned out to be very challenging, especially because going through the literature and trying to understand this in a very thorough way, it took a lot of work. So I kind of split this in two. So what I'm going to do today is mostly focus on the concepts. But I did wrote a blog post or a notebook on which you can actually reproduce everything that's here so that you can still do it yourself. But I think understanding the concept is actually, in my opinion, a little bit much harder than understanding the code for this specific example. All right, yeah. So what's the motivation of this talk? So let us assume that you were given 100 euros and you just need to run a marketing campaign and you kind of identify customers which might turn or might leave your service. So one way of doing that is let us assume you have a voucher and you're going to send an email with that incentive. The easiest way of doing that is just send all the users all of the treatment. And that would work fine. And then how you measure your campaign, well, you just estimate the number of conversions that you had or maybe like basket size or amount in euros and then that will be your success metric. But actually this is a little bit tricky because we have different type of users. So the users that you would like to actually have or like have in this campaign would be be like the green one, which is the , which are the users which were thinking about leaving your product, but they came back because of your voucher. There are two other type of users, which actually you're kind of wasting your money, because on the one hand side, you have the lost causes that no matter what you send, they will never come back. And also you have the sure thing that maybe they were just having like vacations, or they were on pause, and they would have, let's say, come back to your product without needing the incentive. But there's a specific set of users on which this particular campaign actually was helpful. And these are the so-called no disrupts. So this one was someone who could also be kind of in pause, maybe in vacation, but you send that email and you just piss them off. Sometimes I'm one of these users. So you think about, OK, let us use some historical data and try to just train a machine learning algorithm to try to detect, like, let's say, which ones they have the probability of being in the specific groups, and just crack this business case. But the main point is that you cannot send and send an email to a specific user to try to measure the uplift. You won't be able to observe the data. And that's one of the key problems in this type of setting is that you don't have, like, a true label to compare against. So that makes things a little bit tricky. So what we're going to talk about today is precisely about techniques of how to estimate these causal effects. And there's a nice paper, which is the one that I'm going to follow most for this talk, which is Causal Inference and Upward Modeling, a review of the literature. And yeah, what we want to do is to estimate this incremental impact. And as we will see, this has two components. On the one hand side, we have causal inference, again, because we don't have true labels. But on the other hand, we are actually, in some cases, being able to estimate these treatment effects using historical data through machine learning approaches. So let me kind of set up some notation. And I'm going to keep it this very light. But we still need to be very precise about what we mean with these specific variables. So let's y sub i denote the outcome of person i. when it received the incentive, let's say the email, and that's important, this little one. And why not is when it doesn't receive that. And what you would like to compute is the difference between these two, which is the causal effect. But again, you cannot know that. But still, it's worth, let's say, aiming on what we want to estimate. Moreover, if you have a set of covariates, let's say you have information about the users, could be the attribution channel, like how they got your platform, maybe geolocation data, or some other attributes, then you can have that in a feature matrix, like in usual machine learning, which we denote by xi, which are a set of covariates. And you will be interested in the expectation of the difference between those two outcomes, conditioning on the covariates. And that's what we're going to call the conditioned average treatment effect. Again, we cannot measure this, so we need a way of kind of hacking this out. And there's a way, of course, but there always come with an assumption, which is very important to understand. So besides these kind of variables that I introduced before, let us denote by wi. It's a binary variable, like 1 or 0, just indicating whether the person received the treatment or not, let's say the email. So in reality, what you measure is yObserve, which is kind of this combination. So if you replace w by 1, you will get y1i. and you replace it by 0, it's y i 0. So this is just notation, so we haven't done anything yet. And now here comes kind of the most important thing of the talk, which is the assumption on which everything we're going to communicate afterwards holds true. If we assume that the treatment assignment is independent of these observations, conditional on the covariates, that's a big if, then we can estimate this conditional average treatment effect by computing the difference between these two expectation values. So if you think the expectation values actually are think about historical data, you're going to take the outcome of the users which had a treatment and subtract the values or observations for which user which didn't have a treatment. So the tricky part about the unconfoundness assumption is that you need to make sure that there's no kind of hidden variables which can be explained in the assignment. And this is particularly tricky because there's sometimes variables that you don't know that might kind of leaking information or making this assumption fail. So a typical example of doing this is an A-B test when you try to do a random split. But I recently read an article about saying that even if it's random, you could still screw up this assumption. So I'm going to reference that to the end of the talk because they show very concrete examples. So if you think about the next task, so assuming that we have this condition, then we have collected some historical data, let's say, from an experiment. And I'm going to show you how to estimate these two things using any type of machine learning problem. But as in every problem, the tricky part is really the data. So how would you collect the data, and how you will actually make this useful in your specific case? So the first that you need is some training data. And for that, you'll do kind of a typical A-B test, where you split your customers in control and treatment. And then you send emails to one, and the others, you don't do anything. And you collect the outcome. Again, the outcome Y could be a binary variable, like conversion, like the converter or not. Or it could be a revenue, or like basket size. Let us assume, for simplicity, that is going to be conversions. After you do that, you can train your uplift model, which I'm going to talk in a bit. And then for a future campaign, of course, you don't want to do the same mistake of targeting the don't disturb users. You are going to use the prediction of this off-lead model to target specific users which you expect based on historical data and lookalike features from your covariance matrix. I'm going to react positively to your treatment. It is recommended also that in the second step, you still do a control and treatment split so that you keep enlarging your training data, right? Because you still want to, let's say, keep doing this as campaigns go over. So, yeah, the data collection actually is the tricky part because you need to ensure that the hypothesis and the unconfinedness assumption actually holds. So there are many ways of estimating this uplift. Today we're going to focus on what is called method algorithms, and these are kind of, when I was reading about it, like intuitively made sense. The other one seems to be also very powerful and sometimes overperforming these ones, but get a little bit tricky. So as we're going to talk about this in a kind of introductory manner, I'm going to stick with the meta algorithms. And meta just means that we're going to use any kind of arbitrary models to estimate this difference between these expected values. So I'm going to talk about three of these meta learners. The first one is called the S-learner. And what we do here is the following. At the training step, we take, we have all of design metrics, which I denote by X, and then I create a matrix on which I add a column, which is the treatment. So it could be one or zero. And I have the outcome, which I collected from my A-B test, and I can use any machine learning algorithm to learn that. I'm going to call that mu. And whenever you have new customers, or at prediction time, what you do is you take that trained model, and then replace the features for the new users, and then replace the column of the treatment to one and then subtract the same thing by restricting it to zero. So this is really similar to what we saw like from the expectation value that I showed you before. Notice that at training time, I do need the labels, but at prediction time, I don't. So that's the whole point. So there's another possibility and this is what is called the two learner is that instead of having one algorithm that learns this, you actually train on the treatment and the control group separately. And I call this mu c and mu t. And of course, you don't have the column w in this case. And these two algorithms could be very different. One could be a linear regression, and the other could be a tree-based method. And what you do to estimate the uplift is just to take the difference between these two. So in essence, it's very similar to the S-learner. When I was looking into these formulas, I couldn't stop wondering, like, are we actually estimating uplift from noise? Because at this stage, it's a little bit tricky to see if this associated uplift is just the true uplift or it could be just this error, which is just being, let's say, fed by the model. So there are other alternatives. If actually there are many learners, I'm going to just stick to these three ones, which try to account for that. So this is what is called the X-learner. So the X-learner in step one does the same thing as the T-learner, which is just take the control and the treatment group and train two algorithms on them separately. But at prediction time, we do something different. So first we compute something that is called the imputed treatment effects. And these are the following. So for the treatment group, I take all of the outcome of the treatment, and this is why this is the first column, and then I subtract the predictions using the model from the control group using the treatment covariates. And I do this in the other way around for the control. This is somehow to mix the control and the treatment groups. And these by itself are all kind of hinting like they're separately estimates of the uplift if you think about it So if you want a prediction time the final step is once you have you define these D variables Which are imputed a partial treatment effects then you have yet another set of algorithms which is called TC and T tau which actually learned this and These are actually by themselves two estimations of the uplift So, you could actually take one or the other, or take a combination, and that's actually the final step, which is just take a combination of these two with a varying function g. How to select this function? I guess that comes a little bit with experience, and there are some tricks. One of them is actually to use what is called the propensity score. And this is just defined as the probability of having a treatment given the effect. So yeah. When I was reading this, it was very confusing because the kind of trying to explain this to myself took a while because the intuition between this D tilde is kind of there. But the question is like, why is this actually better than the T and the S learner? So what I'm going to do next is to give you a very concrete example, which I found in a paper explaining the x-learner where we can actually get some intuition of the things that could work and that could fail. So yeah, this simulated example actually is such that the treatment effect or like the uplift is actually equal to one. So it's a constant. We know the answer. It's a simulation. So I have two groups. I have the treatment and the control. The treatment, in this case, are the blue lines, and the control are the red dots or Xs, and the X axis is the covariate, so anything that could be used in our model, and the Y axis is the outcome. So if you see now the fit, you have the blue line fitting these treatment points and this kind of piecewise linear one fitting the control group. So this is actually what you would do, this is one example of fitting a machine learning model on the control and the treatment groups separately. So this is the first step of the 2-learner. For the 2-learner, what you would do is actually take the difference between these predictions. And this is what you see in the solid black line. So this is what you would get by taking the difference between these two fitted lines. And of course it doesn't work, just because this area on which the treatment goes up is actually not working properly. And the thing is, we're just looking at them separately. So the dotted one, which we'll see in a bit of how we got there, is actually the estimation of the x-learner. So of course in this example, the x-learner is going to work better. But why does it work better? Well actually it's because exactly this imputed treatment effect. So I might not have the time to go through it, but you can actually kind of verify that, for example, the impeded treatment effect of the treatment group is actually a straight line or like the fit in this case, which is the tower algorithm. Whereas for the other one, for the control, is this kind of valley fit. And you can check that. I don't want to do that because it's a little bit cumbersome and maybe also in the sake of time, also it's explained in the blog post. So in this case, we notice that the blue line actually is what we want. And actually the red one is not working properly. So in this simulation example, I could just eye-pick one, but in reality, I don't know. But again, I talk about the propensity score, which is the probability of the treatment given the covariates. So in this case, on which the classes are very unbalanced, the propensity score is going to be close to zero. So it's going to be small. So it's actually going to weight in this kind of convex sum between these two, the blue line much more than the red one. And this is actually very close to a constant one. So I really hope this example allows you to understand, and let's say that there's some hidden intuition that it has to be kind of obtained by looking this into, let's say, beyond the formulas and the formalism. So there are many Python implementations. So I collected a couple of them. So we have CursalML from Uber, EcomML from Microsoft, and there's another one which is called Scikit-Uplift. Scikit-Uplift has a very nice kind of diagnostics model. So check them out. but they are, in essence, very similar because they give you the freedom to choose any algorithm and try to estimate these effects by taking this difference. So this, for example, is how you would fit one of these meta-learners in Caution ML. So for this specific case, I'm going to use this gradient boost classifier from SKLearn, and I can just select the T-Learner to just use this one for both the treatment and the control group and then I input that into the base T classifier and I just call fit and predict and you can get the average treatment effect which is the average of the whole population and also you can have the conditional average treatment effect which is at user level and you can access the models to do any type of debugging or some quality checks on the models themselves. So again, the code is not that hard. What is hard is really to understand what's going on behind the hood. So now we know how to estimate the uplift, but the next natural question is how do we know if the estimation is good or bad? So when you go into the literature, there are all of these plots which kind of resemble what is called the area of the curve in machine learning, and I naively thought that it was going to be something kind of straightforward to just translate from one to another. But it turned out it's not that similar, and I took a while to understand how to evaluate this method, so actually for the rest of the talk, I'm just going to focus on how to do this model evaluation. So on this side, on the left-hand side, you see how you usually evaluate like a classifier. You have the false positive rate, and you have the true positive rate, and imagine that your model is outputting certain probabilities, but you're interested maybe in the predictions themselves, so in the tags 0, 1. So you can vary the threshold on which you split these classes, and then by computing these kind of quantities, you get this blue curve. And then you have two important things. One is that if you train a dummy classifier, you will just get this straight line, and a perfect model will just go into the kind of corner of the graph, so that the area under the curve for a perfect model is always 1. So a nice measure that people use is the area of the curve. But if you look what you do for the uplift model, you see various things. So it looks, in essence, the same, but on the one hand side, the perfect model looks very weird, and also the perfect model changes with the data. So if you change a problem, that's going to change. So it's not like in this case on which it's always there. Also notice that the axes are different. On the x-axis, I have the number of target users. On the y-axis is the number of incremental outcome, kind of the uplift. So it's kind of the same in essence, but it's actually very different of how this is computed. So hopefully, by the end of this talk, when you go to these papers, you can actually get a better feeling on how these are computed. So again, the problem is that we don't have the true labels. So there's no way we could actually know the truth. But we can still do some work around or some kind of sanity check. So one thing that they do, or the authors in the literature suggest, is to compute the uplift by percentile. And what does it mean? It means that I take my model, and I have all the uplift predictions for all the users. And what I can do is I can rank the uplift predictions in decreasing order. So the highest one are at the very beginning and the lowest one at the end. And this is just a vector of numbers. And then I can group them in percentile, let's say, from 0% to 10%, from 10% to 20%, and so on. And for these kind of buckets, I can estimate which were the number of users in the treatment group, the number of users in the control group, what was the response rate in the treatment, the response rate in the control. And I can estimate the uplift by taking the difference between these two. So I will kind of obtain an uplift curve per percentile after I fit my model. And this is how that looks. Like the red curve, it's the final difference. And the yellow and the green one are actually the absolute, so to say. So if you take the difference, you will get the red one. And a nice model will have like a decrease in curve, right? Because you have sorted your uplift. And so you are expecting that when you take this difference, actually this still kind of conserves the trend, because you organize your uplift predictions in a decreasing way. But that doesn't give you any number. It's kind of a plot. But it's very hard to compare plots if you have different models. We would like to have a measure. But we are going into that direction. So using the same principle, we can actually use something that is the cumulative gain chart. And in essence, this first part, this factor y over n, are essentially this difference that I computed here. So there's nothing new. And I just need to scale by the number of users. And I can plot that. But again, going to the detail is a little bit tricky because it's a cumulative sum. So it's not to take the cumulative sum of the uplift, But you first need to go into the cumulative sum of the absolute values, and then do the computations. And this is something that is hidden in the literature. And just by looking into the concrete numbers, this is something that you realize. So yeah, just be a little bit careful. I, of course, don't want to go into the details. It's not hard. But yeah, sometimes I find it hard to really understand the concept just from the formulas. But if you do it in kind of the expected way, you get something of this sort. So you have the percentiles, again, working with these groups. And for each of these percentiles or buckets, you are actually computing this cumulative gain. So you can actually write it in Python. That's not very hard. And then you just plot it. So this chart is actually giving you a lot of information. So the last point of the plot would be, OK, The percentile is kind of 100, so I target all of the users. And if this is above zero, so to say, this is your cumulative gain. So even if you send the emails to everyone, you still saw positive uplift, which is fine. And this is something that you at least would expect. But notice that there is a maximum around 60% and 70%. And this is telling you, if you wouldn't have sent the email to everyone, but maybe just, of course, this is after fitting the model, but to the 70% or 60%, you have gotten much more uplift. So the rest is just you're wasting your resources, right? Because you're targeting these ones which are not going to react positively to the treatment. So even at this stage, actually, people recommend that this is very useful for marketing managers. Because for the next time they run the campaign, they will have the uplift list, and they will say, OK, let us make a cut at, I don't know, 60%, 70%. And let's go through that if we assume that the kind of samples between the split are similar. And again, this is why the data collection is super, super important. So if you're a marketing manager, I think this plot should make you very, very excited. Because if you think about this, it's not just about kind of effects in general, but also money. because sending an email costs money, and also losing customers just because they were not served is actually quite painful. So this we did for a percentile level. And there's nothing that prohibits us to do this at user level, right? So instead of having 10 buckets, let's do 20, or 50, or actually take all of these users. And this is the generalization towards what is called the uplift curve. So we can just do the split as granular as we can. And this is precisely the uplift curve, which is the blue line. So this is already zoomed in a bit, but this should be kind of the smooth version of this one. So that's why, let's say, you see that if this curve is high, then this is going to be a good one, just because we're expecting there's going to be a kind of high lift with respect to the point which have given emails to all of the users. But then there are two lines of reference, which are the purple one and the red one, which are the perfect model, so to say, and the random one. So let us try to understand how does this work. So I spent a ridiculous amount of time trying to understand this plot because it has to do with how do they come up with the perfect model if there's no perfect, like there are no labels. So the way I did this was to go into the source code of the scikit-uplift package and try to see how they were computing this. And the idea is actually quite simple but quite nice. So here's the key point. A perfect model assigns higher scores to all treated individuals with positive outcomes than any individuals with negative outcomes. So that seems trivial, but we can actually call this in a kind of pseudo-deterministic way. So what we need to do is to make sure that when we do the errors, it's as kind of small as possible. And that's the whole point of the computation. So what are the errors? Well, the errors are the control users which had a positive response. and the treat ones which didn't. So these were kind of the bad cases, so to say. So this is going to be a correction term. So what they do is to take, let's say this is already when you're evaluating your model, you have some true labels, like they convert or not, and you have the treatment. So the model is working, like the uplift model is working good when these two are the same, meaning that you predict there is a treatment, there was a positive outcome, or there was no treatment, there was no positive outcome. And then you need to correct that by just taking the errors, which are, let's say, the biggest ones, so to say. And it's quite clear, because if you add these two, then think about it, and the statement actually holds true. So if you plot this, and again, what you need to do to compare all of these plots is to take the outcome of this perfect uplift vector, sort them in decreasing way, and plot that, you will see this step function. Which, if you think about it, take your time afterwards, please. It makes a lot of sense. This is an example of the models that I use with real data. So they capture, in essence, something. But of course, the models are not as good as the perfect one. And this is one of the things that I'll comment later, getting to the perfect uplift curve is very very hard but now if we plot this step curve on top of the uplift curve that is output from the diagnostics we see that exactly this is this has if this coincides with the points on which there's a jump in the perfect uplift so in this plot that I show you here this is kind of hidden but if you go and mark the points, which are not interpolation but real points, this is actually the case. So to go from the green one, which is the perfect uplift curve, to the purple one in the uplift curve space, so to say, you just need to apply this formula. And this is something that you can do relatively easily. All right, so this solves part A of the mystery. The other part would be how to evaluate the random model. So, I mean, one way of looking into it is like, okay, it should start at the same point and end at the same point because I'm targeting all of the users. But I like simulations instead of sometimes thinking too much about the math. So what I did, I actually ran a lot of dummy models, which would do very random uplift. And you see in this plot, in this upper one, that it does fluctuate but in the limit let's say or when you zoom out at the scale of the perfect model it's essentially a straight line. So I also recommend that when going into this type of literature to work some simulations yourself because it makes all of these computations much more transparent and I have a background in mathematics but I often get lost in this formalism especially because of how they presented in the literature. All right, so in practice, if you're doing this in real life, what you could do for example is take various learners, like the S, the T, and the X learner, and you can do in sample and out of sample prediction. You can still take your data from the A-B test and split that in such a way that the right-hand side is an out-of-sample prediction. So notice, for example, that in this specific case, the T-learner was best on the training set, but actually the X-learner was a little bit better on the test. And I think this has something to do with the fact that the T-learner might be capturing just a lot of noise when it's taking these two differences, whereas the X-learner will try to combine the treatment and the control groups in a more kind of efficient way. In my experience, a very short experience playing around with this model, it's actually very hard to get a good uplift, just because, again, these are kind of proxy metrics, and the literature often says, okay, actually you can plot this without having a perfect model to have a better visualization. I just left it there. So usually comparing it against the perfect model, it's maybe unfair, but, yeah, it's worth comparing against the random one. All right, in the literature, you often see also another type of curve, which is called the Kinney curve, and this looks also quite strange, but the construction is exactly the same, but you just change your target metric that you're going to compare. So you don't take this cumulative gain, but you take this kind of change version, which is actually not that different, to compute this curve. So why would you like to do something like this? Well, some authors suggest that even though the uplift curve can be more intuitively, let's say, easy to understand, this one actually depends on the quotient between these two, like the number of control and treatment. So it's not going to be that biased against if there's like a very unbalanced class. But you can still run the same type of analysis on which you have the green curve, which is the perfect model, pass it through this Keeney curve and you get those plots, and then the best model or like a good indication of a model would be the one which has a better Keeney curve in maybe the out-of-sample set. All right, so you can still go into the notebook, which I can maybe probably do, to guide you through this. So you can find the presentation there. This is also linked to the GitHub repo. And then I took one of the tutorials from the SK uplift kind of a tutorial set but I used casual ML for it so hopefully this can give some kind of different flavor on the results the blog post is a little bit long just because this is real data so I do exploratory data analysis and that takes most of the part but at the very end you can see all of the simulations all of the plots that I presented today and specifically how to compute this from scratch so this is taking a lot of code from SK uplift and try to extract the essence so you can of course use the package themself you don't need to reinvent the wheel but if you really want to have like a complete understanding of this I will still recommend to do this in your free time so to say so yeah here you have all of the code to generate the specific plus that I show here. So yeah, that's about the notebook. I strongly recommend if you're interested in this topic to go and check it out. And also I'll be around in the conference if you want to just talk about this or discuss about any other topics in marketing, which is where I'm working mostly right now. Yeah, here are the references. I collected just the ones which were very useful for me. That doesn't mean it's exhaustive. I would still like to recommend the one by Gutierrez and Gerardi, which is about this kind of review. There's some specific about the meta-learners, but this one, randomization does not imply confoundness, is something that I found last week. And I think it's a very interesting paper. So people that are running A-B tests or plan to run A-B tests, please check it out, because there are some hidden kind of challenges in the data collections when doing this model. And that's it. Thank you very much.

Speaker 2 [36:08]

So, thank you, Ron. So, I feel I have a couple of questions here. So, yeah, so this is the first question on about... So, you play with statistics, and you need to have statistical powers. So, actually, what is the order of, like, what is the minimum amount of sample that you need to start to play, basically?

Speaker 1 [36:32]

I don't have that on top of my mind, okay. I think the, because it's not really, because in an A-B test, you want to compute the change to a specific metric, like conversion rate. But this is something that you want to estimate at user level, right? And you just have one observation per user, so you say in the training set. But I don't have any kind of threshold on what's the minimum sample size you need to do this to make this reliable. Again, we still don't know the target itself. But I think that's an interesting question.

Speaker 2 [37:09]

I assume it's maybe like machine learning, you shouldn't start before like 1,000 points because otherwise, you don't know what to look at.

Speaker 1 [37:14]

But if you have good covariates, and usually in marketing you might not have a lot of quality data, so if you have good covariates and domain knowledge, linear regression is not something that can be discarded from the start.

Speaker 2 [37:29]

Yeah, simple model so second question here is Could you like compare this with what you have in reinforcement learning basically could you like

Speaker 1 [37:43]

I haven't worked with reinforcement learning. I'm more like of a classical machine learning based experience. But if you know, just reach me or write me. I will be very interested in knowing more about that. My impression is that it requires a little bit more machinery and maybe data, but I could be mistaken.

Speaker 2 [38:03]

Okay, so next question, we estimated the uplift, how would we estimate if it is statistically significant or not?

Speaker 1 [38:17]

Again, think about, I think this is a tricky question, but as for now, which is the reason I'm trying to learn this, is there's still an opportunity just thinking about optimizing this without going kind of very deep. So if you have this type of plot from an experiment, it's already hinting you that sending the treatment to everyone could be just not the best solution. So sometimes you just need to optimize on the fly. And that's one of the reasons on which, for example, marketing managers would be keen to understand this, even if they don't kind of reach a p-value level, so to say. But definitely this is something that if I find out, I will write in the documentation. I didn't see that in the literature also, I never saw this type of discussions, which made me wonder.

Speaker 2 [39:14]

So another one is about repeated tests, so If you want like to repeat your experiment or to like to repeat the test And that you have already like you data How do you mean is it possible to under that like do you repeat the experiment by mixing people or like changing the groups?

Speaker 1 [39:34]

Yeah, I think this is actually very tricky and in practice. I have it here. This is from the SK uplift part I think what? Like for example if in your data collection part or a step you send a treatment Which is very different from the one that you are going to a send in the second one Then this probably is not gonna work. So you need certain type of consistency, right? and also, let's say, collecting the data in one country and applying this to another, actually would probably violate the unconfirmed assumption. So it could be that you need to make sure that what you did in the first experiment for the data collection is actually very representative with respect to what you were going to do in the second iteration, which you still want to collect data, that's what they suggest, so that you keep learning this, because, of course, there's kind of a time component there that you want to maybe capture through just enlarging the training set for your uplift model.

Speaker 2 [40:37]

Okay, so one last one here. Yeah, so apparently you mentioned at some point that you could filter some of the data. I don't recall, but apparently. So the question is, if you do that, are you going to create a kind of co-founders and then break your things because actually maybe if the selection could be a latent variable, let's say, to your finger and and like yeah

Speaker 1 [41:07]

Yeah, I think that's quite interesting, but also call into the application. So I have certain type of strategies for different type of users. So if I have users which are the ones which I'm going to apply the certain campaign, maybe there's an incentive of doing that. And this may be because I know there's a potential of churn. Actually, there's a nice, and this goes into this direction, a YouTube video from PyData on which they kind of see this holistically. So many companies or many use cases, they predict churn, right? Maybe your company has a subscription-based model and you predict churn. That's not very useful by itself because, okay, you predict that. So what? So this is something that you could potentially put on top of that because if your churn model outputs certain probability scores, then most likely you will have campaigns to re-engage those users based on that, right? If someone is using already your product a lot, why would you send a voucher, right? So this would go on top of the churn prediction model so that you actually want to prevent churn. And actually, in this model, now that I mentioned it, they go a step further later in the kind of funnel of the project and they use something around reinforcement learning. So maybe if the one interested can check this out, this PyData talk, which is very interesting.

Speaker 2 [42:39]

Maybe there's a time for one last question in the rooms. Okay, first. Very nice talk. Let's say I have 10 candidate emails.

Speaker 1 [42:49]

candidate emails and

Speaker 2 [42:51]

And I've tested them, and I want to send them out to customers given their covariates. Does this generalize to such a situation where it's not to send or not to send, but how to engage with this customer?

Speaker 1 [43:03]

this customer? Yeah, I guess in this case it would be like if your treatment variable has many kind of not just one zero, but many of them. I wonder if we could view thinking about the machine learning problem of multi-class classification, you can do maybe once versus all and try to do that, so this is something that might be interesting to see. So you still do that and then you have various for like say one versus all and try to compute like the final one, but I don't know, this is something that I, yeah, actually the literature is not that advanced in that case for these specific introductory talks, but I would do it this way. I would at least, that's what my intuition suggests.

Speaker 2 [43:46]

Thank you. So thank you. I think, so if you have other questions, we are on a break now, so you can go and see Prae run during the lunch and ask him more questions. So thank you, and we can again applaud. Thank you.

Speaker 1 [44:01]

Thank you.