Intuitive A/B Test Evaluations for Coders

Speaker 1 [00:08]

Yeah, thank you very much for showing up. I know this is also the last session, so I hope there is something to take out from this for you. As already said, I'm working at Holiday Check, so my current role is I'm leading the machine learning team. And in this position, I have a lot of management duties. And part of this is about interacting with stakeholders, be it on C-level, but also on software engineers level. to talk about evaluations of A-B tests. And there is something that I want to share with you, like some experience that I've had, and I hope you can take something out of this. So for those of you who came to this talk and are not really aware what A-B tests are, so usually A-B tests are about so-called randomized controlled experiments. You have your variant A, and you have a variant B. And then, of course, you want to figure out if variant A, for example, performs better than variant B and there's a lot of challenges attached to this right and there's also a lot of discussions about this and I think some of you who might be in this business might also see like there's a certain pressure about releasing early because you already see some good results first right or you see some bad results and then you don't want to do it just to keep it very simple and I tried to do this so this is supposed to be a rather light talk for the end of the conference. Let's say we have a very simple A-B test, which is like two buttons. So usually it's something different. I can promise what we're doing, but you have a blue button and you have a green button. So the blue button would be variant A. The green button would be variant B. And you want to understand if one of those actually, like, for example, now the blue button and variant A, leads, for example, to higher click rates. So, just a warning up front, so the target audience of this presentation is not so much like people who are very much into statistics. So these are also not the people I usually talk about when it comes to the decision making. It's more about people who feel more comfortable on other areas, and in this presentation I want to focus on software engineers. So if you're more familiar with for loops and random number generators, then there is something to take out of this presentation for you. If you're more into statistics, there might be as well something to take out, but it's more like on a discussion level, I would say. So really, like, this is the target audience here, and I thought, like, since we're here at the PyCon and PyData, that's a nice venue to actually talk about this. So if you're into software engineering and you don't have, like, a very thorough statistical training. So again, coming back to the target question that we have, so we have a button that is blue and a button that is green. So one of this is variant A. The other one is variant B. And we have some data attached to this, right? So we kind of randomized the users. For the purpose of this presentation, again, it's only 12 users per group. But as you can imagine, there are way more usually. And they also don't need to be kind of exactly the same number, right? So the sample sizes can also differ. But they should be more or less, like, rather equal. There shouldn't be a huge discrepancy between the sample size numbers. So you have all these numbers, and this could be, like, in the context of a website such as HolidayCheck that could be, for example, like the basket value of booked hotels, right? And then you have the numbers in Variant A and the numbers in Variant B. And then one very simple approach, of course, is you just take the mean of those two. And since you're assuming that Variant A is performing better, you just subtract the mean of Variant B. So you just want to identify if there's a higher basket value in Variant A than in B. And what you end up seeing is that you have 172 more on average, which in this case would be Euro since we're based in the German speaking or in Germany. And like 172 basically less in variant B. So now the question is, is this big enough? And you might look at the numbers and say, actually, like considering all these numbers that I see, that might be actually big enough. But then, of course, I mean, this is not news. That might be just a bit too easy, right? Because if you just do a very simple example and for here we just toss a coin here. So what you end up is like what I did here in NumPy basically is I have 100 coin tosses and I run this experiment exactly 100 times. So this is what you see here on the size parameter in the random binomial function from NumPy. And what you see here is all the results that I get, so all the 100 results. And in an ideal world, because I put in 0.5 as the probability, all of them should be 50, right? Because it's equally likely to have heads or tails. But they're not. I mean, some of them are 50, some of them are a bit more, some of them are a bit less. But what you also see is some of them are quite different, right? So you have 68 and you have 37. And especially 68. I mean, this is just one experiment, right? And the point is, like, what would you say, like, if you just saw this one experiment with 68, would you say that this is a fair coin? Or to put it in other words, if you do an A-B test, you cannot afford to run so many experiments. You just have this one, right? Because it always takes time. It might be also kind of expensive, depending on what you end up testing. but you just want to know like if we were just unlucky in this kind of case that we got or like in put it in other words if it was just by chance how often would you see this and there's a very good book that i can recommend by leonard mladenoff which is called the drunkard's walk how randomness rules our lives so the question is we as human beings we underestimate how often chance events are taken as patterns, right? So chance events are ubiquitous, but we as human beings, we're pattern searching, we're pattern seeking animals, right? So we always try to interpret something into those random events and we identify patterns when there are none. I mean, oftentimes we also do see patterns, but in a lot of cases it could just be attributed to chance, right? So the point is, you need to do it more than just looking at some numbers, right? And then you end up, of course, looking into statistics. And then there's different kind of fields, right? So in North America, there's a lot of people going into the Bayesian approach of A-B test evaluations. Over here in Europe, or at least maybe the times are changing, but at least in the past Or when I did my graduate classes, then we also had a lot of frequentist approaches. So hypothesis testing with p-values and not like Bayesian factors taking into account. And then you just go on and think about this. But again, you easily get into a rabbit hole. You talk about things like null hypothesis, about p-values, about significance levels, t-tests, sample sizes. And I'm not saying that what I'm going to show you now is kind of getting rid of all of this, but the point is you're talking to somebody who's not like statistically trained, right? You're talking about like somebody who knows how to write code. And then you might argue, okay, but luckily Python has you covered because there is, for example, the stats models library, and you can use the independent t-test out of this library. And then you get something like a t test statistic, a p-value and degrees of freedom, DOF, and as a software engineer you know how to plug these numbers in. So what you then get is a result and people tell you okay just look at the p-value and it needs to be below 0.05 or below 0.01. But then again the point is what question did we actually want to answer and this is gets easily lost in this or is it like you being a coder is it more like how likely is it to see such a result purely by chance right what i'm going to present is basically two things out of what is usually called resampling methods the reason why i want to present it is because i think especially in introductory statistics they're not so often shown at least I didn't come across them and in my opinion at least I think they're very intuitive for somebody kind of who has this background. For those of you who are interested I give some additional references at the end especially like one talk by Jake Vander Plaas who is basically also this presentation is highly inspired from. So let's start with shuffling. So again what was the point? The point was, we have two variants, A and B, and we want to see how likely are the results, like this 172 that we found, how likely is this if this would just be chance, right? And then, I'm just putting it here, null hypothesis, if you don't know what it is, just forget about the word, but just say, like, you assume that they are the same. So you assume that variant A and variant B is the same. In our case, it would be like saying we test a blue button against a blue button or a button that is exactly the same, right, which is usually referred to as an AA test, which we also sometimes do. But in our case, actually, there is something behind this. So it's kind of blue and a green button, but we assume that they are the same. And what we can do is what you see down there is we can just shuffle it, because if they are the same, it doesn't matter if something ended up in variant A or variant B. It was just coincidence that it ended up in A or in B. So we can just shuffle these numbers around, right? And then, because you also know how to write for loops, you can do this a couple of times, right? You can shuffle, then you can rearrange. So this is no longer the original variant A and variant B. It's just rearranged according to the shuffling. And then you compute the means. And now, of course, the mean here is different. It's no longer 172, it's 23.8. And this is because we shuffled it, right? And then you can plot it on what is to become a histogram in the end. So this is the code. I mean, I used NumPy for this, but you can use, especially for the permutation here, but you can use direct, like you don't need NumPy for this. You can directly use it in pure Python or directly code it in pure Python. And then, as I said, like you just take a number of iterations. I just take 10,000 iterations. and then you just go through it and since this is a bit small I will just skip this one and just show the video so what you can see is you shuffle you rearrange you compute the means and then you plot it on the right so you shuffle this is for each iteration you do the same thing you just remember what was the mean that I computed for this particular iteration and then I plot it And what you end up doing on the right is basically you're building up a distribution, right? So this is a distribution of the differences that you would see in the means if A and B were the same thing. So again, this is the basic assumption. And this is probably the most important thing in this presentation. You're assuming that they are the same, so you can use that information. And if you're done, so this is kind of 10,000 iterations, then it looks something like this. And this might remind you of the normal distribution. It's centered around zero, which kind of makes sense if you shuffle it. But this is what you get, right? But then again, don't forget about what was actually the question that we wanted to solve. We wanted to see like this 172 that we saw in this one experiment that we had, right? So the experiments that we simulated, these are just simulated data based on the assumption, but we only had one experiment where we really got a difference in the means, and this was 172. And we just plotted on this histogram. Remember, we got this from here. And then we just say, like, if you look at it from a histogram perspective, how big is the area that you see that is 172 or more extreme, which in our case is higher because we're interested in if it's performing better than this. compared to the overall area, and in this case it's 2.5% or 0.025, you want to do it in a fraction. Which means the rest is 97.5, because everything is 100, of course. And now you have this number, and then like the frequency, so okay, that's the p-value. Which, basically, if you just use the resampling methods, that's it. And now, the point is, if you look at the definition of the p-value, if you read it up in Wikipedia, for example, see, in null hypothesis significance testing, the p-value is the probability of obtaining test results at least as extreme as the result actually observed under the assumption that the null hypothesis is correct. And this is exactly what we did. But I think what I just showed you, at least in my mind, is far more intuitive than just reading this definition. Then you might argue, okay, but we had this kind of t-test statistic that you showed us before from the stats models. And yes, if you look at the result, it's 0.02472. Compare this to what we got with the shuffling approach. It's pretty similar. It's not exactly the same. I mean, as you can imagine, if I do this test again, it might be also slightly different. This is what you get if you look at the t-distribution that you actually use for the t-test with the degrees of freedom that you have from the data that we have, which is basically 12 minus 1 times 2, which is 22. And then you look at the t-statistic, which is basically the standardized of what we had with the 172, which in this case is 2.08. and they look kind of similar right and in Python with its ecosystem you even have another way of doing this or like the same way of doing it but you already have it in scipy stats in the library scipy because you also have this independent t-test and you have a parameter which says permutations where you can also give the number of iterations where you can also get a result which is basically even more similar than what I had before so I hope this is kind of clear in that part just to remind you like we just looked at the overall hotel like in our case basket value numbers but you say usually I'm interested in conversions like how many people booked or how many people actually click this button you can do the same thing you just have here the number of users you have the number of conversions you have conversion rate you assume B is this time better you just take subtract a from B you have like point zero zero nine seven and now again the same question is this big enough for me to say that B is actually better than than a and what you do here is kind of also very similar. First of all, you just construct, maybe I can just, does this work? Over here, you can see that we take the total conversions as one, and we take zero as the total conversions, total uses minus the total conversions. So we're building up this long array of zeros and ones based on how many that convert if we just lump them together. And then we just shuffle, which is again the same thing. I use NumPy as a random shuffle here. And I take, and this is important, always the same size for A and B. So they could be different, but here you need to always take the same sizes. You can shuffle them, but then always take the same sizes. Subtract the mean and just plot it. Same thing. In this case, again, it's pretty low. It's 0.9%. Again, the same idea. You can use this as the p-value. And I think there is a certain understanding, because now if I need to make a decision, I would understand as a user, okay, if I run this experiment 10,000 times or 100,000 times, a million times, like in this case, in 190 cases out of 10,000, I see a result like this or even more extreme. Just for the rest of this presentation, I am going to also demonstrate a second method from the resampling tool belt, which is bootstrapping, which is a fancy term of sampling with replacement. You can use this to build confidence intervals, so you might have also heard about confidence intervals when you are analyzing or evaluating A-B tests. And it works slightly differently, but it's kind of a similar idea. In this case, again, we have our original variant A and variant B. And we just want to sample with this. And with replacement means, I just take randomly numbers of this. It needs to be the same size. I mean, of course, as you can imagine, usually this is kind of much larger. And then the important thing is if I would do it without replacement, then I would always end up with the same number or with the same array, right? So we do it with replacement, which means now the 722 is two times in here. But that's fine. That's actually what we want to have. And the bigger the array, the more often you find these things. They could actually occur also twice the same number, but then you have it like two times that you can select from. You do the same thing for Variant B. This time you don't shuffle, you just select and select and select. And you see the same thing, right? You can do this with random numbers and iterations. So in this case I used a choice just to take a group size and I put in replace true. So just to put it back. And what I get is like then I just take the mean from sample B minus sample A and I just dependent and then I take like in this case the percent odds which is basically I cut the two point five percent of the histogram and the other two point five from the other direction and then you end up with something like this you again have such the spell shaped distribution in the middle like the green one this is actually the one that we had before right so this is what we observed is 0.0097. So this is in the middle. And then from the other, from the rest of the distribution, I just take what I get from the 2.5 and 97.5 percentile. And then one interesting thing that you see is, because what you don't want to have, you just want to have like that, in this case, B is performing better, has higher means, right, than A. Then you're comparing it to because zero is actually the tilting point where A could be higher than B. And this is what you don't want to have. And usually what the approach is, if this confidence interval of 95 is outside or it doesn't include zero here, then you can just take it as a clear signal that in this case B is better than A. So this is what we basically get here. So, there are some differences in doing this, like when you need to apply shuffling, which is basically when you want to have something like the p-value, when you just want to make a decision, when you want to have some estimate about, again, like with the resampling, it's very easy to understand because we're doing the same thing here, then you need to resort to bootstrapping. Just a warning towards the end. I'm not claiming that this is to replace the traditional methods that statisticians have been using. All I'm saying is, especially for introductory classes or when you need to get a certain intuition for the evaluation, I think the resampling methods are very good at giving this, at least in my mind. So if you start with this and then also seeing, okay, what are the pros and what are the cons, There are also advantages of using resampling, quite a lot actually, but there are also some disadvantages compared to the analytical methods. This is not new stuff, statisticians have known about this for ages. There's a certain reason why what we currently do in the frequentist approach is more common because first of all, there is a lot of studies behind this, so this is a well-studied area about the certain distributions that you assume and also what you need to do and this is also what people sometimes forget right did you check for normal distributions in your sampling data before you apply a certain test so this is something that you need to be aware of anyway but the most important point at least for me is in the past like these resampling methods they were not so easy to do and the main idea of statisticians was that you don't need to do this because you didn't have any this was before the advent of computers right so you didn't have the possibility to just run these numbers time and again right that was kind of you had to do this by by hand back then and the idea with the distributions was that you didn't have to do this by hand so as i said like this is well known but of course times have changed right and this This reminds me of, I'm a bit older, so in the 90s what my math teacher told us time and again, right, you aren't always going to have a calculator in your pocket. So times have changed, right, and now we can approach it differently, of course. So the key takeaway from this presentation is basically use common sense and don't forget about the actual question that you're trying to answer. And if you're a software engineer, use your coding skills, and then you're basically done. I already mentioned it. It's just a starting point. There's a lot more about resampling methods and what I was presenting. I actually thought, and since I'm now in time, I thought this would be enough. If you're interested, I highly recommend this talk by Jake van der Plaas from PyCon 2016. It's rather very old, but it's still, like, pretty relevant. You will see that I took a lot of the ideas basically from this presentation. But I hope I did justice to this one. So please also look it up. There's also some more stuff in there. And then there's also like some books, like one that I actually also read is Statistics is Easy, where there's also some more explanation about the pros and the cons of these approaches. And with that, I'll thank you for your attention and have a nice trip back home and a nice weekend. Thanks a lot.

Speaker 2 [24:50]

Thank you, Thomas. I have to come here because there are quite some interesting questions. I think you should answer them.

Speaker 1 [24:57]

I hope I can.

Speaker 2 [24:59]

Given enough sample space, do you think statistical A-B testing always yields the best answer? Or, wait a second, no, it switched on. Or have we just left something to our gut feeling? Or we have something left to our gut feeling?

Speaker 1 [25:16]

I mean, the whole point about this presentation is, again, I wouldn't say like gut feeling, but in most cases, right, you need to take a decision. And this is also like what I need to do, like on a regular basis. And then if you do have some understanding and some statistical training, then you know about this. But the point is about these statistical methods. They should help you not to have like a gut feeling, but to better assess the risk of whatever you're doing. Because there is different risks and different opportunities with whatever you're doing. So there is added complexity maybe, but there is also an added chance of more revenue. And in my opinion, all these tools and the resampling is especially one where you're interacting with somebody who is not statistically trained, but who also wants to understand what you're doing. And I think this is also very important. You shouldn't just be saying, yeah, you don't get this. right, let me take the decision and just assume like everything below 0.05 is fine because this is also what I hate, right? And we all know this p-hacking and all these problems with peaking in A-B tests, so I didn't talk about this at all. But this is something where these methods just help us to get a better risk assessment or opportunity assessment of what we're doing.

Speaker 2 [26:35]

Thank you for the answer. I think the next one ties in a bit to that. And still stakeholders will go back to falsely interpreting P equals 2.5% as a probability of B being better than A. So why not use Bayesian statistics? Yeah.

Speaker 1 [26:50]

that's a good point and I think this is going back to the slide that I had right about the thing so I'm also like a huge fan of a Bayesian approach but then I would argue for somebody who basically gets in touch with Bayesian evaluations in the first place it's also not so easy at least in my experience so you also need to I mean for those of you who kind of have this training and for those of you who understand it it's very it might be very intuitive but in my experience it's not so much for people who just heard about it for the first time and they just think okay but how can I relate to this but again I'm not saying that Bayesian approach to A-B test evaluation is wrong quite the contrary I'm just saying like this is what usually what I get in touch with right and And if you use certain tools that are also on the market, then this is also what you come across.

Speaker 2 [27:51]

There are three more questions. Maybe we can do one more. I think it's going to be quick. How about non-normal distributions? Could you elaborate on bootstrapping from these and then testing on it?

Speaker 1 [28:05]

Yeah, I mean the point this is actually like the good thing about the resampling approach, right? So you're not assuming anything about this distribution. It doesn't need to be normal. I mean, I just put in the cases here just for the illustration purposes, but like resampling is especially kind of irrelevant if you don't have a Distribution that you can resort to and you just need to build it yourself and this is then mostly non-normal, right? It could be a bimodal distribution or it could be something else that you might need to kind of you can also analytically get there by combining certain distributions, but The point in here is like if you have certain Cases like also for the confidence interval since you were talking about bootstrapping if you want to do for linear regression Right then also like the point is if you have the error on the linear regression curve like how would you like which distribution would you assume for that one and And with resampling you basically can get there.

Speaker 2 [29:06]

Thank you so much. There were two more questions which we couldn't answer, but please feel free to find Thomas later. So with that, thank you, Thomas. Thanks a lot. Thank you as an audience for staying here Friday afternoon, last session. Well done.