When A/B testing isn’t an option: an introduction to quasi-experimental methods

Speaker 1 [00:03]

Hello, everybody. Good afternoon. My name is Inga, as it was mentioned. I will be giving a talk entitled, when A-B testing isn't an option, an introduction to quasi-experimental methods. Perhaps before we jump into the topic, I will just say a few words about myself. actually i will repeat uh so um i'm i currently work as a data scientist at olx in berlin but my background is in economics and econometrics so i studied economics and yeah i started with that basically and i'm still like at heart i'm an economist and you will see in the talk why I mention this. So, without further ado, let us start. And I would like to start by calling a quote by Leo Breiman. So, in 2001, he published his very famous paper about two cultures in statistical modelling. So, basically, Ryman said that there are two cultures in how statistical modelling is used to reach conclusions from the data. And one culture assumes that data is generated by some stochastic data model, while the other one is most focused on algorithmic models and treats data mechanisms as unknown. So that first culture is something that we would label as economic statistics and econometrics these days, or at the days of Bryman. And this other culture is more something that we would label machine learning these days. So actually, Breiman, he argued that 98% of statisticians and econometricians follow that first culture. This is not true anymore, because these two fields, they converge in, for example, in economics, AI methods, or they started to be incorporated into research. And also, like, in machine learning, there's some interest in causality and structural modelling. Also, like, the fact that we're having this talk today is an evidence for that. So yet about those two cultures, so what they're about. So data modelling culture, theory-driven and allows to identify quantities of interest, for example, treatment effects in causal relationships. On the other side, we have this algorithmic modeling culture, which is theory agnostic and purely data driven. So what it focuses on is prediction. So if you follow that algorithmic modeling culture or machine learning approach, The main goal is to get the Y right, right? So to have the proper prediction. And yeah, machine learning models, methods, they are set to work really well. If they can make an accurate prediction on a holdout set, this is all you need, right? So you have a holdout set, you have still good accuracy of your prediction, and that is it. So, this is like the main focus. In contrast to this data modeling culture, when prediction is not everything, so we rather want to understand how things are modeled. What are the forces and how each of those forces shapes the outcome variable, right? So, more about the process. Yeah, and I said that in the machine learning, like it's mostly predictive considerations are taken into account, while causality, equilibrium, or feedback effects are not. But one might argue that this is also changing, right? Because there's been so much interest in recent years about a model like explainability, interpretability. These things are related with that first culture. Or, for example, there's discussion about bias and fairness. So are our machine learning models fair? But this is something that is discussed post-fact. So once you have your prediction, you can analyze, okay, is my model fair or not? It's not something that it's accounted for, optimized for on the go. On the contrary to this, to this, say, statistics or econometrics way of doing. So, yeah. So, causal inference, so typical application of this first let's say culture is causal inference while for the other it's prediction problems. So let me motivate that distinction with some example. So we, yeah, so in economics and also like if you use common sense, when you think about prices and demand, you should come to a conclusion that when prices are increasing, then sales or demand is decreasing, right? So the higher prices, the lower demand or the lower sales. That's like, that should be natural, right? But this is not something that is always observed in the data. So in the data sometimes, so here is an example. In the scatterplot, you can see we have price and sales which are positively correlated. So in this example, once price increases, sales also increases. This is a bit counterintuitive, but this is, yeah, it's seen in many data sets, actually. To, like, better picture it or, yeah, to visualize it, we can think about that example in terms of, for example, demand for hotel rooms and prices for hotel rooms. So, when, yeah, when you're booking a hotel room, you're booking a hotel, right? you want to go on holidays, or, I don't know, on a wedding, or to a conference, right, actually, we are at a conference at the moment, so you see that, yeah, there's basically prices are high, so what's going on with availability? It turns out that hotels are full, so there's been a lot of sales. So prices are high and sales has also been high, right? Which, yeah. So if you have such data and then you try to approach this, let's say you try to understand, you want to understand the relationship between sales and price so you could try to fit a regression model or any other model and what most likely you would find out is that yeah, price is positively correlated with sales or If you increase the prices, sales will increase too, but yeah, it's, and so here you clearly see that fitted line is, it has a positive slope, right, but that model is wrong, and we know that, right, by economic theory and that common sense, so what is going on there? So what in fact happens is that we have this third force. So we have relationship in this diagram between price and sales. So price definitely impacts sales. But we have this other force, demand here, which impacts both price and sales. So when there's an increase in demand, both price goes up and sales goes up. And it causes basically the relationship between price and sales to be distorted. So with the data we had, which we saw on that scatterplot, we could not control in any way for that demand factor, which definitely impacted the picture. So, in terms of some nomenclature, the demand in this case would be a confounder, confounding factor, and then sales is our outcome variable, which we wanted to understand or predict, and then price would be explanatory variables. And, yeah, they are basically distorted. So, you may ask, okay, so then how do I answer that problem? How do I understand, if I want to understand how actually price is related to sales, or if I want to estimate demand for those hotels, what do I do? The answer is that you would need to basically change the price to generate the data set that could be used to understand the problem, you would need to change the price, but hold everything else fixed. So in the example which we saw two slides ago, there was some variation within price, but what we didn't see is that there were other factors that were changing as well. So here, in contrast in the setup, we want to change only the price and keep everything else constant. and this is something else and we didn't have that in the data so um how that could be how that how that problem could be approached you may ask so uh this is like widely known problem in in economics and economists has been looking into this for decades if not more right because it's like basic uh yeah demand model is the basic basic model and you learn it and you're like not even first semester, first perhaps month of your economics course. So what economists would do, they would build so-called structural models. So these are models, like you might think about them like as a set of equations, functions, which explicitly model relationships in the data. So they could, for example, account for how people, our subjects in the experiment, right, how they maximize their utility, for example, right? so they maximize their utility they have some preferences about how much they are going to pay for given goods like in this case hotel prices and yeah this is something that you put explicitly in such a structural model but they're not those models model for demand is not that complex but once you run into some more more complex scenario the model would also get way more would get joint right so like solving such a model it's not really easy sometimes and also before you build a model you need to have a theory right because you need to like represent the forces which you know what's what's happening with set of equations. So that theory, there might be some contradicting theories, for example. So how do you know which theory is right? It's sometimes difficult to say. So is there other solution, you may ask. And yes, there is other solution. So which seems perfect. So in order to estimate the demand curve, you could offer random prices to your clients. Right? So you're a business. You can do that. You offer them random prices, so basically you run total, yeah. The price is totally independent of all confounding factors, right, including the demand. So once you collect enough data, you will have a perfect estimate of of the demand curve. That's sounds amazing, but no. because like if you think about running such an experiment like it would be incredibly costly such a business would go bankrupt and like some right because there's no like optimization of any sort of prices so it's not really an ideal scenario but in theory could be done In general, experimentation is the gold standard for measuring the effect of an action, and so it's widely adopted and used, so yeah, let's have a look into that. So, especially in business, there's some Silicon Valley label for randomised controlled trials AB testing, right? I assume like most of you are all heard about this, so the so-called completely randomized designs and it's an experimentation technique in which subjects are randomly assigned to control and treatment groups. So I highlighted here the word randomly because it's key. So the assignment should be really the assignment of subjects who are treated should be really random and if we have random assignments we ensure that our treatment is not correlated with any third variables which basically both control and treatment groups they are they are affected with the same confining variables so we can see the difference result of like our treatment yeah so in this so called potential outcome framework we define average treatment effect as in this as here in the slide so average treatment effect is expected outcome given that there was a treatment minus expected outcome given that there was no treatment. So, yeah, just difference between these two expected values. Yeah, but that being said, even though A-B testing is such a great tool, it's not always possible to run, to schedule and to launch an experiment or to design an experiment. So there might be a number of factors that might not be possible. So the experiments can be costly, long lasting, unethical, or illegal. I'm sure you could name more reasons like that. So to give some example, you might think about the following problem, that you want to understand what is the effect of being fired, of somebody being fired on income of that person, on future income of that person, right? This is some question to which economists would be, in which economists would be really interested. But yeah, how could you answer that question, right? How being fired impacts your overall income? So you may have that idea that you could design some experiments. So basically, in order to control for all the factors, you would have to assign subjects to the treatment randomly, which means that you would randomly nominate people for being fired or not. And being fired, it's a big thing. Like, it has impact on, you know, lives of those people who are affected. So this sounds unethical, right? So perhaps it's not the best idea to run such an experiment. Also, there's another factor. So such an experiment would be long-lasting because in order to truly understand what's impact on somebody's impact, like maybe lifetime income or expected income, you would need to run it for years, right? So there are, like, multiple problems arising already, right? So sometimes it's not possible. There is also, like, another scenario in which underlying assumptions for identification cannot be met. So something I talked about before, like, randomization, right? So randomization, like random assignments of subjects to treatments, is really crucial. And, like, scenario in which it's easy to, or which is perceived, like, let's say, almost perfect for possibility of randomization is, like, if you have a website and you want to test some change on the website, right? so it seems like almost perfect because you can have like a large number of subjects which are treated or not yeah you can monitor them and so on and so on however even in that setup however even in that setup there are some there might be some potential problems so for example you might not be it might not be possible to totally separate users or not separate to actually identify a user that's using more than one channel right so you might have users that use some app and desktop version of your site and that That will bring some distortions, right? So basically the same person might land in two groups in control and treatment. So now depending on the size of that issue, you may have biased results of your A-B testing. Another problem which could arise is dependency between treated subjects. So this is also something that should be avoided, and it is assumed that treatment which is imposed on the treated group does not impact the control at all, so there should be basically no dependency between the groups. So given those problems with A-B testing and experimentation at all, so that was the question, is there any other way? Or can we approach the problem some other way? And the answer is yes, some problems you can. So here we are at, we have those quasi-experimental setups, or some call them near-experimental setups or frameworks. So basically those quasi-experiments, they are used to inspect past events, that's important, that resemble laboratory experiments. And they allowed to discover treatment effects when randomized effects were not possible for any reasons mentioned before, for example. Something important to know is that those quasi-experimental frameworks, in order to discover treatment effects, they require some small set of extra assumptions. And the assumptions will depend on the framework. We will go through a few of them, so you will see. So yeah, each one of them has some extra assumptions that has to be made. And also important term about an identification strategy. So how you approach a problem and what assumptions you should impose or what you should assume is called identification strategy. And this is something to think about when you have the problem, let's say, or the data, you should think, okay, given the problem, what should be my identification strategy? How the treatment was distributed or, yeah, this new experiment, sorry, this past event, how it happened, what should I follow? So yes, the first framework, the first quasi-experimental framework I would like to talk about, it is difference in difference framework. It is also known like as sometimes it's referred to as parallel trends design. So what's the idea here? So you need to have two groups still, like in regular AP testing, you need to have two groups of your subjects, control and treatment group. And it is important that the subjects are both, both control and treatment group, they are observed before and after the treatment. So this is key. Here in this example, you can see that we had a treatment on November 1st, so everything that is before November, that will be our pre-treatment period, and then everything on the right, that line, it is post-treatment period. And also we observed two groups which in this chart they are separated, so the orange one one is control and the blue one is, sorry, the orange one is treatment and the blue one is control. Right, so here something happened. So as you may see from like, or think about from the name, there's this parallel trends, this parallel trends term. What it means? It means that we need to, so yeah, so just coming to assumptions, identification strategy. So it means that the crucial assumption in this framework is that before the treatment, we expect that treatment and control, they behave in some similar way. And in the similar way, here it means that they share the same trends. So whenever one is, let's say, growing, the other one should also grow. Whenever one is declining, the other one should decline too. So they should really share the same trend, or even they should be moving in parallel. And here I fitted regression lines, so you can see, I think with just an eye, that in the pre-treatment period, those both groups, they share really parallel trend. And then another assumption which we make here is that now about the post-treatment period, is that if we identified two groups which share trend before the treatment period, We assume that they will also share the trend after the period. Of course for the treated group condition like assuming that the treatment would not have happened. So if there is no treatment we would expect and we would expect both groups to still share the same trend after the treatment date. So this is a key assumption. What is also important to mention, because here in this picture, it seems that the treatment is not supposed to affect the control group, but it seems that for the control group, nothing really changed in terms of the trend in the post-treatment, right? So yeah, this looks like single fitted regression line. But it's not, let's imagine that it's a bit different, so actually that here the line started to like move downwards, right, so this could happen. It's not necessarily, it would not necessarily be like effect of a treatment, but there could have been some other third factors like confining variables, for example, right, which like this demand, which we saw before, that would change the picture, basically, and yeah, so in that case, we assume that, okay, our behavior of control changed, but we also assume that in this hypothetical scenario, that if treatment didn't happen, then the treatment group would also change the slope because they follow, they share parallel trends, right? So, yeah, disclaimer. I think I haven't mentioned that one important property is that even though they must share the trend, they might be on different levels. So, level is not a problem. They just need to move in the same in the same direction so here in this in this chart I picture I put I put basically this hypothesis this this violence violent violet violet line it's that what hypothetically what would value of treatment group would be if it was not treated right so it shows basically the same and same the same trend as control and it's like yeah it would be one line so now identification of treatment effects it's basically graphically it's it's this different set right so what was observed minus what would be observed if treatment did not happen so yeah now you may think like how is the solve how to like you want to use that and how can you solve it so the diff and diff regression is given by like this this this term which you see in the slides right now so yeah expected value of a subject at is equal to alpha which denotes level of the control group, then beta D, which is the difference in the levels between treated and control group, and then beta T, which handles the trend, and then the most interesting one is interaction between the fact of being in the treatment group and the fact of being in the treatment period, in the post-treatment period or not. And really what we are interested in is that gamma coefficient. So this is our effect, this is our causal effect and interaction term. So I put some example so we can see how this looks like in real life, or actually, it's not real life, because that example is, yeah, it's generated with some random processes and assumptions and some noise. So yeah, so here's this visualization of how you prepare the data to estimate such a model. So we have this T and D variable, so T is whether it's post-treatment or pre-treatment or post-treatment period, and D denotes if a unit was treated or not in general. And here you can see some estimate, like output, so we have intercept, which is equal to 6.4, and yes, you can just look at the picture, so this is level of the control group, So, yeah, looks correct, I mean, it is correct, because I know how I generated data. But also, we have intercept for Dt in our treatment effect. So, 1.61 is our gamma, gamma coefficient, which we want to look into. Now, another quasi-experimental design, it's synthetic control. It was like, there was some buzz about it, about that method. It was even by Susan Nassi and Guido Inbens in one of their papers. It was mentioned that it's like the most important development in program evaluation in the last decade. So that's huge, right, to say something like that. So, the idea is quite simple, however beautiful. So, and I like to think about this as some kind of generalization of difference-in-differences framework. So, basically, in difference-in-differences, we had just one group, one control group that was supposed to resemble our treated units, our treated group. the idea is that, okay, maybe we don't have to use just one control, maybe we can have more control and somehow combine them in a way that they, in a better way, resemble our treatment group, right? So we don't have to limit ourselves to just one control. So, yeah, control, what it requires involves construction of a weighted combination of groups used in control. In other words, we want to create some forged control group from the controls which we have, and we want to create it in such a way that it will resemble our treated group as much as possible in the pre-treatment in the pre-treatment period right so um any basically any any weighted combination of the controls you have built in such a way that it will resemble um that it will resemble the treat group in the pre-treatment period so there's obviously, so the method or the idea is beautiful, but there's obviously like a question, like how do I build that synthetic control, like what's the approach, like there are some, like in the most basic setup, you just may try to like run some, let's say, you may try to regress basically your controls on the on the treatment group and then see if it works you can use some regularization if needed but yeah there are some other techniques but this is like out of the scope of the talk of this talk so here is like it's I show basically my treatment group and then my created control group so you can see that in the pretreatment my treatment and control they are almost identical so actually it's it's too good to be true the system yeah this is due to how I created the data in in real life this rarely happens it rarely happens because yeah if you have perfect fit right so So you would just assume that you have overfitting problem and then you will have your synthetic control in the post-treatment periods would not be really reliable, right? One thing also about this post-treatment effect, so the average treatment, the treatment effects are basically the difference between synthetic control and treatment. So this is how you estimate the treatment effect in that setup. In this case, the effect is really close to zero, so yeah, perfect example. One more framework which I wanted to discuss is regression discontinuity. So in this framework, that treatment is being decided based on some thresholds. So here I use example from book, from Matar's book, Business Data Science, and we have an example of a design where, yeah, where basically units with value of some feature which was before, which was above zero, were in one group, and then basically units which had a lower value of that feature were in the other group. So there are some assumptions being made around that setup, and that is basically about that individuals near the threshold, regardless of their position, they are considered equivalent. So basically, so here we have the data, so we assume that individuals like around here and here, they are very similar, right, and yeah, but basically they share the same features. So I didn't say that that feature, that variable which dictates the treatment is called running variable. We also have this continuity assumption. So basically it means that if we change this threshold a bit, then individuals switching the group would behave similarly to their new group right so it's it's an assumption about the fact that really treatment affects affects the behavior and yeah the estimated fact so in order to estimate effects you basically need to take a difference around difference between groups around that treated around the threshold, right? And there are some ways, methods to do that, but it's outside of the scope. And yeah, the estimated effects are basically the local average treatment effects, so they're only local. This is important. They should not be extrapolated to other values of threshold. So, it's the end of the talk. I just wanted to, I hope I gave you some overview of the quasi-equivalential methods. Like, I just would like to, as a summary, say that yeah, what I wanted to, like, tell to you is that remember that prediction and causality are not necessarily the same, and this cliché saying that correlation does not imply causation is actually, yeah, it's actually important, and yeah, remember to consider that. Thank you.

Speaker 2 [39:40]

Thank you very much, Inga, for shedding some light on what we data scientists can do when we don't have the opportunity to do A-B testing. So we received a couple of questions and I'm starting with the first one. So are quasi-experiments the same as so-called

Speaker 1 [39:57]

Natural experiments? Not necessarily. So natural experiments, of course, when there's a natural experiment, it might be analysed with one of the quasi-experimental frameworks. So, for example, for synthetic control, I can give some quick examples. There is this, actually we were in Berlin, but there was this case about like Germany being unified, right? So one could ask a question like, okay, what GDP of West Germany would be if not the unification, right? And this is some kind of a question which you can analyze with synthetic control, example. Actually, this is the example that synthetic control was introduced with.

Speaker 2 [40:58]

for clarifying this.

Speaker 1 [41:02]

next one

Speaker 2 [41:03]

How do you decide whether a diff-in-diff or a synthetic control group is more suitable for a given task?

Speaker 1 [41:09]

So, with diff and diff, it's simple because you need to have parallel trends. So, if you do not have parallel trends between your control and treatment, you should not use diff and diff estimator. As simple as that. So, you need to check for assumptions, basically. And for synthetic control, it's all about building situated combination of controls that will depict well-treated group in the pre-treatment period, so you can validate that.

Speaker 2 [41:47]

And a little bit adding to this, if diff and diff regression assumes that the post-treatment trend has to be parallel, how would one handle possible long-term effects of the treatment?

Speaker 1 [42:03]

Perhaps Yeah Like like maybe this diff and diff should not be used for long-term For really long-term effects. It's more like local estimation I would say but Yeah I'm not sure maybe if the person that asked wants to clarify or

Speaker 2 [42:32]

Yes, and the second one, I think it goes in a very similar direction, and I think you already answered it. So, diff and diff regression, we assume the relationship is linear.

Speaker 1 [42:44]

It's linear. Yeah, the trends are yes

Speaker 2 [42:53]

I'm not sure if this was the intent of the question, but in theory could one

Speaker 1 [43:00]

So, actually, like, when you think about how data looks like, like, right, so you may see some, like, for example, seasonality. It can handle seasonality. But so if you have, like, seasonal data, like, for example, weekly trends, right, so you analyze some feature which has weekly trend. which like if you look at if you like plot that variable then perhaps it will not be like linear as single line but yeah as long as the two series are moving in the same direction it can handle that

Speaker 2 [43:45]

Okay, thank you so much Yeah, maybe one last question So is there some way to mimic the AB testing if there are no events in the past resembling the treatment?

Speaker 1 [44:01]

Tricky one.

Speaker 2 [44:06]

Let's assume you want to A-B test something, but...

Speaker 1 [44:18]

an A-B test if I don't have like I don't know like how can I analyze something if I don't have data

Speaker 2 [44:33]

Time is up. I have one quick announcement So please if you signed up for the evening social activity Don't get your a batch like right before the event that it will be very cute and crowded But try to get it sometime now maybe and have an hour or so So there won't be such a long queue Yeah with that. Thanks a lot in there for shedding and light on a