Leveraging the advantages of Bayesian Methods to build a data science product using PyMC3

Speaker 1 [00:03]

thank you for the warm introduction um yeah i'm corbinian i work for dalia research um and welcome to my session um so a bit of a disclaimer for my talk and so you're in the pi mc session but i will mainly talk about um how to use bayesian methods to build a data science product we used um pi mc for that but my talk will not contain like a single line of code but we have another talk afterwards from cory and she will actually talk you through the more technical details on how you basically implement the whole thing in the end um yes so let's get started um so a little bit of a background um on our product and so i'm working for a company called dalia research and we are market research company um so we are around like 70 people and what we do we run a platform that distributes microservice across the globe so we operate in around 100 countries and we do like five million interviews a month um so one of our main products is um obviously if you are in market research um you do brand tracking for um small and medium and medium scale enterprise enterprises um and yes so um this talk um will mainly be about how we built that product so how we basically built this next generation brand tracker and why we decided to use Bayesian methods for that and what is the advantage we basically or like the value we got by using Bayesian methods and then in the end I will talk a bit about how PyMC basically behaves in production so yeah if you're interested in that stick around so the reality is in market research that you have like hundreds of consumer startups and scale-ups and everyone is investing into brand so if you're right in the um u-burn in berlin you see like basically advertisement everywhere and the big problem those companies have is that they are rather small so if you're a startup you don't have like a big insights team and you have a marketing team and they spend a lot of money on marketing, but you actually never really know does this actually have a positive effect on your brand. I mean, in the long run, you know it because if your sales go up, you know that you had a positive effect on your brand at some point. But it's really hard to kind of tie the spending on marketing campaigns to actually positive effects of your brand. So as a young startup, we decided to go into that market and we launched in 2017 we launched a product called brand tracker um and what brand tracker basically the main value proposition compared to um other markets so the traditional market research was we are faster we are cheaper and we have a fancy dashboard so um in traditionally market research you get you pay a half a million dollars and then you get like a spreadsheet in the end or like a pdf presentation from um yeah some some senior accepted some senior person um yeah so if you look at the dashboard it has like a lot of graphs um we were not really sure if our clients actually use those graphs but it looks good um and if you look in the side it has a filter so you can basically so you have you track your kpis for example like brand buzz or brand awareness and then on the side you can switch female versus male old people versus young people so you have like this four switches which basically means that if you're a brand manager you look at the dashboard you switch those two things in one minute and then you basically leave and then in the next quarter you get another one and you see okay if you saw like last quarter okay our brand awareness was a 10 percent and it's very unlikely that it moves a lot because like young people are pretty big group and female and male are a pretty big group in a population so three months later you pay 20k for our brand tracker and we still tell you okay your brand awareness is 10 percent so this was really uh i mean our clients liked it mainly because it was cheap and it was fast but um they couldn't really get like much out of that product so what we found out pretty fast is that um brand managers always interest in niche audiences so they wanna so they don't talk like basically with their marketing campaigns they don't target the whole population but they focus on specific target audiences so for example if you are app that develops um if you're a company um that develops an app for um for parking your car um you are obviously interested in people who use apps and have a car um or for example if you run a lot of marketing campaigns on like twitter or linkedin you are interested in what's the effect of on those people who actually use those channels so why is that a challenge so assume that you do like a survey with 3 000 people and you want to target um young females who use twitter and let's say 1500 in your sample are female thousand are twitter users and 600 are young people and if you take a combination of all of those two you may end up with a very small sample so if you do the traditional market research approach and say like okay what is this subgroup in my population thinking and let's say seven out of twenty say yes i know your brand then your estimate would be 35 and this is how your confidence bonds look like so um what we figured out is that okay if we want to do that we need to fundamentally rethink the whole thing because we can't just take like old methodology from traditional brand tracking and apply it and just say okay we make it cheaper faster and it looks more fancy so we were experimenting with a method that's called multi-level regression and post-stratification um short mrp and mrp is mostly used by quantitative political scientists in the u.s um for election forecasting or other kind of polling and how does mrp work or how is it different from traditional um so-called quota sampling so you still have the same sample but what it does it doesn't basically it's not calculating an average out of the small group but it's directly modeling the assumption that all of those characteristics are independent so you i mean you all know or have kind of like a machine learning background maybe so you just basically fit a model and all your parameters are assumed to be independent so that means in the end that if you don't do an average but you do a little bit like you're a little bit smarter and use like a model um you use a lot more information because you use basically information from all the 1500 females 1000 twitter users and 600 young people to estimate that small group in the population so this is an idea that was already introduced in the 90s by gelman and little probably many of you know andrew gelman he We wrote like every second book about Bayesian methods. Yeah, I mean, it's not like a novel thing, but Andrew Gelman likes a lot Bayesian hierarchical models. So, of course, his original paper had Bayesian hierarchical models as this model component. But we as a startup, we wanted to build an MVP, so minimal viable product. That means that you always start small. You don't build like this huge model or like complex thing. you always want to start as like small as possible or easy as possible so what we did is basically we just use logistic regression and you can use whatever package you want you use just like scikit-learn or you can just code it down yourself and it worked so brand tracker was dead our marketing team rebranded so now we are latana everything is yellow um and yeah so um the main value proposition of that product at the time was that you can really zoom into this like niche audiences so for example if you're a brand manager and you have this target audience victoria who is young a female high income high educated living in top three cities and um is a music streaming user so create your audience and you get like yeah a more stable estimate if it's as if you would just like basically look for specifically those people in your in your sample so that means in the end the product is for our clients just a lot easier to use because you can click a lot more you can create your audiences you have a lot of things to track um and so if there's also like things changing over time because the more you basically zoom in into niche audiences the more likely it is that you actually see those those small changes um and yeah that was like how the product looked at that time um so why should we bother with patient methods so we had a product it worked and it was really easy to code and to maintain so why should we should we get more complex so the reality is that even if you look at only one characteristic you sometimes get too few people in your in your example and the second thing is that you basically have no estimate of uncertainty so if you have like a campaign and you jump your kpi jumps from five to eight percent you have basically no idea okay how likely is that change was it really a change or is it like due to some noise in my data and so there is no such thing as that so the first problem you can solve with learning from prior information we do wave per wave per wave you always do 3 000 samples 3 000 samples 3 000 samples so every time you basically build your model from scratch you just use the information from the past on your parameters as prior information then just update it with the new data and that gives you a lot more stability even if you have like really really small groups or like very few data on some of the characteristics the second thing is um yeah since you use patient methods you and you always work on distributions you get this quantification of uncertainty for free um so who of you knows blinkist okay niche audience so 7.5 percent of people in germany know blinkist so who is blinkist they're an app um that summarizes key ideas from best-selling non-fiction books in 15 minutes and they're one of our clients and we use their data to basically test this approach how we basically so how we improve our estimates with using prior information so i have some graphs for you they're a little bit confusing but i will explain them um so the dotted line you see this is the population estimate of 7.5 percent and then the y-axis is low educated people so basically what percentage of low educated people in germany know this brand and what we did we had a sample of 1500 people and we bootstrapped from that sample so we created smaller samples and each time we fitted a model and we did that two times with prior information without prior information and what you see is that if you have like really low sample sizes what your model will do if you don't use prior information it will basically just shrink so you have to space and shrink it you shrink your parameter to zero so if there is no significant information in your sample because the sample sizes are so low it will just basically say okay low educated people are the same than the overall population whereas if you use prior information from the past it will basically start off with the information from the past which is for low educated people somehow use the blinkist app which is a reading app so a lot less of them notice and if you go on and sample up to basically converge at some point so the the benefit you get from prior information gets less and less over time and you can look at like several parameters for example people who say they don't want to be productive in their free time the same story we actually have a lot of those people in our sample so that's why it drops quite drastically um so yeah so how does this help so i talked about if there's like really few information and this is classically the case with elderly people 56 to 65 year olds because we do mobile surveys and so we have less elderly people filling out our surveys and out of those 1500 people you only have around 10 of 150 people who are old people and it's a little bit too less information to the model to actually find like a difference but if you use prior information you've kind of offset your model from like with your prior information then this information you kind of gathered in this wave is enough to kind of find find a difference here so learning from prior information really helped us this is the first thing the second thing is this uncertainty quantification so imagine you're doing um two surveys so you have a campaign in november you do a sample of 2000 in october and in december and you have a jump from five to eight percent so as i said you don't have any idea if this jump is true when you use bayesian methods you get the full posterior of your estimates so you get the thing for free so what you can do then you can just look basically at the probability mass so the overlap and check which areas are not overlapping and then you can create this beautiful measure here which is nice to read for marketing people you say if there's an overlap for more than 50 to 60 fairly good 61 to 80 good and you did a really good job if you have 80 to 100 so how does this look like in the dashboard so um you have for example a kpi brand awareness and then you look at like a niche audience here and you have brand awareness of 70 and from march to june you have an uplift of two percent but since the bounds are pretty wide it's like load load change probability really so if you have an uplift here details seem to be like a little bit more narrow you have like high probability there was actually a change so it doesn't actually always um relates to the magnitude of change you have it also relates to kind of like how certain you are okay so as i said we used pi mc3 in production um to build our model and what i just want to quickly say is that i had the feeling that um like many data scientists are uh familiar with like traditional things like scikit-learn everyone can basically run like a whatever random forest um and kind of reason about it but whenever it comes to bayesian method it requires a little bit more theoretical understanding so that why it's kind of like sometimes keeping people from trying those methods um but i would want to really encourage you um if you think it can add like an additional value of like uh additional value of um yeah additional value to your product then um yeah try those methods and in the end it's it's implemented in in in a package so it's kind of like ready to go and it doesn't it's not really different from using like models and scikit so you basically have for our case it's data input so it's our surveys and then you have you pack your model into like a web framework and then you deploy it and then in the end you have some like output thing where you show the uh show the data to the client and so in i mean in the end it's not really like different from using like other um other packages okay so as a summary um so quantify and the probability of change in brain kpis for that really this measure of So uncertainty in Bayesian methods really helped us. Then using prior information was a complete game changer because that basically means that we can go even more niche. So some clients really want to track really, really specific stuff in specific regions that's otherwise impossible. And the last thing is that PyMC is no magic. So try it out. Deploying is not that more complicated than using other frameworks. Yeah, I would really encourage you to look at it, okay. Thanks Thank you very much. Do we have questions? I'll bring the mic

Speaker 2 [17:51]

Hey, thank you for the nice presentation.

Speaker 1 [17:53]

I

Speaker 2 [17:54]

I just wanted to ask if you could share a little bit more about your experiences with using PIME-C3 in production. If you said in the past, I've seen that the API wasn't always totally stable, that you always had to convince yourself about convergence issues, and that it actually did converge and did converge to meaningful posterior distribution and so on. Can you share a little bit about your experiences?

Speaker 1 [18:19]

Yeah, so those are really good points. So there are basically two things you need to decide. So if you do something in production, it needs to be stable and it kind of needs to be unsupervised in a way. So you can't, I mean, if you have a dashboard updating live, you can't always look at the Bayesian diagnostics and check if your model converged or not. So what we did, we pretty fast realized that doing the full Bayesian inference didn't work for us because it's just not stable. I mean, you have, you sometimes, you need to imagine like you have like a survey and you track 400 KPIs in 10 countries at the same time. That means you basically, you build 400 hierarchical models, which means computation wise, it's just crazy to do like a Hamiltonian Monte Carlo. And also in the end, it's sometimes, I mean, sometimes your algorithm just like kind of gets stuck in those really, really tricky areas. and so if you have like whatever you do like four times thousand draws it may basically just end when you're like in an area that is really like unresolved and so your estimates look really weird sometimes so what we did is um we looked at a variational inference and tested that against full hamiltonian and we figured out it worked really well for us and it's actually reasoning quite well with um what we expect the model to do so um so what it basically does it's like it's it's kind of assuming that you have like a normal distribution and just finds the the best fit in the end it's always a decision how your product look like if you're really going for the 0.5 probability and you really want to be exact so you work in finance um you whatever you need to calculate some risks that something happens then go for full Bayesian thing but if you do survey research which is in general a little bit noisy I wouldn't go with full Bayesian thing also I mean you you can't even do it because it's I mean if you fit this this kind of models we are fitting in like computing one model can take up to seven days so that's not not feasible in a way and then the second thing is that um pi mc3 is running on tiano and tiano is not supported anymore um yeah i discovered some some things that were a little bit tricky to resolve so i hope pi mc4 which is done on tensorflow um has a bit of a better like community behind it um yeah so So there are some things, but I guess always when you have like a package, I mean, if you run TensorFlow or if you like scikit-learn, you also always rely on some dependencies. So in the end, I mean, yeah, there's always some things, but they're not unresolvable in a way. So once you wrote it down, it runs and it gives good results. Can you share with us more remarks regarding how do you set and define your priors, especially

Speaker 2 [21:30]

priors, especially for small groups.

Speaker 1 [21:32]

So, I mean, what we do is basically we assume that our parameters are normal, so Gaussian with zero mean, and then you have this kind of like hierarchical hyper-prior that kind of relates your hierarchies. It's pretty similar to basically we just took the stuff that Gelman suggested in his papers. um yeah and then when you go on um to a second wave and third wave you basically use as a mean you use the full posterior of your past wave so it basically means you when you like you do one wave you assume that you have a random variable which has some kind of distribution if you go on into the second wave you assume that it's a process in a way um yeah so that's this kind of the trick Any other questions?

Speaker 2 [22:36]

Thank you for your talk. Did you also evaluate other packages before instead of PyMC, like this AdWord and Pyro?

Speaker 1 [22:45]

Yeah, so if you look at Bayesian packages in Python, there's basically one other big thing that is PyStan. I don't say that it's not good. We just choose PyMC because it's a little bit more PyTonic, so you have more control over your types and things you do. PyStan is more like a Python wrapper around the compiler. so what you so you basically you you write down your model and then the model you make a string out of this model and then you give it to some function and then basically go somewhere it compiles and gives you the results back so for me this idea to write down my model and then make a string out of it wasn't really like something that convinced me to use it in production and then I also have the feeling that Stan is more backed by like an academic community so whenever you google problems it's really really I mean they really really really go into detail and those things and I just had the feeling that like PyMC was a better fit for production so yeah but there's also another one it's called Edward and I haven't really looked at that one but I mean the biggest support definitely have PyMC and PyStan okay last question thanks for your talk so you you showed some nice visualizations how you display the results to your customers could you comment on correlations between the ins or the correlations between the parameters that you get as a result of the Bayesian methods and maybe how how you would present them to your customers so yeah so if you do this kind of modeling at scale that we do you don't really have time to look at every single parameter or correlations in the end but you kind of indirectly handle those because if you have strong correlation between two parameters so your algorithm in the end will probably not really know where the signal comes from and it will just increase the confidence bounds so if you build a model with a lot of correlation you will see that in your confidence bounds in the end and then since you see it in a confidence bounds you will also see it in the change probabilities so in the end you basically get punished by not building like a good model so what we try to do at least is that we think that some things are heavily correlated for example education and income those kind of things you can kind of directly model that but um so if you have this general thing it needs to fit like basically all varieties of survey data um i wouldn't go more into like kind of looking at like interactions and model that directly because it also means that you need more sample in the end Okay, let's thank Cobinian again.