Productionizing your ML code seamlessly

Speaker 1 [00:02]

Hi, so today we're going to talk about productionizing your machine learning code seamlessly, but first a word from our sponsors. Hi, I work for a company called Yelp and they were nice enough for sponsoring the conference and me coming to this conference. Yelp is not that well-known in Europe, unfortunately, but it's a great website that allows you to find great local businesses around you. What does it look like? What is this? It's like, probably if you type it, Especially here in Germany and here in Karlsruhe that should really really pretty much look like that and it allows you to find restaurants businesses and this microphone is not well placed and almost everything else you need and want to find around you and everyone What everyone has been saying about it. How good is it and that way you will never go to a bad restaurant ever again promised In terms of scale what kind of company are we well actually we're a fairly big company The people on Yelp have written an insane amount of reviews that is written behind me and I will not try to pronounce. We have a lot of monthly users, a lot of developers, a lot of services, a lot of features, and what this means is we have a lot of data that is generated by all of these systems. Which is why I'm here to talk to you today about, hey, productionizing machine learning systems because we have them, we do them, and I think many, many people might have heard before the time, ah, yeah, let's just take it to production. It's like, yes, sure, and what does it mean? So what's on the agenda today is trying to understand what does it mean, and what is it to take a machine learning model in production, and then some of the good practices that allow you to do that with much less pain than it can be otherwise. Great. we're talking about machine learning, we're talking about machine learning model, machine learning code everything starts in a notebook, usually let's be very very honest and what is this notebook? Well usually we don't write notebooks just for the sake of writing it, unless you're a student and that's your assignment you have an objective that you're trying to reach when you write machine learning code you're slicing data, you have something you want to learn, something you want to action about You might want to be trying to predict, hey, is there a fault in the pipeline? Is the stock going to go up? Am I going to get rich? No. And whatever it is, you have a goal, something that needs to be achieved. And you do this. Usually you gather some data, you have labels, you have nots, you try to figure it out. You can write a very simple notebook, like the one which is passing way too fast here, don't try to read it. Or something absolutely crazy with complicated data preparation. and you do all of the work and at the end of it hey you did it it works like in your notebook you run it and you think you cracked the problem you found a way to put to yield a result and good job that's when your problem starts usually because it's it's everything is nice and easy in your notebook but real life sometimes doesn't just work you have new data coming in that you might need to be able to handle. Data might come in regularly. You're not always sure how it works. Is your model that you just did, is it going to continue working in time? And how often should I train? How often should I predict? It can be hard. It can be really hard. Also, you had an objective at the beginning. And is that objective met once you have finished your notebook? Well, not always, unless you're selling the notebook. But that's another question. And there was this graph from Google at NIPS, which I really, really like. And in a way, once you have done the notebook, you're there. And that's when the next step starts. And what I'm here to help you is to say it's OK. It's going to be OK. You still did a lot of the work. So what does it mean going from the little box to kind of integrate with the rest. And that's what I'm going to talk about. Productionizing a machine learning model is like productionizing any code. You go from something that works really, really well when you look at it to something that you're not going to look at ever. And if something goes wrong, you need to know about it. And that's the same problem as I wrote this nice Python script, but now millions of users are using it, and I have no clue in what kind of crazy ways they are using it. And you have a lot of documentation and good practices that you can probably reuse from general practice software, though I will give a few which are more relevant right now. This presentation is not going to be too much about tooling. To be really honest, I'm not going to say use Spark. It's the best thing ever. Youth Panda. I mean, we have all the conferences that do that much better than I will. The goal here is more to give a general idea of what the process is and how you can just have a mindset to slice and dice the problem into smaller, easier to understand steps. And talking about taking something which is bigger and more general into smaller steps, let's kind of agree on a fairly standard view of what a machine learning model is. It's here. The green is better on my screen, unfortunately. So you have different steps that we'll just run through quickly. All of the steps are probably you have them in your notebook, because I think if you write any Escalon pipeline, it's going to be roughly the steps. So you have data sources that you maybe sample, maybe not. You calculate features that are interesting for your model from them. Then you train your model. You evaluate that the training went well, otherwise you might loop a bit. And in the end, you know how well the training went overhaul. And you have a model. Yay, training. Once you have this, you need to predict, which is roughly the same operations, which is you have data, you sample, you calculate the features, you use your model to get predictions, and now you have predictions. And if your predictions or anything, I will call them predictions because I like the word, it's easy to pronounce, but it can be many, many other things. You might want to use them to product or answer your final results. As I mentioned, all of this is separated steps, which will help you do things like unit test, because you have already logical things and logical units that are easy to understand and recognize. Great. OK, everyone is sleeping. Maybe I should make more jokes. The data sources, actually, it's the part I'm not going to talk too much about. It can be almost anything, and the world is full of weird data sources. It can be S3, Kafka Topic Database, something completely exotic. I do not know. What we're going to assume for today is that you have new data coming in from time to time. And how this new data is created, whether it's user clicking or a robot exploring the moon and sending magnificent pictures, I do not know. You might have noticed that you can have other ways of thinking about a pipeline, especially if you need to do the prediction online. I will take something simple, because many of the things can be reused. Great. So now, training. And the main question is, when you need to go with a production model, take a model into production, is how often Am I going to train once a day, once a month, once a week, never, manually? I don't know, but you should. And that's the first question. And it depends on your data very, very often. It's something you need to test for, account for. You need to know if it makes sense or not. And the second part is rerun strategies. And what this means is things fail. It's life, it's software, things always end up failing. But what you need to think about when you put your system in production is, before you do it, what happens when it fails? Because you want to know beforehand, to know it's like, oh, the model didn't train today. I can just reuse the one previous version. Maybe it's fine, maybe it's not. And maybe you should be rerunning. Maybe you shouldn't. Is there a problem with the data? No new data coming in. Do I still do a retrain? Do I don't? I don't know. You should. And the last part is scale. And that's something which can beat you quite quickly, which is if the data you did your development with initially is not in the same scale as the data you will want to run it on a regular basis, you are going to have delicious problems. personal experience was an algorithm which was working with the holes. The feature extraction was done with database. And so it was just a gigantic SQL query. And it was working well until the amount of data became 100 times this and just didn't stop working. And you had to rewrite thousands and thousands of lines of SQL query into Spark. And that made me very, very, very sad. So if you think about what scale of things you're doing just a bit before, you can save yourself a lot, lot of time. And it's always about knowing what the right tools are. I'm asking a lot of questions and telling you you should get the answers. I'm really, really sorry about this. One step should retain your attention a bit more than the other, and it's evaluation of the model. I think we kind of all know what it means, this very nice thing that appears at the end and telling us how well training went. But we need to spend a bit more time thinking about this, especially if you really want to do something with your models. And the first is, does the evaluation metric reflect how the thing is used in production? Which is, you have many, many ways of evaluating how well your model performs, but you should always remember to pick one, not that they give good results, but that actually represents what you're trying to do. And what this means is, all the default functions are really good, they have great mathematical properties, they are interpretable, and when you put them in papers, they can be compared with one another. Yet again, sometimes some more, I will say, stupid metric, which don't make a lot of sense from a scientific perspective, or have a lot of problems, can actually just answer the question you're doing in a much closer manner. For example, area under, if you have classification problem, And you're just trying to identify. And you kind of, I will mention this again a bit later, but you did something a bit weird. And you do, every other rock curve kind of gives you a general performance of your algorithm. But in our case, just knowing if the first result of a whole ranking was good was what we were actually using this thing for. So it was better, which means that some parameters or hyperparameters tuning needed to be done very differently. The last part is think about feedback loops, and what this means is, once you evaluate your model, you can evaluate it against itself, or against its previous selves, and it's a good way for you to know if something goes wrong. And once you productionize your model, you should also know that the new data which is coming in might be influenced in some way by what the model has been predicted and influenced on the data before. And your model just can end up looping on itself, predicting what it has already predicted before. How can you guard yourself from that? Doing feature importance and keeping a copy of this and checking that, hey, things are always the same. Or checking having an extra data set that you always test against, just to be sure things are sane and nothing too crazy is happening. Everything is within expectations. Prediction, beware. I think I have three very, very novel ideas here, which is, hey, how often do you need to run your predictions? What happens when you can't generate your prediction? Is it like a critical failure? Do you need to page someone in the middle of the night saying, those things are not predicted. It's red code, red code. Everyone needs to wake up. OK, no. No one is waking up. I'm trying. Sorry. Or, again, scale, because maybe the scale at which you need to predict is even bigger or faster or stronger as what was happening with your training algorithm. You need to take care of that. I'm going to again zoom in into one part, which I think is a bit more important. It's putting your prediction into your product. Until now, I've just told you everything you should be doing, and I never told you in which order. Actually, that's probably what you can do first, because you have already the result from your notebook, and you can test them against reality. And what's important with this is that it allows you to check that you were right, which is when you had the intuition and the result that your model was predicting something which was realistic, test it against real life. Go for it. need to confront your hypothesis to a reality, you might also realize that, hey, there was a bias in my data, I need to go back to round it up. It will also make you realize how hard it can be sometimes to move the prediction and use them in practice. It's not always trivial. And if you start with this, you will get a sooner result, which means you can sooner convince people that you should invest all of the time to build the rest of the infrastructure. The last part I have put here is to beware about predicting what will happen anyway. And that's something which is very important. And that's why when you think about how am I moving the metrics the right way, you should really, really think about it. And it's not always easy. And you might need several times. I work for a team which is called BizGrowth. And our goal is to acquire a certain type of users which we can sell advertisement to, which is basically the core of our companies. These specific users really can be found among the regular amount of people which are just browsing for restaurants and food and are just hungry. So our goal was to detect them, show them a pop-up saying, hey, you, we know who you are. Please try to create an account. And if they create an account, it's a win for us. And otherwise, well, too bad. And so the metrics we had was, hey, is this model going to be successful if the user basically sees this pop-up, click it, and creates an account? And what we were doing before is, what are we using as the training data? Well, we are using the users that we know are already good and are, for a label, it's like, hey, they actually created an account. They were going to create an account. And we take all of this data. And so the model was really, really good at predicting which user would create an account. And so we deployed it. And the results were great. It's like 50% of the users that have created these new accounts, they went through the algorithm. Everything is great. We should so go to production with this. It's awesome. Except no. When we look at the rest of the numbers, it's like nothing was going up. And so can someone guess what happened? I'm going to try to wake up the audience even more. I tried. What happened is basically the algorithm was predicting what was happening, which is, whether we show them a pop-up or not, these users were going to create an account. So we were just saying, this is going to happen, oh look, it happened. And yeah, and that was not the goal. The goal was still to create more things, so we had to go back and change how things were measured, and quite a few things. Great. Enough big theoretical talk. Some tips and tricks. I think, again, ML code is still just code. Use every good practice that you might have heard from any other Python microservice talk, which is, hey, use containers so you know how much memory you're using and you know it. Use virtual environments. The other one advice is debugging and engineering time is very, very expensive. And on the other hand, storage costs are really, really low. So the more you save, the more you log, the more you keep trace of, the easier your life becomes when things don't go right. And you can add TTL on S3, for example. Using production technology from the get-go something I've mentioned before, which is if you build for technology for the scale you intend to use them in the end directly, you save time. The counterpart for this is you need to make your data science and all the people the big tools available from the get-go. Some people say it costs more money, but from the whole length of a project, it doesn't. It's always easier to have the right tool which is going to really solve the problem directly. Yeah, the last part is kind of the obvious, which is don't reinvent the wheel. Probably you're working some, unless you're alone and you're working in a vacuum, nothing is in a vacuum. Use the libraries that already exist. If you're in a company, you might have software that is there to ensure that some regular things are scheduled. So when you need to rerun your training, You don't need to reinvent the wheel. You can reuse all this people's work and infrastructure, which means that they get paged when it gets down, and not you. Great. Just a clarification point. When I put these two things on two different sides of the algorithm, I didn't mean it. It should be the same. Don't duplicate code. Don't have something different. Have your training and your prediction algorithm be the same. And I'm sorry, not algorithm, but obviously the feature extractions. If you have some difference, or they're implemented in different language, but I assure you they are the same logic, that's how you get bugs. And these kind of bugs are really, really hard to understand, see, and it's all weird and you don't get it. It can go from something as silly as how non's value are handled and transformed to zero by one language and not the other. So it's just easier to have everyone on the same page. Unit test is great, especially for feature extractions. The other thing I would recommend is use tools like Hypothesis, as these functions basically need to work reliably, but you do not know if your data is going to always be the way you expect it to. It's better to be ready for surprises. So try to send the right tests, and try to have no values popping here and there, see how well they are handled, and how poorly they are, or how poorly. Try to think ahead of what everything goes wrong and what everything you think is true and might not stay. Again, giant SQL queries are evil. Try to have as much code as possible. Try to not take too many shortcuts. We have SQL alchemy. If you like to write queries and you interact with databases, there are many ways to do this cleanly. Spark too. Evaluating models. this. Yeah, actually I mentioned that before. I will just save myself a few minutes, because I'm running late. The other one is, log everything. It's nice. Among the things that should be logged is like, how long does it take? Because that's how you can realize that, oh, maybe you start ingesting a lot more data than you thought you were. And, or, hey, I have a memory leak here, and it only hits once I hit a certain number. So the more information you log about your system, about what they are doing, about how many rows they are processing, and how long it takes them, the easier it is to debug problems and to also realize what is going wrong when things do go wrong. And persist all of your logs. All the advice are kind of one whole. They all go together. This one is another way to save a lot of time, and it's around versions. And it's really nice, because usually you have the right thing, and you think it's perfect. And a few days later, you have another great idea. But you already put this in production. And then you have things which are not consistent. And you don't always remember how many versions of the feature extraction did you change, how slightly different they are. If you have versions, if you version all of the steps, you know how the prediction was generated exactly. You know that it was made with feature extraction in this version, that the model was. And when I say version, I really mean something. It can really be just a string with a name. But it means that if you go from, if you add new features, well, you will have the old features, which will have the old name, and the prediction will be generated with the old one, and you will know. And it's always about knowing, about data traceability. And all of this allows you to also try several models at the same time in production. Be sure of what is happening. It's always the less you look at it, the more you need information to look. Once things go wrong, you need all the information you can to look at. And sometimes it can be really hard. The more you do of this, the less problem you have. Another good thing to add to your version is the date. So you know if it was generated with a model on this date, on this other date, and which ones are working well, which ones are not working well at all. This allows you to really optimize and monitor your system in time. Talking about monitoring, how do you monitor the pipeline? Well, there is not really a mystery. It's all about setting alerts. Tracking the number of prediction maids and or data row ingested in the beginning is very easy. If you do not have a system to do alerts, it's very quite easy to send emails with Python and receiving an email in your mailbox is a good form of first level alerts. You don't have to go for Sensu. If you have something better, use it. Again, use the tools which are already there. If you work for a web company, it's fairly sure there is an alerting system in place. Keeping track of timing, alerts on errors, it's a bunch of sound advice that you can find in any good deployment runbook of sorts. Which leads me to the last point, runbooks. You made a bunch of assumptions, and you think that things were in certain ways. Some of them you tested against. The others, it's very nice to write them down because your system can run for one year, two years, and then it starts failing. And you're like, I wrote this? And when you can actually refer to your past self for all the explanations you hope you still remember, it's great. And if it's not you and it's someone else, it's even better. You will bless the people that have written all the information you need to understand what's happening. In a way, and that's probably where I will stop, which is putting ML in production is not magic at all. It's putting code in production. So the main thing which goes with making things in production is seeing your hypothesis never stay true in time, or never as much as you would want to. So you should think about things that can change, and how you're going to handle it Much more than, everything works, it's fine, it's done. ML code is code. What's great with conferences, especially like this one or your Python, is you can also go and see what the people do in other domains, and especially production deployment, microservices, all of these. And you can learn a lot, because it's still just code. There is a lot of practices. There is years and years of books of good practices that can be reused and just slightly transformed. And the last part is verify your assumptions. whether it be for your code or for your inside the code or about your data or about how you think your data and your models are going to influence reality. Trust, but verify. We are hiring, if you like these talks or if you want to work on interesting data projects, we will be hiring for San Francisco, Hamburg, and London. We have a booth down there, and you can come. I thank you all for your attention. and attendancy, and I will be taking in questions. Thank you very much. Any questions for ? Yes. Once that first development is done, you've got your container ready and everything. Now somebody has to do the production and do the monitoring and everything else. Do you think it should be you? It should be some people on your team? Obviously, it should be .. So the question is, who does it? If I can sum it that way. I think that depends a lot on how many people are available to do this. If you, I think the worst case scenario is you're alone and you do everything. So that's one answer. At Yelp, we have actually different roles. So you have a data scientist, which to produce this initial work of thinking and exploration, and it's just going to give you the right parts. Then you have a data engineer, sorry, a biz mining, no, data mining, sorry, engineer, which is going to do the putting it in production, writing the runbooks. And then you have a third person, which basically, if the runbooks have been written the right way, can handle it. In general, it's better if it's one person, because they have the same context. But if you're good at it, having runbooks, and if you have a procedure for writing good runbooks and you have good practices, like the one which I mentioned here, anyone can answer an alert if they have all the information needed to be able to know what went wrong. Or at least to be able to say, I think the problem went wrong in this specific step, which means I will call an engineer which is specialized in this to help me. I think it kind of depends on how to kind of wrap up these answers that have been running out for way too long. If you follow the advice which are here about writing runbooks, writing documentation, and have things which are separated and monitored, anyone can actually do the monitoring part and answer the alerts when they come. If you don't, well, it's on you. Yes, there is a question all the way in the back. So the question is, what happened to the poor notebook with everything is gone? I didn't really mention it. I should have. It's better to have everything in code because for simple reasons that it's... Have you ever written a unit test for a notebook? I didn't, and for good reason. It's really hard. It's also easier for... Once the system is in production, you don't need all the visual interface, all of this, so it just doesn't really make sense. So what happens usually is you get a notebook. All of the cells are usually one of the steps, which are here. So then you transform each of the cells into a function, You write, put everything into a few Python files. You write unit tests for every function and every file you just created. And you transform the notebook into something usable. I think there are softwares that try to do this automatically. I have never tried them. Usually, I do believe it's better to have another look just for also you find bugs and things, and it's always better to really understand what you're doing. And the step of transforming the notebook into code is actually a good way for you to learn what is it, in a way, and to have a second, educated look. Does this answer the question? Awesome. I would have one question. It's really not working for me. So everyone said, but come on, it's frozen. Yeah. How do you do that? If you have more programming from up here in the end? So that's something you would actually see with the logs. Because if you log how many rows are going through each steps and how long it takes, you will see that something is actually different here. The other part is if you have an evaluation step with a test, just a sanity check, which is the order in which the features are used or how important are the features used is not very different from one day to another or from a standard. Usually, when these kind of things happen, the model looks weird. Like, if a human looked at it, it would be like, that's weird. And the goal is just to give enough information to a machine to be able to say, that's weird. All right. Let's take a look at the game.