MLOps in practice: our journey from batch to real-time inference

Speaker 1 [00:02]

So, can you, yeah, okay, it's weird. So yeah, welcome everyone. Today I will talk about MLOps in practice. And yeah, without further ado, let's dive into the agenda. So let me start with an introduction where I'll introduce myself, the company, and what we do in my team. and then I will explain how we de-risked the migration from the batch to the real-time inference by breaking it down in multiple steps, where each of them added some values and helped us learn, basically, on the way. Then the second part, I will explain the challenges that arose from this new architecture and the key design decision that we took to overcome them. And finally, I will conclude and open for questions. So, yeah, I'm Theo, I'm a data science manager at GetYourGuide and I will give you a small snapshot of what we are doing at GetYourGuide for context. So GetYourGuide is a marketplace for travel experiences where we connect customers with activity suppliers worldwide and so you can book on our website or on the app experiences basically, or activities. More precisely, on my team, in my team, we're responsible for ranking these activities. And that you can see on the website or the app. And we have different type of pages with different designs, so which come up with different flavor of ranking. So it's a critical component of our website and the major data product that we have in our company. So with that, as you can imagine, trying to change such a system is a sensitive topic for many people, and let's dive into this migration and how we handle it. And first I will explain where we started from and where we wanted to go. So the first thing, at the start, around two years ago, my team was responsible for the scoring of every activities in our inventory by doing some kind of daily batch job. And then we just pushed the score and another team was responsible for ranking the activity using the score that we provided. So it's a pretty simple to maintain and we have a simple interface with the rest of the company but every time you want to do any kind of testing for a new scoring logic then we had to we relied on another team and it was very hard or nearly impossible to do any kind of segmentation personalization with this approach so where we wanted to go was having our own service that was responsible for the ranking of the activity in real time with their then And the nice thing with that is that we control the language logic so we can use it for any kind of scoring or we can have a formula or even better, a model. But that's a new service that needs to be maintained by us. So what our vision basically was to have an ML model doing live inference inside this ranking service. But that's not what we started to build, actually. We wanted to de-risk this migration by breaking it down into multiple steps. And I will go through the steps now with you. So the first thing that we did is to migrate the ranking to the service. So basically what we just did is just having the service that was doing this lookup to the score and then ordering based on the score. Then with that, we were able to prove the feasibility of having such service. And we could increase the speed of launching a new experiment as we were not relying on another team. And also it helped the team to learn the technical challenges of operating such a service. Then on the second step, we started segmenting or ranking depending on some user signal like for example the platform they are using or the language. So we were able to generate different scores for every type of customers, and we were able to get a significant improvement of our ranking that helped us prove to our stakeholders and reinforce our hypothesis that the ranking need to be contextualized and personalized, and we were going the right direction basically. Then what we did, next step was to train a model now to inside, to rank the activities inside the model. So we will use actually the segment scores that we generated in the previous step and now we could train our model to optimally combine these segments based on some past data. Here the nice thing was we were able to stabilise the model in production with this new framework of having a model, a ranker more specifically, and also we learned a lot about how to operate this machine learning model in a real-time fashion and also how to generate the data to train it. And finally, we went into improving the model, so what we started adding is adding more relevant features from the user and from the activity, like the price, the reviews, these kind of things, and we fine-tuned the model. And here we also were able to prove significantly improvement in our ranking through A-B test. And also we helped us to learn how we can improve our offline evaluation basically to select which model we want to learn, we want to test in production for further iteration. Okay. So that's how we had this step-by-step. But as you can imagine, operating such a service has its own set of challenges. And what I want to go into in this second part is the key design decision that we made to overcome them. So first thing, let's go into the requirements that we have. So first, the apps and the website need the ranking to be able to show the relevant activities. So that means that we have a service that needs to be available 24-7, basically. Then second thing is no one likes slow websites. So that means that for us, we had pretty strict latency constraint under 100 milliseconds to do the prediction. And finally, there was a constant stream of change coming from our MLOps engineer and also a data scientist, and that's the make, like, we want to make it work so that people like these two type of experts in our team was able to make change in parallel without blocking each other and having every time requesting a review from the other side for every little change that we're doing. So yeah, in practice, I think that sounds like a pretty nice thing, a pretty nice receipt for incidents or big problems, you have a complex service with strict operational constraints and where you want to change constantly by different types of people that have different areas of expertise. But actually, we manage to keep our system up and without any incidents, we're able to launch experiment in days compared to months as we had with the previous setup where we were relying on other teams. And so for that, I put like three pillars that I think help us the most to be able to achieve that. The first one is to have an explicit ownership among the experts that we have in our team. one we use extensively or end-to-end test and where we use real data instead of dummy data in particular and finally we automated all the workflow to test train deploy as part of a CI and now let's dive into each of them okay the first one we used MLflow to be able to clarify the interface between our data scientist and the MLOps engineers. For the ones that do not know, MLflow is an open source library that is able to help manage the model lifecycle end-to-end. With MLflow, basically for us, what it helped was to clarify this interface and then the data scientists can focus on generating the feature and the data that we that are needed then define the pre and post processing as part of a scikit-lead pipeline so that everything is packaged together in one one big model for on the MLflow sense and finally train and store the model in MLflow and then we have the MLOps engineer that are responsible for loading this model from the storage so from MLflow storage then provide the feature for the inference and ensure the service stay healthy during the during all the time basically great that's the first pillar the second one is about validating our machine learning pipeline And what we struggle for quite some time about that, and what we come up to is the ML pipeline is mainly about interaction between our code and the data. And also the ML data set are very unstable, because you are constantly adding new data sources or new features as you evolve, and so every few weeks, basically, you are changing something in there. Finally, if you are using any kind of dummy data, we try to stop using them because it's cumbersome to create this dummy data. It's also hard to maintain these hard-coded values or this kind of dataset generation to have this dummy dataset. And finally, there is a risk that that it drifts from true production data. It happened that you expect to have a timestamp and actually you get the daytime, or this kind of thing that you suddenly, the data that you will have in production is different, so your test looks fine, but actually when you go to production, boom, it breaks suddenly. So we came to the conclusion that we needed to use production data for testing. But we needed to be able to sample it to be able to run it in a reasonable amount of time. Okay, and so for that, we needed some more tooling and we developed an open source D data flow. So it's a tool that help us to sample and run end-to-end tests as part of our CI. Yeah, you can check it out on PyPI and here on this diagram I will have briefly explained kind of the philosophy behind the library. So basically with this library, you can specify the data sources that you want to sample in a config, and then you can use ddataflow object to read the data. And when you run your code, basically, you have some setting that specify which data you want to use. So if ddataflow is enabled, then the code will run fast. Wait, maybe, sorry. Let me add. So it's enabled and the code will run fast because you will use the data source that are sampled. But if it's disabled, which it is by default, then you will run the full data sources basically and your model will run on everything. And finally, you can also use this data flow to download the data so that you are also able to develop locally on a small subset of real data. The thing is running these codes on the sample data make them like if you run a test, it makes them deterministic. You cannot expect 1.43 of your output of your model or something like that. So here we also needed to tweak a bit our approach around our assertion for testing. What we came up with is adding some kind of thing that we call health check. So in other words, some kind of expectation that we have for our data set and our model. So as an example, we expect that the data set have a certain share of booked activity to be in a certain range, or, for example, for the model, that the accuracy to be above a certain threshold. So we can build all this kind of health check to validate that the data and the model is kind of reasonable before we deploy them. The thing that we wanted to avoid is that we forgot a minus somewhere and basically we rank them on the other things and put the worst activity on top. That was kind of the big fear that we had. So then we can run this health check as part of a CI based on some sample data. And of course then we use it in combination with sampling with the data doc. But also this health check can also be run as part of the production pipeline on a full data set and on the production model. And also if you remember the definition, the example I gave about the health check, they based on some kind of metrics like accuracy or share of booking in our data set. So we can also send like compute these metrics and send them to Datadog or any other dashboard so that we have a dashboard that can get an overview of our service and also we can also send alerts about if some metrics have some dangerous trends basically. Okay. So to be more visual how it looks like about this data health check in end-to-end test so we first have the we sample our production data with the data flow then we can generate the data set we can validate the sample data sets with this health check then we can train and bottle with this validated sample data set and validate the sample data the sample model in the same uh all together great and that's I mean you see with these buildings love actually that you see you we can reuse them and we can reuse them for also production data set and the production model and that bring us to our last point the automation that we introduce in our workflow and so for that what we ended up doing is that we automated every step in our testing and deployment process in our CI so that That means that every time we have a change, then we run the unit test, like normal, then we run this end-to-end test with data where we'll have all the steps running where we'll sample the data, run the health check, train a model on sample data, run the health check again, and at the end what we can do is push this model into MLflow. And this step basically then we can have some integration test where we can load this model and see if we are able to build a service with that and run some integration tests with the rest of the service. Once it's done, if it looks fine, then we can trigger start on the deployment. That means that we can train a model now, here again we can launch a training job one more time, but this time we're using production data, all the data sets, and also as the building blocks can be reused and we say okay we can also reuse these building blocks to validate after that model with production data is sane. And finally we again push it to MLflow and then we can start our deployment process. Great yeah so now we can deploy the new services so what happened here is we will load the model, wrap it into a Docker image, and then we can send it to our orchestration deployment or orchestration tool, Argo. And this allows us to have a, we also put some additional safety here around canary deployment because it makes sure that our model is not breaking with real requests and And also that we are still respecting our latency constraint. And so if a model that we launch suddenly is too slow, then we will start breaking the request and the canary step will fail, basically. And also nighting on the side. And also, finally, the packaging of our model into this Docker image make it easy to roll back as we have a Docker image and which is very similar to any other services that we have at Get Your Guide. Finally, we also have an Airflow DAG that is triggered every day so that we generate a new data set for the new day, basically. Here again, we can train some, we can also verify that this new data set is correct and with using the same health check and actually we always build this health check for the CI and we thought like well you never know we can also do it in production but if it buys the CI should not work but actually we had an incident recently that on a pipeline be above and thanks to this health check failing we avoid cascading to our model and starting having a rubbish data set that will have trained a rubbish model to go to production so yeah that's basically how we do so we make sure that we automate everything and every time we have a new change basically we will automatically run all these steps which means that we'll test end-to-end that the data data set is still correct that we can train a model that we can integrate the model and if it passed and we'll train a new model and deploy it it and after verifying it's able to handle the production traffic. And that's make sure that we remove by automating everything, we remove a room for manual error, we simplify our deployment process and that increases speed of development for everyone. So with that, let's conclude. Yeah, today I showed how we improved our ranking and the risk of migration. I'm way too fast. Sorry I was, sorry I'm, I will try to slow down, too late. So yeah, I show how we improve our ranking and de-risk the migration by moving step-by-step. So I explain that the three design decisions that we take that help us go to the state that we are. So we have a clear ownership with MLflow as the interface between our data scientists and our MLOps engineer. We use production data for testing and with health check. And finally, we automated our full deployment with the CI, including some data set generation and the model serving. Of course, our journey was actually full of surprise. We struggled for multiple iterations, for example, to improve the ranking by segmentation, and the learning curve also for our ranker model was not as smooth as expected, and we I think also we could have made some integration tests earlier in the process, but at the end we have a pretty robust process and service. And finally, I think this work was actually work not from me only but from the whole team, And yeah, it's for many months, and even more than one year. So yeah, special thanks to some people, especially Jean, Xingting, Olivia, Max, and Mikhail, that's also helped me on this. That's it. Thank you.

Speaker 2 [22:18]

Right there, thank you for your talk you were very fast But that gives us fortunately time for some interesting questions First one how are you sampling the data are there additional steps to check that your data is representative?

Speaker 1 [22:34]

Yes, so we also struggled with this, we came up with the best idea was actually you are responsible to define as part of the data source how you want to sample the data. And so for us, what made the most of sense was actually to look at some specific key that we use. So most of the join that we're doing are made on some visitor ID which represent for every browser basically you have a kind of visitor ID assigned to in your cookie and that's what we use along the chain to to create the data set and so for us what makes sense is to actually join on have some kind of pattern on this visitor ID to say oh you have to start with AAA as a this on this unique key to be able to to be in this sample data so I may allow us to sample and we expect this this to be a representative sample. What we wanted to avoid is we say oh we just sample the first 10,000 but then when we start joining the data set together then of course it's unlikely that a sample data set from this and this will when they join them and it would just take a random sample it's unlikely that we'll be able to join them together if they are just randomly sampled. So that's the the main thing that we had to take care of.

Speaker 2 [23:52]

Yeah, that makes sense definitely Next question is how did you ensure data privacy when using real data for model testing?

Speaker 1 [24:01]

Still, everything is in our cloud, actually, except when we load the data, but that's also on some personal laptop, basically, sorry, not personal laptop, but the laptop of the company. First, we have no sensitive information in our model as a ranking. We are not able to extract things, but to answer the question is, it's part of the CI, so if you have the model there, it's... We just automated this process of doing this sampling, but you have access to data, and if you don't have access, then you have access to nothing, or if you have access to everything, then you also have access to a sample subset of it that you can also train to create a the sub-sample model, basically. Does it make sense or? Audience?

Speaker 2 [24:59]

Does it make sense?

Speaker 1 [25:00]

Okay, thank you

Speaker 2 [25:03]

Next question First of all, I have one One here that says could you show what the data scientist and ML engineer responsibilities? Could you show that on the diagram of workflow automation? I think there was a

Speaker 1 [25:20]

Ah, this one I suspect.

Speaker 2 [25:26]

So...

Speaker 1 [25:28]

Is it from the audience?

Speaker 2 [25:30]

Whoever wanted to see can look at it now while we are continuing with the other questions.

Speaker 1 [25:35]

I will share the the slides also in the channel afterwards so you can see them but was it a question about what who is responsible for which step basically also

Speaker 2 [25:46]

Yes, I guess that was the question.

Speaker 1 [25:48]

Okay, so basically then the unit test is a shared responsibility depending on the area of the code that it is Then this end-to-end tests are definitely on the data science to define What how the data and unit tests that you want to want want to do for the data set generation and the model training? That's part of the data set of the data scientist then pushing to ML flow then loading this model and doing this integration test Is on the ML on the? Sorry, it's on the ML Ops engineer Then, training this is then on the data scientist, pushing it, the data scientist, and deploying and these steps is then on the MLOps engineer. Building this data set is also part of the data scientist.

Speaker 2 [26:31]

All right, very nice next questions. What might you switch from Benton bento ml? Sorry you Apparently apparently presented it at a conference last year. I'm a southern core. I guess you have a fan here

Speaker 1 [26:47]

I removed it for the sake of brevity because I didn't think it was important for this talk but we're still using Bento ML and it's part of our deployment. Actually when we load the model, we load the MLflow model and the wrapper that we use to create from a model, transform a model to a web service is Bento ML. I didn't want to, that was not a key part of this talk, so that's why I removed it. And I'm sorry if I offended some people.

Speaker 2 [27:23]

All right, next one. Do you have development or production accounts in your cloud environment for proper testing and development? If so, how do you move the data and models around?

Speaker 1 [27:34]

Yes, at this time we have Databricks where we have a lot of access, so basically we are able to move the data around. But we mainly, I think the thing is, if we're not familiar with Databricks, basically they provide a kind of UI that helps you, that's a data platform service that helps you with having a nice way to have analysis and do machine learning also, and then that's what we can use to easily create a cluster and be able to start moving the data around and things like that. So yeah, I think I answered all of the questions. So we don't have directly access to AWS to be able to spin up AC2 instance or things like that, but actually through Databricks we can create clusters and be able to move things around.

Speaker 2 [28:35]

Alright, makes sense. Next question. With respect to the explicit ownership, we have problems to find people for the MLOps part. Is there a typical background where your MLOps engineers come from?

Speaker 1 [28:49]

Yes, we struggle also, we, no, I don't have like a magic answer, I think if I had a magic answer I would keep it for myself to be honest, but I can already tell you like it's hard to find people with the right skill on this MLOps engineer side.

Speaker 2 [29:17]

How did you decide on your MLOps tooling? So Airflow, Argo, Databricks, MLflow? Have you encountered problems that will make you switch in the future?

Speaker 1 [29:27]

So actually, Jean and I worked quite a bit on this ML platform, and we decided a lot of this, what are the tools that we're using, and actually what's the thing that we end up doing is we wanted to use as much as possible what was already used by the company already, And that means that our deployment pipeline with Argo is basically all before what we had was Spinnaker, but basically we just decided to keep with them so that we have one deployment tool that is the same than the rest of the company for any other services. Something are not possible. One example, this is MLflow. There is no way your classical company has things to be able to store models. And so here we introduce the tool that we're making the most sense. And MLflow is really well integrated with Databricks, so we made the list of pro and crons and decided to do that. So that's how we went through this. And most of them actually have been things that we start reusing and we're just combining them with things that we just reuse the thing that is in the company and just tweak them or really push them to limits. One example is our CI that is taking a bit longer than most of the CI of the other services, but still just a CI like any other services.

Speaker 2 [30:54]

All right, very nice. Is Airflow only used for building the daily data set? What is the advantage of using something simpler for this, for example, a cron job?

Speaker 1 [31:06]

Yes, please use Airflow because current jobs really get messy and it will be already very messy for us to use something as an Airflow because to be able to have our daily data ready, we also rely on some previous pipeline that are scheduled on Airflow to be able to clean the data of the previous day and put them into the right formats And only when it's done, then we can trigger our data set training. So here again, we didn't want to reinvent the wheel. We already use Airflow, which was used by everything on our core data engineering. And we just added those things on top of it. Otherwise, I have no idea how much work. Yeah, I will still be working on this.

Speaker 2 [32:01]

Okay, even with real data testing it should be the same each time Does the data drift problem still applies if so or?

Speaker 1 [32:11]

So I think of two things first thing is if you use real data, it should still be the same Yes, but the thing is the data Basically, we use the data of the last week and the data of the last week is changing depending on the day you run The pipeline so we our data is changing Every time you run the CI depending on the day you will have different data and do we still have data drift? Yes We are looking into that that so we recently introduced a new tool to look into ML monitoring and this data drift in particular but the best thing you can do and we know we do it by default is actually you retrain your model regularly which is for us every day and that makes sure this data drift cannot go too fast but we are also looking into some ML monitoring tool but yeah This small hand will make me high at this time.

Speaker 2 [33:08]

How do you sample data from different sources which have to be joined later?

Speaker 1 [33:12]

Yeah, I think already answer is for us the thing that they are trying to visit ID And so we have some specific condition on this visitor ID Which is just like some alphanumerical string and we say we are on alpha things then You just look at the one for that start with AAA which make it like basically All the others we ignore and just just a random sample then we make sure that we can get them through the list of events

Speaker 2 [33:40]

I can't believe we made it through all the questions. This is the last one. How do your models interact with the front and back end apps?

Speaker 1 [33:49]

Yeah. Good question. I will need another talk to talk about that, I think. I mean, at the end we just have an endpoint and then we basically provide this endpoint to any services that want to, need to rank activities. So basically they need to send us a list of activities and send it back. But there are many steps in the process of like getting from the activity to getting the picture, the title, for the right language, the price, etc., so there are many steps that happen afterwards, but yeah, I'm not the best person to explain everything around that.

Speaker 2 [34:30]

All right. Thank you very much. Give it up for Theodore.