Staying Alert: How to Implement Continuous Testing for Machine Learning Models

Speaker 1 [00:03]

Thank you so much for being here today and thank you for connecting to us. I'm excited to talk about monitoring because I believe it's very important to make sure that all services we deploy to production works safe and sound. So today my talk will be dedicated to staying alert and I'm going to share with you some thoughts on how you can implement the alternative way to monitor your model, alternative to this traditional one when we build live dashboards. And I hope maybe someone will be inspired at the end to try and apply this to your production machine learning based services So, my name is Emily. I will be here like till the end of the conference So if you have some questions, we can discuss it later or even later later So I will be here And now I'm CTO and co-founder at Evidently which is the startup where we build a tool to validate and monitor machine learning models and And honestly, this work was inspired by my previous experience of helping companies to build machine learning-based solutions and deploy them to production. Together with many different companies, we faced a lot of issues related to data quality, data integrity, models performance in production. And this is why I decided to invest a lot of time into building such tools. And today, I'm going to share some ideas on how we can implement performance monitoring for machine learning. Everybody knows that machine learning models need to be monitored because of data drift, concept drift, and many, many potential and real problems with the data and models quality. So, basically, to make sure that models work, we need to implement monitoring. Luckily, we have production software services for ages, so we know how to build monitoring for a software system. Basically, the traditional scheme for monitoring looks somewhat like that. If you have our REST API service in production, basically what we do is we are logging all the requests and our goal is to implement something like realtime dashboard with a lot of metrics and alerts so that we can understand in realtime or near realtime how our service operates. So basically what we generally do is we have our request data, we try to calculate a lot of metrics which we can basically derive from these event logs, then we have to expose endpoints, for example metrics, and we really want to expose an endpoint because we want to pull our metrics more or less in real time because we need to understand what is happening right now, not like five minutes ago, right? And we basically use something like Prometheus service or maybe something else, the service which can pull data from our production service and then store it as the time series metrics because based on this time series metrics, we can implement any dashboards we want. And we also have a lot of tools to do that, for example, Zabbix or Grafana where you can query your data source and build any metrics that you want to monitor together with alerts, Of course, tools like PrimiTools not only supports pool, so if you prefer to, you can push data there. Of course, it will be more delay, but that's also possible, so it's not limited to pooling. And that's how our traditional monitoring solution looks like. Basically, the most straightforward idea, I believe, is to just reuse the same scheme, but add a couple of data-related metrics. When we look at this from points of metrics, You will see that together with the standard service health metrics like latency or memory usage we have something related to data, right? It's data quality, data drift, concept drift, model performance and a lot of such metrics. So the idea is or the question is the follows. Can we just calculate a lot of machine learning related metrics and add them to our metric storage so that later we can just query and visualize those metrics in our dashboards. Basically, the idea sounds very good, but as usual, there are some potential problems here. First, let's quickly take a look at what data we are dealing with to figure out the source of potential problems. Basically, when it comes to software monitoring metrics, we don't really calculate metric. Basically what we do is we derive some information from our requests. And it's a lot of technical metrics like time of response or amount of used memory, right? So we don't really need to use any algorithm to calculate it. We just need to parse our request and like store all this data. But when it comes to machine learning models, it's not exactly like that because it's not always enough to just parse our logs, right, because we have a lot of raw data, we need to transform it into features, we need to calculate metrics, we need to have ground truth sometimes to calculate some error or quality-related metrics, so it's not that straightforward. And now I want to share some potential list of metrics one might want to calculate and then discuss where we can potentially have some difficulties. So first of all, I believe everybody agrees that when something is wrong with our machine learning model, that's because the data is wrong, right? This is why we need to monitor data quality and data integrity metrics. We can start from something pretty straightforward like number of columns or share of missing values and move to the distributions of features, out range values, top features, drift and stuff like that. Together with data, we are quite interested in model performance, right? So this is the reason why we need to analyze models output, models output distribution to make sure it stays exactly as we expected, target drift if you have access to target data, and a bunch of machine learning quality metrics like precision, recall, log loss for classification problems, or mean absolute error and stuff for regression models. So quite a lot of metrics. What's interesting about those metrics that we not always can just go to our event logs and calculate them. Why so? Because for example some metrics are pretty often delayed because for many, many cases we do not have our ground truth from the very beginning, right? Especially if we are trying to predict something, for example, predict the demand for some products or some services and our prediction horizon is quite long, for example, a month or two, Then we need to wait until this time passes and only after that we can derive this target and calculate things like precision, recall or even target drift because we need to have this target. So it doesn't really easily integrate in the traditional monitoring because in traditional monitoring we are used to getting data without any delay and just visualize it in our dashboard. So delayed data is the first issue here. So second is that there are a lot of metrics that need to be calculated and we need to implement this algorithm for calculating of those metrics. And the problem here is that when it comes to memory usage or latency, we just derive those data from the logs. When it comes to precision recall or detections, we cannot do that exactly because we need to perform some post-processing of those data, right? And there are basically two places where we can do that. in our service when we send this data to the Prometheus, for example, or when Prometheus pull this data from our service. But we don't really happy with that because we do not want to use any memory or other resources to like calculating drift because, well, why? We prefer to save as much time as possible for and resources for our service. And there are second option. We came to calculate all those metrics on the dashboard side, right? So in this case, you need to be like SQL guru or PromQL guru because when you try to write an SQL query which calculates drifts, that's hard, believe me, that's hard. So when it comes to calculate just average or something or max or something, SQL is perfect, but when it comes to something more complicated, then you need to validate those queries. And this is why basically it's much better to already like log the value in the format when you just need to then calculate average or other statistics on top of already calculated values. No like raw data. So we have more issues here. Sometimes you need to have not only current logs from logs from most recent moment of time, but also some reference data to calculate metrics. Because for example, when it come to stuff like data drift, basically data drift is about comparing the behavior of our current data with the ones from the previous moment of time. For example, from our validation test, or handpicked golden set, or even sometimes from training data, which we do not recommend, but still sometimes it's happening. So we need to have two data sets, and we need to compare distributions from our features on top of this reference data and current data. And it doesn't very nicely lies into the monitoring scheme, because loading huge reference data, you build in some metrics in real time, that's generally not what we want to do, especially if our reference data set is quite huge, right? And moreover, when it comes to calculating general metrics, what we're trying to do basically is to use incremental calculations. I mean just collect more data from logs and then update our counters or just plot those data in time, something like this. But when it comes to data drift which is pretty often equals to some statistical test calculations, it doesn't make sense to update your p-value after each new object you get from the logs, because it doesn't change that fast. Basically, you better collect a bunch of data and only after you get this bunch of data, you update your drift estimation. That's why it's not always that handy to recalculate those metrics in the real time. Finally, I just want to mention that there are a lot of customization here. Because when it comes to traditional software, if you have, like, several production services in your company and you want to monitor for, like, latency and response time, you can basically reuse the same thresholds, right? So if you even if you have, like, a lot of production services, you could possibly split it into two groups, supercritical and just critical, right, because everything in production is critical, and specify, like, different expectations or thresholds for this memory and time-related metrics. But when it comes to machine learning models, it literally doesn't work like that, because for some models, like, 90% of accuracy is great, for others, 90% of accuracy is disaster, so we really need to customize all these thresholds for different models, and if you have thousands of such models in production, you really spend, like, many hours into customising of those old thresholds, and now you actually want to do that. So it's really rare when you can do it automatically, but there are some solutions which could help with that. So now I believe we see the potential problems with the implementing monitoring in the very same way comparing to traditional software monitoring. And I want to suggest a kind of alternative way to doing that. So together with traditional monitoring, and I'm not telling like instead, it's more together because anyway we are talking about production services, so we need to monitor service health, right? So this dashboard is like mandatory, but together with that, we can add one more. And I want to suggest try to use test-based monitoring, which basically looks pretty interesting. Instead of having the online updated calculations, we could implement it as the pipelines. So basically what we can do is schedule a pipeline which calculates tests for us so that we can run them in a schedule and then log the results to, for example, the same data storage as we use for traditional monitoring and then visualize it in some dashboards even in the same dashboards comparing to traditional monitoring because tools like Grafana or Zabbix or many, many other BI tools supports different data storages, right? So if you use, for example, Prometheus for service health monitoring, you can also use the PostgreSQL for your machine learning related metrics and visualize it in the same dashboards. of course in different like tabs or different panels, but the tool will be the same, which is pretty nice. So the idea is to implement the monitoring as pipeline jobs, which we run in a schedule. And now let's quickly see how it can help. Basically, when we have the issue with the reference data set, what we can do is just implement it as a pipeline. In this case, we will have much less memory and latency requirements because we already moved from real time to batch, so we can spend quite a lot of time on calculations. So just read all those data, calculate all those metrics and be happy with that because we have quite a lot of resources for that. With ground truth, it's much easier because we can easily split our tests into separated jobs, right, and jobs that do not need any ground truth, we calculate with a schedule like once per minute or once per hour, and other tests which requires to have this ground truth, generally it's tests which depends on the target in order to calculate some quality metrics or error-based metrics, we can just make some dependency like we calculate all those tests and metrics only after we successfully loaded the target data, right? For pipelines, it's pretty simple because there are a lot of tools like Airflow and Prefect which supports dependency between tasks, so we can implement that. When it comes to threshold customization, there are some tools which can help with that. So basically, tools like great expectations, tools like deep check, like evidently, we allow to use reference data set for threshold calculations so instead of manually selecting threshold for each for example drift detection for each feature or for quality metric for each model you can spend more resources into creating very well reference data set and ask tool to derive all these test conditions all these thresholds automatically which is quite nice especially if have a lot of models. Then you can just start to use it and then later just customize a couple of thresholds which you don't really like because you missed too much problems or because it's alerts too often. Finally, when it comes to pretty complicated metrics, it's always good to have some pipelines because when you implement your pipeline, you're not limited to any specific, very specific programming languages. What I mean here, you don't really need to implement each task in your pipeline by writing a SQL query, right? You can use some general programming languages like Python and implement any complicated algorithm as a general Python function, which is much easier, right? Because we have a lot of tools and we have a lot of libraries, packages which can help with that. So basically import whatever you want, implement it as the Python function, and here you have your metrics, which is really very nice because when you log to your database those metrics, the only thing you need to do finally is just calculate something like simple statistics on top of those metrics and you don't need to write any complicated SQL queries, which is quite nice. And I want to encourage everyone who is thinking about adding some machine learning or data-related metrics to monitoring, try out open source libraries and see how we can do that using tests because basically it makes a lot of sense to calculate metrics which are designed to be calculated on top of batch data in a batch mode because when it comes to quality metrics, even some straightforward metrics like precision, recall or error, we don't really want to update it after every single object, right, because, well, we need to have some batch of objects to see how the statistics change. And if it's designed to be calculated on top of batch model, batch objects, why not to calculate it this way? So in Evidently you can use it, you can build it having Evidently installed as the Python package, so here is a link, it's totally open source, all tests and metrics are available, and together with some metrics and tests which you can calculate and output in the JSON format which is the best for automation because basically we need to log it somewhere and then visualize for example in Grafana if you still do not have monitoring you can just visualize it in Python and see how it looks like but that's not the scenario I suggest for monitoring. It's just for like testing validation for monitoring better JSON of course. And it has very simple syntaxes so basically you You can just create a test suite, list all the tests you want, specify some variables, I mean some features, some conditions and it works for you. But if you don't want to do that, you have pretty nice presets which are quite good for some specific case like getting new labelled data or getting new batch of predictions and just call that and see what tests there are and then later customise it. The final architecture might be very, very similar, so basically together with the existing service health monitoring, you can just add a couple of pipelines, which works in a schedule, and basically write the result data in the separate storage, so you can just, for example, in Grafana, create button and panel, use another data source, and it will be in the same dashboard, So no need to create more and more architectural solutions, just add some results of the tests. And finally, I still have like two minutes, I want to say that it works not only for real-time solutions but also for batches, right? So I started from real-time because it's a bit more complicated, I would say, and when you have your machine learning model deployed as the REST API service, you really have like a lot of pressure for monitoring. But when it comes to batch, it also works very nice because, well, if you use something like this, like you collect and combine your data from many, many different sources, then you perform something like join and preprocessing, derive features, calculate model, create the outputs and then use it, for example, like in the slides, it might be some marketing campaign that you are trying to optimise with help of machine learning model, so you generate some scores and based on the scores you are like creating your marketing campaign and doing for example email sentencing. So in this case you can also use continuous testing. It will be even more straightforward comparing to the rest of the API services because you can basically split your pipeline into steps. will be split into steps because it's generally implemented like that and basically if you start from the loading of new data batch we can implement the tests like after each step of the pipeline for example after new batch of data arrives you can run the set of tests for data stability like data scheme check like column types check like missing values checks so something pretty straightforward. I believe it solves 90% of problems as usual. Then you can, for example, check your data for drift if you know that you have strong seasonality or like you assume that might be some shifts in your data. Then when you get new labels, of course check for performance, right? I mean quality-related metrics. And basically if all checks has passed, you can retrain your data on top of new models and check model quality. So that's how you can integrate this continuous testing in your batch models. And this is the good news, because you can reuse the existing testing scheme for both batchly deployed and REST API deployed models, which I believe nice, especially if you have both models in production in your company and you don't really want to spend as much resources as you spend for model building for monitoring. It should be simpler and faster. To sum up, I want to say that even if you have your machine learning services in real time deployed with API, you might still monitor them in batch. That's okay. I know that sometimes sounds a bit strange because if you have model deployed in real time, you prefer to have monitoring in real But when it comes to machine learning model, there are a lot of metrics designed to be calculated in the batch mode. This is why I strongly believe that for machine learning services, we can just add part of monitoring as a batch testing and it will be perfectly fine. So when it comes to testing, it's much more lightweight in terms of architecture because when we calculate something in the batch mode, we have much less limits in terms of resources dependencies and memory usage so we can allow ourselves pretty complicated algorithms for example for drift detections like statistical tests, distances and many, many different things. And finally, if you have not only REST API but batch models and you don't want to customise your testing pipeline, that's used very well because basically it's much easier to use the same scheme for batch and realtime models, why not to use that? So my final thought would be at least consider using continuous testing for REST API models and if you have any questions or suggestions or arguments why you shouldn't do that, I will be happy to discuss that. Thank you so much.

Speaker 2 [23:54]

Do I have sound? Yes. Thank you so much for a very interesting talk and we have multiple questions on Lido for you. And the most upvoted question currently is what do you think about auto retraining models based on drift or other metrics going the wrong way in quotation marks? Any problems or challenges with that?

Speaker 1 [24:17]

Wow, that's a very nice question. Yes, I think there are quite a lot of challenges with that. First of all, I'm totally for out-retraining models, because when you have a lot, you don't really want to have your data scientists manually retraining those models, right? So it needs to be automated. But if we use only drift detection for retraining new models, we can quite spend our resources for training models because together with like shifts, we need to make sure that we have enough data to retrain model, right? Because if we just detected the shift, maybe we do not still have enough data for model to learn the pattern, right? So maybe we still need to wait for quite a lot of time to collect new data. And I believe it can be implemented in an automated way, but together with data drift, I would analyze more metrics to make sure that we have enough resources, enough data to train a new model and to make sure that it's actually something that makes sense because together with retraining model, we can react differently. For example, if we have like a fallback system, we might switch from the machine learning approach to some statistical-based or rule-based or even send our data to manual review if you're working in the medicine sphere, right, for some short period of time until we collect enough data. So I believe it should be implemented with respect to problem statement and considering other two retraining model approach to react on drift. Yeah, I think like this.

Speaker 2 [25:52]

Thank you very much. Also, quite some people seem to be interested in how to get ground truth for your metrics.

Speaker 1 [26:01]

Thank you.

Speaker 2 [26:02]

The sentence here is, does somebody check batches at random?

Speaker 1 [26:07]

Okay, so let me start from the fact that it really depends on the task and I love data science because you pretty much can say it for any question depending on the task, right? But generally, we get our ground truth from some automated systems. For example, when it comes to some demand forecasting, generally, we get those data from other data sources when the horizon has passed or there might be some pipelines which calculate for us ground truth by querying the external data sources and load this data for us. So basically what I would do from the pipeline automation point of view, I would implement this ground truth loading as the task which is scheduled by time and then implement and the metrics and test about the data quality as the dependent for this loading ground truth and run it only after we get this ground truth. I think somehow like this, but of course it depends. If we load data manually, it doesn't work obviously and in this case, you just need to try, I believe figure out how often we do that and maybe come up with some solution where someone can like drag and drop these targets and run test calculations. I'm not a big fan of that, but anyway, it's better than just ignore those metrics, right? So better to automate that, but if cannot, then some user-friendly interfaces.

Speaker 2 [27:41]

Thank you again. The next question is that you made a distinction between model quality and performance. Can you explain the difference here, what you meant by the distinction?

Speaker 1 [27:54]

Yeah, sure. Basically, when it comes to model quality, we somehow thinking and talking about error-based metrics like precision, recall, and absolute error and stuff like that. But when it comes to production machine learning-based services, we are generally interested in more product-related metrics. For example, something related to cost efficiency or reduction of human hours and stuff like this. And sometimes it might happen that our model still behaves quite well in terms of, for example, accuracy or precision. But it's already started to work much worse in terms of our business metrics. For example, our model might be trained to a little underestimate your demand and that was okay in terms of your business problem. But later, with data drift, it starts to overestimate your demand and this might be a disaster from a business point of view, right? And in this case, you might miss it from model quality metrics, but if you calculate also business-related metrics, you will definitely see that and react.

Speaker 2 [29:04]

So we might have time left for one more question

Speaker 1 [29:08]

Cool.

Speaker 2 [29:08]

Do you have advice on how to detect data and prediction drift in computer vision applications?

Speaker 1 [29:15]

Oh, that's hard. That's hard for me, at least because I'm more work with the tabular data and a little bit with encoders, so this is the source of my answer. I would say that when it comes to computer vision models, I personally would work with encoders, because generally at some point we use this, we work with those data as these vectors, right? And when you come your encoders, there are many ways how you can compare that. Basically it it moves us to the problem of multivariate variable drift detection, right? Because these encoders, they are quite large nowadays. And basically, there are several approaches, like using machine learning models for those. I mean, if you have your data as encoders from reference data and from current one, you can train a binary classifier model, which will try to distinguish between reference and current data. And if model is successful enough, then there is some differences, right? and you can treat these results as a drift, like rock receiver operation curve as the drift detection, right? And there are more statistical-related ways. I don't know if you have time to...

Speaker 2 [30:25]

Unfortunately not, no. We need to come to an end.

Speaker 1 [30:28]

Then just a list, like MMD test, ratio of drifted features, if you treat each vector component as a separate one. Something like this.

Speaker 2 [30:39]

Thank you so much for a very interesting talk.