The State of Production Machine Learning in 2023

Speaker 1 [00:03]

Great, so I'll just wait for the screen and Yes, great. So welcome everyone. I'm really excited today to present to you very interesting and very packed Presentation on the state of production ML. I hope you're all are enjoying the conference so far And yeah, just a reminder that you will be able to find the slides in the link above so if you will want to dive into any of these areas you'll be able to do so there so a bit about myself so my name is Alejandro I'm engineering director at Zalando I'm also scientific advisor at the Institute for ethical AI and machine learning and governing council member at large at the ACM so another thing to mention is that the talk slides are going to have a link at the bottom basically providing a previous talk or another talk that will be able to cover that particular topic in more detail and the reason why is because this talk as the state of production ML will be bringing together a lot of insights in the ecosystem into an attempted single session of course this will be a whistleblower tour so you will I will recommend you to actually you know, for anything that you find interesting, do a deep dive yourself. So in this session we're going to be diving into the motivations and challenges of why do we care about production machine learning concepts. We're going to be delving in some trends in industry and in the more broader ecosystem, as well as then some technological trends, organizational trends, and then wrap up. So we're going to be starting at the very, very high level, very abstract, and we're starting to go kind of like lower and lower into the stack and finishing even with the organizations. So let's get started with the motivations and challenges. So why do we care about production machine learning, right? Because, you know, when you train your model and you reach the relevant metrics, whether it's, you know, accuracy or whatever you're aiming for, you know, the job is finished, right? Well, what we realize is that the life cycle of the model actually begins once it's put in production, once it's actually being used. Because there are considerations with the model, right? Things like data drift, concept drift, just changes that need to be considered, retrainings. So there's much more than just, you know, training the model and then you're done. And with that, you know, taking a step back and asking the question of why is it so challenging, you know, a couple of things for us to, you know, understand kind of like the challenge that we deal with compared to traditional software is the fact that we have to deal with specialized hardware, right? You know, not only GPUs and TPUs, but also crazy amounts of RAM, right? Or, you know, multiple different types of resources in a very heterogeneous environment. We also have complex data flows, right? So we have dependencies that if something goes wrong, it affects down the stream or up the stream. There are compliance considerations at the use case level, which we're gonna talk about. And there's a need for reproducibility, right? It's not just about we can run it, but if something happens, we want to be able to run it again to have, whether it's a respective audit trails, et cetera. And also then we go a level higher, right, to the use case and the potentially high-profile cases. There are challenges that we have heard even in this conference on algorithmic bias when it comes to the machine learning models themselves, misuse of the data itself, so personal and tribal data. There are challenges on security, which we're going to be talking about, and also other challenges with just general, you know, reliability of software, right? Now you're dealing with systems that are powering infrastructure-critical use cases in society, right? So considering that as practitioners, we have to, you know, realize that the impact of a bad solution can be worse than no solution at all. So what we've now seen is a convergence of skills that have allowed us to tackle this, right? You can't kind of like, you know, open a job post and try to hire this unicorn with, you know, a PhD, years of consulting experience, business experience, data science experience, DevOps, and everything for the salary of an intern, right? I mean, ultimately, this actually converged into being able to leverage the skill set of different practitioners. practitioner. So that is data science, software engineering, and DevOps engineering into basically the practice of machine learning engineering still interacting with these different areas. And now this practice of machine learning, you know, expertise in machine learning engineering also intersects with the use cases and the impacts that are involved in that area. So industry, domain expertise, policy expertise, and leveraging kind of like whatever is those best practices or industry standards. So those are some of the motivations. Why should we care, right? Now let's dive into some industry and high-level trends. So one of the things that we have seen is basically the rise of principles for ethical AI, right? We've seen every single company release their own sort of nice blog post saying, this is how we are responsible, and this is our five principles to rule them all, right? But one of the things that is interesting is that, you know, the principles themselves are very much only guiding, you know, Northern Star, right? Because you can have as many round tables you want, but if you don't have the infrastructure that actually is able to enforce and match those principles, it's not going to make any difference, right? So that is why it's important to make sure that you have, or that, you know, there is a mapping between not just this high-level principles guidelines, but also the underlying, you know, standards, frameworks, and then at the very bottom, the software, right? And even more specifically, in some cases, you know, the open source software, that if it's not built by design to be able to implement these things, it's not going to be possible to achieve those, you know, high-level principles. And in the same way, you know, ethical challenges to that extent should not fall in the shoulders of a single data scientist or a single software engineer, right? Because of that, it's worth considering that it's not just about a practitioner being ethical or responsible, right? Because you can have a group of ethical or responsible practitioners that doesn't make the compound also ethical and responsible, right? And we have seen high-profile use cases where, you know, I don't know, use cases of machine learning have led to scandals. It's not to say that like every single specific individual had malice in their heart and wanted to build this, you know, racist algorithm, right? It's ultimately at some stage a lack of touch points and best practice that then results in those undesired outcomes. So with that, it is important to understand what is that structure that allows for the distribution of accountability at the right touch points, right? So the individual practitioner, right, is making sure that they're using their, you know, competence in the field that is relevant to them. They are using and leveraging technology best practices, professional responsibility. Then at the next level is a team level, right? The cross-functional domain skill set that is pertinent for the particular context. The key domain experts and the interaction and bringing them in where relevant, and then this also devolves into the organization, right, the high-level governing structure, the aligned objectives, etc. Of course, this is very, very high level, but, you know, bear with me, we'll delve into it. Now, also, we're seeing some trends in regulation, right, we're seeing, especially in the European Commission, a lot of really interesting and innovative policy documents that are trying to regulate and introduce best practices, you know, by by regulation, right? So things like the AI regulatory proposal, the Data Act, as well as other specific countries that are creating their own. And before we were saying how regulation was playing catch-up, but now we're seeing the opposite, like tech companies playing catch-up into innovative regulation. And one thing that we have to also consider is the fact that it's not just about these principles and this regulation to be in place, but, you know, the practitioners in this room are actually key part of this, right? If the underlying foundation is not, you know, capable of ensuring that those high-level principles are carried forward, then it's just not going to be possible to even have, you know, those perspectives. So, you know, that is the emphasis of principles are useless if the foundation is not in place. And that emphasizes that it is software practitioners, machine learning practitioners that are going to have a big voice, A very important voice in this area. So those are some very very high level areas. I know that it's still very high level So now let's start diving a little bit deeper on the more practical side So what are some technological trends that we're seeing mapping into that? One of the things to consider is what are some of the MLOps frameworks when you know back when when it started, right? We had a couple of machine learning frameworks, you know, it was you know, you can count them with with with your two hands Well, actually even even then it was it was it was quite a lot But this is how it's going, right? Like we have a pretty large list of very different or very similar tools that do very similar things across very similar areas, right? They just have different fancy names, right? That's the main difference. So how do we make sense of this, right? So how do we actually navigate? And the way that I'd like to think of this is if we take a step back and first ask the question of what is the anatomy of production machine learning system, right? What we tend to see is that it boils down into a very sort of like high-level abstraction in these components, right? So you have maybe, you know, these three areas of resources, you know, your training data, your artifact store, and your inference data, right? Meaning a lot of, you know, different things in the nuance. You then would have kind of like the first part of experimentation, right? Where you have either your Jupyter notebooks or your distributed training, ETL, hyperparameter tuning, evaluation, in order to, you know, convert this training data into models. right, whether it's Docker files or artifacts, whatever that is, then you want to be able to deploy these models, right? This could be someone pressing a button or this could be an automated pipeline that allows you to move those artifacts into the production environment, right? And that production environment may have your machine learning models, right, whether it's batch, you know, real-time, semi-real-time models that are currently able to be consumed by users. And then on top of that is the question of monitoring, right. You may have some standard monitoring, software monitoring, but also advanced machine learning monitoring like drift detection and explainability. And then finally, the data that you have that is actually going through your production machine learning, it's also something that you want to somehow leverage, right, that is your inference data. So there's the question of how do you then consume that data that is being generated, those unseen data points, right, different to your label data points or your your training data. So, on top of all this, you also have the question of metadata, right, is what is the information about the resources that you have deployed here. Now, with all of those things into consideration, now we have a concept, a high level sort of blueprint, right, a blueprint that we can think of and reason about. Now, we've actually started to see some tools that allow us to then think of this blueprint as something that we can actually, that we can take action from. And this is basically what is converging to the name of a canonical stack, right? Is what are the tools that I use in each of those blocks? Right, so I have an architectural blueprint, now what are the fancy names, the tools that I'm going to be using for that? And what is the relevant tools that are gonna be fit for purpose for my particular use case, right? And I think at that case, once we start converging into an understanding of the architectural blueprint components, we can start creating the contracts that allow us to state what we expect from those, right? The base sort of requirements in order for us to ensure that one of those components is reliable, right? So this is something to now reason about in how to kind of like start delving a bit more detail, right, and we talked about the high-level principles, the regulations, the governance, the governance frameworks. This then starts mapping into that, right? It's the question of what is your architectural blueprint that then allows you to be compliant to whatever is necessary at the higher level. Now of course, now you have something running, there's the question of how do you then, how do you then ensure, you know, robust operation, right? And of course, when it comes to general software, that would be through monitoring. And this in itself could be thought of as, you know, machine learning reactive monitoring, right? Like dashboards, reactive in the sense that you would look at it, right? So this is things like performance metrics and tracing, so things that you have in standard software, you would want to know what is your request per second, your throughput, your latency, but that affects your machine learning as well, right? It's like, perhaps you see a dip in throughput or in latency, that may mean that multiple of your models are being sort of concurrently used within your hardware, right, so that's perhaps something that you may need to think about. There's also statistical performance, so understanding what is the machine learning performance your models not just the operational side but perhaps things like accuracy right and the thing is this is more challenging that you would think given that you don't have the actual labels in production right so you have to think about well how do I approach that so if you're curious again you know just a reminder that we have like a link at the bottom there is a you know 60 minute 90 minute talk on production machine learning monitoring so if you don't want to dive into a bit more detail as opposed to just you know hand way be high level as in this talk you know do check it out and then of course you you delve into the drill down on the metrics, right? You also have not just the real-time metrics, but the ability to be able to ask questions of, you know, what are my distributions? And am I seeing something that is perhaps an anomaly in terms of the data that I have? And then finally, you also have the question of explainability and interpretability, right? It's like being able to explain why a model actually predicted something. And, you know, there are some interesting talks that are delving specifically into each of these areas. Now we move into observability, right? This is less of the reactive and more of the proactive. This is on the things like alerting, right? Instead of staring at a dashboard, it's how can you create, you know, more sort of meaningful automated means of understanding when you have to look at something, right? So alerting on top of those metrics that we talked about, things like service level objectives and service level indicators, right? To be able to say, hey, this is my commitment for the metrics. I expect that it's not going to be lower than an 80% accuracy or that my throughput is going to be consistently on X number of requests per second. And if you kind of like break that objective, then you get notified. Things like progressive rollouts is basically saying, hey, I have this new machine learning model version two that I want to deploy, but I don't want to deploy it by putting it there and manually press the button. I want to be able to put it there, run a test, and if basically it runs correctly based on the heuristics for a day, then it automatically actually rolls out without interruption. And the final one is things like drifts and outlier detection, being able to, you know, identify concept drift or data drift, you know, in a non sort of like manual perspective. So those are some things to consider, you know, in terms of the monitoring side. Now let's delve into the concept of, you know, what we have, right? So whenever we're now talking about machine learning models, we rarely now deploy a machine learning model. We're now actually productionizing a machine learning system, right? The reason that we mention it this way is because now you have to consider that sort of data flow consideration, right? You have not just a single model that you're dealing with, but often you have a multi-component machine learning system, right? And this is an example from a recent paper from Meta on embedding-based retrieval, so also search and they show kind of like their multi-stage machine learning sort of system that allows to be able to you know retrieve and search right and you have basically the indexing creation on one side that happens offline then you have kind of like the different sort of stages of the retrieval etc etc so the ranking and this actually shows you right that you're no longer just putting a single machine learning model into production now you have to consider how do I then manage from a conceptual perspective something as complex as this and this is why you start having to consider those questions about what is my architectural blueprint so you don't end up with like a single sort of like monolith for every single one of your of your you know productionized machine learning systems there is also an interesting area in the intersection of data engineering and machine learning engineering right and this is something that is that is not yet touched upon as much but in the data engineering world there's the concept of data meshes so this basically kind of like stating that you know the concept of the data lake that was sold as the silver bullet back in like you know 2015 or I remember but it was supposed to be the solution to all problems and then they realized actually yes centralizing everything doesn't really make sense especially because that then means that you don't have flexibility to then have specialized use cases right so then this concept of data mesh basically says hey now you actually do have a data lake to dump all your kind of like raw data but then you require hubs of specialization and this is something that we're now seeing kind of in the intersection also with machine learning because that's a similar requirement right you can't just have kind of like a single sort of like ML platform that just serves everything in a very rigid way just because the amount of overhead and red tape that you need to have for use cases is often proportionate to the risks involved right so that also means that you don't have to have that level of scrutiny for every single thing that you deploy right so this is actually a very interesting area that is going to be growing in the next years that intersection of data engineering and machine learning engineering and the last thing that we didn't touch upon is the question about metadata right so metadata is important because that's the ability for you to know what what are the digital assets that I currently have running right so what are the the machine learning models that I have in production and this goes into the question of things like what is the risk involved with each of the models what is the modality of each of the models both for discoverability but also for you know risk assessment this way it allows you to be able to understand where should you actually focus your your your efforts but also it's not just for you to be able to you know ask the question of what do I have there but we talked a little bit about the question about when you have your data being processed in inference time in production you want to also be able to use that data for training later on but that data may have a different shape it may have a different complete different concept right because now you don't have just one model but you may have like a pipeline of multiple sort of models with an input shape being very different to the output shape right so it's the question of how then you access the hops of between this this of the models so yeah again you know this is something that you can dive into into more detail in the in the talk for end-to-end machine learning metadata in systems but this is a very interesting component as it will become an open question how do we then standardize the metadata of our data and of our models in itself and the final thing to cover in the in the technological side and the trends is the question of security right and when it comes to machine learning in itself we're dealing with end-to-end lifecycle right you have the data cleansing the the model training and optimization and then the deployment and fine-tuning that we just discussed. One question to think about is, well, what part of this pipeline is susceptible to security vulnerabilities, right? And the challenge is that every single step of your machine learning lifecycle is susceptible to potential vulnerabilities and exploits. And for that, one of the things that we started doing is delving into the question of what are those vulnerabilities. So we've published what is now a resource called the ML SecOps Top 10, which is analogous to the OWASP Top 10, which is like the open web application security project, which explores kind of like on an annual basis the top 10 most common vulnerabilities in web development, right? Just like SQL injections, you know, things like that. So we've explored a similar one for machine learning and we basically delved into a showcase of how to exploit every every stage right so so this delves into things like loading model artifacts accessing models the dependencies across your machine learning models the code vulnerabilities of the servers the runtime images and the infrastructure so just kind of i want to see a show of hands so who here uses pickles for for your models nice okay well so check out the talk please But yeah, so these are some of the things to explore on the questions of why do people actually tell us all the time that using pickles is not safe? So you'll find out. Yeah, so okay, so this is on the technology side. Yeah, and of course, the talk is kind of like at the bottom. We're also running a working group on machine learning security with the Linux Foundation. Anyone that is interested, please join. Our next session is next month. So now the last section is organization. organization-wise, how do we approach this as a team, right? As your team in your company, right? So one of the things that we're seeing is at the company level, there is this trend that is kind of adopting concepts from the traditional software development lifecycle, right? Your SDLC that you have to follow and that is painful, often, you know, for compliance, right? But when it comes to the machine learning side, this machine learning development lifecycle, the MLDLC approach, is very different, right? Because you cannot have like a blanket process where every single use case is going to have CICD pipelines or every single use case is going to have this specific thing, particularly because there are cases where machine learning teams are embedded as analysts within business units. Other cases where actually machine learning engineers are embedded within medium-sized companies or low-risk use cases. And there are others which is like large case, sort of high risk use cases, right? So this actually emphasizes the need for, of course, this architectural approach, architectural blueprint approach, but with the flexibility that is, you know, as we have seen, risk-based, right? So that's one thing that we have seen as consideration. The second thing is the need to shift the mindset, the mindset from projects to products, right? And this is often seen in the world of data engineering as the topic of, like, data products, right? So it is a similar extent where you have machine learning products, right? And your machine learning system, your machine learning model itself can also be visualized and thought of it as a product in itself or as a service in itself. And that is now seeing a trend where now there is specialized support from product managers, project managers, in the same way that we're now seeing that for data products. The next thing is to be able to ensure you're able to map kind of like the different sort of technology roadmaps from the outputs into your business outcomes. And this is something that we are seeing kind of, especially in larger organizations, mapping those roadmaps of the machine learning platform, the machine learning engineering and data science teams, and then the actual iterative tooling to be able to kind of like improve the flow of each of these, you know, different areas and teams. And then, you know, finally, in terms of the actual, you know, cross-functional capabilities, this is, again, something that we're seeing emphasized in terms of the thought process of thinking this machine learning as products, right? So this embedding of the teams as squads to be able to continuously and iteratively deliver on those capabilities in the same way that you would have in, I don't know, like a full stack use case. And then the final things that I want to mention in the organizational piece is how are we seeing also organizations approach this, right? One of the things that we're seeing is that whenever their machine learning and data science practice is smaller, the type of roles that are necessary are very different, right? So whenever you have like only a few models, you have kind of like a set of data scientists that are all being able to have the full stack capabilities, right? Like full stack data scientists. I don't really like that term, but, you know, let's just think about it like that. As you have, you know, some more models, you start requiring the needs to specialize that role into the machine learning engineer, like that individual that is responsible for productionizing the pipelines and dealing with some of the operationalization. And then as you have those many models, that's when you start breaking down into the need for those machine learning platform engineers, right? The MLOps engineers that are serving, you know, multiple sort of machine learning engineers, each with a use case and then, again, each for a group of data scientists. And also as it goes, you are increasing kind of on the number of, you know, automation, standardization, control, security, observability, and reliability needs, right? So you go from a data scientist kind of like running everything on their SageMaker or kind of like Jupyter Notebook into something that actually needs to scale significantly more. And we're seeing this also kind of like at the sort of relationship between those teams, right? So whilst we have kind of like those data scientists serving kind of like a use case, Then you have kind of those machine learning engineers being able to serve the productionization of that use case. And then those use cases can be thought of like use case pipelines that are able to continuously deliver more value from those kind of like intelligence or models that are being productionized. And this mindset of the products themselves, of the machine learning as product, is what then allows for this to iteratively deliver value that reduces that manual input with, you know, the combination of that SRE mentality that is brought into machine learning operations, but also the product mentality that is brought into the higher level use cases. So, to wrap up, you know, we've already covered a lot of content, so I do recommend you to go and check out what you find interesting. I do want to kind of like remind everybody that not everything can be solved with AI, right? So when you're running around with a hammer, you know, everything looks like a nail. And I know that right now there's, you know, so much hype around machine learning and I'm giving a talk on production machine learning, which is ironic. But yeah, it's just to emphasize that a lot of the cases, you know, you just don't need machine learning. You can just, you know, do it with, you know, other tools, right? So this is just kind of like a reminder to ourselves and even to myself, right? And the final thing is that to emphasize the importance that we have as practitioners, right? Because critical infrastructure is increasingly depending on machine learning systems. And, you know, people in this room are involved in the deployment and creation of these models. And regardless of the number of layers and abstractions of software and machine learning, you know, the impact is always human and to the individual. So that's just something that I want to, you know, kind of like, you know, finish with. And with that, yeah, thank you everybody for joining this talk on the state of production machine learning. And if we have time, we'll take some questions, if not, in the social today. So thank you, everybody.

Speaker 2 [28:01]

Thank you, Alexandre. We have two minutes left. OK, perfect. If you have some questions, time to show up. There is no question on the slide, though. Yes? Yeah, you had one slide about regulations and how software can force them. Can you give some examples? I will repeat the question. So can you give some examples about regulations?

Speaker 1 [28:24]

On how software and the stuff

Speaker 2 [28:25]

And the software frameworks.

Speaker 1 [28:27]

Yeah, so yeah, I mean, the question of regulation and how software frameworks can enforce it. So it's not specifically of software frameworks enforcing the regulation itself, but it's more around the software frameworks abiding through the regulation. And when you think of regulation, even the name regulation is like in biology, in organisms, to regulate, right? To avoid kind of like the organism to consume itself into destruction, right? So it's in place to avoid malicious players to take advantage of loopholes. And regulation, especially within AI, what it's doing is it's basically saying, hey, look, we need to see some best practices and even more when it comes to high-risk use cases. So in terms of the technology involved for those high-risk use cases, there needs to be, you know, reproducibility, explainability, et cetera, et cetera, auditability. So, yeah, it is more kind of like regulation of the technology as opposed to through the technology. Yes.

Speaker 2 [29:28]

Thanks. Is there any other question here?

Speaker 1 [29:30]

Yeah, so the question is, so there's a mention of explainability in the talk, but it's mentioned in the context of monitoring. So how to understand the quality of explainability as well as why is it in monitoring. Yes, that's a really good question. So it was it was currently on the slide of sort of reactive monitoring in terms of like looking at dashboards so this is sort of like thought in the sense of being able to have a model that run ran perhaps a Inference input and a domain experts wanting to ask questions deeper about that specific Inference instance, so it could be you know two ways. The first one is of course course, you know, the individual manually actually going and requesting that explanation and then being able to judge it, but also one of the things that, you know, we had been exploring is the question of whether the outputs of the explainability models themselves, because explainability tools often are models themselves, right, with outputs. So the question of whether you can actually have the explainability models actually running in the background to all of the models, and then if there are particular sort of thresholds or heuristics that are met, then that actually gets flagged for somebody to go and have a look. Now in terms of the quality of explainability, so that's an important question because of that same point, right? Explainability themselves are models, and they themselves not only can be biased, but can be wrong. So it's important to be able to also bring that domain expertise to understand what type of explainability techniques are pertinent for the particular use case. So it would be very specific to the type of models that you're dealing with, but it would require just the same level of scrutiny and process as you would be training and evaluating a machine learning model. Yeah.

Speaker 2 [31:35]

Thank you.

Speaker 1 [31:37]

Okay, thank you very much, everybody.