Production ML across 2015-2035: A Journey to the Past and the Future

Speaker 1 [00:37]

Hello, before the last session, we have the last talk here. It's production machine learning across 2015 to 2035, a journey to the past and the future. Let's give a warm welcome to Alejandro.

Speaker 2 [01:06]

Great. Great, great, great. Welcome, everybody. I hope you're having a great conference so far. So, today we have a very interesting topic. We're going to dive into production machine learning across an entire two decades. So, the past and the future. So, I want to have a show of hands here to get a feel. So, raise your hand if you have ever operated a production system, right? It doesn't have to be machine learning. Production system. Yes? Good. And then raise your hand if you have operated a production machine learning system. Let's see. Okay. So there's a lot of pain in the room. I can already see that a lot of this stuff is going to resonate. So it's very good as this will actually be reminiscing. A little bit about myself. So I am director of markets AI data and platform at Zalando. I am board member at the ACM and AI expert at the UN and the EC. So at Zalando, we are a leading fashion organization, 18 billion Euro GMV, 61 million customers, 2 million articles. So the reason why I mention this is because through this and many other roles, I have been able to learn the hard things about the hard things of ML, especially production ML. And I want to actually take you to this journey. A shameless plug, of course, we are hiring. If you know of Zalando or want to know about Zalando, check out some of the roles. We're looking always for great people, especially in the ML space. Particularly some of the use cases, right? I mean, I don't want to delve, but within organizations of this size, AI is used across the board, right? Things like search and recommendations, pricing, fraud detection, and now with the shiny gen AI, there are now starting to be some use cases like fashion assistant, conversational AI, machine translation, et cetera, et cetera. So some of the use cases here, we're not going to talk about them directly, but we're going to talk about the underlying foundation to enable this kind of engines. So what we're going to do is we're going to take this journey, right? There's going to be a lot of AI-generated pictures throughout the presentation, of course. We're going to go through the genesis, the messy innovation, the LLMOPs era, that's how I categorized them. I found some high-level, you know, kind of like blocks and trends, and then from there we will actually dive into some reflections after covering some of the outlook. So let's actually dive into each of them, right? So we're going to first start with the journey to the past. So jump into my DeLorean, right? We're going to first travel to the past. I don't know how many of you know that reference, right? But indeed, we're going to start with each of these in themselves. So let's start with the Genesis, right? Where it all began. I mean, it probably began much earlier, right? But I'm just going to claim that it began there, right? So we started to realize that actually the life cycle of a machine learning model, it doesn't finish once you train them. It actually begins once you train them and once you start using them, right? Once your model is deployed, once the first outliers start appearing, once your models start drifting and your accuracy starts falling, and once you actually realize that you need to supersede that model, that model has, quote, unquote, expired and is no longer fit. So this is one of the initial realizations, which was in a way arguably solidified in one of the most popular diagrams that you've always seen in every single presentation on production machine learning, which came from this paper. So high hidden technical debt in machine learning systems, which basically brought this image. That said, machine learning is that tiny little blip of sand, and the operations around it is what is needed to actually drive the value of those machine learning systems. Now this also brought in the question on the taxonomies, the data dependencies, the hidden feedback loops, configuration debt, entanglement, et cetera. If you're interested, I mean, I recommend not just this reference, but many of the references that were going through the presentation. So we started with some of the titans defining this architecture. So one of the first few ones was Uber's Michelangelo, which was the platform that came and transformed and set basically some of the high bar. I mean, maybe some of you wouldn't remember back then, but this was a big thing for folk in the machine learning space, saying, okay, you can actually have a platform that covers the end-to-end, right? Covering both the batch, but also the real-time, as well as also the metadata management, the continuous delivery, the coding environments, Jupyter Notebook environments. So back in the day, revolutionary, right? And many of other platforms started to follow, right? SageMaker. You probably heard of it. Love it or hate it. Started also in 2017. Other companies also came in. That's back in the day where TensorFlow used to be cool, right? And we also started to see things like TFX, et cetera, et cetera. So some genesis kind of like view from that perspective. And then we started asking the question, okay, well, so we're starting to see these different components in the production machine learning stack, right? But what is that architectural blueprint? What are those components that we can all all agree on, that we can actually solidify into a set of contracts that then we all know they are ubiquitous in any production machine learning pipeline, right? So this is one of the ones that I actually drew back in 2017. It takes basically your training data, your artifact store, and your inference data. What that means is you basically build models using training data. You create artifacts, right? Those artifacts are created through either interactive Jupyter notebooks or through distributed training like hyperparameter optimization, et cetera. Then you have those artifacts that need to be provided and used, whether it's for real time or batch processing. And then finally you want to add a level of monitoring to it. As part of that, you have now data that is processing through production. That is basically data that, you know, is going through, again, potentially back to your training data, and then you have also the metadata that flies horizontally across your stack. So, I mean, this was some of the first kind of like, you know, few iterations of thinking about what could that architectural blueprint would be. You know, we're much at a better position, for example, at Zalando, we have defined and we are defining what our architectural blueprint is at an organizational level, and then making sure that kind of like teams are able to benefit from that. Now, there was also other emergence, right? So, we started to see what I like to call flawed machine learning, right? The acronym is better, right? So, flawed machine learning, right, started to see that when you put machine learning in production, the challenges that you face are much harder than the ones that you would face with traditional software. Not just because we have probabilistic systems, not just because you have complex data dependencies, these complex graphs, and also reproducibility considerations around your code, data, and configuration. But also, you have security risks. Now you have challenges of potential injection at the artifact level. You have dependencies that your scientists are now realizing actually, well, security is also an important element, code vulnerabilities, and traditional and non-traditional potential loopholes. So obviously, this is where other initiatives started, like the committee that I actually started back in 2017 on machine learning security with the Linux Foundation, which actually came in a timely manner after the OWASP published their first ML top 10 initiatives. So this is an interesting set of insights of basically when it all started. Now, we move to the messy innovation, right? So yes, OK, we have the foundations. We've set some of the taxonomy. We set some of the guardrails, like where some of those components can be classified. And then what I categorize 2018 to 2022 is the golden rush of ML Ops. That was actually a pretty fun era. I loved it. It's when we went from this, of how it started, to this, right? So, all of the vendors that started appearing and arising. And I want to just point out, this is back in 2021, right? So, this chart is much bigger now, right? So, there were some major milestones as well, right? So, we had some of the releases like Databricks coming in with MLflow, right? MLflow being experiment tracking. And then others followed suit or, you know, came more or less at the same time, like Weights and Biases, DVC, et cetera. Netflix also brought in model orchestration with pipeline coordination, Metaflow, right? Came out of Netflix and then was open sourced. And then others followed like Ray, Prefect, Flight, Argo, et cetera. So there's actually a reemergence on Prefect, for example, as they're the ones actually developing the fast MCP. Then other tools like Kubeflow came out. We were actually part of the core team within Kubeflow, particularly on the machine learning serving. So we're going to talk a little bit more. But there was also the race about different components. So the orchestration with things like Kubeflow, obviously Airflow, et cetera, et cetera. But also the race towards model serving, real-time model serving. And when it comes to that latter one, this is one of the more interesting parts, right? This is where I was hitting my head the longest throughout this part of the journey, is how do we crack realtime machine learning, right? And by that, what we mean is how do we make sure that a science team can go from experimentation to production in a way that is consistent, standardised, and that you get all the free food by not having to have all of this customized custom fast API wrappers every time that you want to deploy a machine learning model. I can tell you that right now, fast forward to 2026, we kind of still haven't figured it out, but back then, you know, we managed to actually progress quite a lot. The reason why it was so hard, it was because each of these different components we realized that was obviously very different to a traditional microservice. In this case, there is semantic context in terms of the inputs and outputs. You benefit by knowing if it's text, if it's image, et cetera, et cetera. And I would argue the reason why we have been able to advance so much more when it comes to the agentic systems, which we'll talk a little bit about that, is because that tackled a vertical slice. It said we're going to look just at generative text, and we're going to look at generative text across the entire stack for inference, whereas here you're trying to tackle absolutely everything. So, there were a couple of different, you know, attempts, schools of thought, approaches, etc., and also different interesting perspectives on that. So, with that, there was also the question of we have now real-time machine learning in production. How do we monitor that? How do we ensure service performance, machine learning performance, explainability, outlier detection, drift detection is also standardized? Again, a couple of players started coming out, and then And the industry was abstracting and exploring abstractions on how do we standardize this into architectural components, right? Machine learning monitoring is another of those areas that, as of today, still not cracked, right? There's not a single silver bullet that will just be, like, you plug and play, and then you sort out, you solve all of your end-to-end machine learning problems, obviously on a slice level that is solved. 2020. Something happened. Can somebody tell me what happened in 2020? Unexpected accelerator. Anyone? COVID. We don't remember COVID. That was, you know, an era, wasn't it? So the COVID pandemic kicked in. And that actually, like, pushed forward this already fast pace into another level of acceleration. So we started to see a large amount of some of these either vendors or systems coming out to a larger extent and driving either much more volume within their customer base or whatever, or actually new players coming into the space. So the need for machine learning also scaled massively, right? Like a lot of organizations were trying to capture the value of ML within their domains. So within that as well, we started solidifying the domain of ML Ops. Finally, we started to see some consolidation. And this is where I would see the nexus, right? A lot of startups started being acquired. A lot of basically end-to-end platforms started to emerge like Databricks, like SageMaker, et cetera, et cetera. So, now we go into basically the other comet that we were not expecting, the GenAI, you know, comet. Which then brings us to the last era, the LLM ops era, which is a little bit closer to where we are today. So to set the context, I mean, we all heard like thousands of talks probably in this conference about GenAI agents, so I'm not going to talk much about that, but this is when somebody thought about the idea, why don't we take a model, right, that is already connected to a system, but instead we just put it on a loop and we allow it access to other things, right? A little bit more involved than that, right? But at the end, this started unlocking not just a lot of opportunities, but it started bringing some of the foundations back into popularity, right? The boring data is something that started being brought back and said, okay, how do we make sure that we avoid this garbage in, garbage out, as now we can get the acceleration at a significantly larger level. This acquired a set of new protocols that are driving standardization. And this is where I argue that a new stack is emerging. Similar to back in 2015 when MLOps genesis was starting, this is where we started to see a LLMops new stack coming out. This strategic investment into the boring data as well as standardized interfaces is what is leading into an explosions of protocols. So, obviously, we've all heard about things like MCP, right? But there's all of the letters of the ABC, right? There's A to A, there's blah, blah, blah, right? So, there's now an interesting era of messy innovation that is happening. And I would argue that we're going through that same journey in a very similar way, as I also mentioned. Like for the MLOps era, we're also seeing a similar set of security things surfacing, right? I mean, probably some of you heard about the announcement of Mythos, right? I had a bit of a controversial take, right? That was like, okay, yes, the next model, but all of the potential risks were already in place, right? And that's also something that we're starting to identify, that many of the cybersecurity vulnerabilities would have also been identified through less powerful models. And the way that I see that is now that we are catching up with the technology that has just gone running, we're starting to see the potential that it's unlocking. And part of that, it really is challenging the status quo and the societal foundations that we have built so that, you know, the technology that we have is sustainable and stable. So as part of this, we also have started to look a lot into things like security for agentic systems. Together with OWASP now, we published a set of taxonomies that allow us to define some of the challenges from a security perspective that you tend to find in this LLM ops stack. So things like data poisoning, memory poisoning, et cetera, et cetera, et cetera. So that's an entire talk in itself, right? So if you're interested, you can dive into that. And one ask that I have is let's please not make SQL injections cool again, right? As we've started to see a little bit of emergence of that, and that's a trend that we don't want. So as part of this emergence of the agent stack, this is another attempt to standardize it, right? So, as part of this connected data ecosystem, we have the different components like agent orchestration. So, actually, yeah, I just recently released one framework on that front called Chaos. There's also the different MCP routers or protocols for communication. There's the guardrails, moderation. There's the gateways for routing, like LLM. There's model orchestration, which is what you would actually have a little bit more of your traditional ML ops. There's the hardware scheduling, right, especially if you have models in-house. There's the observability and also the security and metadata. So that is kind of like maybe like a take into the new stack as we start to see it standardized. That I presented last year I think is going to evolve soon. Now, there are some current limitations. We actually ran a survey on the state of production machine learning operations, which you can actually find the results. As part of that survey, we found that we are still not there, right? We're still very far. We still have about 50% of organizations without machine learning monitoring. We still have a large heterogeneity of providers when it comes to model serving, right? So we are still not there yet. Have we progressed? Yes, we have. And it's been quite a fun journey. Are we now opening another can of worms? Yes, we are, right? So, I think we need to make sure that we're progressing across both of those eras as we actually dive. And this is, you know, one of the examples of, you know, for production, machine learning monitoring, you know, what do you use, you know, 50%, none, right? So that is the story so far, right? All the way to this year, well, 2025. Now what I want to do is try to go to the future, right, and this is something that I don't really do very often, right, because I, you know, probably will get 90% of this wrong, but why not, right? So, let's make some predictions, right? Like, what's going to happen in the next 10 years, right? Of not just MLOps, but production ML, right? And this including kind of like also other agentic systems. So, the way that I thought about it is I want to give a couple of predictions, right? And rank them on high, medium, and low, right? So, I would say 2027, you know, not too far away. A little bit, you know, into the future. You know, the certain piece is that most organizations will have hired their machine learning operations engineers. They will have already introduced at least one personal agent across their developer ecosystem, and maybe even non-technical. And then the less probable unified standard for machine learning monitoring emerges, right? So this is something that I have been wanting for a long time. I want to give a shout out to OpenTelemetry as they are now introducing at the very least standards for Gen AI telemetry, right? And we're starting to see some great advances there. And I also think the LLM ops heterogeneity will peak next year. So consolidation begins. Median time to production decreases to less than a month. The survey told us that the actual median time to production is still more than a month, right? Sometimes three months, sometimes six months, right? So this is really something that to me is absolutely crazy, right? Like the fact that it takes us that long to put a machine learning model in production, obviously, in corporate environments, right? But we have seen some acceleration on that front. Now, 2030, right? So, by 2030, the title is LLM Ops Unifies. So, most organizations have a central machine learning platform, right? This is one that either they build or buy. Inference subsidies stop, right? Cost per token peaks, right? I mean, so, this means, right, like, you know, the amount we pay for our cloud code tokens, right? Right now, it's not the full pay because there's a lot of backing behind that. I mean, maybe, maybe not, but my prediction is indeed that we'll see a little bit of that. Perhaps then, popularity of self-service models starts exploding as well. And this is where I see a new stack emerging. So right now, we're talking about LLM Ops. I would expect a new stack. I don't know how to call it. Autonomous Ops, Auton Ops, I don't know, right? New stack emerges, right? And machine learning monitoring finally becomes ubiquitous through the standards, hopefully 90% as a MLSRE side. Time to prod decreases as well. 2033, operations for autonomous systems. So this stack starts standardizing. And this is where we have already seen glimpses. Even within our group, we saw some interesting projects with auto research applied as the real auto ML. And to me, that's a super exciting field. So definitely, this will be where we're starting to see intelligent progressive rollouts, self-healing systems, root cause analysis, and system optimization also finally started to be done through maybe not agents, but support agentic augmentation, let's say. And then the ML Ops and LLML Ops stacks start to align. And then finally, I mean, 2035, you're making stuff up. I mean, it's too far into the future, even 2030. But self-healing systems commoditized, next bubble. We started to see another bubble, hype. Again, everybody starts talking about these self-healing systems in Christmas dinners with the family, right? Autonomous rollouts begin to reach mass. And then finally, GPU and inference costs are optimized, right? So that's going to take some time. So I'm still waiting for AGI, right? That's my prediction. That is where still glimpses are not yet to be seen. Good. So the final section, some reflection, right? When it comes to ML Ops, when it comes to LLM Ops, right, this is always proportionate and relevant to the context and the time, right? Not all teams, not all data science teams need to have fancy end-to-end machine learning operations platforms, right? It has to be basically also relative to the maturity of the stack. So for some small number of models with a small science team, they can self-service. Larger number of models, maybe machine learning engineer is able to support many, many more models. Maybe the organization can actually standardize the MLOps engineer, right? So this is actually an entire talk in itself, so I've given a little bit more of an organizational shape as kind of like your machine learning operations evolve. Now also another takeaway that I want to mention is that as of today, right, we see that there's a lot of things happening throughout the software development pipeline, right? So writing code, testing, operating. There's a lot of tools that are emerging for producing code at the top of the funnel. But there's not as much at the bottom of the funnel, right? So that's why we're not seeing increases of productivity gains. That and other reasons, right? Things like your cloud code, et cetera. But as you go down that funnel, you start seeing less and less maturity on those tools. So a call to action to us all is how can we actually invest in the tools that are needed to actually accelerate the end-to-end. And of course, you know, make the humans happy again so that they are not just reviewing code, right? They're also, like, creating things. And then obviously when it comes to this, there's always an element of going from low complexity, low scale, to actually larger scale, but with added complexity. Now we need to move back kind of like the next level of scale, but also, you know, increasing the complexity, right? So this is really something that I like to think about it, right, for us to think through these waves, right? So first it emerges, you have your messy innovation, you have your consolidation. And that's it. So if you want to actually dive into some of the other talks, so these are links, right, to previous talks. This is a compilation of that. If you want the slides, you can access them through the link on the top right. And with that, thank you all, and I hope you enjoyed the rest of the conference. Thank you all.

Speaker 1 [25:06]

Thank you so much. Thank you so much, Alejandro. There are some questions. The first one is why did Zalando stop maintaining the Spillow? Do you know? The what? Spillow. I don't know what it is. It's an open source library. I don't know.

Speaker 2 [25:25]

I don't know. So I can tell you I have no clue. So maybe you can come and chat afterwards so we can find out what that is.

Speaker 1 [25:35]

Okay, the next one, do you think the chat language interface will dominate the stack or are solutions that are more integrated with the data interface directly will take over?

Speaker 2 [25:49]

That is such a great question. So so so in other conferences what I do is we need to go beyond the chatbots, right? Absolutely. No, the future is not a chatbot. That would be a very sad future, right? So I think this is just a stepping stone, right? Which is just gonna be legacy We're gonna look back at it and be like, oh my god, you know, that was that boring, you know 1990 something websites, right? So I do believe it will be integrated workflows, right, as part of experiences that are much more personalized, and not just from a coding perspective, but also from a user experience, right, when going through, whether it's like shopping or any others, but yes.

Speaker 1 [26:28]

Okay, thank you, and what do you think will be the next big shift in MOPs, and where do you think most teams are still underprepared?

Speaker 2 [26:44]

Yeah, so I mean, I would say that's very interesting, and it's also a little bit of what I referenced. So I would say that as the waves start, they start to mature, and they start to consolidate. So the MLOps side, I think it's already starting to consolidate a little bit more. We will be, as my predictions, let's call it, we'll be picking on this LLM ops explosion, maybe in the next couple of years or in the next year or so. I do think that that stack will also standardize and mature, but we will obviously see as we actually catch up with what is possible with that, and once we actually reach that level of maturity and scale, and we actually know how to build that next level of abstraction, we will unlock what could be that next stack, right? So I called it this AutonomOps or AutonomousOps. I don't know, right? But there will be something that goes beyond that that I personally am super excited about, And it will see a similar wave, right, I expect, yeah.

Speaker 1 [27:45]

I think the next question is similar to what you said. What about the predictions if the AI becomes so autonomous that it will deploy itself?

Speaker 2 [27:57]

Yeah, I mean, I would see that from a non-existential perspective, right? I mean, I'm really like, look, I look at the current state of the production machine learning through the survey, and just seeing that we haven't even reached beyond 50% of production machine learning monitoring, and you're telling me that there's going to be a Skynet that just like self-heals itself when they don't even understand like, you know, what's going wrong, right? What I would say is having a setup where you perhaps have a self-healing system or a automation on productionization or even automation on experimentation, right, and you actually do the entire workflow of a science pipeline, that just means that you enable your scientists and your domain experts to operate at the next level of abstraction, right? So for me, the real question is, what can we build at that next level of abstraction, right? It's not how can this AI just like replace absolutely everything. It's more of that former, yeah.

Speaker 1 [28:52]

We take one more one last question How do you see a machine learning governance process in the coming years giving speedly MMA growth and how? organization like EU and Union are responding to it. Yeah the governance. Yeah. Yeah

Speaker 2 [29:11]

Yeah, you know, that's interesting. So, I would say, obviously, we know the EU has been one of the first, yeah, kind of ones to tackle the governance from a regulatory standpoint for AI, right? And there's already a lot of organizations that are starting to introduce some of that governance. Examples include the fact that now you need to have what they call an AI inventory, right? So it basically means you need to know all of the machine learning applications that you have deployed. And if you ask me, I mean, that sounds basic, right? I mean, should we know what AI models we have deployed? Yes, right? So I think for me, it's certainly something that you could also see as how do we make sure that we are scaling best practices in a way that we can...