Maximizing Efficiency and Scalability in Open-Source MLOps: A Step-by-Step Approach

Speaker 1 [00:05]

Welcome to my presentation. Glad that so many came and are interested in MLOps. I appreciate that you didn't read out the title because it's actually a lot of stuff in there, I guess. And you probably all wonder what this talk is gonna be about. And I hope to clarify this a little bit more during my talk. So it's called Maximizing Efficiency and Scalability in Open Source MLOps. And yeah, what this is all about and what efficiency and scalability are and how you can use them for MLOps. And that's what basically I want to explain to you today. So I want to kick things off with a plot or with a graphic that came out already in 2015 in a paper called Hidden Depths in Machine Learning Systems. So I just ask all of you who knows this and who has already seen this plot before. Okay, seems like I would say half. To me, it's a very seminal paper, which is already eight years old now. And basically the point that the authors at that time made was that if you want to put machine learning into production, there's actually very little component that is only about machine learning. So you see this small little dot, this black box in there, that's your ML code. And then there's basically 95% of stuff that you have to build around it. And that's basically one of the big challenges since then. And I think this challenge still exists. And it's basically all about how to put machine learning into production. Just to give you some context from which perspective I'm looking at MLOps, I want to quickly introduce myself and my company. So a few words about me. I'm Paul. I studied systematic musicology and did a doctor fill in this domain. I did a lot of cognitive studies in musicology and so at the intersection of cognitive sciences and music which was very interesting but at some point I decided to move into data science i worked at as a data scientist in some in-house positions then later moved to ai consulting or in data science consulting and i'm currently leading the ai and data science team at data drivers and we are a small smallish consulting company based in hamburg germany which is pretty much one and a half hour train ride from berlin and the second biggest city in Germany if you don't know much about Hamburg. We were founded in 2015 and based in Hamburg and we have around 40 people working here and we do a bunch of stuff mostly about building data platforms in the cloud for various clients and to show you some of the clients we have been working with you I am I'm guessing you're probably familiar with at least some of them and what is interesting if you work in IT consulting and particularly if you work in something that is has to do with data science consulting and also AI or machine learning consulting is that you basically face the the challenge of building an ML production system for like very different kind of setups so basically every client has their own history every client has their own data their own IT system their own IT infrastructure so you basically learn a lot and you see many different things and also the way of how you build out the machine learning system that you want to run into production can be different of course because you should try to adapt this to the client needs. And what this means for us is that it's basically a very good way of also trying to abstract what kind of things matter and what is valuable and also sometimes some very tedious and heavy learnings that we had to take when building ML production systems. So I'm trying to build on this, on our experience, what we have seen so far, and try to give you some of these experiences in this talk. So basically what can you expect from this? A very opinionated take on MLOps based on our experience. And I want to kick this discussion off with basically a statement that I'm going to try to convince you of, and this is the following. So I assume that your MLOps problem is not so much a technical problem, but it's more an organizational problem. There was a blog post which I kind of came across very recently by Kostas Padalis, who claims that MLOps is 98% data engineering. I thought that's a bummer. I'm a data scientist trying to do MLOps or like at least I was kind of a data scientist and now I'm told that I'm actually not the right guy and this can all be achieved by data engineers but when you read through this blog post you actually are kind of convinced I would say at least I was kind of convinced so there are many things that you have to tackle if you remember like this Google plot with all these boxes a lot of these components have been already there so for instance things like data and pipeline management infrastructure management deploying and monitoring applications all of this has been there before before people were trying to put machine learning into production and people called data engineers or maybe software engineers, backend engineers, whatever you call them, they have been dealing with these kinds of problems for a very long time, and there are many standard ways of how to do this. But obviously there are these remaining 2% that are somehow important, so obviously you also need to understand machine learning and data science, and it's a lot more complex then just calling fit and predict on on a model and if you don't like understand the logic behind the models then obviously you are also kind of lost and not even to mention the business case that you always also need to take into account for instance if you think about monitoring monitoring the the relevant business KPIs for your model is very important and this has to do with your machine learning application and also with the business case you're trying to solve. But I think the main point here is that whatever the composition of skills are, the skills are available. So I think in most data teams, at least in many data teams that I have seen and in many companies, you have people who know a lot and who have different skill sets. But it is all a matter of combining these skill sets very efficiently, and that's going to help you a lot. And I think another point to consider here is that we don't need more tools, actually there is an abundance of tools already available. So if you look at this, this is a plot, it's a bit hard to read, but I can just give you I can just read it out. So this is from a blog by Chip Huyen, who is a professor at Stanford, or at least a lecturer, and she basically did a survey on the ML tooling landscape, and this is from 2020, so this is already like light years old in machine learning time. But But already since then, she collected more than 284 different tools for things like data pipeline, infrastructure, serving, all-in-one solution, hardware, model training. So basically all the components that are out there, basically you can find tools. And if you want, you can look for her blog, and she has a list. She even has a Google sheet with all the tools available. So I think it's not so much a matter of finding that there is a tool missing in your tool stack. It's more a question of not using the wrong tool for the wrong job. And so basically, what do you need to do? Obviously, you need to get your skills and tools organized. And how can you do this with efficiency and scalability? so now you might wonder what are these things then efficiency and scalability and how can they help you and i think like to start with efficiency you can think of these things as kind of design principles or maybe even some kind of meta design principles that you can use to make informed decisions. So we know efficiency already from systems design where we basically try for instance when we try to implement an algorithm so we want this algorithm to be as efficient as possible. It could be like a sorting algorithm but could also be machine learning algorithms so you want this to use very very little resources you want to be this you want this to be fast and and so on. You can also think of efficiency in the systems design perspective in terms of data structures. So for instance, yesterday I heard a talk about the Spark Connect component, and there were RDDs mentioned. RDD is like a perfect example for a data structure that is like very narrowed down to a very proprietary data structure, basically to make things more efficient. And then you can also think of efficiency in terms of parallelizing your work. But I think for MLOps, efficiency also has another meaning that is very important, and that's basically the efficiency of the data science developer. So usually a data scientist wants to be very quick. they want to try out things very quickly and see if something works or not and usually our work mode as a data scientist is not by having like very deterministic assumptions and then you know what to build but you have to try out many things you have to experiment with your features with your data with your transformations and if you have high iteration speed and high experimentation velocity. This is actually gonna matter a lot for how successful your whole ML project will be. So to put this even a little bit more bluntly, one very efficient system could be the Kaggle data scientist. You probably heard of Kaggle all, I guess. So like a data scientist who who is working on a Kaggle problem has a very well-defined business problem, has very well-defined data sources. You can just download your data from Kaggle as a CSV file and then basically work on your local machine and iterate very quickly on your own in your Jupyter notebook or IDE. And you can make very rapid progress in the way how you solve this data science problem. And you can also see it in the results of these Kaggle competitions that people are usually like the benchmarks are very high and it's very hard to beat each other and very close race. But obviously, there's also another twist to efficiency that you can put into the picture. So you can also think of maintainability as being a feature of efficiency. So, for instance, if you build something and then you need to change something, you want this code that you have previously developed to be maintainable. And the Kaggle data scientist probably has a hard time to, I don't know, hand over the notebook to another person or to maintain the code that he has been writing. And maintainability is also antagonistic to many other design principles. So obviously that's not the only thing that counts to be rapid, to be efficient, but there are other design principles that you might want to consider when building a product. And these are sometimes antagonistic to this idea of being very efficient. So if you think of robustness, of the robustness of your system, you don't want things to be You want to be sure that things are deployed in the same way. You want your code to be tested before you deploy it. So these are also things that will in the end kind of be relevant for your system but are somehow like slowing you down in your development speed. That's about efficiency. Now let's switch to scalability. we also have a notion that we kind of know from systems design so we have perhaps like some things like horizontal and vertical scalability in terms of compute resources so you can basically scale your system by adding more resources for some kind of task or you can make your resources bigger but also here i think we can make a twist to it and adapt this to the ml ops sphere and i think what here is most important is that scalability in terms of MLOps has to do with organizational scalability and the possibility to share knowledge across users and roles. So for instance it's also very important that you not only enable other data scientists to collaborate with you on your data science problem but also make your work in a way shareable with people who have other roles for for instance, like data engineers or cloud engineers who help you by deploying the solutions or who you are collaborating with on your MLOps problem. And yeah, then also a few things that also matter here are also things like automation, validation and versioning. So these will all factor in or like help you help you with with scaling things so automating tasks making them less like making work less redundant is obviously helping you in scaling your system validating data and like early on and validating your system is also very important and versioning will help you as well so efficiency and scalability can be thought of as some kind of design principles that you can think of that are important and they are somehow like moving into different directions I would say but they can help you make decisions so now you might wonder what is he talking about and how this can actually help so and I will try to make this a a little bit more clear in the next section, where we are actually talking a little bit more about tools and evaluating tools. So first of all, this is kind of the thing that we at DataDrivers believe in. So we think that for perhaps 90% of ML systems, the best solution is to combine open source software with cloud platform services. I could probably give another talk on why this is the case, so for now I will just leave it like that. But what I want to do is just to pick out a few of the open source components that we think are very important and that will kind of nicely add up to some cloud managed services that we are also using. Okay, let's start with the first one, MLflow. I guess some of you have already been working with MLflow or perhaps some of you are using this on a regular basis. We think this is a very, very good tool. Obviously, there are other alternatives to MLflow, but we stick with MLflow because it's open source, you can deploy it in your own cloud environment, and you can do a lot of stuff with it. And why is it good? So it is an experiment tracking tool. It basically allows you to compare different experiment runs for your machine learning application so you can compare for instance different sets of features you can compare different hyper parameter tunings or you can even compare training different models against and compare these experimentation runs against each other and then basically decide which results are best and which you want to then later put into production but it also it also provides you a model registry and it's very cool collaboration tool and now our friends efficiency and scalability can come in handy because we can now think okay is MLOps something interesting and I would say yes because it will basically enhance our efficiency so it will allow us to be better at experimentation it will increase velocity because of this experimentation tracking component and it will also help to scale us so it will also help us to be better at collaboration with each other because we can basically for instance see what other people are doing we can like we have different contributors who can like submit their experiments for the same problem and we can compare them so it basically provides a benefit for for these two components yeah just to mention one more aspect so what I really like about MLflow is the tracking is nice and the experimentation but what I really like are these two later components MLflow models and the model registry because this is going already like a step beyond just experimentation tracking where you have functionality of basically also reviewing what kind of model you want to put into production based on your experiment runs. So this is sometimes referred to as the data science pull request or data science merge request and it goes a lot into this direction of deploying models is different from deploying code because you basically want to look at visualizations you want to compare metrics and it's it's it's just different and Emma flow allows this to do to allows to do this and then another tool that we almost always use is terraform so terraform helps us to deploy infrastructure as code or to have our infrastructure as code and deploy this via Terraform it allows us to scale resources and also to be able to deploy this in various cloud settings so for instance it also helps you against like vendor lock-in effects this obviously again is not so much contributing to your efficiency because it will make things more complicated and more complex but it allows you a lot more scalability in terms of how you can like add resources or you can also like scale out to I don't know different cloud environments for instance and then the last tool I want to mention is feast which we also really like so feast is a feature store it's also open source and it has I think two features that contribute a lot to scalability and that's like the one thing is that it it allows you to scale to real-time serving so it makes it easier at least to go from batch prediction jobs to streaming jobs. And what I also really like about it is that it basically allows you to scale features across teams. And let me talk a little bit about what that means. So Feast is also considered to be like a feature store version three or whatever you call it. So it has kind of register for your features and I really like this register part about it because similar to kind of a data catalog that you perhaps might have come across it allows you to store and register features and especially if you want to scale out your machine learning system to kind of a like like to like collaborated with many teams or across different data science use cases you can basically share the same feature the same kind of transformation that you want to apply to data with many other data teams so for instance if you work on one use case and you basically implement a feature for this use case and then you I don't know start another project and you want to reuse this feature feast makes this a lot easier and will also help you kind of reduce a boilerplate and redundant code. Yes, so in the end I think MLOps is still hard and even like having seen many things and trying to put many machine learning models in production, I can say it's still very hard and difficult. And I think one of the reasons for why it is so hard is because we have all these contradicting design principles. So we want to be very quick in our experimentation but also we want to make sure it's robust and yeah this is hard to handle but I hope that I at least kind of convinced you that efficiency and scalability may help you to solve the organizational burden of MLOps because they can be used as a guideline for tool selection yeah so with that I can just say there is no single best approach to MLOps and I think finding your own way and finding what works best for you is like the the way to go and I hope like using or thinking of design principles and finding your way may help you deploy all your models into production. So thank you and good luck with your MLOps journey.

Speaker 2 [24:20]

Thank you, Paul. Nice overview of machine learning production and MLOps. We have some questions on the Slido, and you can put them on Slido still. If you cannot see them to upvote, I'm sorry because I'm approving everything. So put them, try to vote, and I'm going to ask the questions to Paul. So do you have a feeling that there is a serious fear of missing out in MLOps space, And in reality, people do not need most of those tools at all.

Speaker 1 [24:54]

Yeah, for sure. I gave another talk on MLOps where I basically said, like, beware of the model zoo. And I think that's very true because it can be very overwhelming. You see all these new tools popping up and you quickly have to decide whether or not this is something that is important or relevant for you. and this can be very challenging and I mean what we try to do is basically stick to as much to build things as simple as possible and to use as little tools as possible and to be very like thoughtful about putting some new model into our system.

Speaker 2 [25:35]

Nice, you don't want to be changing everyday tools and so on.

Speaker 1 [25:38]

tools and so on. That's not so nice. Yeah.

Speaker 2 [25:41]

So there is a question about cloud provider, which one are you using and Which service are you using to deploy a mental? Okay? Yeah good

Speaker 1 [25:49]

okay yeah good point yeah i didn't mention cloud services so we mostly work on gcp and aws and we are using both sage maker and vertex ai my feeling is that gcp is like the services i like them a little bit better for machine learning and for mlops so like if you have to make the decision i would probably go with gcp right now

Speaker 2 [26:15]

Okay, nice. Managed service is good to end. Next question, what is your recommendation for model serving with Terraform?

Speaker 1 [26:25]

I haven't served models with Terraform, usually we only serve infrastructure and the models are served separately.

Speaker 2 [26:31]

Yes. It's more for deploying infrastructure with the code. Yes. And next one, what kind of interfaces do you define between teams like ML flow artifacts or what kind of what interfaces?

Speaker 1 [26:46]

between teams that really depends on I guess the project so like and I'm also struggling if I get the question right so I don't think we have interface between teams or if it's like the deployments I think

Speaker 2 [27:05]

I think you can use, let's say, models as artifacts between the training part and the serving part.

Speaker 1 [27:15]

yeah yeah usually yeah usually so we have usually we have like docker images for the models and then basically but the model artifacts themselves we deploy then via either ML flow or sometimes we even use like SageMaker model registry

Speaker 2 [27:34]

Okay. There is a couple of questions about data tracking and versioning. What are you using?

Speaker 1 [27:42]

So, sometimes we've just built stuff kind of by hand, using also cloud-managed services. We've been looking into DVC, which looks very good. So, yeah, so that's probably my recommendation, but it's a lot of work.

Speaker 2 [27:58]

Yeah. True. There are a lot of bunch of flows, Kubeflow, Metaflow, MLflow and so on, but how would you go on about selecting a specific one?

Speaker 1 [28:11]

Yeah, I think these flows are very different from each other. Like Kubeflow is nice, but I think especially for teams who are not yet using Kubernetes, I wouldn't advise to just if you want to solve MLOps to go into Kubernetes because it's just too hard to manage. So you can use Kubeflow on GCP, which is good. But if you're using Kubernetes, then Kubeflow is obviously good.

Speaker 2 [28:45]

Yeah on top of it and Have you used fist within data bricks if so, how does it compare to data bricks building features store?

Speaker 1 [28:54]

That's a good question. No, I don't have any experience with data.

Speaker 2 [28:59]

OK, sorry, I'm picking the last questions. How do you choose between using features from the cloud provider, like the serving, tracking, and so on, and using OPRSource tools?

Speaker 1 [29:19]

That's a really good question. I think that's becoming more and more difficult because cloud providers are like more and more trying to Integrate functionality from open source tooling. So if you look at What Google has been doing in the last I don't know year or something They have like a lot improved a lot so that they you can also find like experimentation Tracking system kind of and you have you have model registry So that's a really difficult decision. I think the main benefit of using managed services is obviously that you don't have to manage them and that's a huge advantage and I would always kind of use this in doubt. So my advice would be to use open source only when you really need it.

Speaker 2 [30:11]

And maybe when you don't have the skills to manage by yourself.

Speaker 1 [30:15]

Exactly. Yeah, obviously. Yeah, if you have skills to manage all of this by yourself then go for it

Speaker 2 [30:19]

Okay, based on your ranking you seems to have optimized for scalability Was this a conscious choice?

Speaker 1 [30:31]

mean kind of like I think like after every project and every model that you want to deploy you would you have like the feeling that you next time you will do something different you take some learning out of it so I think for now we are like leaning more to scalability I guess yeah I mean there are other people who are like using notebooks in production which is actually a lot more towards efficiency but yeah we don't do this

Speaker 2 [31:03]

Okay, great. Thank you for your sharing your experience.