Managing the end-to-end machine learning lifecycle with MLFlow

Speaker 1 [00:03]

Yes, good morning. Everyone up and running from the party last night? Looks like it, so it's quite crowded. Yeah, my name is Tobias, freelance data scientist. You can find me on Depends on the Definition, where I write occasionally blog posts. I'm going to talk about the machine learning lifecycle and how you can set it up with MLflow. Yeah, a lot of the inspiration for the talk came from the freelance work I do and for my desire to sleep well and don't worry too much about machine learning experiments. And I hope this will set you up for or give you some ideas how you can integrate it into your personal work. Yeah, so first few questions. Who of you tracks or occasionally tracks experiment results in spreadsheets? Or worse, yeah, in some text file somewhere. So who of you fails, like puts machine learning models to production and fails to reproduce or even find out how the model came to be after four weeks? Okay, so these are some of the questions we will address here. and many more, and we will also miss some questions, but I hope I give you an idea. So, this tutorial is set up like, I will give you a short theory overview of what MLflow is and what are the ideas behind it. It's like around 10, 15 minutes, and then I walk you through some example workflow you could set up for a machine learning experiment and deployment pipeline. Yes, so, and this will, yeah, the second part will be quite dirty, so you have to get your hands dirty, try this stuff out, the goal would be that you would be able to use these tools on Monday in your organization, say like, okay, I can set it up, I understand enough to know what I have to be careful with. Okay, so it's not super well polished, but this will help you, I hope, would help me. So yeah, so this is the machine learning life cycle as I view it, at least for this talk. So you start out with some raw data in the beginning, normally stored in some database or some file storage somewhere. Then you have a pipeline that moves the data somewhere, transforms it to some machine learning, adjustable data format. You can view feature generation and pre-processing also in this step if you want, or you view it as modeling. Then you move on, try a lot of different tools or software packages to create machine learning models out of this pre-processed data. And then hopefully you're happy with your experiments, you try a lot of stuff. And then you move it to a production environment to solve your business need or your use case. And then, in general, it starts from the beginning again. So you think, how can I improve the model? What did I do wrong? Probably you missed some things. You get new data, get a new understanding of the problem, and you start again. And there are certain challenges arise in this cycle, as you can guess and probably know. Like your raw data lives somewhere. In a different form, then you have to have a defined process that moves this data to some preparation environment. And also preparation is kind of an experiment, right? This probably could be Spark or could be another data warehousing solution like a Postgres database or something. And then you move to the modeling environment where you probably try different tools. Maybe in your team, people use different programming languages even. like R or Python or Spark, like Scala in this case, and you want to pick the best thing for this case, or what you know so far at least, and then if you're done with this, you have a model, you have an understanding of the problem, and then you move it to some cloud deployment in general, or some on-premise deployment with Docker, or to Amazon SageMaker, or to Microsoft Azure ML, or whatever comes to mind. Yeah, so, and this can be quite messy in how to reproduce every step, right? And another important thing is like, in general, you want to go back. Like, you want to be able to run the whole cycle the other way around at every step of your process, especially for model governance. If you say, okay, I have this model in production, I want to be able to know in four weeks what exactly happened for this model came to be. and how can I reproduce it, and how can a colleague of mine reproduce it in half a year or something? Or if you move your environment, you want to be able to take the modeling part from one environment to another environment, and reproduce the whole thing, yeah, quite easily. And these are kind of the things MLflow addresses. Yeah, so, What is MLflow from a high-level perspective? MLflow is an open-source platform developed by Databricks for managing the end-to-end machine learning lifecycle. That's a little bit high expectations, managing the end-to-end machine learning lifecycle, so it's not that easy as it sounds. By the way, I'm not affiliated to Databricks and I'm not a developer of MLflow. I'm just a happy user and it's free and it should be free forever, I guess. They also offer it as a service, but, yeah. Key features are its library and language agnostics, so you have interfaces for Java, Python, R, Scala, a common line interface, yeah. And it works with basically any machine learning libraries. There are special support for some libraries, we will see later, but it also works with raw Python functions, whatever you can write in Python. Then it is, the aim is that it runs the same way everywhere, so you can run it on a Raspberry Pi, like wherever you can run Python, basically you can, like my perspective is mostly Python, so I don't know how it works in R, but it should work. It's quite easy to use. Okay, probably I shouldn't have said this, but yeah. So if it's not easy to use, it's not your fault, it's fine. And probably I will fail using it in this talk, so we will see. Yeah, that scales from local environments to large servers, so that's quite a nice feature. So it's super easy to set up, so we'll see in a second. And you can run it locally, use it for whatever student project, you can set it up at home. and yeah if you have a large organization you can easily with the same tools scale to like a thousand people running experiments and putting it to the cloud or running a thousand experiments per day and putting it in some central storage yeah so how is ember flow organized ember flow comes in three components that you can use independently or combine as you need them. So, by the way, there's a great presentation about MLflow by Databricks, by these two people here, Mani Parke and Thomas Nicot. Check this out. So I borrowed some of their slides because they're really great. I don't try to reinvent the wheel here. So now we have these three components. We have the MLflow tracking that offers you a server to record and query experiments that tracks code, config, results, and all the stuff you want to know about your experiments pretty flexibly. Then we have MLflow projects, which offers you a simple packaging format for reproducing your machine learning experiments on basically any platform. And we have MLflow Models, which is a model format that you can use to deploy wherever you need it, to Docker or to some cloud environment, for example, or locally. So let's look into this a little bit more in detail. So as I said, MLflow Tracking offers you a tracking server that can run locally, that can run on some cloud, and you can access from a distance. We will see later what you can connect to the server. And you can just send experiment results and status updates about your experiments to the server. The server hosts a nice UI where you can query the results and compare experiments, also retrieve information about the experiments. you can query, for example, like some artifact store. I will talk about it in a second, yes. Then we have the ML projects again, MLflow projects. They're basically, you write a simple, you can think of it as like a simplistic way of Docker in some sense, I would say. We have a simple configuration, and that generates some execution environment for you that you can run locally, or you can just ship to your colleague, for example, and say, hey, please run this experiment, or you can send it to some standardized cloud environment where you need to run the experiment again, for example, for deployment. Yeah, it can run in Docker, Azure, Databricks, Kubernetes, for example. We have MLflow models and the aim here is to have a standard format to put together all your tools you might use for modeling and make it easy for everyone to run them and also offer a standardized interface for deployment of these different models. So, for example, you could take like a PyTorch neural network, write it, like compile it, not compile it, put it together in this MLflow models format, and then deploy it as a Spark UFD, UDF, for some stream scoring, for example, okay? So, this is quite powerful, and we will see how to use this now. So, again, this is a resource that's basically used for this talk, the talk by the Databricks people. It's on YouTube. And the MLflow documentation, this is quite nice documentation that's really helpful and covers basically everything. Well, so, this is it. Let's get our hands dirty. hence dirty I hope you all got the repository someone didn't get it and you were all able to kind of set up the environment I heard there's problems with Postgres but it's not that important so this will be an add-on in the end if we have time well so let's see we just start I don't see it on my screen that's yeah so it must work like this so we just start the environment alright and then go into the Jupyter notebook not this one where is my not here but here so slowly getting there so so you have all this hands-on example notebook running or at least in the way of running it But, so yeah, so I will just walk you through. You can play with it a little bit. It would be the value that you recognize all components and all things that go into, that happen in the code. And if you have any questions, just raise your hand and then we can probably talk about it, right? Yeah, we have the normal imports, as you would expect them, and here is basically MLflow imported, and this is some special feature of MLflow. You have some tools specifically for some machine learning frameworks. For example, here we're using the scikit-learn, basically functionality of MLflow, But, yeah, you can also write this functionality quite, yeah, you can write it for yourself if you need it for your own machine learning library that your company developed, for example. Then we have a quick look at the data set we use for this use case. So we have some wine data from, like, I guess, some wines from north of Portugal. And we have some chemical properties of these wines and a rating and we try to find out or like we try to predict the quality score of this wine um yeah i mean in this tutorial it's now set up as a regression problem but it ranges from zero to 10 so could also be a classification problem feel free to change it if you if you go through it so and of course we start now by by setting up like the tracking server like where to where to store all the experiment information and for this we have like two kind of back ends we need to store the information of our experiments so one is like the so-called tracking store where you put for example, your metrics and versions of your code and stuff like this. And then you have different flavors where you can put it, like a locally, you can put it to some Postgres database, for example, or like a HTTP server where your tracking server's running, or for example, a Databricks workspace, because of course, Databricks offers a Databricks workspace for their tools. And then you have the artifact storage, where you would save model binaries, plots that you generate in the run, or for example, also data sets, whatever is a file, basically. This can be done to Amazon, S3, Azure Block Storage, or also, of course, locally, wherever this is not in this list. Exactly, and you can start the server basically with this command, so you run MLflow server, then specify the backend store, the artifact store, the host, and the port, okay? So I have a command here, where you can start it. And now we try to start it, I need probably a new... You also need the environment active here, so... Well, and then we have a Junicon server running. Does it work for you? Like, it's in the notebook, also, the, you can probably see it like this, in the fourth cell. You have this MLflow server that specifies the backend to the ML runs directory and also the artifact storage there and holds it on your localhost or Paul5000. So does it work for you? I know it's a stupid question, but you have to have the environment active in the terminal. So you should be able to run MLflow. You can check it if you go to your browser now and go basically to this URL, port 5000, and then you should see this. This is basically the UI of the tracking server. So, hands up who can see this? Okay, then we'll wait a second. So, again, who can see this now? Hands up, please. Okay. Does anybody need help? So please raise your hand. Okay. So everyone who sees this now can, if you want, just go on with the notebook and check out what's going on there. Yeah, I offer some help. It's okay. Now you, so everyone seems at least busy. So good, good, good, good busy or bad busy? Okay, so we now move on for a second. So you have to server running, I expect. And now in the notebook you have to specify where the server lives. and then you set the tracking URI for MLflow. You can also, if you run it as a script, for example, as I would recommend to you, then you can set up an environment variable for this. Let's run it, and you can also retrieve it and check where it's running, so it's pointing to this URI. Next, we set an experiment that will also create an experiment if it's not there yet. So you can also only create an experiment without moving to it. It's a little bit like Git branches, like you can create a branch or you can go directly to the branch. And this is created and go directly to the branch. We call it Elastic Net Wine because I'm going to use an Elastic Net model. And then, for example, if you go now to the UI, you should see here a new experiment with the ID 1. And you see here the artifact location where it stores its artifacts. This is a relative path to where you started the server, basically. So in my case, you see now here the ML runs directory. in the PyData Berlin 2019. If I'm too fast, please interrupt me or try to interrupt me. Okay, so, well. And now we have to think about what we track. So now we have the server running where we can dump all our information. We want to persist. And now it's a question, what do we want to track? Yeah, so we want to, the version of the code, right? Then we can go back in our Git repository and say, like, this was run with this exact code. Yeah. Then you have to find another identifier that uniquely identifies the code, or you just, like, it saves an environment, right? So the environment is reproducible, and if you have, like, custom code in this environment, you could for example just collect all files in this environment like that's what i like to do you collect all files all custom packages that are not in some package repository and also dump them with it because code is small right you can just collect all the code you're using basically all your dependencies that are not in your environment variable of python for example so So there are other tools actually that are doing this. Like there's this sacred, it's a similar tool and they are just collecting everything that's in your environment. But this is more like set up for use with Git or some similar solution. Okay, great question, thank you. and then we want to start in n time the source like where did this experiment come from and which code started it with jupyter notebook this feature is not working well so you will only see that hey this comes from a jupyter notebook yay because it recognizes the ipython kernel behind jupyter so you don't know which jupyter notebook run it so better to run with a script yeah then we have things that are called parameters this is like key value stored values where everything like where all values can also be um strings so basically everything will be strings here or it will be um parsed to a string and then we have metrics and also key values but the value is always numeric and you can always up and you can also update those values throughout your run so for example you can have like different epochs of your neural network training you can add to this metric for a certain run and we have artifacts where you can dump basically any files you want code model artifact plots you produced um or what do i dump later we'll see um yeah yeah of course exactly that's what i'm doing later yeah i forgot yeah exactly you could dump your data there or you can yeah of course you could also just you can also track i mean these are the basic things that i can be built in but you can also track whatever you want you could also treat like data as a parameter right if you have a data pipeline that exports your data from a production environment for example to some file storage let's say s3 or you run another file server then hopefully your data has a unique name that they are not changed right the data so you could save this as a parameter for your run? Good question. Always do this. Yeah. So now we come to the code finally. So we have some simple function that creates the metric or computes the metrics for our model. then we have a function that loads the data and splits it in train and test split for evaluation and we have the interesting part here that's the training function for our model so we have two parameters regularization parameter for the linear model and like a trade-off between L2 and L1 normalization like you have in Elastic Nets. Then we read the data and then the interesting things happen. So we use MLflow start run in like a with clause to tell MLflow, hey, here is the code where the run is happening. Keep track of the stuff that's happening here and we tell you what's the stuff exactly. You can also run it like just mflow start run without the with, but then you have to be careful because you can only have one run then. If you have a script, it's fine. You run it, you're done. If you do multiple runs like a grid search, for example, in a script, you would do it with a with clause. So with clause is the way to go here. then we have the elastic net model okay so i could learn everything's nice then we compute the evaluation metrics and um we will print them because we can and now we we um tell the tell ml flow basically what we want to log exactly, so here we add parameters to our tracking server with key and values. And you notice here, this is not a string yet, but MLflow will persist it as a string in the end. And then we have metrics with a key and a value again. You can also log your stuff as you might notice with metrics, or metrics, or a single metric can be logged like this, or you pass a Python dictionary to log metrics, and it also works for parameters. If you add an S here, so you can track multiples at the same time in one line of code. And then you log artifacts by just giving it the path to where the artifact is saved. So in this case, we save the data. and then we'll print out where the artifact lives now. You can also see this in your tracking server UI later. So this is interesting if you, for example, want to get artifacts, another one in your team, for example, produced, right? Or also you produced and want to see them and get them five weeks later or something. And then we use the scikit-learn MLflow tools to lock the model in a way that it's kept being a scikit-learn model. Like it uses the inbuilt tools of scikit-learn. It's called the scikit-learn flavor in MLflow. So MLflow thinks about model formats as flavors. And the most basic flavor is the Python function flavor. So everything is like for Python. Everything is runnable as a Python function in the area of Python model flavors. So this, but you have also like a PyTorch model flavor, TensorFlow model flavor, XGBoost model flavor that uses the inbuilt tools of these tools to save and persist and load the models again. Yeah, and here we locked the model with the scikit-learn model flavor to file name model. All right, first run it. Oh, yeah, just get it. You can also have metrics with some steps that adds, for example, a learning curve while you're training. Yeah, it's not useful for this example, but we have later a Keras example where you can use this, for example, to track certain values over some multiple steps that are happening. And now you can feel free to run this training thing. And we see, okay, it's doing something. And here the artifact is stored locally on your... where the ML runs. This is the first experiment. And we have here a lot of stuff going on. So in here the artifacts, and we have the model directory and here's our model with some description we'll talk about later. Also, the data is dumped here. Yes, question? No, no. In general, not. The goal is it saves like, it's a little bit like his question, right? So it tries to save the git hash. The code you have to persist yourself if you want it. That's a little bit missing, sadly, so far. Yeah, or optimally, you have your experiments set up like you only change the input parameters and nothing more, and then, yeah, it's a dream, but, yeah. Other question? No, they're optional. I'm not sure and I'm not sure if it at least it's not intended to do so I would say if you create an artifact it should be in best case specific to this run and everything else should not be an artifact. I know I'm misusing it here already because I dump the data every time I run it. But in general, the idea is that everything that belongs to this run specifically can be an artifact that is produced by this run. But I guess you can set it up. Good, so this is basically the artifact. stores and we also see now if you run it or if you have run it we can see now in the ui now you have to refresh it probably yeah you see like only two runs here but yeah so you see um the runs comes this is basically a source and as i told you it's only ipython kernel ea um git version the parameters stored and who run it and if you click on the on the date time you see like more details you can add some notes um see the metrics see the artifacts and you can also click on the artifacts for example you can read like what's going on in the conda environment probably can also see the data it's probably not a good idea but yeah have you run it okay can you have it so okay um you can also click here on the metrics for example to to compare them and look at them visually this is not that interesting if you have no steps and nothing going on of course because you can of course look at your metrics and say like okay nice i have okay this is like visually comparable but you don't compare any you don't see any progress here right because we'll see later that can be more interesting if a proper learning curve or if you have experiment that you're setting up is more interesting in this regard so yeah so interesting thing also you can write here in the search runs you can basically write a SQL where clause to to retrieve runs that obey certain qualities, or you can just order it here by some values. For example, we could say, okay, order it by mean absolute error, and then we have some ordering here and can pick the best model by this metric. Does it work for you? Any questions? Everyone's happy. Yeah. Thank you. Yeah, I mean, this is what I did here now to have some easy accessible artifacts. But in general, you hopefully not do it. You can do it if you have small, often changing data. you probably can do it because it might be convenient but if you have a large data set you hopefully have a proper data store somewhere and you would retrieve the data from there and you would only save the unique name of the data set and don't produce a lot of duplicated data okay has anyone tried using a different model? really good it still works very good okay so what did I want to say here and you can also see this is like the file storage for example because we specified that we want to save it in also the metrics in some files you could also send them to some Postgres server or any other relational database and you can go to these directories this is basically in the ML runs and we can look at the metrics and here it has just saved as as text files with an ID to specify the run of timestamp basically timestamp same for the parameters and with text we will see in a second okay so So, we did this, right? So this is basically the basic work through for the tagging. So are there any more questions for how to tag your experiments? the naming of the model no you would have to store it you can also name runs so you can for example here in the ml flow start run you can name your run and that can part of the model or you say like the experiment is the name of or contains the name of the model or you add it as a parameter it's quite flexible but also doesn't of course from the pickle file you can just load it like if you that's basically the beauty of these model flavors so if you save it with the scikit-learn tools you can retrieve it as a scikit-learn model you could also save it just as a python function yeah no no no if you go here you get just just blob or nothing and so you probably tried yeah it's also not here no yeah because also yeah it's not standardized how the model naming is done between tools, yeah. Okay, so then we move on to packaging experiments. to packaging experiments. So the goal here is like to be able to run the experiment in a reproducible way basically everywhere you can run for example doc or conda or some standardized environment manager so that enables you to move your experiment for example to your colleague or your colleague is able to reproduce the experiment easily or you can move it for example to cloud because you need the artifact there for deployment whatever you want to do with it or you can just go back in two months and say like i run this looks the same nice for this you need to create an ml project file in the respective directory that specifies the entry point for what to do in this experiment basically it's like a simplified docker file in some way and the conda environment you want to use for for this case, because it's conda use case. You can just get the conda YAML from your experiment logs, like this one basically. We want, for example, wait, where is it? Yeah, so in the model artifacts, you have basically the conda YAML, And yeah, so this you can just take. I have some thing put together. Like this is one conda yammer, right? So I could learn everything. So I have the Python version and MLflow, what we need, okay? And then we create this ML projects file. There should also be one in your repository. Let's walk through it in a second, so you can name your project, your experiment, you specify the environment you want to use, in this case we used the conda and you can use docker and for example also, then you specify the file where the configuration of the environment is stored, and then you specify the entry points for the experiment, so in this case we only have a simple main entry point so every every experiment should have a main entry point at least but you could have multiple of course you can have like an experiment you can have a certain validation entry point that you run after you run the experiment or run the training for example or you can have a create plots entry point that creates some plots off of the trained model in the end, for example, so it's quite flexible. And yeah, each entry point can take some parameters that you have to specify here. There are multiple ways to do this, so you can just write as key values, basically. We say like, okay, this is the name, and this is the type I expect there. Then you can have, this is basically JSON, because it's where you can specify the type and the default value if you want one, but you could also, this is like a short version, you could also write it like, okay, this is not a proper manager, but you could also write it as a YAML still. And then you have a command for the entry point that is run in the environment, and here you specify the parameters. And of course this function or the script should take the parameters there. And this is actually the not so nice point about MLflow and I don't really know how to address this probably because it's of course not working from your Jupyter Notebook environment. So you have to have a script that is running the experiment. So Jupyter Notebook's nice for showcase but I would recommend you writing your experiments in scripts because then you can use it easily in this workflow. So we have, I have created for you the same function as in the Jupyter Notebook as the script. Oh, this is ugly. So this is basically a command line runnable Python script that basically does the same thing as the Jupyter Notebook, okay? Yes. Okay, and this is basically specifying your experiment and your project, yeah, one question? Okay, so yeah, I will come in a second, let's just do the next step, close without saving. So you can now just run the experiment like this. So you have your MLflow environment set up, of course. And then you can just go to this environment and reproduce the experiment. So this P specifies that after there are the parameters, basically. Okay? So let's try it. Yeah, so I need another one. Hey, crutch. So now we can just run it like this, hopefully. so yeah i already have the conda environment it's not that interesting um but you can try it run different experiment it should also lock the experience now to the ui but to the default experiment because we haven't specified any experiment in the in the training script here okay so can you can you can you run it like this can anyone not run it so far okay for you okay interesting so we'll i will i will uh come by and check it in a second okay Okay. Oh, you debug it yourself. Yeah. Okay. So, we move on in a second. We move on. One thing, it's probably unclear, so here we are not taking a safe model from some experience, a safe model from some experience. So this is about reproducing experiments and creating a new model in a reproducible way. Okay, just to clarify again. Here I don't set the name. So this is the interesting thing, kind of. But it locks it to the default experiment if you have a look. You can also, yeah. Well, where am I? So the new run is basically here in the default experiment. but you could set it the same way I just didn't do it but if you have like if you go to the training script you could set it for example in the you can have the set experiment function right and then it would point to this experiment yeah I don't know yet yeah so yeah I don't know ah nice to avoid if it's already created okay but if you run a new experiment it would still create a new content environment for it so it's a partial solution so it would not try to create a new one but it would take the working environment but otherwise yeah you have to clean it up yourself by probably killing the server or something okay so Yeah, and it also can pollute your Docker repository quite quickly. Just to. So, wait, this is the wrong browser. Well. So, now we have enough models, I guess. At least most people have enough models. And now we come to the deployment part, I quite like, so you can just specify basically your model and then MFLow takes care of the whole serving part, basically you say MFLow model serve, specify the model and the house and the port and off you go. This is like the simple local deployment thing. Let's do this first, but there are also tools to deploy to Docker, you see down here. I'll build a Docker image from it, or you can also directly deploy to SageMaker, to Azure ML, or similar tools, or to Databricks, of course. So, let's try to run this. you have to change your model ID, right? Because this is like some model ID, you have to hear the experiment, model ID, change it to some of your IDs and try to deploy it. Please. So, just paste it here. Go to the UI, go to the UI and find an appropriate model ID. Let's take this one. We find the model ID, for example here, the run ID. Copy it. And put it here. Very good, of course I did it wrong, so it has the experiment ID 1, so one interesting thing to note here so to deploy locally it selected automatically the python function flavor so you're not interested in knowing that it's a scikit-learn model at this point right so it's like okay you didn't specify anything better so i just take the most basic thing i run it as a python function um and don't use any specific um functions or methods that are offered by scikit-learn. It's basically here, right? You can add different flavors if you, for example, want to use some tooling that is available for this tool, for example, like TenderBoard for the surf model. Yes, yeah. No. Ah, yeah. I'm not sure we can find out so zero zero zero zero yes so good that only offers actually like the invocation the invocations and point here let's try what happens if we do this invocations not allowed yeah so because it only accepts post requests so yeah so this we would need a new window for this I pasted you some some curl command in the notebook if you have curl available you can just try it Whoop, in another terminal. So let's see, basically here specify the invocations endpoint of this URL. And then we hope something's happening. This is quite not what I expected to happen. Ah, yeah, something happened. So we got back a number. So the standard ways for MLflow deployment is to accept like chasens and in some, and CSVs, of course, in some pandas defined. format like you have split and I guess record are the pandas you have to tell it how to load it in pandas afterwards and csv is just loader standard way but you can also change this if you need it for your application for example if you want to work with images or other types of data so did it work for No, I should ask the other way around. Is there someone for whom it didn't work so far? If you could create model artifacts, because... Okay, so everyone doesn't want help, or it works. It's in the Jupyter notebook also. So you can find it here, test the endpoint, and just copy it if you have curl available. It should also work on a MacBook. Any questions? No. That's good in time. And you could also build a Docker image from your model. It's also in the notebook. I'll also show it. And you can also try it. Ah, where is it? Kill this one. kill it and with the mfl models build docker let's see if it works here because it will try to build a docker container obviously um i need the url again the model id hmm and okay everything stored basically here mm-hmm okay trust download some stuff can come back to this later so please don't all run it run it at home and then you can deploy it of course like you deploy a docker container Um, what is it? Yeah. Question. Yes. I mean, of course you, you would view the feature engineering as part of the model, right? So, for example, if you have a scikit-learn pipeline for your feature, standardization, normalization, it would just work like this, right? You have a scikit-learn pipeline, that's basically the model. If you have some custom code, some custom objects, you can still save it as a Python flavor. You cannot use the scikit-learn tools, but you can still save it as a Python model, and it works the same way. I don't think it would just follow the same process. No, it's just for the purpose of this tutorial, really good question. So you can, yeah, the preprocessing should and could also be part of your modeling, so to say. So in the best case, this process starts with the raw, or at least only partially processed data, and most of the processing happens in this part. if you can afford it, probably you can't, yeah. Yeah, another question? Your experience, how are the logging and monitoring options for the model? I mean, of course, this workflow doesn't take care of the logging and the monitoring of models in production, so you basically, I mean, there's the next point, like tagging the runs, but that addresses a little bit of this, but of course for your production environment, you would have to have another logging and monitoring in place. So... No, I mean you can also deploy from this. I mean, you create a Docker image, or you put it directly to SageMaker, but then, for example, like, I work with SageMaker occasionally, and then SageMaker takes care of the logging and tells it some stuff. And then, of course, the consumer take, like if you get some feedback, for example, you have some users and you get feedback, like how well your machine learning model performed, then this is not part of this workflow. You have a different story and it's a different story, sadly. So yeah, of course, I mean, that's what I meant in the beginning, right? Emberflow claims to manage the end-to-end lifecycle, but there are some disconnections still. You don't have the production monitoring in and you don't have the raw data handling completely in one workflow. Yes, sorry. . No, no, there's a different tool specifically for SageMaker with MLflow. If you go here to the happy documentation, MLflow models, I guess, and built-in deployment tools, you can find a deploy of Python function model to SageMaker, and that basically is obviously like MLflow SageMaker. They can just deploy from there. And you can also find other things like to Spark or Azure. Yeah, do you have a question? Yeah, I have a question. Basically, how do you get to start a Flask API? Yes, yes, a Junicon server with a Flask API. yes in a conda environment exactly well so yeah so the next next step is so at least it addresses a little bit of your question like now you have a model that is ready for deployment but you hopefully in your tracking server you want to know what happened to the model Like, is it just an experiment that lived there forever in the experiment repository but never went to production, or if it went to production, right? So, what I do, and I don't know, it's not a standard workflow, I guess, but you tag your experiment automatically if you run a deployment script. So, with MLflow, you can just get the respective run from your tracking server. and then you add a tag to it in my case i would just tag it as like deployed and the date and the timestamp where it is deployed and at least you can go back from your production environment and say like okay this is the exact model that run at this point in time okay so this is a little bit more in depth that uses like the model the emma flow internal python api so you create this emma flow client you retrieve a list of the experiments let's just do this so i have two experiments these are the artifacts location id still active as a name stuff like this then Then we try to get a run, by run ID. For example, this one, because we built a docker container for it. And now we get all the data of this run also in this run object, like the metrics, the source. And yeah, even some things that are not in the UI, sadly, Like, for example, the status is not visible here, right? I don't know why, actually. So, you can get it from the API. And then we can just set a tag for the respective run. And now we should have a tag here. You can see it here, for example, in the text. We have a deploy tag that's basically also a key value store with a timestamp. Yes. So this is basically, from my point of view, kind of the workflow parts. Of course, they are still disconnected. You would need a script or a scheduler or something to put this all together. works neatly with Airflow, for example, or you have just deployment scripts to do this. So are there any questions at this point? I hope you're all a little bit confused and have some impression what you can or cannot do or can try to do with this tool. So we have a little bit of time for questions or you could walk through some Keras example also, but we can do questions first. Yes, please. So, yeah, so that's the hot topic at this conference, I guess, like data pipelining and reproducibility in some sense. like airflow is like different tool that's for for for etl and not for specifically for keeping track of the things so you could you could use this in combination with with airflow to track what exactly happened with the models and you could easily go back in your life cycle another tool i like i don't know if it was talked about in the conference so far is sacred um it's i guess by a german company um it's a little bit i mean it has no built-in ui there's an open source user developed ai ai ui available um called omniboard it has has different features but it's quite similar it works more with python decorators but it addresses the same things at at least for the tracking part, but it has nothing for deployment and reproducibility built in. Then there's like another angle is like the data version control thing, like for example, DVC, or something that could also play neatly with MLflow because you can add some hashes for your data and stuff like this, yeah. But there are a lot of tools and it's still unclear and would be a good topic for talk actually like tool selection and I have no silver bullet solution, sadly. Yeah. Good question. You had a question? Yeah. I don't know if it's just me but I find the UI very . Yeah, it happens sometimes. I don't know why actually. It can be slow. I guess there's some issues in GitHub for this. I'm not sure if there's some progress. Any other questions? Yes, please. What is tagging run? So it means basically you add a tag here to the specific run of your experiment. So here, you see like, okay, this has a tag now. And for example, if you think about it, like you have in your organization, for example, you have this server running somewhere in a completely separate environment, or you just store information about all experiments in your team, for example, there. And it's like, okay, we moved something to a production environment, and you want to see it in one place. This is the use case I use tagging for, but you could tag it with whatever you like or your workflow enforces you. Yes. Any experience regarding the response time to deploy the model? I mean. So if you just put it there as a server container and you use it as a service? I mean this heavily depends on your model and your infrastructure, right? I mean there is some, I mean I know there's some delay if you deploy to Azure ML or SageMaker because obviously they build the new stuff and start infrastructure, but otherwise that completely depends on your infrastructure and your model. Yeah, so what's the time? Should we? Okay, so I can give you an introduction to what else is in the notebooks, kind of. So there is, what was here already? So there's basically like a small, small example how you could connect a server with a Postgres backend. Not much Python here, but yeah, it can be convenient because like dumping all your experiment stuff and information from a thousand experiments to some files might be annoying to retrieve. and also you can have your custom queries there, right? So, yeah, you create a database and a table, and then you run it and just change, basically, the backend store here with some Postgres URI, and then you can just go to the database, it creates the MLflow database there, yeah, database, and we created it, yeah. And then you can select from experiments, select from runs, and check what's going on there. And you have some more information available there that you would not have in the UI for some reason. For example, the status, if you want to know if your experiment is broken or dead or finished. For some reason it's not in the UI. Yeah, so that's also a thing that is not addressed yet. So if your experiment fails, then you have an incomplete dump of your results. You'll probably have the parameters, but no metrics available. That's not nice from my point of view. So are there any final questions? Nice, so you're all properly confused. yeah so so the adventures of the flavors are you can retrieve the model or you can have a deployment that uses tools specific like methods specifically for this model type for example you could use the tensor board integration in in your experiment run in your produced experiment run and this is not available if you use it as a python function and it just functions python this is the best example i have i in general only use python flavors because if you have a custom-made model you only have python flavors so you could write your own flavors but yeah you're welcome so to wrap it up i hope hope it helps you a little bit it was not too confusing not too annoying um yeah it's the workflow in general is quite flexible and you have to find your own workflow for your team your needs and your organization and emma flow can be suitable for a lot of use cases i guess so i hope this helps you a little bit gives you understanding what is possible and what is there what you can think about