Making MLOps uncool again

Speaker 1 [00:04]

Thanks everybody for coming So my name is David. I work as a software engineer at iterative.ai which is a company that pays me for this trip, so Let me just ask one slide about it We work on building MLOps tools, open source tools tools, trying to fill the gap between the existing software engineering stack and machine learning practitioners. We are a fully remote team, as you can see from the map distribution. We have open positions of all kinds, if you would like to join, check the website or talk with me later. I have a bunch of stickers. You can pass around like if they were drunk. All right. So what do I mean by making MLOps uncool again? I am not sure, but I have to ask one slide to explain it, because it's the title. So I think the idea came from the evolution of DevOps. From my perspective, DevOps started from an uncool era with Jenkins and maybe other tools that I am not familiar with, and this is the uncool phase because not many people were interested in this matter, and then eventually the interest started to grow exponentially, And so a huge number of startups, open source approaches, big companies working on providing specialized DevOps solutions. So I call this the cool era. The issue with this is that I believe that the end user experience starts to degrade because now you need to pick one of many of these tools and you need to discover how to integrate with your workflow. And then the current status, I think, is the uncool again era where the major Git providers, Git server providers like GitLab and GitHub have consolidated their own approach to DevOps and the other tools are kind of left behind. I think this This is the uncool era because while it's not so popular to talk about DevOps because it's kind of solved, I think it's a really good state. In my experience, GitHub Actions are the first tool where I have seen a researcher actually carrying or being able to extend a CI or CD workflow. Maybe it's just my circles of people that I know, but this is the first time I have seen this kind of interest. So yeah, a little overview of what is MLOps in my perspective. I don't think there is a universal definition of MLOps, and I think it's very dependent on the use case, and what I am about to explain is truly from my experience. So I got into MLOps coming from a researcher background. As researchers, we didn't really care about DevOps at all, we were worried about compiling Ciano or whichever framework was at that time, and we didn't really care about CI, CD, software engineering best practices. I was the one that started to worry about it, getting interested in this, and I guess I was a little annoying to the DevOps team asking for stuff that they didn't deliver. So that made me a good candidate to get the title of the MLOps guy, and I had to go into this quest of discovering what this means. So I asked around to my fellow researchers and to myself what we really want from MLOps. And from our perspective, the first thing I discovered is that this thing called reproducibility Reproducibility actually means that it was very annoying for a researcher taking over a project that was from a previous researcher that for some reason is no longer in the company or has been assigned to a different project, it was very difficult to catch up with the current status and discover how all the configuration data model and pipeline was set up. So I guess this is what people call reproducibility. And then there is automation, which is basically that ideally the researcher should be able to focus on doing research and not having to manually trigger a pipeline in order to retrain the model when new data arrives. But we were in the situation where researchers were also in charge of retraining the model, and this was a manual process, and ideally we would like to automate this part so we we could focus on doing research. Yes, so I think pretty much all researchers I met agree on these goals. So I went on a new quest of discovering how to do this. Yeah, first option we found and we tried was end-to-end machine learning platforms offered by the big cloud providers. It kind of makes sense to try this first, because we were already using many of the services from these cloud providers. I found quite a few, actually, issues that I didn't like at all. The most important one, in my opinion, was that these platforms isolate the researchers from the software engineering stack. And I really wanted, actually, researchers to be involved in software engineering best practices. So this didn't feel like a good option to achieve that. In addition, we have some projects where these scalability features provided by these platforms make sense. But then we also have a majority of small projects where this was completely overkill. But you cannot opt out from these services in this platform. You have to buy the full package. And finally, for obvious reason, because it's their business model. Once you build your stack around one of these cloud providers, they are not going to make your life easier if you want to integrate an external tool or if you want to switch to another provider. I am not saying that using an end-to-end machine learning platform is not the right solution. Maybe if you are a big company which already have a lot of services in one of these platforms. So you are already attached to this vendor locking. Maybe this is the right solution for you. All right. And then the next alternative was exploring the widely-based open-source space of MLOps tools. I think the image talks for itself. It's just that there are so many tools for so many different things, and nobody really explains you how to connect these tools, or even what tools do you need. That this was just a cumbersome to explore all the possibilities. So I took a step back, went to the DevOps team, and ask them, OK, if we are using DevOps for software engineering, why can't you, DevOps team, extend your tools so we can do MLOps with them? Which is basically what this meme is about. And they ask that you already have some sort of similar workflow tools available in the MLOps space, for example, Kubeflow. But the issue is that researchers were really proficient using GitHub Actions. For some reason, I have nothing against Kubeflow. I think it's a great tool. But for some reason, they were not so proficient trying to do the same things around Kubeflow. So many blog posts from I think it's called. It's linked at the bottom. And the first reason they give for DevOps being not sufficient for machine learning is that you, in addition to source code, you also need to version the data, the parameters, and probably the model. And it turns out that I started to sell our tools again. And it turns out that there is this fancy tool called DBC or the version control that allows you to version data code parameters all together through it The other point Made in this book post is that? many machine learning projects require hardware that is not usually available in the CICD Space and there is another tool made by the same company called continuous machine learning learning, or CML, that acts as powerful machines with views or whatever inside your existing DevOps CI CD workflow. All right, so with this, if someone wants to follow the workshop, we are about to start. The toy project that I choose to build this workflow around is an automatic issue labeler. Basically because I wanted to try having face library. And because we have an interesting data set in the actual DPC repository where we have over 3,000 issues, and we have a bunch of labels that we use to categorize. So the idea is that we are going to build a model that takes the title of the issue and is going to predict which label should be assigned. Here's the link to the workshop. I am going to give you a few seconds to go there. And if, while doing the workshop, you face some issues, please raise your hand, and I can stop and try to solve it. But for the questions please ask them in this platform slider so I can answer in the end. So I can assume that everybody who wants to follow the workshop is in the GitHub repository. Please raise your hand. Yeah, all right. So a quick overview of the high-level idea of the project is that we have a data source which lives in GitHub. In this case, it is the DBC repository where people open new issues every day. In this repository, we are going to use GitHub, DBC and CML to first construct a data set to train a hanging face model that we are going to eventually ship inside the Docker container so it could be deployed anywhere, including the original GitHub repository. The tools we are about to use are DBC, DBC Live, and CML by iterative, and the high-end phase transformers. So the current status is what I call local reproducibility. If you click on this link, in this current status, we have a DBC pipeline. There was a great talk yesterday about DBC pipelines. I don't think I have the time to cover how it works, but there are links to the documentation and you can ask me questions later. The idea is that the pipeline is defined in a DBC YAML file. You can click and explore how this pipeline looks like. I hope that it looks simple enough for you to understand how the pipeline is being built. The pipeline has a first stage that is going to get issues, issues from the DBC repository. There are a bunch of parameters to configure this stage and all the other stages, and they are defined in the params.yaml file. So the first stage, named get data, is going to use all the parameters defined inside the data section. We are going to fetch issues from the iterative. How is this size? Right. So we are going to fetch issues from the iterative DBC repository. We are only going to select a subset of the labels. This is what you are supposed to do also in your real life projects and don't try to solve the whole problem from the beginning. Start with a minimal problem. So we select these three subset of labels, and we are going to get the issues from the first day of 2021 to the first day of April of this year. And this is just a way to indicate where we are going to store the outputs. So this is the data that is configured by those parameters. Then we have another stage, which is called split data. And I assume that you understand what it's about to do, which is split the data set into a train and evaluation splits. Then we have the train stage that uses to train the model. And finally, an evaluation stage. I think these pipelines enable local reproducibility in a way that if you clone these first steps are optional, you can run locally, it might going to take a while and we are all using the same internet connection so you are guessing that fetching the issues from GitHub might collapse the network. But don't worry if you don't run these steps, you are still able to follow the next steps. So if you clone this repository locally And you stole the requirements which also might collapse the network if you are wrong at the same time You can reproduce the full pipeline with a single command called DBC repro so I have this workshop Repository clone locally and I have already installed the requirements and And I already have run the pipeline. So the thing with this DBC pipeline is that once you run for the first time your pipeline it's going to store the outputs in a local DBC cache. So unless you modify one of the dependencies of one stage, that stage is not going to be rerun because between the last DBC repo and the one I have just run, I didn't change anything. The command does nothing because all the states didn't change. If I go ahead and change the number of epochs from 15 to 10, and I run DBC repo, it's going to skip the first two stages and rerun the train stage with these new parameters. It doesn't take very much to train the model, it's a very small model and a very small dataset. So it's going to rerun the stages from the pipeline that are dependent on this change. In this case, because the evaluation stage depends on the train outputs, even though Train I only change a parameter from the training stage is going to rerun the train and the evaluation stage I'm just a simple pipeline that is built using dependencies and outputs So once I have run this pipeline You can see in the local changes that it generates a bunch of of TSB files and JSON files which are basically what in DBC is called metrics and plots. These metrics and plots are playing text files and because they are small enough, you can track them with Git. The plots you might or might not want to track with Git because they can a little big, but in this case, for the sake of simplicity, I am tracking them with Git. The funny file to care about is this DBC log file, which is basically storing the hashes of the outputs that we just generate, so this file is shipped inside your Git repository, So as long as you have a local cache, which I have in my computer, or you are going to have in your computer if you run the command, these outputs can be retrieved from the local cache, so the stages don't need to be recompute. This is why I call local reproducibility, because you can reproduce the pipeline as long as you have access to the local cache. DBC plots and metrics are already shown, and I have to make this publicity. We also have this tool called Studio, which is a way to visualize all these DBC-related files in a nicer way. So this is just to use this, all you need to do is to connect your GitHub repo with Studio, and it's going to parse the commits and look for the files to show here. So as you can see, it can differentiate the values of the parameters and metrics from other outputs, and it's also able to render the plots that are just plain TSB files in our local Git repository. All right, so I have run the pipeline locally and now I can track all the outputs with DBC Sorry with it and share with other people So I just ask all these files to get and push them So, later on, if I clone this repository, I could reproduce the pipeline and nothing will happen. You might notice that we have some outputs that are being Git-ignored because they are too big to be tracked by Git, and that's the reason why we need a bigger ship. So this is the next step, which is the first you are about to do something, which is called shares reproducibility. In order to be able to have access to these large files and directories that we have generated, DBC provides a thing called DBC remote. And how it works is that in Git, you are going to track and push and pull the small metadata files, which were the DBC log I showed you before, and they are going to be linked to the actual big files in your local cache. Introducing a DBC remote allows you to run DBC push and DBC pull commands to this remote so you can keep your Git repository and your big data files in sync. In order to as a new remote I think that the simple thing we could do is I assume that you all have a Google account so you can go go to your Google Drive account and create a new folder which is going to serve as our DBC remote you can Deleted later and it's not going to take much space If you don't have a Google Drive remote then I am sorry I didn't consider that So the important thing is that Once you have this new folder created you can go ahead and copy paste the last part of the URL Which is the identifier of the folder So if you are following I I forgot you are supposed to be working on a fork of the repository Maybe I forgot to tell but I didn't fork it myself And I already have a fork which is going to be out of date sorry about that alright So, in your fork, this is not mandatory, but you can open your repo changing .com by .dev and you will enter the web editor that GitHub provides, and I find this more convenient for editing files. All right. Going back to the docs. I think we are here. So we have a new Google Drive folder that we are going to use as DBC remote. In order to add it to our project from the web we can edit this file called dbc slash config and we can ask this content so inside dot dbc config i copy paste this content and i replace this by my google drive url So, this is how your DBC slash config is supposed to look, and this is what connects and tells DBC that you are going to use this Google Drive as a remote. Just checking whoever is following is at this step. No? Okay. Is there an issue or...? The connection is terrible. Even for the editor? I mean, you can skip running the pipeline locally to save connection, and you can just jump into this step. If the web editor is not working, I guess you could also edit directly from the GitHub interface. So, has anyone reached this point? Raise your hands, please. Not many. All right, so I guess I can take one question if there is some. To give some time. There is one? All right. And so, maybe you have a question. My mic is not working. Yeah, so the question is what why not get a large file system instead of a DVC remote and that's very good question And we have a page in our documentation Which is called Related technologies and And here is the answer that I can just read. So the difference between Git LFS and DBC remote is that DBC does not require any special server like Git LFS. So you could not set up without additional work a Google Drive remote as a Git LFS server. You have to do a bunch of stuff. DBC does not have any Git hooks in your repo if you don't want to. And yeah, another thing that is not mentioned here, maybe I can improve the documentation, is that the DVC hashes and remotes are connected to the pipeline and the stages, and you cannot have this direct connection or reference in a Git LFS by default. I think that's the answer. All right, how are we doing with this, adding a remote? Yes? No? I wonder what's the issue, or what's the question? The IP has reached the rate limit. When you run the pipeline locally, yeah. I forgot to say that. Because you are fetching issues from a GitHub repository, if you want to run this locally, you have to generate a personal access token. So why not all can just skip the first step of running this locally and jump directly into working in the browser and adding this DVC remote in the browser? To remind you, you just have to create a new folder in your Google Drive, copy the URL, and create this content in dbc folder config file, replacing this stuff with the corresponding ID of your Google Drive folder. I am sorry? Why do we do it on the fork and not on the drive? Because you are supposed to each one connect your own DBC remote. Otherwise, we will be editing all the same repo. The idea is that in your fork, you put your own DBC remote with your own Google Drive folder that you can delete later. We are all going to do the same, but we are going to duplicate all the results in each Google Drive account. And I think, yeah, I am doing this on my own fork. So how are we going? Have we added this remote? I see some yes, yes, yes, yes, okay. So once you have these changes, you can commit the file, adding Google Drive Remote. All right. So this is how you ask a DVC remote from the web. If you are running this locally, I encourage not to do it because of the connection. You could do this also from the command line interface with this command. There is a link if you want more information about how this works. So because now we have added this DBC remote, we have enabled what I call shared reproducibility. The idea is that before in our local machine, after we run DBC repo, we could ask and commit the changes, track it by Git, and we could push them with Git now we can also run DBC push to upload the associated big files to the DBC remote so in if now we go to a different machine we can run git pull to get the changes and we can also run DBC pull to get the data set and the model generated by the previous run so we have enabled a way to share the full pipeline including data and outputs but still working in this git like workflow. The good thing is that if you want to use a different remote you just have to go to the documentation and check what are the credentials required. We support pretty much all the major cloud providers as remote including Amazon buckets, Azure, Google Drive, which we are using, and you can also go creative and use custom remotes in your own servers with SSH, HTTP, and a lot of different protocols. If one of the remotes is not supported, the only requirement is that it's compliant with with FSPEC file system, so if you know what is that, good chances is that you can implement your custom remote. But the commands and the workflow would be the same. The only thing that would change is how this config file looks. All right. So that's shared reproducibility. Raise your hands if you have completed this. You are not supposed to run the commands. you are just supposed to set up this for the next step. So everybody has like, it is this config file. Yeah, the content of what you are supposed to copy paste is here. Is there another question, sir? Sorry, can you repeat? Okay, what we have accomplished with this is that we have enabled the DBC push and DBC pull commands. So before, from my machine, I am researcher A, if I update a hyperparameter and I run DBC repo, there are these small files that can be tracked by Git and pushed by Git. We could not share we didn't have a way to share the data set and the model, right? Yeah Yeah, this is how you would do it from the command line Well You can actually have multiple remotes per project and Each individual file can have a custom remote, but in this case is very simple. We are going to put everything in the same Dbc drive Google Drive, sorry so locally Because I was editing in the web in the web. Oh This is not my fork All right So I'm going into online reproducibility. We have to grant, we have to create a GitHub personal access token because of the error you faced before which is that our first stage is going to fetch a lot of issues from GitHub so you are going to reach the limit of the API but you can generate a personal token If you go to your profile, settings, at the bottom developer settings, personal access token, I created one but I can delete, and you click on generate a new token, you give it a meaningful name, and you don't need to grant any permissions, you just need a personal token so you can just generate once you generate you can copy the value of the token and back in your repository sorry back in your fork you can go to settings and create an action secret secrets actions I am going to repeat this process okay I'm just doing it myself personal github token and I copy paste the value okay I am going to repeat now the the process I am in github I go to my profile settings at the very bottom there is the developer settings section at the very bottom there is the personal access tokens and we generate a new one we give it a name and we don't need to Click any of the other stuff We just generate the token This is This house is going ID is going to appear here. You need to copy the value Okay, and now going back to your fork I am in my fork now I Am I go to settings We are in secrets, Actions, and we create a new repository secret. We paste the value that we have just copied, and we name it personal GitHub token. we click add secret. I already have it, so I cannot do it. This is just because I am stupid and I created a data set that requires GitHub API to bypass the limits, but here Do you all have your personal access token stored as a secret? Great. So we are going to do a similar step, but now in order to grant GitHub workflows access to our DBC remote. So we need to give GitHub access to this DBC Google Drive folder. So I have two options. If you are working from the web, this is super hacky. But I added here a link to a Google Colab. I am in the Docs folder, online reproducibility document. And in the section, grant GitHub access to the DBC remote, there is a link for a Google Collab. I am just realizing that I made my life so much harder than it could be for you to follow this workshop. But I basically have a Google Collab so you can get the right content of the credentials from the web. Just run all the cells, all the, I think, yeah, sorry. If you reach the Google Collab, please create a copy. And run the cells. Here you are supposed to copy paste the Google Drive URL in the second cell. you are supposed to run all the other cells to finally get, sorry, yeah. Once you run this before the last cell, you are going to be prompt to go to this URL and grant DBC access to your Google Drive. And eventually it's going to give you this authorization code that you can copy paste into the cell. And it's going to retrieve, finally, the access token. So this process is a little tedious. Are you all in the Google Colab? Yes. Are you facing some issues? The important part is that you paste your URL of your Google Drive remote in this cell. What do we do? You are supposed to store a copy of this in Colab in your account. It's actually just because I was focused on being able to run from the browser, But if you actually install the dependencies and follow the previous steps, the first thing you run DBC push, you are going to reach this Google authentication page and you are going to copy paste the authorization code. So locally, you should have this credentials file stored in a githignore folder. And this is the content you need to grant GitHub access to the DBC remote. Is there a question? No questions? Is there in-person questions? Yeah. So what do we do after we, like, on the Google Authentication page, we have the app? All right. Yeah. I am waiting so everyone has this final output with the right access token. Have you all run the Google Colab? Yeah? Yes? Kind of? What's kind of? I mean it's supposed to be in your fork and in this collab is supposed to be on your account I choose not to have access to anything yeah but only for that folder yeah I mean we have been famous so you can treat me sue me but yeah it's only spot for the folder so do you all have the generated this final token with with your Google Drive URL. Cool. So you copy the full JSON, the final output you copied. And back in your fork, we are going to store this in a new secret. So the same process we went before. We go to settings, secrets, actions, we create a new repository secret, we paste the whole all JSON authentication stuff. And let me check what is supposed to be the name. Yeah? All right, yeah, it's G drive credentials data. Thank you. It's supposed to be in the notebook somewhere. So this is how your secret is supposed to look. Have you created this secret? Yes, raise your hand or something. Yeah? All right. So with this, finally, GitHub is supposed to be able to fetch issues from the DBC repository and have access to the Google Drive remote. So we can go back to the online reproducibility step. And we can create a workflow that runs whenever a new PR is created in this repository. The idea is that whenever a new PR is created, we are going to run the DBC pipeline inside the GitHub actions runner. And we are going to pull and push data to the DBC remote. And we are going to use CML to send a report that will be posted as a comment in the PR that will, it's supposed to look like this, containing all the changes of the metrics, parameters, and plots. OK. So the workflow that you will need to copy-paste is here. I can copy all the content. And the idea is that in the web editor, you create a new file under GitHub slash workflows called onPR. The name is not important. The important part is that it's located under GitHub workflows. So I copy all the content from this snippet. And I am back in my editor. And inside GitHub workflows, I create a new file. And I name it onpr.yaml. I paste all the content here. Now I am trying to explain very quickly what's supposed to happen. So this is the syntax for our GitHub Actions workflow. And we are telling that it's going to be triggered when a new PR gets created. And we can also trigger this workflow manually. What this workflow does is that it installs the requirements, and it runs the pipeline, pulling all the necessary data if there is some data in the remote. The first time you are going to run this, it's not going to pull anything. but subsequent runs. It's supposed to pull the required data. And all this stuff about secrets, it was all for this step to be able to fetch and put data into this Google Drive folder. And the GitHub token is just so the GitHub API doesn't complain. So this is the same thing that we run locally. And then what we do is inside the workflow, we push the changes to DBC, to Google Drive. Because we, yeah. So if you do that because of the secret, the workflow wouldn't run on PR to get some of the fork? No, you're supposed to create the fork on your own, a PR on your own repository. I mean, if I'm an organization with a ton of people working on the same project, I cannot accept PRs from their folks. Yeah, I guess if you want that set up, you are not supposed to work in separate forks. You are supposed to work on the same repository. Or, I mean, there must be ways that you can trigger workflows from other repositories. I think there is a repository dispatch options, but I am not super familiar with GitHub Actions. So all this is happening in the GitHub runner, and after the pipeline has run, we push the changes to Google Drive, and we have the CML utilities, which are meant to track, because we are updating files inside GitHub Actions, we have on the one hand files that we want to share through DBC, and this is the DBC push, but we also have these files that we also want to update, and this is what the CML PR functionality does. It's going to create another PR targeting our original PR that we can just accept, and it's going to merge these changes, which are basically the files that before when I ran locally, I Git added and committed manually and finally all this code is only below or is just to generate the fancy report that is going to be posted as a comment so just copy paste the content into github workflows on PR and save the changes so if or PR workflow just commit the changes in your work in your fork now you are supposed to have a file with that content how are we doing do we all have this file yes yes more yeses yeah so because now we have this file from the gig from the github github interface I go to the parameters to jammer file I edit and then I'm going to chase epochs from 10 to 8 I don't know important thing is that you create a new branch on a PR instead of directly committing the changes. So at the bottom you select create a new branch for this commit and start a pull request. So I click this. And I create a new PR. When you reach this point, GitHub actions is supposed to trigger the workflow that we have just created, which is happening right now. So with this, we have allowed anyone with access to the repository to trigger the same pipeline that we were running locally. And they are able to run custom experiments from the web interface. And the experiments can be managed with regular Git workflow, you can decline or accept these PRs. This PR is going to take a while to run. You can just check that it's going to run the DBC pipeline, share the changes, and all that. Because run the DBC pipeline, oh, beautiful, unexpected error. Yeah, it was supposed to work. Let me quick check if I messed up something. Has anyone created the workflow? Yeah, I don't know, what is this? I actually think that I say too quickly that this though no it cannot be Yeah, it's failing in this step, Actions Checkout. So I don't know if it has to be with a fork or something. Did you raise the CR against the original repository? No, it's here. Yeah, funny stuff. So luckily I think I have an actual repository working somewhere. where I can show you how this is supposed to look. I have a question. I mean, I need something, but can we add a custom runner to all of that? Because I'm using it as a custom runner. A custom runner? Well, it just doesn't run, because we don't run it in my time. Sorry? They don't have a runner, so actually, So we are entering the click ops approach. Are you saying that if you trigger manually, it works? Yeah, I mean, I am going to unarchive this repository. So, I don't know what is going on. That's the issue when a researcher is doing DevOps. But yeah, if you go to this link, workshop on cool ML Ops solution, there must be a lot of closest PRs. And actually, I know that it works because it works in this repository. So let me try to do it here. It works great. Well the idea is that if you have in your team someone that knows GitHub Actions can help you get this workflow running. I don't really know what's going on because I tried this yesterday and it worked. But going back to the narrative, the idea is that if you update the parameters, create a new branch and trigger a new PR, this pipeline and workflow that you were running locally is supposed to be running in these GitHub runners, but for some reason it's not working Now it might be because of I run inside our organization and something is different. But yeah, I am going to have this PR running in this solution repository that you can access because now it's public. And with this, eventually if you get it working, you are going to achieve online reproducibility because anyone can reproduce the pipeline from the GitHub interface as long as there are no errors. The thing is that the GitHub runners might not be powerful enough. That was one of the things that the DevOps versus MLOps blog post was saying. And for that, there's this feature that I was smart enough to not include in this workshop because it's adding another layer of complexity, but it allows you to use self-hosted runners inside GitHub or GitLab. And these self-hosted runners are supposed to be a cloud instance from any of the major cloud providers. So your workflow would be the same, but instead of running on the GitHub runners, you add this step that uses CML runner to create a new Cloud instance. And then the steps below, for example, the running the DBC pipeline, is going to run on this self-hosted runner that lives in the Cloud instance. So CML is going to take care of granting GitHub access to this Cloud instance. And after you finish this workflow, the Cloud instance is supposed to be killed. So this is CML self-hosted runners. And you could do this same workflow we are doing, but we will be relying on Cloud instance for the computation and training of the model. Now this is also not working, and I am pretty sure now that this is a bug with the Actions checkout or something. All right, yeah. Sorry about that. I swear to God that I ran this yesterday, and it worked, and I didn't change the code. and you can find are you serious mother fuckers apologies nobody's fault we all go from we all break things but come on All right, it's not my fault. I guess I could say Yeah in the solutions. You can look for PRs that before this amazing change We're working. I Close all of those but yeah, here's an example where I create a similar PR updating the parameters The workflow actually run. It's the same that you copy pasted and the CML report looks like this So it's a GitHub comment with the difference of the metrics, the parameters, the plots. And it actually contains the all. So if you merge this, on the one hand, the Google Drive remote is going to have the right versions of the data and models. And your GitHub is going to have the right metadata files and parameter files and everything. So the idea is that you can handle this experimentation in a similar way as you handle source code, I guess. Right, so the issue is that the next steps rely on having a successful run of the previous workflow. So I am going to walk you through, and whenever this bug or whatever is fixed, maybe you can try it. So the idea is that because we have successfully run a workflow in GitHub, and GitHub has access to our DBC remote, we can also include a new workflow that is also not going to work because it uses the same Actions checkout stuff to deploy the model. So the idea is that the model gets automatically deployed from the main branch without manual intervention. In this case, what I choose to do is to use a Docker file inheriting from the Hugging Face Transformers image, and what this Docker file does is it copies the model, which is located in the output stream. This file is tracked by DBC, so it doesn't exist in your machine or wherever you are running this unless you run DBC pool. So if you are in the main branch and you run DBC pool, DBC is going to get the right version of the model from the Google Drive remote and is going to copy into this Docker container. Then we have an inference script that is very simple. You can check it out later. And we provide an entry point for this. So once we have this Docker file in our repository, we can use, I am reusing an existing GitHub actions workflow, that's the thing I like about GitHub actions is that you have marketplace where you can copy paste existing workflows. So if you search for a GitHub workflow for deploying Docker images, it looks exactly like this. The only change is that before building the the actual Docker image, I run DBC pool to get the right version of the model from the Google Drive remote. And with this, I am building a Docker image that is going to be stored in the GitHub container registry containing the latest version of the model that someone trained and merged into the main branch. I think that because of this solution repository, You can actually try, so whenever you run this workflow, once you merge and once you create a new commit, this workflow is going to be triggered, a new Docker image is going to be created and registered into this GitHub package. So from your local machine or whatever, you should be able to run these commands and get the Docker image that contains the latest version of the model. I think I can run this, actually. So this Docker image has an entry point with the symphonies script. The idea is that if you pass an example issue title, it's supposed to return the prediction of the model in JSON format. So if you run this command from whichever part you want, and I actually launch Docker, this works. If you don't have the image locally, it's going to get the latest version from the GitHub package registry, and, yeah, you have access to the latest model that some from the searcher to Enix with this Docker image. I assume this is not working yet. The URL is wrong. So the deployer image is at iterative, workshop on cool ML Ops. So, this is the URL that you can see in the GitHub. Whenever you are in the solution repository, like this. The latest version of the image is accessible through this URL. So I am running this locally in my machine, telling to use that image to predict a label for this issue. And it predicts that the label is a data sync issue, which is correct, because, yeah, it's related to DBC pool. I can try this with other inputs like dbcx files on github run. And it predicts that the label is experiments. So this provides researchers with an easy way to make the latest version of the model available for whoever wants to use it. The final step of the deployment in this use case is that you can create yet another GitHub workflow that whenever a new issue is created, is going to run the same commands that I have just run locally using the latest version of the model, and is going to predict and assign a label to the issue that was just created. workflow is supposed to be triggered whenever a new issue gets created so in if in the solution repository I create a new issue with the same stuff same title. The workflow that you are supposed to commit and push in this repository is going to trigger a new Actions, a new run of the issue laborer workflow, because we just create a new issue. And this is going to run the prediction and eventually is going to assign a new label to the issue that has been created. I am not sure how much this is supposed to take. There are a lot of stuff that, because I am not a DevOps guy, I am not doing, but you You could use GitHub cache to cache some layers of the Docker image, you could cache the requirements of your project so you are not constantly installing the same dependencies each time a new PR gets created. All right. So this workflow has finished, so the issue that I created now has a label called experiments. And this was created with the latest model that is available in the main branch that a researcher could have created via PR. If this stuff with the checkout action didn't happen. So this is the deployment part. And finally, just a minor extension of the workflow we already have, is that because GitHub Actions workflow actually allows you to run PRs on different triggering, we can use a cron schedule to automatically run the same process at the end of each day. Because new issues get added to the DBC repo every day, this workflow is going to be very similar to the one that we trigger when we manually create a PR, but in this case we are using the set param functionality of the DBC expram to update this parameter that we have that sets the boundary of the final date that we want to fetch issues from, and we we are going to set to the current date that this workflow is being run. So this is going to fetch a new data set from the DBC repo. It's going to train a new model. It's going to publish that model. And it's going to create a PR. So every day, we are going to create a new PR, a new version of the model, using the same configuration that is matched on the main branch. Whenever a researcher finds a new parameter that works better is going to get merged into the main branch. So this new model is going to be retrained with this new parameter that is supposed to work better. And this is like the final step of the automation, which is where the researcher is never going to have to trigger the model again. You could leave the same model with the same configuration, just being retrained as new data arrives. And yeah, I think that's the end of my workshop. I am truly sorry that we faced this issue all together but the solution repository is publicly available and there is proof history that this works at some point and And whenever this issue gets fixed, your fork is also supposed to work, and you can modify wherever you want. So I think that's everything. All right. So ask your questions, either via Slido, or just raise your hand. All right. So the CML provides a different set of utilities. The major one is the self-hosted runner. The idea to have a powerful machine to run inside your GitHub workflow. In addition, it provides another utilities for publishing markdown reports as part of as GitHub or GitLab comments. And this is the sense comment utility that we were using. We were also using this PR functionality that all it does is you provide them with some paths of files that you updated during your workflow, and it's going to create a new PR with those changes. Otherwise, all these changes to Git tracked files will be lost inside the workflow. So it's a convenient subtlety to enable this workflow to still be only on GitHub. Okay, next question. So, the idea is basically to also have some kind of link to your data, tracking changes, et cetera, also in the GitHub repository, the way you would track the source code. So basically, you're seeing this also as part of the source code, because the model also depends on the data, right? What I didn't get yet is if there is changes in the base data, you will have, like, a lot of stuff on your file system, right? And the links are just created by these DBC meta files, right? But you will still have multiple copies of the data, right? Yeah. It's not like an iterative approach where you just always add new data to it, but it's really like separate data sets. Yeah, also because in this case, the nature of the data set is incremental because we're fetching issues. But on the one hand, DBC handles deduplication at the file level. So unless the title of the issue has changed, even though I am fetching these issues, whenever I do the DVC push, because the hashes of the files, individual files, are still the same. Those files are not going to be duplicated in your remote. So this is all handled by DVC? Yeah. Okay, thanks. Yep. Yes? Would it be easy to integrate the check and the sign-by? Yeah, I don't have it, sorry. Would it be easy to integrate checks into these pipelines? For example, checking that the accuracy has actually increased in the news portal? Yeah, I mean, because all this is happening in GitHub Actions workflow, you can ask any step that you want. And so after I have created this report, you could ask an additional step that runs a custom Python script that checks that the accuracy has improved or something. So all the data is here. The model is here. You could evaluate and ask a custom check. you could set up an automatic merge of this PR based on that rule. Okay, another question? Okay, thank you very much, David. Yeah.