Practical DevOps for the busy data scientist

Speaker 1 [00:03]

Hello everyone, I'm Tania Lard, I'm a Developer Advocate at Microsoft, and I'm going to be presenting Practical DevOps for the Busy Data Scientist. You can find me on Twitter, if you use Twitter, under the hashtag check, sorry, the handle is check, and there is where you can find all of my slides in case you want to download them now to follow if you need to have some text, or later on. And I'm going to start with what I'm actually going to cover in these slides. First, I'm going to do a very brief introduction to what is DevOps for data science or MLOps and why us as a data scientist should care about it. I also going to dive into who should be responsible for implementing MLOps into our workflows. And I will give you some practical steps for you to get started. This is not going to be a deeply technical talk or going very deeply into the technical or deployment aspects of machine learning models, but will give practical recommendations on how to start building from where you are currently to start adopting some of the practices. And a tiny bit about me, I love Python, I use Python pretty much every day, and I'm I'm also a Google Developer Expert for TensorFlow and all machine learning stuff. And I also love mechanical keyboards. So if anyone here loves mechanical keyboards, I can see my friends out. Come talk to me. So just to start with a bit of background. If you're not a machine learning person or a data scientist at all, then probably you're used to this paradigm in software engineering. where you have your algorithm and your data as an input, then you have something that processes that, and then in the end you have your answers. And you have normally well-defined answers, something that you know that you can start testing and know what you're expecting. In machine learning, things are inverted a tiny bit, where you will have your data here and your answers, And normally, for example, if you are doing some sort of classification algorithms, you would have those answers in the form of labels. So if you want to identify, for example, from different pictures, whether what you're seeing is an apple or an orange, then you will have to have a lot of images that are already tagged for you to identify that. Then you put your answers and your data together and go and pass that through to create the algorithm. or a model, depending on what you're actually doing. And eventually, you will then use that newly created algorithm or that newly created model, along with new data or data that you have not previously seen, to generate new predictions. As you would imagine, this creates an additional layer of complexity because you not always know what you're meant to get in some cases. or in other cases, you might need to have your algorithm and will accumulate more data or collect more data over time. And this will help you to make your predictions more accurate or more reliable or faster. So you might need to use that algorithm that you created or that model that you created and retrain it or new data to then give better predictions. so how do we actually get into here normally if you're a data scientist or work as a data scientist you will start by doing some research and development and this is a very iterative fast development processes where development process where lgo try probably different approaches or different models to get to something that you can actually then process as something consumable once you have identified the best approach or the best model or algorithm that you want to create that you want to use then you have to think about taking that from the research and development stage into something that can be production ready that can be consumed by your users and here's where normally data scientists start working a bit closer with software engineers to bring that into production. But what happens then? How do we actually make sure that that model is ready, that my customers can go, for example, and upload their Apple versus orange images and get a prediction? Or in other cases, how are doctors using models to predict or use image to predict whether there is breast cancer or colon cancer or brain cancer? And it's usually this part of the process that seems to be a bit magical in some scenarios. How do I actually take that research and development model into production ready? And very so often we see this kind of scenarios where someone developed a novel algorithm or a very useful model, and it's been sitting somewhere for six months, 11 months, for it to be taken into production. And this is because usually data science or data scientist teams tend to hit this wall on fire. That is, once you have prepared your data, built your model, then train it, produce some evaluations and a lot of testing, how do I actually ship it? And the thing is, because all of this pipeline, all of this model generation pipeline is very iterative, it also allows for a lot of failures in the intermediate processes. So when I'm doing my data preparation, I have to first make sure that I know where my data lives. I need to know that I can track it, and I also need to overcome some storage bottlenecks. When I am doing the iterative processes of trying different algorithms and training my models, I also have to overcome sometimes slow processes to do this, and that will depend a lot on the kind of compute that you're using. Also sometimes it can be very very hard to explore the hyperparameter space, because if you're not familiar with machine learning, the hyperparameters cannot be deduced a priori but you will have to do some long extensive search for you to actually determine which are the best hyperparameters that will give the best optimization on your models and then if you have very compute intensive applications you also want to distribute the loads or distribute the compute but actually managing these distributed clusters can be very very hard so you need help from somebody else and when we're finally ready to put this into production and serve we encounter the common problems of it worked in my machine but now it doesn't work in production or it's not scalable enough what happens when I have only one person trying to access my resource versus having a hundred or a thousand people trying to access that resource at the same time so when we think about how we can overcome all of these bottlenecks i want you to think about what your wish would be and think big how would you normally overcome it for me it would as a data scientist or a machine learner what i would want is for me to spend more time focusing on the R&D, productionizing my code, and not having to worry about that wall on fire that is taking it from R&D into production. And here is where MLOps and robust pipelines actually come into place, because this gives us streamlined, secure deployments. And this takes the abstraction from all of that magic that I showed in the previous slides. This assigns proper responsibilities and proper automation tasks for taking these models into production. So, okay, who should be responsible for the actual automation and for the actual deployment? That's a question that people often ask. The problem is that in many companies, we see that data scientists and data engineers overlap quite a lot. And we think that we share certain responsibilities that part in the middle, in the Venn diagram. And software engineers seem to be perceived or are perceived very, very far having very different set of skills. So they're involved in the conversation or involved in the process very late. And the truth is, when we think about skills, it is not just that Venn diagram and that overlapping part between data scientists and data engineers. But actually, it's a convergence of skills where data scientists are really, really good at doing all the machine learning, AI, data learning, advanced analytics and statistics. And data engineers are very good at pipeline state integrity validation and the readiness or the skill betterment of each one overlaps as in a gradient. But we still have that part in the middle that is underserved, that is underlooked. And this is where the role of a machine learning engineer or someone that takes care of the operations, productions, and optimization comes into place. and here's where we have all of this ability to integrate devops into our machine learning pipelines because devops is actually the intersection of these roles and the processes and products so we can deliver better value to other customers to our company to our teammates. And truly, it is a true team effort. And by adopting these practices into our machine learning operations or our machine learning workflows, we are trying to reduce all of these end-to-end cycle times and eliminate as many frictions as we can. So I hear it that all of you say, I don't work in a big company with a lot of machine learning engineers. I don't work in a big company where we have these very distinct roles, but I work in a place where everyone is pretty much responsible for everything. And I am a data scientist by day and in the evenings, I'm trying to ship my products and I do a lot of stuff. The beauty of open source and the beauty of having things in the cloud is that you can start building your machine learning ops or your MLOps platform with what you're already using. You do need to start integrating the tools. So you can start if you have your code on GitHub or GitLab or Bitbucket. That's fine. That's a good start. Then if you're using things like Kubeflow or machine learning services or machine learning workspaces in Azure, that also works. And then you can integrate that with Azure DevOps or Jenkins. or CircleCI or GitHub Actions to give you the capabilities of continuous integration and continuous delivery. So that way you can bring all of the beauty of our vast data science ecosystem and machine learning ecosystem to build robust pipelines, to build robust ML platforms. And this platform should take care of your environment, of your development, of your testing, training and retraining of your models, deployment, and all monitoring and alerting. And you can put your stack together with all of the tools that you're using. The previous talk integrated was about Airflow for beginners, so you can also have things like Airflow and Luigi or Qflow to orchestrate your pipelines or schedule tasks. So that way you can have these three distinct roles of your data scientist, your developer or your software engineer and someone working in operations all working together towards the same goals using the tools that they normally use to develop their job or to to their workflows and get a reproducible, automatable stack. So you as a data scientist, you can still use Jupyter Notebooks, Version Control, Azure Machine Learning Services, and Azure Pipelines to integrate with your model management. Your software engineer friend or your colleague can use their favorite tools, whether it's VS Code or PyCharm or anything else. Again, use Version Control, build the pipelines for continuous integration, and build Docker containers. And then the operation person will take that pipeline that was created before it to take into production, for example, using Kubernetes. So how do I go, or how do you go from what you have now to this automatable MLOps platform? First, start by having everything in version control, in source control. A lot of the data scientists use Jupyter notebooks, so if you already have your Jupyter notebooks in source control, that's a good step. Even better, if you have everything, all your code, all your comments, sanitize your Jupyter notebooks when you push to GitHub, make sure that the outputs are not there. Every single part of your pipeline, also your infrastructure and dependencies, try to have your infrastructure as a code, and that means capturing your environments, and maybe a subset of your data. If you're providing additional data for your folks or your colleagues to reproduce a minimal working example, make sure that that is there. Accept your training data. Your training data version control or GET is not intended to work for versioning and controlling of large data sets. And also keep in mind that all your training data, your raw data, should be a known shared data source that everyone in your team could access. And very important, your raw data should not be changed. It should be immutable. You don't touch it, not with a stick, with nothing. Your raw data is there for a reason. The second step that you can start introducing is having deterministic environments. How often do you pull a Jupyter notebook or a script from one of your colleagues, or even a notebook that you ran six months ago and you end up with things like this, like import a package and that package doesn't exist, or something has changed, or you're using an outdated version that has significant API changes and all of your results are changed. Whatever environment you're using, whether it's the cloud or you're developing on a Jupyterhub that is self-hosted and self-maintained or you're developing locally with your editor of choice make sure that your environment is as deterministic as possible because your laptop is not a production environment so you need to bulletproof yourself for reproducibility for example i use cond environments because it gives me a lot of flexibility for me to be able to try different versions of packages, use different Python environments, but then when I want to take things into production, I normally have to convert that into a Docker container because that works on my machine, on a virtual machine, and on Kubernetes. maintaining both a cond environment and Docker containers can be very very problematic. And that's where libraries like repo2docker also helps you to abstract that. So if you already have your code into GitHub and your environment in the form of an environment.yaml file or a requirements.txt or things of the such, you can use that to generate rate version control repo, Docker images, and Docker files. Again, when you're trying to define your environments, make sure that you specify the versions as much as possible, especially those that are critical for the operation of your models. The next step is using pipelines for repeatability and repeatability. It's very very hard when you have 1000 Python scripts to identify what tasks are being run in which order. If you have a very well defined set of steps and tasks that should be run in a specific order, it's easier for to trace back and stop any tasks that need to be done again or need to be modified. So when possible, try to use things like Kubeflow, Airflow, Luigi, Azure Machine Learning pipelines where you can have as a step one, for example, your data preparation, step two, direct training, step three, model registration, and so on and so forth. And most of these options or most of these tools have a visual interface that also allows you to track every single run. For example, here it can say that I have a pre-processed step, a training, a registering. And it also tells you when it was done, the duration, and whether they are suitable or set up for reuse. So this gives you a very, very good overview of all the tasks that you're running and all of your pipelines. So it doesn't matter how complex your task is. In that case, I had preparation, training, and registering, but you can also integrate other tasks like creating the Docker image there that you're going to use and use a Docker image to attach it to your compute and run that pipeline. But if you have all of this defined as code, your infrastructure defined as code, and in version control, it's much easier to keep track of how these tasks and these pipelines evolve over time. And the next step is automate wisely. Automation can do wonders for you and your workflows, but you also have to have a very critical eye about what you're going to automate. Sometimes we can spend a lot of time trying to automate a task that takes us five minutes and we only do once every year and then you have to think whether that automation is actually worth it. If you in turn are performing a scheduled task every week that actually takes you three hours or half a day or a full day then probably that's a task that you could be automating. Something that you could definitely automate is orchestration and continuous delivery of all of your machine learning products. If you start integrating Azure DevOps or GitHub Actions or Jenkins or CircleCI into your workflows, it also allows you to detect errors and bugs earlier, way before they reach production stage. And you can also integrate with many other services. You can start having smart alerting when your model is not performing the way it should be or when a merge request is introducing breaking changes into your code bases. And you can also set it so that it triggers on code events like branching, merge requests or pull requests and also non-coding events so imagine that you have a model and you want to update your dependencies every three months or every six months you can set this automated tasks that will upgrade your dependencies test your code base or your model against these new dependencies and report and only update them if they pass the tests and you can have your continuous integration in your full machine learning pipeline starting from those initial steps that i showed before that is preparation training registration integrate continuous integration to do your containerization and your deployment to kubernetes or whatever it is that you're having and your deployments in and everything starts becoming much more traceable so you can start tracing again your pipelines every single step is code every single step is a version control traceable and you can analyze every single triggers every single trigger that are it that are executed on your code, sorry. From again training to verifying the deploy. So in this case this model is being tested first, creating the container and then all the way on to deployment and testing my infrastructure once my model has been deployed. So in brief, what can I start or what can you start doing to go from where you are onto up to automatable fully robust pipelines? Start doing source control right. Create deterministic environments and however they are defined, make sure that they are deterministic. Start using pipelines for repeatable workflows and start adopting continuous integration and continuous delivery as part of your deployments. If you want to learn more about any of the resources that I put in here, there are the links and you can download all their slides. And just before I finish, also come talk to us at the Microsoft booth. We're more than happy to help you with your deployments if you want to integrate Azure DevOps or AML. And we're also running a group of face competition. So, oh, and that's it. Thank you. right thank you for being right on time so up to a second and you you know the drill by now just raise your hand if you have a question we have five minutes so maybe just four five questions at most way back i see anybody closer Hi, thanks for the nice talk I do have a question about inside the team my experience would be that if you have 10 engineers or 10 data scientists, there will likely be 10 different ways of doing the work. Would you advocate trying to standardize that, a single tool for your pipelines, a single tool for continuous deployment, or would you, or how have you guys approached it? So in previous teams that I've been with, what has worked best is standardizing as much as possible. Because let's say that a team prefers using GitHub, and another one prefers using GitLab, and somebody else uses, they use Azure DevOps, or somebody else prefers Travis or Circle CI. It starts becoming very, very complicated. If someone from another team needs to take over a project, there are a lot of tools and a lot of things to maintain and duplicate code bases everywhere. So we try to standardize as much as possible. Start with something that could be replicated or easily maintained by all the teams or by everyone that should be responsible and then move from there. Obviously you'll have specifics per project or specifics for different things but I would try to to standardize as much as possible. Very well, next question. Don't I see a hand or is there none? There's one. Oh, there's one, sorry. Just take the mic. Hey, so my team has been sort of hesitant to actually end up using notebooks due to the sort of issues when notebooks change a lot in version control. And I figured, can you dig a bit deeper in how you handle those? Are there tools or practices that help you essentially prevent the Git repository from being bloated? Yeah, that's excellent. Well, I have other resources on that topic precisely. Jupyter notebooks are amazing for outreach, disseminations, faster iterations, very excellent, but they don't play nicely with version control. And unless you have very good practices, it can be very, very problematic for production readiness. So what I normally suggest is when you need something that needs to be into production, to have actual Python scripts or modules. If you're going to be heavily reusing parts of the code, you'll have to go back to creating a Python package or a Python module that will give you that flexibility and place nicer with production-ready systems. If you're planning to use, for example, notebooks to create parametrized reports in conjunction with Luigi or Airflow, they're amazing. But again, you have to be very, very careful that every time you're checking a notebook into version control there is sanitized meaning that all of the output are cleared and there are tools that will do that for you and i can't remember the name out of my the top of my head but i can provide the links to other resources or my other slides that have a lot of best practices when working with notebooks and and tools to integrate them in your workflow but i would not recommend having Jira production ready things or Jira productionized models in Obix. Very well. My phone says 1255 and with that we have to release the room and go grab lunch, have a break, whatever, but a round of applause for...