From idea to production in a day: Leveraging Azure ML and Streamlit to build and user test machine learning ideas quickly

Speaker 1 [00:05]

So from idea to production in a day, leveraging Azure Machine Learning and Streamlit to build and user test machine learning ideas quickly. In the future, how is the job that we have as data scientists going to change? What are we going to do now? Generative AI is maybe leveraging some of that coding work away from us. What do you think matters? I think it matters what to do and how to do it fast. And at work, we have this perfect practice playground for this. So we do these hackathons. Who has participated in a hackathon? Okay, nice. Many. So the idea is this. We get some data from the business, and we have a business problem. And just within a single day, we try to come up with an idea of how to solve this problem with data science. And the goal is that at the end of the day, we have some kind of product that is usable. And I think this environment fits very, very well to the challenges that we as data scientists have in the future. How to do things and how to do things fast. So today, I want to take you through this. I want to take you through the question, if you just have one day to go from idea to production, what should you build? I also want to tell you how to build it. I want to explore a stack with you. We're going to build an app that enables you, I hope, to build something in a day. Definitely, I hope that at the end of the talk, you go away and you have learned something new, a new way to create business value from data quickly. My name is Florian Roschek. I'm a senior data scientist at Henkel, the company behind brands like Loctite, Schwarzkopf, and Purcell. I'm also an instructor for Apache Spark, and I'm also volunteering for NumFocus. If you want to reach out to me, then do so best on LinkedIn. I'm pretty active there. Now, let's go. If you just have one day, what should you build? I'm advocating for building a minimum viable product. And a minimum viable product does two things. It has a minimum and enough amount of features to be usable. And it has the ability to collect user feedback. So a minimum viable product is all around generating value for the user. How do you do this? So when you build a minimum viable product, you can build out this so-called build, measure, learn loop. The idea is this. You build something. A piece of code. you train a machine learning model you release it into the world to your users and they provide feedback you measure what they do and based on these measurements on this feedback you learn and then you get new ideas about how to improve and you build again you improve your model this is called build measure learn loop and for data people there's another loop that is inside embedded in that loop it works like this you have some users who use your tool once again you collect feedback and data and then you can improve your models you can roll out these models to users and you make them happier and this actually happened in one of those hackathons we built a product and we had some people who are not data scientists use it and they showed it to their colleagues and their colleagues came back and said hey this is really cool we also have data that is very similar can you use this in your model so we can also use this product and this is called the data flywheel. So when you just have one day, I think you should focus on building out this build-measure-learn loop on building this data flywheel. So are you ready? Think about the day, the moment that you start building. All right, what happens? You've got a lot of issues. Data not in place. Or you have environment issues where I do something on my computer, my colleague has something they do on theirs, and suddenly our environments are mismatched, and we can't even run the same code. Or, especially under stress, I write code really fast, my colleague writes code really fast, and then our code bases don't match up, and then one day we just don't have the opportunity to collaborate there and get together. You get lost in modeling. I mean, how exciting is feature engineering? Very exciting. Hyperparameter tuning, very exciting. But it's also a huge time drag. Or you build your model, you know, you package it up maybe as a pickle file. Now, the people that I work with, they work in chemistry labs. And when I tell them, hey, I have a cool model and a pickle file, they say, oh, yeah, I like pickles. What's the file? I really don't know. So somehow you need to get around this, and you need to build something that is appropriate for the end users that you have. You throw something over the fence, and you never hear back. You don't know, are people using this or not? How do they like it? So I want to help you with a lot of these things. These are all issues that occur when you try to build out this build, measure, learn loop in a day. But there's one thing I can help you with when the data is not in place. that can take a lot of time. So you need to get the data beforehand, before you start building in a day. But for the rest, let's now look at how to do it, how to build this out. So I want to introduce this time-saving stack to you. Environment issues and difficult collaboration, we're going to use Azure Machine Learning and Azure Machine Learning notebooks to get around this. Then, before you get lost in modeling, let's try automated machine learning. These interface issues, I'm going to talk about Streamlit and show you how to use it. And then lastly, if you are not getting feedback about use, there's one really cool thing that you can do with Azure Application Insights and Streamlit, and I'm going to talk about this later. So I also want to tell you, I'm not a salesperson for Azure, for Microsoft, or I think you can do this with a lot of different tools. But here's how we do it in Azure. So let's go, and let's go and start building. What are we going to build? So we're going to build an app, and imagine we have customers who are waste management professionals. And what they want to do is they want to evaluate different computer vision solutions with their workforce. So the people who are using this, who are evaluating, they are not people who know a lot about data science. The idea is that we present them with an app where they can see, similar to how you have it here on the bottom right, images of trash that are labeled, that have boxes around this. This is what the machine learning model predicted. And they need to say, okay, this prediction is correct or this is wrong. They need to provide feedback. So that's what we're going to try and build. How are we going to build it? Let's make a plan. So we have five things we want to do. We want to get data. We want to train a model. We want to build an app. We want to deploy the app with a model. And then we want to collect feedback. Let's get data. So for this kind of problem, I want to give a shout out to this open source data set. It's called TACO, Trash Annotations in Context. And it contains 1,500 images with over 4,000 annotations. And probably it's a little bit small here on the right, so let's zoom in. What does that mean? So here you have some context. You have a street, and there's some trash on the street. And the trash is labeled with these outlines and text labels. And this is the data set that we're going to use to build this app. Now, how are we going to do this? We're going to do it on Azure Machine Learning. What is Azure Machine Learning? It's a cloud-based machine learning platform by Microsoft, and it incorporates a lot of open-source tools. So, for example, you can run Jupyter Notebooks on it. You can also run and track experiments with MLflow. You can version data. You can version models. And this is what we're going to use to get data. Now, how do we get data into Azure Machine Learning? Here's how it works. When you get data into Azure Machine Learning, you want to create a data asset. Here, I've called this data asset tackle annotations, and we're going to look at it in a second. A data asset on Azure Machine Learning is like a mask for files. So imagine you have some files and you want to share them with your colleagues. Usually, you don't only share the files. You also say, hey, by the way, this is where I got them from. This is the license. So Azure Machine Learning enables you to do this. It's like a data asset. It's like a pointer to the files with metadata attached to it. And it lives in this Azure Machine Learning workspace. Now, how do you create such a data asset? You can do it via the Azure ML web UI, or I like to do it via code. So you can do it via code. And I've prepared a notebook here for you. You get access to it at the end of this talk when I'm going to share the repository. But the point is these Azure ML Notebooks is just a Jupyter Notebook that is running on the Azure ML Cloud platform. And the cool thing is they have a managed environment. So you and your colleague, you can both use the same environment. So how do we get this data from Taco into the Azure Machine Learning platform? There's this repository, and you can clone it onto the platform. And then in the Taco repository, there's a download script. So you need to download files from the open internet somehow into your subscription. And the way to do it is this. Attached to the Azure Machine Learning workspace, there's a blob storage. So you can download all the pictures from the open internet to the blob storage, and now it's in your, in my case, you know, your company network. And then you can work with it. How do you create this data asset? Here's how to do it. You can write a piece of code and you say, hey, Azure Machine Learning, use these files that I just downloaded and create this mask and store it here in form of a data asset in this Azure ML workspace. And then I can share it with my colleagues. So you get two giant advantages. You now have a reproducible environment and you're ready for collaboration. So you can work on it as a team, which means you're faster. So this is what the data asset looks like when I go into the Azure Machine Learning platform. And I can click on explore here and I immediately have this interactive preview of the data set. And you see there, very small, there is also the outline of the object. So not only are the images now here in the preview, but also the labels. So this is very easy then to work with because you know exactly what's in it. The next step is training a model. It's unicorns, it's rainbows, it's automated machine learning, right? This is going to solve all of our problems as data scientists, right? Well, as a data scientist, I'm skeptical because, come on, like, feature engineering takes so much time. talking with people who know something about the data takes so much time. The thing is, we don't have time. We want to go from idea to production in one day. So we need some kind of baseline model. And I think automated machine learning is a good basis here. What is it? So you try different hyperparameters. You try different model types in an automated fashion. And Azure Machine Learning offers a variety of automated machine learning models for a bunch of typical data science issues. The point is, this is a modeling time saver. You can get started really fast. How do you do this? So I like to set up automated machine learning through code. You need three things. Create a compute cluster, define a training job, and submit the job to the compute cluster. What's a compute cluster? A compute cluster is a collection of virtual machines that is managed by Azure machine learning. And you create this here. I've called it taco training. And in the second step in this notebook, by the way, which I will also share with you, you say, hey, take this taco annotations data set that we just defined and run automated machine learning on it. And then lastly, you submit the job. Now, when you create a compute cluster, I want to give you some tips there. So the first thing is you can choose between a variety of different compute types with GPU, without GPU, and they will cost you different amounts of money. And maybe you want to save money. right like think about it if you go from production from idea to production in a day you save already a lot of time is it really smart to also save money at the same time but maybe you have no choice right so you want to save money here are two ways you can do this whenever a job is run this virtual machine in the cloud is going to sit idle and that is consuming money so you want to shut them down as soon as possible and you can set here this 120 second timer two minute timer when the machine isn't used it's just going to shut down and the second thing is you can actually also choose from a so-called low priority tier which means and in our case we saved 80 percent of cost there running this the only downside is when azure ml needs to compute it will just kick you out but stuff is reproducible so you can then run it again for running automated machine learning here are two things you should know so you can use the data science knowledge to restrict the search space let's say for hyper parameters maybe you know oh the learning rate for this kind of problem is like more or less like this so use this knowledge put it into the limits of this automated machine learning job and the second thing is you can also set job limits i mean when we go from idea to production in a day maybe we only really have three hours for modeling so we want to be done in three hours and you can put this there let's say we did it we submitted the job now it's three hours later ideally you don't sit idle and to down to anything, you know, one day is not so much. So you start building the app, but we're going to get to that later. So three hours later, my training job is done. This is what it looks like. So in the Azure ML UI and the web interface, this is what I see. And on the top right, you see the link to the tackle annotations. So anyone who looks at the job results, they will also notice, ah, okay, this is the data set. So now you know exactly which data you use to train this model. And the second thing is there's an MLflow model. That's cool, right? MLflow models, interchangeable. You can easily transfer them from one machine to the other and all that stuff. Let's look at this later. Let's look now at the models and child jobs. So when I click there, I get into this list here. And you see Azure Machine Learning has tried different things. It looks like only one kind of model, but they're probably different hyperparameters. You see the models ordered by performance. In this case, mean average precision. Let's click on this first model. there on the top left and investigate so this is what you see and when you click on the overview tab on the right you already see some training metrics so pretty easy there and there's also this metrics tab on the top now you see per epoch metrics you see precision recall curves nice interactive floods you can zoom in zoom out do whatever you want but you know as a data scientist i want to get skeptical like what am i actually looking at so here's the tip go to the output and logs tab and there you will see in this std log txt a lot of different outputs that were used during this automated training job and then we can also figure out this riddle what is mean average precision so the key here is on the bottom it says validation metric type voc now you need to read documentation to understand what this actually means but it's there so my tip to you before you lose time over this riddle go and look in the documentation i think the other thing is Even though it's automated machine learning, you still need data science knowledge to understand what's actually going on and to really use this. The good news is we do have a model now, a model that we can work with. So let's go. Let's go and build the app. So let's go back to this MLflow model. Well, it looks good on the surface. But when you dig in, you find that the Azure Machine Learning produced MLflow model has 204 subversion pin dependencies. I tried to install it on this Mac, and it doesn't work. Somehow it doesn't work. There's a dependency, I think, that doesn't compile on the system. But the open source community comes to the rescue. There's something called ONIX, so Open Neural Network Exchange. and it's a file format that enables you to transfer deep learning models between different frameworks. And there is another open source project called Open Onyx Runtime and this enables you to run this kind of model which has a single dependency. You don't have 204 dependencies, you have a single one. So you can then run inference. And this is what we want to use. So you have now the opportunity to deploy a model flexibly. And now we can finally get back to building that app. I want to introduce Streamlit to you. So Streamlit is an open-source app framework for creating database web apps in Python. And, you know, it's relatively easy to use. It looks great, and it works. And as a developer myself, I think this is a really hard thing to put off. Make something that is easy to use, looks great, and just works. So the open-source community, open-source community, they really delivered. So this is really nice. You should also know when you use Streamlit. when you have more complex stuff with like state management in the front end or yeah, this is harder to set up and users may perceive it as slow. It's just not as fast as like a typical consumer grade web app. But you know what? For what we're trying to do, going from idea to production in one day, this is perfect. So let's look at what a Streamlit app looks like. Four lines of code and you get a web app. You have an input mask where you can put in a number and it gets squared. That's all it takes. Let's go back to the app that we actually want to build, the Trash Recognize app. It looks not much more complicated. You have an upload button, and then people can upload photos, and then we run the model inference, and then you get this table. On the left in the column, you have images with the bounding boxes where your model made the recognitions. And on the right, you have another column that says, okay, this is what I recognized. The code is in the repository I'm going to share with you, but here's what it looks like. So, I click on browse files, upload a couple of images, and then you see the person there on the top right involved in various ways of locomotion. This is the loading bar. So, we wait a little bit and hear a prediction. So, it detected something there. No trash in the second picture. And in the third picture, it looks like a plastic bottle and maybe a straw. So, great. This is really an easy-to-use interface to your model. What's next? Let's deploy the app with a model. And here I have to thank the data science engineering team at Henkel because they built a pipeline to do this stuff. Feel free to scan the QR code now. This is the link to our blog. And my colleague Roberto actually wrote up and is currently writing up this kick-starting data science project in Azure DevOps article series. And in the next month, we're also going to publish how we do the Streamlit deployment because they figured out a way to deploy a Streamlit app inside a corporate Azure subscription. You're now behind all the cybersecurity concerns. And I think this is really cool and definitely has been super helpful to me. Now, we still have one problem, and it looks like this. This poor cat is definitely not a piece of trash. And somehow we need to provide this feedback from the user perspective. So let's focus on this. Okay, so there's this really cool way of getting feedback from the Streamlit app. So the open source community, they came up with a tool called Streamlit Feedback. And Streamlit Feedback enables you to have these thumbs up, thumbs down button in a Streamlit app. Now, when this button is submitted, you have a callback function. And this callback function you can use to write to the Python logs. From the Azure side, there's a way to stream Python app logs to the Azure cloud. So now you have your logs in the Azure cloud. And there, there's a way to produce dashboards, live dashboards from logs. so now you get from some users clicking some buttons to live logs in in your azure subscription let's see what that looks like so here now i can finally say yeah a cat is not trash boo submit and then like i think it takes about 10 seconds you see this in your dashboard so there you see a cat is not trash and also this pie chart and the bar charts they're automatically generated from the locks so cool right now you have an easy and fast way to get feedback out of your app. We've come a long way. We've built, measured, and learned. We're almost done with the loop, just like with the Pac-Man this morning. We're still missing a little piece, and this is not technology. Now you need a way to learn, to learn from the user feedback. So the organization that you work in, they need to provide you the space and say, okay, you know, we did this once, let's do it again and again and again and then you build your product this has been tons of fun and now it's time for you to scan this qr code and get into the code and there's also this giant and interesting reading list like everything that i talked about there is at least one or two links okay i'm just looking at the cell phones here thank you so much and i'm looking forward to your questions

Speaker 2 [20:45]

Thank you, Florian. And yeah, we have a bunch of questions for you. The first one, I think, belongs to the data you use to train your model. And the question is, what file format do the annotations need to be?

Speaker 1 [21:02]

Yeah, this is a good question. So in the TACO data set, I think you have the so-called COCO annotations. But in Azure machine learning, especially for this automated machine learning, they actually have a format called JSON-L. So you need to convert the labels from COCO to this JSON-L format. There's code available that is also in the repository to do this. And I think Microsoft also published recently some notebooks, some Jupyter notebooks that show you how to do this. Thank you.

Speaker 2 [21:34]

Okay, thank you. So from your expectations, that's a quite fast thing to create labels for my own pictures, my own data sets. Okay.

Speaker 1 [21:44]

So if you want to create labels for your own data set, you need a solution that enables you to do this. And of course, a picture is not a piece of code. So you actually need some kind of UI to do this. There are a variety of open source labeling solutions. Azure Machine Learning itself, they also have a way to label data, to label image data. And you know what I like is they also have keyboard shortcuts. So you can be really fast because it's kind of boring, but you still need to do it. So yeah

Speaker 2 [22:17]

Okay, good. The next question is quite tricky, I would say. So imagine someone from the management also listened to our talk and now have a quite high expectation. So any recommendation how we can manage the expectation of the management? Because one day solution might lead to unreasonable stakeholder expectations.

Speaker 1 [22:44]

Yeah, I understand the question. This is, first of all, you know, with a thing like this, you're standing on the shoulders of giants because we have the open source community who enables this. You have the data science engineering team that has all the tooling in place to automate a lot of stuff. And then you can start doing this. And this already took a lot. And then I think for me, what I've learned, when you do these kind of things, you can only always talk about a prototype. hype look we're testing something we're testing something um and especially for stuff like this where i think a lot of now now that you have a lot of pre-trained models also a lot of things become possible but i think you know with some technologies we might be hitting like a peak of inflated expectations it's important um to not over promise and um yeah so we have we consistently files when we do something like this or when i do something a hackathon i felt as a prototype which means a lot of things don't work but maybe one thing works

Speaker 2 [23:48]

Okay, I see. Thank you. As a question, you said you are not trying to sell us any particular cloud, and the question is if we are already using in a company some other clouds instead of AWS. So do you have any information? Do we have the same, like any major cloud providers? Do we have the same functionality for such quite fast trade?

Speaker 1 [24:15]

training yeah i mean probably yes yeah i think that you know there are so many different cloud providers and they have so many different tools and i think this is probably also possible in other clouds

Speaker 2 [24:29]

Yeah, so at least from my knowledge, the Google have a vertex with the same functionality and also Azure have a solution for such task. Okay. The question from our audience, if you could use any tool, would you use the same as in your presentation or would you like to change something?

Speaker 1 [24:57]

Yeah, this is a good question. You know, I think it's not about the tool. It's about providing user value. So you should use what you know so you can work fast and create this user value. Even if you know how to use it and you can create user value, this is probably what you should use.

Speaker 2 [25:18]

Okay, good. The next question about tools. Why do you use Streamlit and not Dash, for example, or other tools?

Speaker 1 [25:26]

Ah, yes, tool competitions, tool stack questions. Yeah, there's a lot. I have worked with both, and for me, it's always very hard, and we actually have some tangible experience in this, getting off a Dash app, getting it off really quickly. There's like a callback structure, from my experience, a callback structure that you need to follow, and sometimes when you don't do it every day, it can be hard to build in one day. And, you know, as much as I try to also use different solutions, for me, it's always this, when I then try to do it in Streamlit, and this is not a complicated app, yeah, then it's just much faster. And then when you just have one day, I always default to this after trying all the different things, because I know, okay, I can get something off the ground.

Speaker 2 [26:16]

Okay, maybe to finish this question about Streamlit, what is the maximum amount of data I can upload? Probably that's a question about this UI interface. Does it have any API?

Speaker 1 [26:30]

Yeah, I think this depends on, so this is not an API. Okay, I think the professional way would be to build like a backend API and then have a separate frontend, but this is not an API, so I think there are limits, but I cannot say which ones these are, like as a specific number. From experience, the more data, the slower.

Speaker 2 [26:52]

OK. Thank you, Florian, for your talk. It was extremely useful. Thank you.