You don’t think about your Streamlit app optimization until you try to deploy it to the cloud

Speaker 1 [00:08]

Let's start this talk with an analogy. It all started as a good love story full of hope, dreams, and a sense of endless possibility. A data scientist built a Streamlit app, proudly said it works on a laptop, and then hands it over to an MLOps or ML engineer. And then ML engineer takes a quick look at it and see security holes, security holes, massive modern loading times and quickly realizes this relationship is toxic. So it turned into, you know, claims and whispered curse at 2 a.m. during the debugging. fear not today we will try to bring this gap and make the relationship functional again and you will hear about how to how to overcome these connections when you are trying to deploy your app from your laptop to to cloud so our agenda is first I will talk about briefly what it was streamlit why do we need it and then we will go over different disconnections like modern loading, autoscaling, authentication, security, sensitive CRUD storage and maybe happily we will have a happily ever after and happily Q&A. My name is Daria, I guess I can skip this part because I had a very nice intro. So first question, how many of you have heard about Streamlit yeah okay once you came here and how many of you worked with it touch it yeah pretty many hands off so yeah I will just keep this part simple and brief Streamlit is an open source Python library to create and share custom web apps for machine learning and data science so why is it good why is it popular because you can have a quick way to show your models or your data to stakeholders it has very simple functionality to use to quickly start building and even if you're not working with external stakeholders it's an easy way to taste to test your models and if you like me cannot write a back and front-end ads can be a lifesaver and also it has a lot of predefined elements so it's hard to make it ugly but I know people who managed so let's go over to the first disconnection as a model loading and as example here we have a lot a bit of Japanese here I just a fun project I built a Japanese learning app so if you know a little bit about Japanese it has a kana alphabet and it is it looks nothing like our like Latin script right and you need to memorize these characters and there are a lot of them so to be able to write them and to be able to quickly test and understand whether my writing is good I decided to build the app and to recognize what I have written I need to some OCR model which can parse the character and map it to a Japanese kind of character so I tried to use some LLMs but they were not good enough to parse a single character and maybe they were expecting for like whole words or whole phrases that's why I decided to use OCR model from a Hagen-Face hub which does optical character recognition so I have a model and I need to download this model somehow from the Hagen-Face hub and how can I approach this task so I can write a simple function which checks whether I I have a model and download this model each time in my Streamlit application and save it to a session state. Why do we need session state in Streamlit? Because every time user clicks on something page is going to be re-rendered. So but this means that models is actually downloaded every single time. So maybe it's not a a big deal when I'm playing with Streamlit on my laptop but you can imagine if we have multiple users we have like a lot of pages with our application opened and this is slowing down the app performance user have to wait they have to wait until the model loads and also it increases costs and And why costs? Because I don't pay anything when I downloaded model from Hugging Face. It's free. Like why costs? So if we take a look at a simple architecture, when we try to deploy our Streamlit to, for example, AWS or to public cloud, it may look like this. So, we have our application deployed as a task running on Elastic Container Service Fargate tasks, and we have it in a private subnet. So, private subnet, it means that it doesn't touch public subnet. It's due to security reasons. And in public subnet, we have NAT gateways. This is something that ensures that your app has outboard network connection. So if you need a model to be downloaded, the download will be handled by NAT gateway. And here is where costs come in because, okay, you need to set up NAT gateway. You pay per hour, but you also pay per gigabyte of data processed by your NAT gateways. Yeah, here you can see that it's not a big deal, it's only like $2 versus $20, but it was, you know, just for sampling purposes. But if you imagine the scale, if we have several models and they're all like more than gigabytes of weight, and we have multiple users, that this number can grow and grow, and we can avoid it, yeah? So we use the Streamlit cache to reuse loaded models, but we also can preload models during the image build and maybe even the better solution would be to deploy models and call them as APIs. But the last option is heavily depend on your setup, on your company and your MLE team. So let's try to have a golden middle what we can do is like a data scientist this was our previous code which we used in our streamlet application but we can move this code actually to a separate separate module which is preload model dot PI and we can have this function like we can have this model called during the docker container building time so we just run this module and we're just downloading and saving the model into separate directory and and we just check this directory in our streamlet up so we're just assuming that the model is there it's being downloaded and placed into into correct directory and we can we can use it during our app run. So this hopefully makes the life of our ML team easy, ML Ops or MLE team easy and this disconnection is being solved. The second disconnection is about auto scaling and authentication and we think about when we work with AWS and we try to build something on a streamlet we have some different options of let's say deployment first maybe they like the quickest one is to try to use Amazon search maker studio proxy with Jupiter lab 3 all then you can use a virtual machine or ec2 elastic compute cloud instance to it's like do-it-yourself your self-approach to host your Streamlit app there or you can use our EWS foregate and elastic container service managed containers as a serverless option and in that case the first option is not about auto scaling it's about like building the app and sharing maybe with your team of developers second option is challenging because you have one instance and you need to do maybe vertical scaling you need to you need to just make your instance bigger and this sort of option despite it involves more services it actually does a auto scaling easy so we have we have our app running as a containerized app using elastic container service and fargate forget does it serverless so you you don't need to manage service servers underlying service under an infrastructure and you can use auto scaling based on threshold and you can put your stress threshold and make it automatic and you also can use a combination of reserved and sport instances here so also authentication it's it's kind of you know like boring type of tags that should be done yeah and it's a big question who should to take care of it and one way to take care about it it's to you know hold it over to data scientists but they need to build everything like login widget the logic of authentication privileges so everything everything everything and MLOps engineer have just they just don't care what's going on inside your streamlet but since we are running for example on AWS you have service like Cognito and here we have a like a split of tasks so MLOps engineer can help to create a Cognito user pool like a user directory then configure up client setup domain name and configure callback URLs where your user will be directed so in that case data scientists just not just need to have proper Cognito authentication in this initialization inside the app they need to authenticate user and check check the permissions so in that case it is it is becoming pretty simple for a data scientist so they just can use credentials and can connect to to the up to the setup up by MLOps engineer and then can inside the code check if the user is authenticated and also depending on your business use case if there are different roles for example reader or administrator so by splitting up the tasks hopefully this relationship became better the next one a very important one is security and yeah you see this setup we have application load balances that balance the load between two availability zones so our app is highly available we have our streamlet app running on the serverless Fargate it is like auto scaled and it seems that it's it's pretty good but we can do better or we can use services like Amazon Route 53 web application firewall and Amazon CloudFront to enhance the security and how does it work so mainly is that there are tasks for MLOps engineer but data scientists should be aware about them for example the first one when we setting up the HTTPS with route 53 and TLS certificates for secure connections role of the data scientists so first to be aware about the difference between HTTP and HTTPS and inside the app try to avoid like insecure API calling and everything that you know put the app back in terms of security. The second task is to configure CloudFront to protect against DDoS attacks and improve performance. So CloudFront works as a content deliverer network from one side it protects the app from like a lot of malicious traffic and from the other side it it it has caching so here's the role of data scientist it's to understand and collaborate with MLOps engineer to establish the best caching strategy to balance between performance and and content freshness and the last one is using web application firewall to block malicious traffic so web application firewall can prevent SQL injections cross-site scriptings bot attacks and the task of data scientist is to validate and sanitize user input inside the application so by understanding the like shared responsibility of security between MLOps engineer and data scientist hopefully we bridge the gap between them and the last disconnection we will discuss today is to how to store credentials securely using AWS and in our architecture you may may have noticed that we're using AWS secrets manager in that case it has been used to store secure headers so Amazon CloudFront uses these headers to communicate with application load balancer and it attach the CloudFront attach the secure header to each call which is being done to application load balancer and application load balancer on its side checks the value of the secure header and all the values are stored in AWS Secrets Manager to prevent their exposure. But in terms of data in if you think about this concept from the data scientists perspective yeah you know that we have a lot of situations that you know you need to do a quick check whether it's working and you can exposure like token creds API keys just just forget and put them happily we have we have a AWS secrets manager and we have just you know five lines of code set up to be able to use this as a secret is a reliable and insecure storage for your API talk for your tokens for your credentials for your API keys you just need to you just need to use bota3 to to connect to AWS secret manager and load the secret value and you're using environmental variables to get this value by the secret name and yeah so this is happily like bringing the gap because MLOps engineer don't struggle with your secrets exposure so happily we we approaching happily ever after so key takeaways maybe the one biggest takeaway is to is to communicate yeah because when you are walking as a data scientist and trying to build application you very often have no idea what is being done when the app is deployed. You just think that it's like a green button, you need to press it and it is like magic is happening. So you need to think about problems like performance, scalability, authentication, security during the time your app is deployed. And for MLOps engineer, you need to understand that the app development can be challenging and there are like easy and quick solutions that they can offer to data scientists so all the compliance levels are being met. So again, yeah, communication is key. If you're interested in my Canna app that I showed you in the beginning and if you want to have you know hands-on experience how to how to build this architecture on how to try to deploy a stream that to AWS you can you can use my report not to start start from scratch and yeah we have time for Q&A right

Speaker 2 [19:05]

Thank you for the wonderful talk and yes, we have some time for Q&A session and the first question How do I handle cold start delays when auto scaling hugging face model in streamlit? Especially for large models then exceed lambda size limit

Speaker 1 [19:25]

Okay, you are talking about Lambda. Oh, okay, okay. Cold stars delays. You know, the question, here is a question I see that Lambda is mentioned, but it is like, at the moment, it is challenging to use Lambda with Streamlit because of web sockets. so so far I haven't seen any like example of lambda usage with trimlet it's mostly about having this containerized and with with the help of fargate and elastic container service about cold start delays I guess I guess if you cannot like quantize your model if you cannot if you cannot like make it shrink it yeah I need to think about it yeah let's meet afterwards and discuss it

Speaker 2 [20:38]

What size of the image, which grows with embedding the model, has a negative impact on the performance?

Speaker 1 [20:47]

okay so I would say that that I would say that the image size if you're working with elastic container service I wouldn't put this this way I wouldn't say that it would have like a huge negative impact on the performance I think that it it will depend on on your use case and how do you set up the auto scaling and all the threshold and also and also actually here the perspective of data scientists is also valuable because you can for example if you have an auto scaling configured using using ECS and forget you can set up not only like threshold you can set up also some kind of rules because you can think that during business hours that are like more people are likely to use the app so yeah it is it is I would say it is highly dependent on how you can configure the autoscaling itself

Speaker 2 [22:16]

From your perspective, is being able to fetch the secrets via the environment of your container a secure enough way of managing runtime secrets?

Speaker 1 [22:44]

would say it is it is considered as a like better practice to use AWS secrets manager and yeah so far what I've seen during that unfortunately there are not a lot of you know tutorials and example projects how to how to actually deploy your streamlets to AWS so but so far what I've seen using AWS Secrets Manager is the best way to do it

Speaker 2 [23:24]

Who should be driving the discussion between MLOps and data scientists to bridge the gap?

Speaker 1 [23:31]

you know both because it is good when you have a good product manager or you know good guy in the team who can like a team lead or someone who can set up a table and invite both teams or both both people to a conversation but I think that the key here is to maintain like interpersonal relationship and just to and just just to you know try to speak and talk to each other I don't know whether you work remotely or in office in office that would be easy you just grab a coffee together and you can you can talk about struggles

Speaker 2 [24:15]

Okay, unfortunately, we need to finish right now because Daria should be a superman flying down to the other room. So if you have additional questions to Daria, please, I think you can find her. And thanks again, speaker.

Speaker 1 [24:30]

Thank you. Thank you.