Next Stop: Insights! How Streamlit and Snowflake Power Up Data Stories

Speaker 1 [00:06]

So, hello everyone, and thank you for the nice introduction. So, I'm really happy to be here today, and excited to present you my talk, Next Stop Insights, how Streamlit and Snowflake power up data stories. Before we dive into the topic, just a quick introduction about the company I'm working for. So, this is Innovex. Innovex is an IT project center that is driven by innovation and quality. We are located throughout Germany and we offer a wide range of services for our clients. So today at the conference, several of my colleagues are attending it as well. So if you are interested in learning more about us and InnoVEX, feel free to catch up with us. Yes, and a few words about myself. So I'm Marie. I'm a data scientist and my background is mathematics. I'm interested in everything that is related to machine learning and AI. And one part I'm really passionate about in my job is to generate data insights. So, and this is exactly the reason why I have experienced the following process a couple of times. So, when it comes to generating data insights and where we affect the way of communicating them is to generate or prepare data story out of it. Why? Because complex metrics can be transformed into a much simpler language that can be more easily understood by a broader audience. And this overall helps to drive business value. So in order to prepare a data story, the very first step is to have a specific goal in mind, right? So to be very precise of a question that needs to be answered or a problem that we want to solve. The next step would be then to collect relevant data that serves our intended purpose. Which brings us directly to the next step in which the data is processed, cleaned, explored and analysed to get the question answered or the problem solved. And we might even apply some machine learning techniques on it to gain a deeper understanding for the data. And overall, this helps to generate insights and key findings from which a presentation or a narrative can be told. To structure the presentation properly and to support it, it's beneficial to plot some visualizations or you might even work on some data application in which the visualizations are integrated or even on a dashboard. And with these steps together, we are able to provide an appealing data story, which is able to drive business value. But now here it comes. I have encountered the common scenario for some time. So when I revisit the data story, but the problem was that the data story has evolved since my last presentation. And because the goal has slightly shifted, all the data has changed in the background. And this required me to rework through each of these single steps again, which can be very time consuming. And yeah, unfortunately, this slows down then the business value delivery. So what can we do about it? A very promising solution for that is Streamlit and Snowflake. Probably you know these two technologies, but let me introduce these quickly to you. So Snowflake is a cloud data platform which is offered as a software as a service software. And it's well known for its architecture and scalability. And it offers a variety of different features of how you can work with your data. Streamlit is an open source package which is designed to facilitate the process of building data apps and the best of it is that no frontend experience is required because you can build as a data app just with a few lines of code. And in 2022 Snowflake has acquired Streamlit and offered it now as a feature which is at the moment or which is available now in public preview. And the idea of combining these tools is also quite simple because it should help to accelerate the business value delivery. So how does it work now? We see again the process of preparing a data story, right? And now the difference is that every step can be completely performed within Snowflake. So the data is stored in Snowflake, the insights can be generated in Snowflake, and the presentation part is completely handled by Streamlit. Which benefits does this provide? So on the one hand, as everything works in Snowflake now, we make use of a single platform. So we do not need to move the data and also the application code is hosted at this platform, which results in an instant leveraging of the data that is stored in Snowflake. And further, also Snowflake provides a fully managed environment which ensures that the underlying compute and storage for the application is handled and optimized. And it's also very easy to keep track of data changes because Streamlit directly accesses the data in Snowflake and so if there is a change then this will be reflected also in the data app. And on the other hand, with the Streamlit feature, Snowflake offers a Python editor in its web UI, which allows you to iteratively develop the Streamlit app. Then the interactivity of the data apps allows you to tailor the data apps according to your needs, so for some specific audiences or for business users, and it's also very easy to to share the app within a Snowflake account, which makes collaboration very easy. And in the following, we will take a look on a small POC, which I have developed. It's just a hypothetical use case, and we make use of an open source dataset of the Deutsche Bahn for that. And we will go through each of the steps to prepare the data story and then finally implement a small data app with Streamlit and Snowflake. So let's start with a goal. For that, we are pretending now that we work as a data analyst at the Deutsche Bahn. And as the last time was very exhausting for us, we want to take a break and want to plan a weekend trip. So it means we want to get out of Berlin and spend our weekend at a different city in Germany. And of course, to get there, we want to go by train. yes and therefore we are addressing now the question how good is the Wi-Fi internet connectivity on the trains because it's highly important that we have a reliable internet connection throughout the entire journey yeah and so I mean in the following we will analyze some different routes and will then to then hopefully conclude which is the best destination city for our weekend plans. Okay, for the data set, as I said already, we use an open source data set, which is called Wi-Fi on ECE. It contains details of mobile connection measurements for around 250 routers in the trains. These measurements are recorded at the five-second intervals for three days and include parameters like the number of locked-in devices, send and receive data rates of the train, the modem gateway connectivity that belongs to a router and the respective GPS locations for these all recorded parameters at the five second intervals. And we have prepared already some key findings and these are the following so we know the start and the end station for a certain route, we know if internet disruptions have occurred at certain parts of the route and we have also the send and receive data arrays grouped into categories which are associated with some internet-based activities so with some labels and we also know the number of locked-in devices and with that we have everything what we need to put it together and now we can code a a little bit. We will build a simple Python script and start with the import statements, so probably you have already worked with Streamlit, then you know how easy it is to build data apps with it. The same holds for working with Streamlit in Snowflake. Here are just some slight differences that I will highlight in the following. Let's start with the import statements. We import Streamlit of course, Pandas for some data manipulation, Pydex and Plotly for visualization purposes and here's the first light difference and we also need to import the get active session function of Snowpark. It represents a possibility to get access to the data in Snowflake from the Streamlit app and yeah and we also need to import and we also import some other Snowpark functions for data manipulation which we will use while the loading. If you have never heard of Snowpack, it's quite similar to PySpark. It's a data manipulation library, but just with the difference that it works in Snowflake. So and a few more information. So the provided environment by Snowflake comes equipped with Python, Streamlit and Snowpack, which are pre-installed by default and other packages are supported by the Snowflake Anaconda channel. So as we imported Pandas, Pydex and Plotly, these are as well supported by the Anaconda channel. Okay and with that we can continue with the main part of the data app. So the main part is given by these few lines of code and they contain the main functionality and the app layout. So let's go through it step by step again. First of all we initialize the user session with the get active session function. This is important. Then we give the app a nice title and a header and then after that we load our first data set. The data set includes all the different routes, all the different destinations that we want to offer to analyze in the data app. Yeah and we need this to set up then a Streamlit multi-selection widget from which a user can choose the different destinations he or she is interested in and the variable selection contains a list or is a list that contains all the different selected values. After a separating line which is implemented with the streamlit markdown function, we integrate two streamlit containers. In the first container, we load some filtered and aggregated data that is then used to plot this bar chart here. The bar chart illustrates the distribution of different feasible internet-based activities a traveler can engage in for the selected route. And then in the second container, we load again some filtered data, which is then plotted on a map to analyze the route also on a spatial or geographical level. The last part which we will cover here is that we also take a look on the data loading function. One point I should mention as well is the visualizations are not covered further here during this talk. You can check out the code as well on GitHub, it's available there. I will share the link at the end of the presentation if you're interested in how the visualizations are implemented. But now come back to the data loading functions. We again will go through each of it. So the first one, we load these different destination cities. For that, we are just applying a SQL statement on the user session, which is then transformed to a Pandas data frame. So this is quite easy. In the second one, we query the entire table and then make use of the Snowpark functions to filter the data accordingly and then it's transformed again to a Pandas data frame on which we apply some Pandas data manipulation. One side note is that we can here combine Snowpark functions with a Pandas data frame to manipulate the data. I think it depends a little bit on your taste and also on performance considerations. So if we would work with a very huge data set, then I would rather comment to go with these NOPARC user-defined functions because these are faster in performance than using Panda's data manipulation functions. But our data set is little, so we are fine to go just with Panda data manipulation. Yeah, exactly. And then in the last data loading function we are doing nearly the same. We again query the table then apply some aggregation and filtering and afterwards or at the end some Pandas data manipulation. And with that we have covered the main important parts for building this little app. And we now check it out in Snowflake. So we are here in the Snowflake web UI interface, which is called Snowsight, and under projects in the Streamlit tab we find all the apps to which we have access. If we now want to create a new Streamlit app, we could simply click on this button here on the right, give the app a nice title, choose an app location, and also a warehouse on which the app is running. And then we could simply click on create and this would bring us to the Python editor where we can implement the app. We did this already so we just can simply jump into the app. And now the warehouse is starting in the background and as long as it's starting we can go a little bit through this interface here. You see it's separated into three parts. Here on the left we see the object browser which allows us to invest all the databases, schemas and tables to which we have access. In the center we see the Python editor so here we can implement the data app. On the top we have a package manager where we see all the installed packages listed below. One can also choose different versions for the packages that are supported by the Anaconda channel and just select them by clicking on them and then they are installed and ready to import for the data app. And here on the right, we have the Streamlit preview where we see the app running in action. We want now to focus a little bit more on our data story question because we want to finalize our weekend plans, right? And for that, we will take a closer look onto this bar chart here. So just as a quick recap, this bar chart shows how much time a traveler can spend on a certain based activity. So for example, let me make it just a little bit clearer. If we're now looking for on the route starting from Berlin to Cologne, this would mean a traveler could spend a little bit more than one hour with simple messaging. This refers rather to an unstable internet connection. So this is something what we don't want because we want a stable internet connection. and the remaining part for this journey or for this route one could spend with video streaming which is very good because this means we would rather have a stable internet connection based on the underlying data and our analyzers there. Okay, what we now can do is just simply select also some other destination cities we are interested in. So for example we could click on Hamburg as well and on Stuttgart but then we see that this bar chart is just growing and this does not really gives us a good overview which city we should choose for our weekend plans and we prepared something already for that so we can click here on the second tab where we left a to-do for ourselves that a better comparison view would be great. So let us quickly fix this in the code for that we go to the to the function so we see it here here's a sub header given the two tabs which are implemented the first step with the first blocky shot and now that here's a second tab what we can now do is comment the to do out and integrate the second a second blockly bar chart so and now we can simply click on run and we see the immediate change which is great. So and now we can compare it much better to which destination we or which destination we should choose for the weekend plans and it seems like that Cologne is still our favorite choice as here lots of green is given in the bar chart. Okay great so as we are now happy with the result for this little data app and we convinced that also our colleagues are happy to get a look on it because this would also help to finalize their trips or plan their weekend trips we can here on this blood button click on share and now share it with different roles and so then they would also have access to this data app okay and with that um jumping back to the presentation we are almost there so um just a few more information if you're now also interested in trying this feature out in Snowflake. First of all, I think it's obvious you need a Snowflake account for that, and the feature needs to be activated. Activation means here that some external offerings needs to be accepted. For example, for maps, Mapbox is used, and this is a package which doesn't need to become completely free, especially for commercial purposes, and the same or something similar could hold for other third-party applications that are imported via the Anaconda channel. So the next point is that the apps are streamlit objects with follow role-based access control, which means that your user role in Snowflake needs to be granted to certain privileges and permissions to either develop the data app or to have access, so to view the data app. And we have seen this as well, that also a warehouse needs to be selected on which the app is running as well as on which the SQL queries are executed on. And of course, dependent on the different sizes from which you can choose for a warehouse, different costs can incur. And it has also an impact on the performance for the app as well as on how many concurrent request can be handled by the app. The recommendation is here to start with a very small warehouse and you can resize it afterwards if necessary so if you haven't enough power for your data app. Okay and with that my impression when I have worked with this feature in in Snowflake. So first of all, I think it was for me, it was a very fun project to work on this little PUC. And yeah, I think it's a very easy and efficient way to implement data apps. So and it got me also the immediate sense of achievement when you edit something in the code and you see the change right away in the preview of the app, which is great. So and this gives also the impression that one is able to create directly value from the data. Yeah, I also really like the share button because it's an easy way to collaborate with your team members or with other business users. So that I think it's great for presentation and live reporting purposes. Just one thing that you should keep in mind. So if you start to develop the Streamlit app in the Python editor, which we have seen during the demo, and you want to check out the outcome in the preview, just keep in mind that it costs something because a warehouse is running in the background. So just make sure that the intended value that you want to achieve with developing the data app justifies the cost that you develop it in the Python editor. And with that, I'm already at the end. so thank you for going with me on the journey make sure to drop by the Streamlit booth so if you want to learn more about it or I'm also happy now to answer any questions and yeah, here are my references

Speaker 2 [21:39]

Marie, thank you for your presentation. We will go to the questions next. Afterwards is the coffee break. I'm sorry for the bad air quality. We will get some fresh air then. Okay, first question. How do you version control your chord or project in Snowflake?

Speaker 1 [21:59]

Yes. So this isn't supported yet by the Streamlit feature, which I have shown here. So I know there are the native app framework also a feature in Snowflake, which allows version controlling. So something like GitHub or version controlling or CICD isn't supported yet for the Streamlit apps. but I have looked up a couple of times the documentation and I see that there is some progress so I think this will come soon.

Speaker 2 [22:32]

How do you debug your code in Snowflake?

Speaker 1 [22:36]

Just try and error, I would say. Just print it out with some Streamlit statements. Debugging is also not that possible, but there's also the other option to locally develop the Streamlit app and then to upload it via some SQL statements. Then you could also show it in this preview, which we have seen. There, of course, you can work with your favorite EDA.

Speaker 2 [23:04]

How much of your presentation was about Streamlit? How usable is it without the Snowflake SAAS product?

Speaker 1 [23:11]

Again, please.

Speaker 2 [23:13]

How much is your presentation worth about Streamlit? How usable is it without the Snowflake SAAS product?

Speaker 1 [23:23]

I think you can of course you can also use a streamlit for other projects right you're not dependent on snowflake to use it I think it's just a very easy way to have direct access to the data that is lying in snowflake so this this makes it very easy and yeah makes you also more efficient to work on data apps but it depends on your tech stack with which you are working Thank you.

Speaker 2 [23:51]

How do you manage, respectively control, your Python environment in Snowflake?

Speaker 1 [23:59]

Controlling again? Does it mean also like the version controlling, which we have talked about before? I think then, I think I gave you already the answer.

Speaker 2 [24:10]

How Snowflake controls usage of CPU or GPU on your local or virtual machines?

Speaker 1 [24:19]

How it uses it directly, I'm not sure. When it comes to some billing and so on, then you have some monitoring tools as well in Snowflake, which you can look up.

Speaker 2 [24:32]

Can I use Streamlit without using Snowflake for my data storage?

Speaker 1 [24:38]

yes for sure so I think somehow you will have access to the data at least you can download it somehow and then you can set up as well as screen it app on it

Speaker 2 [24:49]

Did you encounter any limitations when working with Streamlit? Would you recommend it mostly for POC or mock-up scenarios or also for larger front-end projects?

Speaker 1 [25:02]

So as this is a feature which is in preview, I would recommend to look at the documentation because there you have stated all the limitations for this preview version at the moment. Of course, there are some limitations, just look it up. So for example, I think you can query just less than 32 megabytes of data also at the moment. But as I said, they're working on it. So I think everything else will come with time.

Speaker 2 [25:30]

Are permissions to access the app and the underlying data managed separately?

Speaker 1 [25:37]

Yes, these are managed separately. Yes, you have different roles. You can define different roles. Some roles can just have access to the screenlit apps, and other roles are able to develop and implement the apps. So this is managed separately.

Speaker 2 [25:55]

What are the advantages of snowflakes over cloud platforms like GCP, AVSO or Azure?

Speaker 1 [26:03]

Oh, I think I'm not sure. I mean, I have worked on this POC. I think this is a bigger topic that I can't answer here.

Speaker 2 [26:13]

Okay, then how would you manage changes in the underlying data? How would you manage changes in the underlying data?

Speaker 1 [26:24]

So, I mean, yeah, I think this question refers to, okay, if something breaks in then in the app, right? So, with the feature which I have shown, I think the easiest way is just to edit it again then in the Python editor. If you recognize that the data has changed or the data app is broken somehow for the moment, right?

Speaker 2 [26:52]

How expensive is this to run compared to other analytics in Snowflake? Do you use it often?

Speaker 1 [27:00]

have used it so far for some of our clients project yes but then it was more for some POC or mock-ups as we have seen it here for example and I think regarding the prices or how much it's cost it depends really on the configuration that you wish right on which warehouse you're choosing and so on I think also it's not possible to give you a general answer.

Speaker 2 [27:24]

Do you think Streamlit is better for dev versus prod?

Speaker 1 [27:29]

No, I wouldn't say so. I think Streamlit can be also used for prod. For that, I think it would be just advantage or beneficial to have a CICD pipeline then.

Speaker 2 [27:39]

What are the advantages of using Snowflake instead of Clickhouse, for example?

Speaker 1 [27:45]

I don't know Clickhouse, so therefore can't say any advantages or disadvantages.

Speaker 2 [27:52]

How is the security and data protection in Snowflake and Streamlit?

Speaker 1 [27:57]

The security is handled via these roles, via these snowflake roles. There you grant permissions to who or which ritual or user has access to certain data apps.

Speaker 2 [28:12]

In case I do not want to deploy single instances, what platforms or hubs are beneath Snowflake, including cloud deployment or security, MGMT, etc.?

Speaker 1 [28:26]

question. I just need to read it up.

Speaker 2 [28:29]

In case I do not want to deploy single instances, what platforms or hubs are there beneath Snowflake?

Speaker 1 [28:36]

Oh, I think, I don't know, honestly. I don't know. Sorry for that.

Speaker 2 [28:45]

This was the last question we had. Thank you for answering all our questions.