Large Scale Feature Engineering and Datascience with Python & Snowflake

Speaker 1 [00:02]

Good morning, everyone. Thanks for being here. We are five minutes before, but we are go-to-go, the room is almost full. We will present Michael Gorkow for a talk on engineering data science with Python and Snowflake. He's the CTO of data science at Snowflake, so give it a warm welcome. Thank you. All right. Thank you very much for having me. My name, as mentioned, is Michael. Actually, I'm not the CTO. I'm field CTO at Snowflake. What it means is I have a very technical role, but actually with a very narrow focus on data science and machine learning. So afterwards, we have a booth here. Feel free to come over if you have questions in detail. My plan for today is give you an overview of what Snowflake can offer in terms of feature engineering in the first place, and obviously also about doing data science, model training, model inference, et cetera, in Snowflake. But some basics before. Usually people already know what Snowflake is. For those who not know what Snowflake is, what it does, very quickly, some history. 2014, we started going out of stealth mode. Basically our idea was going to the cloud, separate storage and compute, all that stuff that probably a lot of other vendors today also explained to you. So we skip over that. Four years in time, 2018, we were talking about connecting the world of data. We had a lot of customers already on the Snowflake platform. All of them are interested in sharing data, but not only data, but also functionality in Snowflake. And now, and this is where my talk is about today, we are talking about 2022. Last year, we announced something called programmable data. And really, the idea is to not only have data warehousing in Snowflake, SQL-based solutions, but also give you the possibility to program your own apps, to do data science and machine learning modeling, all that stuff, and of course also with frameworks you are familiar with. One of it is of course Python, and we are talking about Scikit-learn, XGBoost, TensorFlow, PyTorch, et cetera, all that inside of Snowflake so that you do not have to extract data, for example, from Snowflake to other solutions, you don't have to manage these other solutions all baked into the platform now. For those of you who also know Streamlit, for example, something which we acquired last year, open source framework to develop applications in Python, all that is also being integrated into Snowflake, so we are really trying to give you the opportunity in Snowflake to do more than just let's call it classical standard data warehousing with SQL. And how to do that, I will give you an explanation today. But But also some words about why it makes sense. Because Snowflake is not only a data science solution. It is now a data platform. So probably a lot of your team members, a lot of other departments are already working on Snowflake if you're a Snowflake customer. So maybe a lot of interesting data you want to use for your data science projects is already in Snowflake. Of course, we also talk about reporting stuff. Your predictions from your beautiful models will end up in these dashboards. So probably you also want to do your data science where your data already is. This is exactly what Snowflake can do today. We are, of course, focusing today on the data science and machine learning part, but you can imagine that there are a lot of other stuff already happening on Snowflake. What is very important is, as we are working all on the same platform, of course, scalability and workload isolation. Can you imagine, you as a data scientist, you have very high compute demands probably on tasks which require a lot of data, but there's also the manager who wants to have his dashboard, there's also some regular processes running, so obviously you want to separate between them. This is what we are doing in Snowflake with our virtual warehouse concept. And also we want to scale depending on how much compute power you actually need, because provisioning compute power in the cloud is costly, so we always try to match exactly the demand of the compute. And later on when we dive into a small demo of all about this, you will also see how easy it is to actually do that in Snowflake, also for the data science workloads. Long story short, scaling in the cloud is very important because it can achieve the following. Here's just an example of a concept that works with T-shirt sizes, every time we double the compute. And what you can see here is you can use an S warehouse doing some work. You can also use an M warehouse, double the compute power, and achieve the same amount of work in the same time, or you can use two M warehouses and achieve the same amount of work actually half the time. At the end, all of them doing exactly the same stuff, but the runtime is extremely reduced from this very long part here to a very small one if you have more compute, but they all cost the same because actually you're only paying for compute you're using. So in Snowflake, later on you will see it in the demo, it makes sense also to play around with compute. Use more compute if you need it, use less compute if you don't need it. It is at the end not more costly. Coming to the Python part, probably a lot of people already know that we have been offering a Python connector for quite some time, and what people did is the following. have a Snowflake virtual warehouse, you know it understands SQL, and a lot of big benefits come from that. Data is already there, it is secured, you have some role-based access controls in place, it is governed, you have probably something like a role-level access policy, it is very scalable, and it's already connected. All the dashboard solutions are already attached, all the data sources are already attached to Snowflake, but in the past people said, okay, but I cannot work with Python, I cannot train a scikit-learn model in Snowflake. So what they did is they used the Python connector, select stuff from table, moved all the data into the data science workbench, basically in any Python engine, did all the stuff there and pushed back the results. Of course, this introduces latency and cost. Every time you move data, it costs you money, and also from a security and governance perspective. As soon as this data here is leaving Snowflake, obviously all the role-based access controls are gone. We don't know what is happening in this environment here, so there must be someone who is managing that environment as well. And obviously when we push back the results, we are again introducing cost because we have to transfer data again. So what if we said is Python not available in Snowflake? Is that true? No, not anymore, because actually now you can run Python in Snowflake, which gives you the possibility to not extract data. The main goal of what I'm showing you today is not having to move data around. We are trying to push logic where data is, but not moving data to where logic is. Benefits, one single platform, so you don't have to manage multiple platforms. From a scalability perspective, you will see Snowflake is very easy to use here. So you don't have to worry about, for example, managing complex Spark cluster configurations. From a governance and security perspective, it becomes much easier because usually you already have something defined in your data warehouse. So it's a lot less management effort for you, and actually you as a data scientist have a very quick access to all the data you want to have in your data science-related work. How does it work? Actually we introduced something called Snowpark, and if you remove the N-O-W, it will become Spark, so it should already give you an impression where we are going to. It's not Spark, so please keep that in mind, we're talking about Snowflake and its own Compute Engineer, but from a syntax and functionality perspective, it will look very familiar to everyone who already knows PySpark, especially when we talk about the Python variant. For other people coming from the Scala world, I know this is the Python conference, but usually people also know Scala, the same is true for Snowpark, it also is available in multiple languages, and the very first thing we will see for data engineering is the client side implementation or the DataFrame API. I already have to say sorry for all the people working with Pandas, single node, except if you're going to the Dask guys, yes, they can distribute it somehow. Our DataFrame API, similar to what the Spark DataFrame does, is already distributed. So you don't have to worry about distributing stuff. We will automatically do that under the hood, which makes it very easy to work on really huge data sets. This is the client side part. Usually we are talking about data engineering stuff here. comes to the server side. We are talking about something called user-defined functions, or in short, user UDFs, and they also support different languages. As we are on PyCon here, I'm talking about the execution of Python code, and this is where you would run your scikit-learn and XGBoost and so on directly in Snowflake. From an architecture perspective, how does it look? On the left side, we have the DataFrame API. What you are doing is you are coding in Python in a DataFrame API, but under the hood, automatically we are creating SQL statements which are executed in our normal SQL engine, which would already give you the idea we don't have to do any magic here, SQL is always distributed in our case, so yes, all the DataFrame manipulations are already distributed. But not everything can be translated into a SQL statement, like, for example, a scikit-learn model, there is no way of translating it to SQL. We are not going to try that. I know some database companies tried it and failed. So what we are doing is we are coding our function. But actually, we are serializing the object and sending the bytecode to Snowflake. And next to the SQL engine, closely attached is a so-called Python secure sandbox. So Python engine running in our same compute clusters that is attached to Anaconda, which provides all the beautiful open source packages, which allows you to use, for example, the Cycle Learn model directly in Snowflake. What are we trying to do? When you look at the past, this is what people did. You have some data, you are doing your select star, you have data in the Pandas data frame trying to do some transformations, you push it back to an enriched table, you again have to download your data to your final table, calling a fit on the scikit-learn model, and then at the very end you have your predictions and push that back. time, here we are introducing cost, latency, security issues, so instead what we are trying to do is come to a solution like this. We only have pointers to data, which basically is our DataFrame API, so we are not physically moving data, we are physically moving the logic, and at the very end we can also push all the heavy lifting of, for example, scikit-learn into the platform where you already have your data. we are again only moving Python logic, no data. From a distribution process, I mean we have already seen we are creating SQL queries if you are using the DataFrame API, so it should give you an idea that SQL is already distributed. More interesting is the part about user-defined functions and also what we call stored procedures, user-defined table functions. For user-defined functions, it is very simple. We have a DataFrame, we have rows in the data frame, and what we are doing is row-level distribution. Each row gets to a different compute node, we apply our function to it, and at the very end, we assemble all these rows together to result in data frame. It is a scalar function, so one row in, one result out. Very simple. For Python, we have a special case which is called vectorized functions or Pandas user-defined functions. What we are doing here is exactly the same, we are working in batches of data, so instead of having row by row transformations, for example in scikit-learn, predict is optimised to work in batches of data, so we are of course using batches of data. But basically the idea is the same, for each row that gets in we have a scalar reside as an output, in this case our prediction. There is a way to use also user-defined table functions, for example if you want to distribute model training. I'm not talking about distributing a single model but for example if you're doing a grid search or a random search you train a lot of models in parallel and at the very end you combine all of these models and find out which one was the best configuration. You could do exactly the same in Snowflake, you have a virtual warehouse that has a lot of compute nodes, each node can train a separate model, at the end you compare all these models and find out which one was the best. Or another idea, imagine you have different stores and you want to calculate or train a model for each store. You could distribute your workload across all the nodes, and for each node you're training a model for each store. If you have different configurations, this is usually the parameter tuning part, user-defined table functions are also very handy to use. With that said, I will jump to a quick demo to give you an overview of how Snowflake and Snowpark looks like, and I can already tell you if you're interested and running exactly the same stuff, have a deeper dive into the code. I will also share a GitHub repository with all the codes attached. You can create a free Snowflake trial account, takes less than two minutes, and run all that code on your own. Now let's get started with actually this one and we are clearing the output. I am using maybe, is that large enough? Awesome. So I'm using Visual Studio Code as a developer environment here, but basically as Snowplug itself is just another Python package, you can use whatever ID you want. But I think Visual Studio Code is a very good one, so highly recommended, especially because on the left side we have a Snowflake plug-in that shows you all the available objects in Snowflake like for example we see that we have two tables in our database. When we start working with Snowpark we always have to connect to Snowflake. I'm using a very simple approach, I'm just connecting given some credentials in a JSON file but of course there are smarter and more secure ways of doing that. Think about single sign-on, key pair authentication etc. The more important part for us now is we are always working in a specific context in Snowflake. We have a role attached to us, and this role defines which tables can I see, which tables can I edit, which functions can I run, so all of the governance and security around your data is already put into the platform. You don't have to worry about that. In Snowflake terms, a virtual warehouse, as I mentioned at the beginning, is our term for compute power so remember we are using a compute cluster called compute warehouse and we are using it all the time but we will change it in between. I'm not switching between different clusters. When we want to create a data frame and on the left side you see we have already some tables I can just simply reference the table and call it show to have a look at the data. In our case the use case is very, very simple. we are doing is we have some e-commerce data, unique identifier, some features, and we have a spend variable at the end which is our target variable today. And first things first, difference between Snowpark data frames and Pandas data frames because I know people love Pandas, there's Polars coming out which I'm very interested in, and by the way the syntax is very similar to what you see here as well, but the main difference is of course the data type, one is Snowpack data frames, one is Pandas data frames, but the more interesting part is the distinction between the memory usage. As you can see, the Snowpack data frame will always consist of 48 bytes, which is the size of the pointer to the data. If we load data into a Pandas data frame, and we can do that with the toPandas statement, we will load it into memory, we are memory bound again, and my My recommendation is never call to pandas if you don't need it. Scalability. Very interesting part. So of course we have some metadata around our data frames. So for example here we have our columns and for each and every column we know whether it's a string type, a double type, and we have some other types as well. And what I would like to do now is distinguish between the different columns in my data set because probably I want to apply different functions based on the type. So pretty standard stuff that I also can do in pandas, you can do exactly the same stuff here. And I'm also using a describe comment to get a first impression of what my data looks like. We usually have one million rows, but you can see here that we have some missing values, so probably before I start modeling the data, I want to do something around data engineering, feature engineering missing value imputation here. And now comes the interesting part. We have one million rows which is obviously not a lot of data, so hence I'm using the smallest compute size available in Snowflake. And by the way, I'm also turning off any caching, so it's really live computation, nothing read from the background. And now we have defined our compute cluster. And I can iterate through all of my columns, apply a function to fill the missing values and we will see that this roughly takes five to six seconds. We now have one million count each column, so basically we have imputed all of the data, everything works fine. But what if we have 100 times the data? So I have another data set created that has 100 million rows, and if I would run the exact same code on that data set on my current cluster, obviously we would probably wait like 500 seconds, which would be very boring for all of us, so that is the reason why we can immediately scale up to 2X large, increasing from extra small to 2X large, I don't know, eight times the compute power, so we have a really huge machine now attached to us, and as you can see it took less than two seconds to start it, actually it was even less on the server side, and I run exactly the same code on 100 times the data, and as we have more compute power, we would expect usually something like 10 seconds, I think, on this size. Hopefully the internet connection is not broken. All right. As we are also getting some results back, sometimes it takes a while, but I think you get the idea. So we have more compute in less than a second can work on larger data sets, especially regarding the data engineering and feature engineering part. In this case, it looked quite easy, but what about the one hot encoding stuff? Here it is very important to have all the data because if you only do that on a subset of data, you don't have all of the different expressions in your data set in the future, so we have to do that on the whole data, which is very, very difficult if you're using a memory bound Pandas data frame. And here's the way of doing one-hot encoding yourself. A small function which produces a one-hot encoding on 100 million rows, still we are working on a 2x large cluster, and what you can see here is basically we have new columns, for example for the gender column we now have gender female and gender male, and we have membership status column with all the different status. This looks very complicated, obviously you don't want to have to rewrite the logic for one-hot encoding yourself, so what you can do instead is using built-in transformers, and I know a lot of people using scikit-learn to do that, yes, you can do scikit-learn in Snowflake as well, but it will not be distributed. So instead what we want to do is offering an API that looks like scikit-learn but is doing it in a distributed fashion, and this is what we are doing with Snowflake ML, I I know there is SparkML as well, so hence the name probably. And this is what you can do. You can just import a one-hot encoder from Snowflake ML preprocessing and have a distributed way of doing a one-hot encoding instead of the scikit-learn one. So very similar code, but it will run a lot faster on larger data sets. And also, you will never receive an out-of-memory error because we are not memory bound when we use the DataFrame API. This is what I'm showing here is the SnowML stuff. It is currently in private preview and June summit 2023 for Snowflake is starting in Las Vegas. You can imagine this is a good date where we probably announce some things going from private preview to public preview or even going to GA. We have defined our training data set, so now I want to split it, and there's obviously also a splitting function in the DataFrame API, so now we are splitting 100 million rows and afterwards we are persisting all that stuff, because until now we have never persisted anything. As you have seen, this one is a DataFrame API that is lazily executed, so only executed when you request results, and is also not persisting anything except if I have to or or if I call a right function. We now have two data sets, train and test, and my big task, the data engineering part is done, so I want to reduce the cost, I scale back to an extra small, and of course I turn the caching on again, there's no reason to ever turn it off, except if you're doing benchmarking. This is the data engineering part, and there's a lot of examples on the web which tell you exactly what is possible with the data frame API. should just give you an idea, easy to use, very similar to probably a lot of frameworks you already know, except that we are now running on a scalable compute engine in Snowflake. Next step, we are trying to build a machine learning model in Snowflake, and our ultimate goal is to have a block storage location with some model files stored, which we later on can reference for inference. So what I am doing is I am creating two stages, actually. I'm calling one of it functions, because remember when we had a look at our slide here, we are sending bytecode to Snowflake, so this bytecode has to live somewhere, hence we are creating a place on the blob storage for that. And then I'm training a machine learning model, Maybe we start with the Python code first. So this is just a very simple function definition. It uses scikit-learn random false regression model in our case. And the code you're seeing in here should look very familiar to anyone who already coded in Python. Basically, you already have that code sitting on your laptop or in some cloud instances already. There's definitely no change required. We are only adding a decorator here at the very top to tell that this is actually a server-side function. So this one makes sure we create a stored procedure, hence the sprog name, called trainRandomForceModel. It lives in our function stage. This is where the bytecode will end up. And we define the dependencies. Basically, what do we have to provide as dependencies during execution of this function? And as we are using scikit-learn, of course we want to have scikit-learn joblib for the pickling part and pandas, because we are loading data into a Pandas data frame. When we define this one, it takes a short amount of time because now Snowpack Under the Hood automatically packages your function, serializes it, sends it to Snowflake, registers the function. And on the left side, when we have a look at our explorer here, and we are now working in the enriched ones, you will see that we have a train random forest model function available. And here also comes the nice part. As this one became another database object, anyone in the organization who is working on the platform can use your function. It is basically already shared with anyone. Calling it is very simple. I mean, right now, it is just a stored procedure, and we are executing it like a simple Python function with all the inputs, what's my training table, features, target, et cetera. Here I'm only training on the subset of data, so we don't have to wait that much time. But this one is now executing your training job on the server side, so there's no data extraction needed. If you have GDPR-relevant data, for example, in Snowflake, probably you're not allowed to extract it. This way would allow you to push your training logic to Snowflake, work on GDPR-related data that is governed, and make sure that you can use the data science in your organisation. I get some coefficients here back to see what was happening in the model, but more importantly, We now have in our model stage a new file, myrandomforest.sav, and we also see it when we update here on the left our stages and have a look at my model stage. Here's my file. So this one can now be picked up by Snowflake, of course, but you could also use any other tool that needs this pickled file to do predictions. So you can also train in Snowflake and maybe run the inference somewhere else, or the other way around. train in some machine learning tool, I don't know, and have a model file ready, put it into a blob storage and we can pick it up from there, do the inference on top of it. By the way, for anyone who is using MLflow as a model repository, there are built-in integrations to push the model deployment into Snowflake without you having to write any of the boilerplate code. Why does it make sense? Let's have a look at the model inferencing part. So we are basically getting the file object from the stage and running the user defined function, which is our scoring function on fresh data. How does it look like? Very similar to the training part. So I'm defining a scoring function which essentially is just calling the predict function of the scikit-learn model with the data I'm inputting. Some best practices here, first of all, we are using a cache decorator for the load model function. Because when we have a look at this one here, what is happening is each node will get a various amount of partitions of data. And if you would not use a cache decorator, it would reload that model for each and every partition. So we are trying to avoid that. Only the first core to that function on each worker node will load the model, or the other ones after it will load from memory, which is a lot faster. So tip number one. By the way, this is not only true for Snowflake, but for a lot of other engines out there as well. Cache your model artifacts in your function calls. And we are defining our user-defined functions. So this one is not actually a local function, but it is, and we can execute it quickly. This one is a user-defined function, and this one is a special case. It's the vectorized or batched user-defined function. Hence the reason why I can't define a maximum batch size. Very important, it is what the name defines here, a maximum. So when I say 10k rows, it does not have to be 10k because under the hood, Snowflake tries to be very smart and optimizes the batch size for you up to that amount of rows. So if we find out that it makes more sense to use on smaller batches because you can distribute it on more nodes because your virtual warehouse is larger, we will do that for you. The maximum here is more to avoid long-running jobs or timeouts. If we have a look at the type, obviously it's of type user defined function and not a local Python function anymore. And let's start with a small amount of rows that we want to score. And here I'm also scaling up again, I want to distribute my workload, I want to use a lot of compute nodes doing the same work but just in different batches of data. So I'm calling my test table here, and I'm creating a new column called predicted spend and call my function with my feature inputs. And afterwards, I also persist my predictions. So keep in mind, this one is not only executing the scoring job, but it's also writing all the data, and in this case, I think it's roughly, when we remove that limit, it's 30 million rows, so it's quite a huge table, actually. But let's start with this very small one, 10k rows, will run probably for a couple of seconds, not that much, but more interestingly if you run it again on a larger table, I'm not sure why this is taking so long, so now I increased it 100 times, we now had 11 seconds, what will now happen under the hood is we will spawn more Python workers and score more data in a shorter amount of time, so in this case it was even faster, and you can imagine that this one as I increase, I could also increase the compute power behind it, so I don't have long-running jobs, so I basically use the compute power I need given the size of my data here. Last but not least, I can, of course, evaluate my model, have a look at the predictions, I think, do I limit it somewhere? Yeah, I do, so don't have to wait that much time. Get the true and the prediction, have a look at my data, it is fake data, so hence why this model is performing quite worse, but I think it should give you an idea, you can visualize data as well, and this is my last recommendation with that part of the demo, if you want to visualize data, aggregate it before or limit your data set, and afterwards call the two pandas, obviously if you want to visualize data, we need data in memory, so it's good to do that, but only do that if you really need it. Last but not least, what did we do on the Snowflake side? Very quickly, the Snowflake UI. This is what your database administrator, your SQL people, or maybe you as a user would also see. And on the left side we have a small icon called data, databases, we had a PyCon demo, we had different schemas, and we were mostly working in the enriched schema, and this is also a place where you find your stored procedures, user defined functions, and so on, and the nice stuff is, even in the UI, what you can do is provide privileges, so right now as I'm the owner of that function and I have not provided someone access to it, nobody else would be able to use it, but imagine we have a finance department, they should be able to train or at least score a model, maybe not train, you could easily just provide access rights to their role, the finance role, and they would be able to incorporate the functions you have developed in Python directly in the data pipelines. Maybe your data engineer who has no idea about Python, but he is more a SQL guy, he can at least incorporate all your Python developments. How does it look like? I can use SQL as well for that. So this This is the SQL UI, one of the possible UIs where you can develop, and I can simply do a call train with all my inputs. This is exactly what we have seen in the notebook before, and now we are training a model just in pure SQL. There's a feature importance we are getting back, and we now have a new model called 1.1234. For the scoring, exactly the same. Scoring a random forest model function directly in SQL. It's a simple select statement. nobody knows that the Python model is behind it. So a really nice way of giving your non-data science guys access to, for example, machine learning stuff. All right. How are we doing in time? Awesome. Then I would also talk very quickly about SnowML. I had a very short part already in my demo where we talked about the one hot encoding. By the way, this one is private preview, again, so not yet available to you, but if you're interested in and you're a Snowflake customer, let me know. I can make sure that you can access, get access to that. What is our idea behind it? We know that a lot of code already exists. A lot of people already use scikit-learn for their data preparation, for example. You see a lot of scalers, transformers, And we want to make it easy for you to make that happen on really huge data sets. So what we are offering is a preprocessing library called SnowML, as you have seen, which is distributed, multi-node executed, and looks very similar to what you already have as code, just that it is running on a different compute engine that is capable of scaling. And the other part is modelling, training machine learning models. And right now we are talking about single node execution. What does it mean? I mentioned the distributed case for cycle to learn parameter tuning, yes, we can train a lot of models in parallel, but we are not distributing the single model, which is different to what, for example, Spark ML does. There's a big debate whether you ever need it to have single models distributed. I'm happy to discuss with you after the talk, but right now this is not possible in Snowflake, but we are working on it. So this is something on our roadmap to offer. Right now we have it on a single node, but we have very simple APIs to do that. And what I would do last is give you an idea how this will look in the future. Again, remember, it is still something which is going in a private preview. So we are switching back, I have another notebook. It is exactly doing the same stuff. So we are again connecting to a Snowflake, we are again having some e-commerce data, all that is exactly the same. And here comes the transformer part, and of course we can work with pipelines, something you know from scikit-learn, I can define a complex pipeline, I need some one-hut encoding, in this case I also do a min-max scaling, scikit-learn syntax, but it's not scikit-learn, it is Snowpark call under the hood, we are calling a fit on our train data frame, and we are transforming our train data frame with our fitted pipeline, and as a result you will see that, yes, the membership column with statuses, basic, bronze, et cetera, is actually working, and these are also the scaled columns in our data frame. For the training itself, it also looks very similar, we have a grid search cross-validation where you can just provide an estimator, everything is exactly like in scikit-learn but actually is running on the server side, so this one, what it does under the hood is it will automatically creates a boilerplate code we have seen in here, the stored procedure. This is what is created under the hood for you, but you don't have to write it yourself, which makes it even easier I think for people to make a transition from a single node execution to a Snowflake Snowpark execution. And of course afterwards we can again do our evaluation of the model and have our output here. So as I said, currently in private preview we are planning to offer that very soon to just make it easier for you to transition from something like scikit-learn to Snowflake. What else is important? I have seen that there is some interest around Streamlit because I think there's also another session about Streamlit and I want to give you an impression of what this means in Snowflake terms in the future because right now we have created a stored procedure that is able to train a model, we have a scoring function, we have it available via SQL so probably a data engineer who can code will be able to incorporate into data pipelines. But I will give you one example of what this might look like in the very near future. And I go into another account for that where I have it enabled called streamlit in Snowflake. And remember we had this stored procedure where we were able to train a random forest. And how does it look like if we will build an app on top of it? On the left side, you have an editor to edit Streamlit apps, and these Streamlit apps are hosted and executed in Snowflake. So all on the left is basic Streamlit code, right, it just defines what we are seeing on the right page as an application. The interesting part is this one here, and by the way, it's using Snowpark of course for all the data engineering stuff and so on, and we are calling our train random forest model function in our application code. So we are utilising Snowflake here for training a model. One nice thing is we don't have to worry about credentials because as we are already connected to Snowflake and we already have a role attached and we already have provided credentials, you can just get an active session. So nobody has to play around by sharing credentials or have to secure credentials, all that is automatically done. And on the left side, and maybe we click this one, on the right side we have our application. This is what an end user would see after he's getting the link to our application. And he can define, for example, what table I want to use for my training. I have a target. I have some features as inputs. And I can define what the model should do in terms of parameters. Do I want to have 10 or do I want to maybe have like 15 estimators on how many rows do I want to train tree depth and so on, and we want to have a new model, and it should be saved in a stage called models. And now I click on submit, and on the server side, it will automatically spawn a job which is training a model and saves it in a Snowflake stage. So really nice and easy way of sharing your data science work also via UI applications, and you don't have to worry about where do I host this UI? It is hosted in Snowflake. Where do I host the training code? is hosted in Snowflake and executed. And nothing is really changing because all of that is working on Snowflake virtual warehouses. Even hosting this app right now, it is running in the Snowflake virtual warehouse. So really easy for you to share all these results with other departments, for example. And the nice stuff is, this is coming pretty soon, so again I refer to our summit in Las Vegas. So it will be very soon available for all of our customers right now, it is in private preview, so again, if you're a customer, if you're interested in trying that out, let me know, we can make sure that it is getting enabled for you very soon. With that said, some recommendations if you're interested in more information, if you want to have a deeper look at it, I mentioned it, there is actually the GitHub repository for the ones who want to have the link, feel free, I think the slides will be shared, but just scan the QR code, it will link you to the GitHub repository. It is exactly the demo I showed today, which is very high level, but I think it's very good for a first impression. If you want to have more examples, I highly recommend to go on to Google and Google Snowflake Quick Start. And the very first non-sponsored link is the Snowflake Quick Start page. Yes, you're paying some money to Google for advertising, and you can type in Snowpark and you see a lot of tutorials, and not only about basic data engineering and model training stuff, but I think there was also something like, I think computer vision was already in there. If you're interested in feature stores, Feast, Tecton integration, for example, which also under the hood use Snowpark, you will find the tutorial here, or just go on medium.com, for Snowpark, there's a ton of interesting blog articles that explain basically the art of the possible, like, I don't know, natural language processing, computer vision stuff, streaming capabilities, et cetera, all what you can do with Snowpark today. And with that said, I am finished. Thank you very much for the attention, and open for any questions you have. Thank you, Michael. We have time for questions. Anyone in the room? Why did you build your own Spark-like language and not reuse Spark? Actually you are running on a completely different compute engine. So it's not running Spark under the hood, it's running virtual warehouses, so we needed a different API on top of that. Okay, but would that work if you are using Spark from the beginning? Like as a performance difference, like fundamental differences? From what we saw, it does basically the same thing as Spark? Not exactly. I think you probably have seen it, but it takes less than a second to change compute. Okay, so it's faster than Spark? Yes, because we are talking about a shared architecture here, so all the compute is basically a very huge pool around all the Snowflake customers. So yes, you can more or less achieve the same in Spark if you manage your own hot pool, but in Snowflake, I mean, it's software as a service, shared architecture, so there's a big difference in what is happening technically under the hood, hence we needed something like Snowpark. And Snowpark is not only Spark, but as you have seen, we also include something like the transformer stuff, there will be a model registry integration very soon with Snowpark, so it's definitely not only Spark-like framework, but it will be our developer framework around Snowflake in total, especially when it comes to anything related to Python. Thank you very much. Thanks a lot for your talk. It was very interesting and insightful. I have one question regarding the Python sandbox runtime. How can you configure it? For example, can you specify different Python versions? versions. What about CPU and memory? Because I can imagine a case where you run into out of memory issues, which is very common with Python. And also when you use like storing and loading those models, it uses JobLib and unload pickle. And between different Python versions, those pickling processes, they are not compatible. How do you solve this? Yeah. Very good question. Let's start with the first one, maybe. Can I run different Python versions? The honest answer right now is not yet. We are now talking about three 3.8, 3.9 should come in the next couple of weeks, I think 3.10 is probably around summit in June. And you will specify the version you are using during the definition of your procedure, as you have seen, I specified the dependencies, like scikit-learn 1.2 I think I was using, you can also specify the exact Python version you want to use. And in the future, whenever you execute that function, we will never change the version. So if you want to have a newer version, you would, for example, redefine that function with different dependencies. So that is, I think, one way how we would avoid loading a different model with a different Python version. Regarding the CPU memory stuff, I mean, the perspective was always to make it very easy, So that was the reason why we created this extra small, small, medium-sized compute clusters. And yes, there are customers asking for, like, hey, how do I define how much memory do I have? We don't have that because we still want to make it easy, but what we do have is so-called Snowpack optimized warehouses. So if you go into the Snowflake UI and you are an admin and you are allowed to create a virtual warehouse, what you can do is, once you have at least a medium-sized one, you can choose a warehouse type to Snowpack optimize. And I think here is also the explanation of what it does, 16 times the memory, we are roughly talking about 230, 240 gigabytes of memory, and of course also more caching, so if you have larger artifacts like deep learning models, for example, this would be the way to go. One thing to keep in mind when I activate the Snowberg Optimize here, the credit price will be 1.5 times the one of a standard warehouse. Hi. Thanks a lot also from my side for the talk. I have mainly two questions. First one, any plan to move also towards NoSQL databases underneath? Or is there maybe already any plan right now? To what database? NoSQL databases. NoSQL database. I understood NoSQL database was like, do we have a new product? Honestly, I don't know. Not that I'm aware of. So right now only SQL? My question would be more like what are you interested in when using NoSQL? Simply the background of or the advantages of using NoSQL is just dumping data coming from different sources easily into the database without having any type of schemas and taking care of the schemas afterwards. But this is exactly what you can do in Snowflake today. We have a data type called variant which basically can receive whatever data you want and then afterwards you can define the structure to work on top of it. We also are working on something called schema evolution, for example, so over time probably your schema changes, we will automatically detect it and adapt your schemas. So this is definitely something you would do today already in Snowflake. Perfect. Sounds like something I'd look into. And then second question, have you any figures about the overhead of the client server computation that you do? Today I used the local Jupyter notebook to do all that stuff, but when you run this in production of course you would not schedule local notebooks, not even, I mean, I think notebooks are a terrible way of putting things into production, so usually you have some Python scripts and you want to execute them, and one way of doing that in Snowflake is Snowflake stored procedures, so basically my whole notebook code I have I would just put in a stored procedure and it would be executed on the server side, you can schedule it on the server side, so this was really just for demonstration purposes, and once it is executed on the server side, there is no computation over it. Even here right now, all the logic was pushed to Snowflake, so the training was not executed locally and especially all the data frame stuff, this is at the end just generating SQL queries that are executed on the Snowflake side. Only retrieving results, collect, show to pandas, et cetera, this is where you would download data from Snowflake hence I always use like a limit statement to only collect 10 rows or a show statement will automatically only collect 10 rows so it is really very small over at here okay thanks a lot we have time for one more who was first so quick question about the cost structure for the stream It is exactly the same pricing for all of the stuff you have seen today. So in terms, when you create a warehouse, you will see these credits here. That's our way of charging. And a Streamlit app, when it runs for one hour and it runs on an extra small, it will be one credit. If you run a training job for one hour, it will be one credit. So whatever you do, it's only the runtime of the virtual warehouse, the compute you're paying for. So it's running basically while I'm providing it for us, it's running, so if I just want to make it available to my customer for the whole month, I'm paying for a whole month, or just while it's accessing it? So your Streamlit app should probably not run 24-7, but we will automatically when someone is accessing the Streamlit, provide compute, when he closes the browser, I think, I don't have it open anymore. The Streamlit app will automatically go into sleep mode and when I have to provide more inputs, I would activate it again. So you're only paying for what you're using for. Otherwise, cloud will always be very expensive if you're running jobs 24-7. Okay, thank you. Thanks for the talk. Just a quick question regarding, again, the Python sandbox. Is it possible to integrate with non-public repository, so if you have an internal repository where you create packages, can you use them as well, or is it limited to PyPy? So it's not using PyPy, it's using Anaconda. We partnered with them. You are limited to what the Snowflake channel, we have a dedicated Snowflake channel with Anaconda, is providing. But what you can do is provide any code that is pure Python. The only limitation is if you have non-native dependencies, like, for example, NumPy, obviously, yes, we have a Python interface, but it's running in C, C++, other languages. For security reasons, we are not executing these non-native dependencies. But if it is a pure Python library, some custom model you have developed, you can just bring it in as an external dependency, and we will execute it. Okay, thanks. And by the way, one last comment regarding that. If you are also having the need for external access like web scraping, calling web APIs. Right now it's also limited to security reasons. We don't want your data going into a prism function and by accident going to the internet because you are not careful enough to make sure that it's going to the right direction. But very soon we will introduce something called external access where you can provide a whitelist of secure endpoints, your Google image recognition API, you want to call that, You would define it in the whitelist beforehand so that you make sure you only provide access to this endpoint and then you can also use a Python UDF to call these endpoints for example. Thank you very much for the questions and Michael again for your talk.