Building reliable data pipelines with polars and dataframely

Speaker 1 [00:04]

Hello!

Speaker 2 [00:05]

Please welcome Oliver and Andreas. They're going to talk about how to build the reliable data pipelines with Polar and DataFrame.

Speaker 1 [00:18]

Hi Thanks for the introduction. Thanks for being here This sounds really weird to me with like the speakers you guys should complain if this doesn't sound good to you It doesn't matter if it sounds good to me Welcome to the Polaris and data for me tutorial super glad you're here. I'm pumped so many people showed up What we're gonna do in a second, I'm Gonna leave that out. I want to start with introducing us You can see I'm a little nervous. I'm getting my slide mixed up. I'm Andreas. This is Olly we work at Quantco. At Quantco we do data analytics often with big companies as partners. We try to solve data problems that they have in our specific life. That often means that we work with health insurance data, so that's structured data that we need to process in some way, and that means we often work with data pipelines. We write them, we think about them, we maintain them, and we just kind of have to make our life work around them. So So what we're going to do today, that's the slide I thought was coming, is we're going to first talk about Polars. I think Polars doesn't really need much of an introduction. It's a data frame library. It's written in Rust. It's great. We love it. We use it for everything. We're going to try to make you also love it if you haven't started loving it yet. And that's kind of the backbone of all the processing that we do. We think there's a piece missing there, and we try to provide that with DataFrame.me, which is a tool that we developed internally and then open source last year, which kind of bridges the gap between just Polar allowing you to very efficiently transform your data to a place where you can write really reliable software with it that makes it easy for you to also know tomorrow what you wrote yesterday, to document your pipeline, to test your pipeline, to validate your data at runtime so that you don't have some weird assumptions that you violate in your data and you don't notice and your results are just crap and it just goes through silently. So all of these are things that we try to solve with DataFramely and we're going to go through a little bit how that works and show you how you can make that work in your own life. Now, the idea of today's tutorial is we're going to try to give you a mini case study. We're going to start with some raw data that is shitty in various ways because that's just how data often is. It comes to you in weird shapes. It sometimes has stupid columns, it has stupid encodings, and so on. And what we want to do is we want to write a pipeline in order to be able to provide a report that isn't shitty. And that means we kind of need to do the actual reporting analytics queries that we need to do, but we also need to worry a little bit about, you know, how do we transform this data so that our results are reliable. In the first round, we're just going to build that with polars, and we're going to do, I think, a bunch of stuff right already by trying to structure our code in a way that makes it easy to maintain, by making it easier to understand, and so on. And because we're going to write with polars, we're also going to have a lot of efficiency going on. In the second part, we're then going to basically spruce this up with data framing on top, which is kind of a lightweight addition, but should really make it easier for us to maintain this in the future to kind of keep the transforms, the joins, all the queries from kind of growing into this big blob that we can't understand anymore. And while this is minimal, this is kind of what we do in real life. So I mean, the data sets will be different. The data sets will probably be bigger. The pipeline is longer. But all the concepts should kind of be the same. All right. The first thing that we should do to jump right in is to get you started. There are some setup instructions here. You need a GitHub repository that's linked here. It's also in the talk description on pretalks. You will need the Pixie package manager if you don't have that already. Also highly recommended. Super fantastic. We love it. It's kind of the packaging equivalent of Polar for us. And we will need you to run Pixie run post install once you have all of this stuff to get the environment set up. And because this was already in the talk instructions, maybe some of you already did it. Quick show of hands. Who already has this stuff set up? That is way more than I thought. Fantastic. Who doesn't have it set up but would still like to set it up? Okay, also a bunch of people. Nice. Okay, so I think what we're going to do is we're going to give you a few minutes to set that up. Take your time. It shouldn't be very complicated. In the meantime, if you already have this set up, feel free to open the tutorial notebook that's in there. It's called Tutorial, so easy to find. And just verify that you can basically execute some cells, that you have the right Python interpreter, so, like, import polars, import data framely, and then you should be ready to go. And we'll get back to you in a minute. They're smart, they turn it off, so if I whisper to my co-presenter, you don't hear it. If there's anything that's wrong, or if anything doesn't work, just yell and we'll help. Oh, why Pixie? The first time I was about Pixie, three days ago, I had no clue what that is. Yeah. So, I mean, okay, so the world, I mean, there's like a split into people who like to use Pip, right, and people who like to use the Conda ecosystem. And for... Sorry? Sure. I'm going to count that over there. For various historical reasons, Conco is a Conda shop. So we've always been using the Conda ecosystem. And Pixie is just the modern way of using the Conda ecosystem. and it has a lot of the like I think if you use it it should feel in a way to you like UV and then it's like it's fast it has like a lot of you know comfort features so it's just fun to use and that's just really how people use conda nowadays so like don't install like conda as in c-o-n-d-a or micro mamba or whatever that was in the future just install pixie and that's it Oh, this time it didn't turn off, so I can't whisper, okay. All right. I hope that everyone is kind of getting there. Don't worry. You can still, like, if your download is still going, you can still do that in a second. It's not going to turn off. But I want to kind of keep the momentum up a little bit because, as you know, 90 minutes are short. The last thing that I need to tell you in order for you to understand what you will have to do, we're mostly going to go through this tutorial notebook. And the idea is that we use the notebook for us to give you some instructions. We also have them on the slides, but they're also in the notebook, so you can follow along there. And you can like basically write any queries, like really destroy this notebook if you want. Just write in whatever explorative stuff you want. The nice code, like the production code that you're you're going to ship to your production server, that's going to go not in the notebook, but in a folder pipeline and then different files. So we'll try to organize this like a real Python package. So this is kind of the first part of organizing your pipeline well, is to do that rather than typing everything into a notebook. I'm sure you all already do that at home anyways. And with that, I think I'm going to start handing over to Oli, who's going to tell you a few things about Polars. And again, don't worry if the setup is still running. You have a few slides to get that finished.

Speaker 2 [08:34]

So, as already mentioned in the first half, we're going to focus on Polus, and if you haven't seen Polus before, Polus is a data frame library. The main operational unit that you have in Polus is the data frame. You can think of it as a table. You can initialise it as you're used to in Pandas, for example, and then you can print it and then you can see that Polus created the data frame. Now we already have a DataFrame library, Pandas, so why would you want to use Polus? Polus has a couple of benefits. First of all, it's really fast, it's written in Rust, that doesn't necessarily mean anything by itself, but it allows Polus to leverage all the cores in your machine, so you can actually scale up the speed of your data processing pipeline by running on a bigger machine. Second of all, Polus has a really expressive API, we'll get to know the expression API in a minute, and it allows you to write complex transformations in a really, really nice way. So usually people really like the syntax of Polars, and it allows you to easily read through complex transformations. Then Polar is also very memory efficient. Polars relies on the arrow memory layout, which we'll also talk a little bit about later. It generally minimises the data size that you have in memory, so it allows you to process large data sets, much larger than you're used to in Pandas at least a few years ago. What is very unique to Polus in the data frame ecosystem is that it runs query optimisation, so you can build so-called lazy frames that optimise the query before the query is actually executed, much like you're used to in a SQL database. Two things that are also very nice about Polus but we won't dive into today, first of all, Polus has out-of-core support. It even allows you to process data that is larger than your memory by processing it in chunks. And it also allows you to extend Polus very efficiently by allowing you to write certain Rust plugins that you can then use in your Polus transformations. Again, we won't do that today, but we do use that a lot in our daily work, and it's a main argument why Polus is nice for us. All right. So, let's talk about the expressive API, or the expression API. Polar expressions are essentially descriptions of a transformation to execute, and you compose that into complex logic. So, for example, we can define an expression like this. We say that we reference a column, We do that by PL call kind with the previous example of our data frame. We say that the kind needs to be dog, and we alias that to is dog. So all of that is an expression, PL call is an expression, PL call equals dog is an expression, and aliasing it creates another expression. And we can now use those expressions in a variety of contexts. First of all, we can filter our data frame by that expression. So we execute that conditional logic, for example. We can also just select this expression. So we create a data frame that has just this output row with this output column with this transformation. The other option here is we can use with column. So we just extend our data frame with that column. And we can also use it in more complex settings. For example, we can group by our expression and then aggregate data within that group. Overall, Polaris has dozens of expressions that you can use, and it usually makes your queries very easy to understand, very easy to write. And we want you to get to know the Expression API a little bit, and that's what we will focus on in the first task, where you'll explore the raw data. This is the first task in the Jupyter Notebook that Andreas already mentioned. Your task would be to print data frames, to explore the structure, explore the in-memory size, and use the Expression API to explore the contents of these columns. We'll give you a few minutes, and then we'll also share a solution to that.

Speaker 1 [13:00]

Oh, that's a fantastic question. Do you have VS Code, for example? I think you should just be able to open it in VS Code. I don't know if you still need to install something for that nowadays.

Speaker 2 [13:10]

You probably need to select a Python interpreter in VS Code.

Speaker 1 [13:10]

You shouldn't.

Speaker 2 [13:16]

The alternative is that within your terminal, you run pixi-run-jupyter-notebook, and then that opens a page in your browser, and it allows you to use Jupyter from the browser.

Speaker 1 [13:31]

We don't have Jupiter in the environment though, do we? We don't have it in the environment, do we?

Speaker 2 [13:38]

I thought we do.

Speaker 1 [13:40]

I don't know. You don't need it if you don't have your USB-SCOUT. . Yeah. Fantastic. All right.

Speaker 2 [13:49]

So if you don't get VS Code to run the browser option or running Pixie Run, Jupyter Notebook is probably the easiest way to actually open the notebook.

Speaker 1 [14:01]

Let me take a look Thank you. This silent working time is really weird. Thank you. awesome i'm sorry Thank you. Amen. All right. Thank you. All right. I think the sounds of the keyboard's clacking has reduced. Is there anyone who needs a lot more time to get started? Does the setup work for everyone? Did everyone who wanted to succeed in running the notebook? I don't see complaints, so I'm going to count that as a yes. We're going to slowly get started with the solution. So the first task was, obviously very straightforward. If you've ever used Polar before, the idea is that we get to warm up a little bit. We look at this data and kind of get a feeling for what we're even working with. And this part was already filled out, so you should be able to load the data. And you can just look at the data frames in your notebook. I think that should be readable, yeah. So basically, we have two tables that we have to deal with. We have policies and we have models. Those names might be a little bit opaque, but once you look into them, you might quickly get the idea that this is about car insurance. This is not necessarily because we like car insurance, but we often, because we work in insurance stuff, we often think about insurance. So I guess the first thing that we found on the internet was insurance, and we liked it. And the idea of this data set is that we have two tables that are very different in the way that they work. The policies table, as you can see, is really long. It has a million columns, a million rows, sorry, but relatively narrow, only seven columns. And the Models table tells us something about different car models. So obviously, there aren't as many car models. And so it is a little bit shorter, but then it is wider, because cars just have a bunch of different properties. So you might get the idea that the Models table acts like a little bit of a lookup table for you to get more information on the car model that is in each of your policies. And you can see that when you look, for example, model column, and the model column here, you are going to see the same values, and those are also the same, the only columns that these two data sets have in common, so this is not really rigorous or scientific, but we are just going to basically assume that this is a key that we can join on later on. In real life, you should probably do more research on this, and like look at your data more, but this is a tutorial, so we are kind of limited in time. And the thing that we start with might be a little bit like you might wonder why do we care about the size? Who cares? It's mostly so that we understand really where we're going in our pipeline, and I hope it becomes clear later. The first thing that we basically see is, okay, we have about 50 megabytes of policies and about less than a megabyte of models here. And that's kind of our starting point. And we're going to see in a little bit why that matters. Now, in the to-do here, there were a few different instructions. But I'm also happy if you just wrote some exploratory queries like whatever to get your fingers warmed up. A few things that are interesting that we can look at. When we look at, for example, this fuel type column that's called out here, if we just select the unique values of that, we can see there's only three different types of fuel type. That kind of makes sense because how many different types of fuels are there for cars? Not that many. But this stuff is still encoded as a string in our data frame. I'm going to see in a little bit why that might be a weird choice and how we can maybe get this to be better. But this is something that you should keep in mind. The other thing that we can look at is we can also look at the airbags that were explicitly called out. Here, I could have done the same thing as up here. I can just basically select a column and put unique on it, and Poloise figures out what the unique values are. Another interesting way that I often to explore data is to group by and then select the length of the group. Basically tells you this is the same as select count star from whatever and then group by. So you get how many of each of these values you have to get an idea of what there is. And, again, you can see that there's a very finite number of airbag configurations. You cannot have two to the 33 airbags in your car. But we're still using, you know, an integer 64 here. So we're using 64 bits to save this information. We're wasting a bunch of space. I'm going to fix that in a second. Now, that's just data type stuff. I really care about this. You might not. It's a little bit boring. I agree, but it's important. More interesting is in terms of content. If you look at all of these is columns here, so let me just actually select those so you can see what I am talking about. Let me just write a regular expression here. So that's one cool feature about Polars. We don't really go through all of the features in total because there's too many. What I want to plant in your mind is the idea that Polars gives you fancy stuff like this where you can write a regular expression to select everything by name. So it's just really easy to do stuff that you would otherwise have to do manually. And this level of, like, comfort and syntactic sugar is what I find exciting. Anyway, these columns are all basically Booleans. I mean, they're all called is whatever. They're all just always yes or no. Here we actually do the scientific thing of actually verifying that all of them are yes or no, but you can also just maybe for the purposes of this tutorial just believe that. And that obviously is also, like, very inconsistent in coding in that we would probably just want this to be a Boolean so we can also work with it more easily later on. In the same vein, when we look at this max power column that was called out in the instructions, you can see that that is absolutely terrible to work with. If you ever imagine writing any analytics query against this thing, you're going to have a hard time, because it basically encodes two different quantities in one string with an at in between and units. So good luck querying that. We're also going to figure that out in a second. And the proof here is that basically they're all shaped the same. So I basically just take this column, I check whether it conforms to a regular expression, and I just require that all of the rows pass this. And this took me, I don't know, five seconds to write, and I know that all of them have this specific structure. Actually, I kind of cheated, because here I should also have regex start and end markers, so that this is actually true. Can you make this one bigger? MARTIN SPLITT- Can make it somewhat bigger, I guess. If I make it too much bigger, I think Things start getting weird. Does that work? All right. Cool. So I think we now have an understanding of what the data is, what's broken about it. And we're going to fix it in a second. Ollie is first going to tell you a little bit about why that matters.

Speaker 2 [25:11]

Yes. So Andreas already mentioned, well, some data types are not ideal. And if we think about data types in Polas, we want to think about memory efficiency because we want to save space, and for that, it's really useful to know how Polas actually lays out your data and memory. And that is the arrow memory layout. And essentially all columns in Polas are stored in that standardised format, which many tools support these days, and the idea is simple. You have one contiguous buffer with your data, the memory layout of that data buffer kind of depends on your data type, we'll see an example in a second, and you might have also additional data buffers depending on the data type. For example, you might have very complex data types like structs or lists that you can actually represent with Arrow, and then you would have additional data buffers. The very nice thing about Arrow is that in that Arrow specification there is also a validity bit mask. So you have a bit mask with ones or zeros that tells you whether an entry in your data buffer is null or not. If you have ever worked with pandas in the past, you might have had the case that you have an integer array and one of your values becomes null and now everything becomes a float because there is no way to represent that otherwise. In Polars, you can actually have nullable integers, nullable everything, because you have that validity bit mask, and it's really just one bit per row, so it's very efficient to store. To give you two examples about these data types, we want to have a look at uint8 and string. Uint8 is a very simple example, it's a primitive type. You have your data buffer where your integers are laid out contiguously in memory, and then you have the validity bit mask that in this case tells us that all of the values are non-null. So your buffer is contiguous, you have fixed offsets, it's very fast to process in a computer. But it's not always that easy, as for these primitive types like these integers, for strings what Arrow is doing is it also uses a contiguous representation in 16 bytes per row, which at first sight is strange because strings can have varying length. So what Arrow is doing is short strings are actually put right into the data buffer. So the first four bytes represent the length, and then if it's a short string smaller than 12 bytes, then all of your text can be laid out there. However, if it's longer, then only four bytes of prefix are stored in your data buffer, and then you will have additional data buffers that are essentially linked to. Now, traversing that is obviously much more expensive, because you have these indirections into another buffer, and in general, you have 16 bytes per row here in your integer, so if you think about cache efficiency, for example, processing strings is just much more expensive than processing small integers. In general, that's the reason why using the right data type will both save you memory, because your data buffer becomes smaller, and saves you time, because Polar can process your data more efficiently. And to understand that a little bit more, we want you to go back to your laptop, fill are the methods in the preprocessed file in the pipeline folder that allows you to explore some more data types. Cast is the way to translate columns into different data types, and then we want you to compare the size of the preprocessed data frame to the original data to see how how you actually saved memory with your transformation.

Speaker 1 [29:48]

I actually cannot hear you very well, I am just going to come up there and I am going I'm going to repeat your question. Because actually this is not interesting for everyone else probably. Oh, this is PowerShell, exciting.

Speaker 2 [30:41]

Let's see.

Speaker 1 [30:54]

yeah yeah I'm not doing anything else so let's see it should be in here I mean this is where the path is right whoops this is not where the path is for you interesting so I guess it's this one yeah can you somehow find out what this path is and try that as an interpreter I'm not sure if it accepts relative ah yes that's nice if you can just click through your pixie and then and default right and should be right in this folder You might have to reload the window entirely. Yes, there it is, at the very top, there in the sky. OK, thank you. Let's see if it works. Do you want to run some cells? I think we sometimes had the experience that we had to reload the window more than once. So if it doesn't work, yeah, sure. Thank you. Thank you. You I think we're slowly going to start talking about the solution a little bit. Don't worry if you didn't get there entirely, I don't think that's super important. We kind of also try to provide enough stuff so that people who know what they're doing can be really fast and do everything, and if you're a little bit more of a beginner you have a little bit more leeway. So don't worry about that. If you at any point feel like your solution is not on track and you would just rather like to start from a checkpoint, there are two additional branches in this repository. One is solution polars. So that's what we're looking at right now. It's the first part. And there's solution dataframely, which is what comes later. So you can also just start over, check out these branches, and then you should be set up to keep working from there. All right. So the task was to basically implement this pre-processing step. In the end, the interface that we're looking for in organizing this is we want to be able to put our data into little containers so it's easy to ship them around. So some of the stuff that was already in here was this, for example, this raw data container, where if we look at it, it's really just the data class that has models and policies in it. So we would not always have to ship around 15 data frames if we have a big data set. We just pack those together, and you know that they belong together because they're in the same class. So you're never confused about these policies or these models, maybe from later stages where we make this pre-processed data, which might also have models, and so you don't confuse them. One thing in here, if you're not used to generics, don't worry about it. This stuff is currently generic in PL DataFrame and PL LazyFrame, because one of the points that we're going to make in a second is that you can run basically the same code with a LazyFrame and with a DataFrame. So one's eager, where everything is executed immediately, lazily, and once lazily, don't worry about it, we're going to get to it. And the idea was now to basically get to this point where we can run these two steps. So we have our raw data, and we run preprocess on it, and we have preprocess data. So what does preprocess do? This part is also just, you know, in direction to kind of organize stuff. The real magic happens in this preprocess policies and preprocess models functions. And there's really two things going on here. We're kind of getting in a data frame or a lazy frame, and we're trying to modify it by just overriding columns. So what this does is it doesn't change the initial data frame. It just always creates a new data frame that has these columns changed. So you're not never changing stuff in place, which is very good. And what we basically have to do here is we have to do a bunch of casting. So there's like a billion casts in here. And basically, you have to go through and just always select what data type you want for all these different things. For example, I'm not necessarily arguing that these data types are exactly optimized, because we also didn't look at the data just now a lot. We just kind of went through it. But you might get the idea that some of this stuff is a categorical. I'm going to see in a second why that's awesome. We're going to, for some of these things, just use smaller floats. Instead of a float 64, use a float 32. And here, we use a uint64, because we have possibly a lot of policies. Part of the task was also to spruce up this policy ID column, because as you saw in the initial data frame, just always was a string. It had the word policy in it, kind of useless. So what we're just going to do is we're going to use the polar string namespace, which allows you to do string operations on columns that have strings in them, obviously. And we can just strip a prefix. So again, the idea is not necessarily that you have to know this upfront. But if you ever need to do something that feels like it should be something someone has done before, it's probably implemented in Polar, so you don't have to worry about it and can just use this and it's very efficient. The second thing, then, was we wanted to deal with these models. There were a bunch of things wrong that we saw before already in the data. One thing was these is columns. Here we can very efficiently cast them to Booleans. Again, we use this regular expression syntax that's a little bit advanced. If you did this for the first time, you might also just have written every column out individually, also fine, it's just more typing, but just basically selecting on this allows you to very efficiently encode what you want to do. For the torque and power columns, we have to kind of split up this weird format that's in there, so we also want to get away from the strings, we want to get away from having two things in one column, and so what we do is basically we just, again, use the polar string namespace, and here it gets a little bit funky, because after you split the column, you obviously have two elements, or even more if it was somehow formatted differently, so this ends up being a list data type. So now every cell in this column has a list in it, which might sound funky, but after you get used to it, you realise that it works really well in Polar, so you can do really fantastic stuff with it, and it becomes really comfortable to use. Here we do something simple. We just basically, again, go to the list namespace. So this is the equivalent of going to the string namespace up here. Whoopsie. Up here. And we then go to the string namespace. We strip a suffix. And we cast this to whatever data type we want. And personally, I feel like this is pretty straightforward after you have known these things for a little bit. So it just allows you to write very little code to get stuff done, which I always enjoy. Down here, then, we had the enums. I think that's another interesting thing. Compared to the categoricals that we had before, the difference is just that we have to commit onto what different types we allow. We're going to see in a second what exactly that difference makes. And then there's just a bunch more casting, and that's kind of it. If we look back into our tutorial notebook, one of the things that we wanted to do is after we've already run this pre-processing stuff, now I want to see what happened to the sizes here. So the policies just got a lot smaller. We lost, like we went from 50 to about 30 megabytes. That's really nice. For the models, we got like a factor of 2-ish by just encoding our columns more correctly. And I think because the data set, and we're not really doing very stringent benchmarking here, you kind of just have to believe me when I claim that this type of stuff will make it run faster. cluster. One thing that's really interesting about the enums that I want to show you here is what this actually does under the hood. Polaris has this cool two-physical method also on their expressions, where for a column, you can also get the physical representation. The idea being that the logical representation is what you see when you use an airframe, and the physical representation is basically the bits that Polaris stores behind. And before Before preprocessing, it's the same thing. So the logical thing is diesel, and the physical thing is the string diesel, and Polars just stores that. After the preprocessing, when we made this an enum, the cool thing is that before it's a diesel, but it's still diesel now in the logical representation, so the data frame remains readable. You can also write expressions where you write equals diesel, and it will be very readable code. But Polars has done the work of translating this into unsigned integers for you. So it really just stores these u and 8s in a second. And you just save a bunch of data, a bunch of space. And Polaris really manages the translation from these values to these values for you. Now, the point about the categoricals was that they're kind of the same, except that you haven't, for categoricals, you don't commit to the values. So here we said these are the values that exist. So Polaris knows it can use a really small integer data type. For categoricals, because you don't say what the values are, Polaris has to leave some room, and typically will guess a bigger data type. So it's not as efficient, but it's still pretty efficient and much better than just storing this train. Anyway, there was a question right here. Maybe initial, but if you write that to a database, would it write diesel or one inside the database? Okay, I'm not sure if everyone heard that. I'm just going to repeat it. The question was, which of these representations gets written to a database? And I think the TLDR is depends how you write it to the database. Because I think like writing stuff to the database, I wouldn't consider that like a core feature of Polars. So it kind of depends. Polars has a like write database thing. I actually don't know what that is. I would assume that it has to either write the string if there's no enum D type. Or if there is something like an enum in your database, then it might work. But I think some of that also goes then through some code that I wouldn't necessarily admire at this point. I think it was a little bit hacky in the past. But I think the TLDR is, don't rely on this, but think about how you write stuff to databases. Because oftentimes, there's a little nuance there that you want to control. All right. Are there any more questions on this part? Otherwise, I think we're ready to keep going. Wonderful. Okay, then I'm going to hand this over back to Olli.

Speaker 2 [43:53]

All right. We mentioned already before that Polus has a really nice concept of the lazy frames that perform query optimisation. We haven't used that yet. Before actually writing code with that, we want to give you a brief introduction. When we are pre-processing data, there are two modes of execution. First of all, there is eager execution and then there is lazy evaluation or lazy execution. In eager execution, that's what we have been doing so far, all operations are executed immediately. So if we have a transformation like that where we have a data frame, we group by something, we aggregate the data in our groups, and then we select a few columns in the end, then we just sequentially execute what we've written. So we first do the aggregation for the groups, then we select the columns. As you can see here, however, we don't actually do anything with the max weight. We don't select it in the end, it's not an output, so we're doing some unnecessary computation. So in general, when we do EGA execution, intermediate computation can be wasted, and the performance of your query depends heavily on how you write the code. And lazy evaluation, on the other hand, execution is deferred until calling collect. So we might have the very same query, but we just insert two things. We do .lazy at the beginning, and we do .collect at the end, And everything in between is our lazy frame that can be optimised by Polus. So what is happening is Polus builds a query plan of the entire transformation chain, and then optimises all of the transformations before actually executing your code. So in this particular instance, Polus wouldn't actually compute max weight at all because we don't need it in the output. The general promise of lazy frames is that by automatically performing many operations, you can actually focus on writing queries that are expressive. So you don't have to move around stuff to make it faster, but you can just trust Polas to optimise your query in the best way possible. In many cases, when you're using a lazy frame, you don't want to just trust Polas, though. You actually want to see what Polas is doing, and there's the possibility to inspect a lazy frame. So let's say we write a query like that where we join the pets on themselves, so for example we want to combine or we want to have pairs of dogs and cats, but we also want to filter to not have animals that are too heavy in our pairs for whatever reason. So we can then do, this is our lazy frame, we can then run LF show graph optimised false, and that will display the execution graph, what we're doing. So we have two tables, we join them, we filter them, and then we select some columns. And this is exactly what happens when you run this eagerly. If you do LFShowGraph, which implies optimised equals true, then you can see, well, this looks a little bit different. The main thing that is happening here is that this filter operation is automatically moved down through the join, which is something called predicate pushdown, and essentially allows Polars to reduce the size of your data that you're processing as early as possible so you don't have to do any computations that are not really necessary. There are a handful of optimisations that Polars is doing. Predicate pushdown is one of the most efficient ones, and it generally allows you to execute your query much faster. All right. So we want you experience lazy frames a little bit, and for that we will ask you to fill out the methods into report.py to actually generate the report that we have previously talked about, and then we can run the entire pipeline in lazy mode and in eager mode and see what is the effect of that.

Speaker 1 [48:02]

No, I was nodding because I understand the question, not because I was going to say yes. Do you use base mode in production? Yes. Not really. Yes. So you trust Thomas that much that you don't want to see what he's doing and then you transfer it to Perf. I mean, obviously, we test that code a lot with data to see that the optimization doesn't get confused, which in principle can happen. So I think that's also something that people have experienced with SQL databases a lot, right? that you have some query optimization and it goes awry and suddenly your memory is gone. We have had this problem occasionally in bigger analytics workloads. I don't think we've had it a lot as a recurring thing. Usually it's something that occurs rarely, then you fix it once by changing a little bit, and then you don't have to worry about it again. Or the other way around, where don't you use ? Yeah, I think the downside is the debugging. So it's usually useful to kind of like, if you have a pipeline that's like this long, to have some units in it that are logically confined and basically not have the laziness go beyond that. So you would collect at the end of that, often because that already gives you the performance gain, so you don't lose anything. And it's much easier to understand if something goes wrong because otherwise, if you have this whole thing as a lazy graph, something goes wrong here And you notice at the end, you're just screwed. Ah, there's another question. Is it also a problem when I use Eger that all the data gets materialized? So when I use lazy and I have a big chunk of data, it can't process it through? So I always thought this is the reason, or one of the reasons, to use lazy . Yeah, so the question was basically, if I understand correctly, whether the point of laziness is also to avoid materializing really big things in memory? Yeah, I think so. For some things, so I think that's totally a valid trade-off that you want to think about. I think for some things, you never get around materializing them. And then it's just a good idea to just do it and get it over with. For example, when you also want to write this stuff out to some storage or something, that often happens. But yeah, if you have a really big intermediate thing you can avoid materializing it this way and that helps you solve a constraint that you otherwise couldn't, that's probably worth it. If you're just doing that but it wouldn't be a problem to materialize it, then maybe it's also worth just materializing it so you you can look at it. But totally valid concern, I think. Thank you. Let's go. Let's go. I lost one. All right. I realize this might not have been enough time for everyone to get through everything. That's totally fair. But in the light of time, I think we are going to try keeping the momentum up here. The solution for the report part is mostly just writing more polars. So the idea is again that we want to have an interface where we have everything sequestered into nice individual building blocks, we have a build report function, that thing creates a report. If I was a really good boy, I would have written some description here, so like future maintainers will know what this thing is, and basically our task was now to just populate these frames here. Whoops. Those are PyChomp shortcuts. And the idea is that we just write two queries that try to do different Polar things, when We try to find the most popular make and models. Obviously, we have the policies that tells us how many cars there are of which kind. And we have the models data frame, which tells us what make and model is. So we have to join them in order to be able to work with them, which is very similar to the cat and dog example we had before. We can join them. We can group them then to get the counts in each of these different groups. And we can do some extra stuff that really should feel natural to you, like sorting or just getting the first three rows. Okay. I don't really want to go into too much detail here because in the end, this is not like a full Polar scores, but the idea is that I just want to give you a feeling for like what you can write in a few lines. Safest models kind of sidesteps the problem that I said before. You just need the models. You don't really need the policies for this. And the safety score here is just kind of counting how many of these Booleans are true. And so here's another cool feature that you can do. You can, by doing PL call and then PL boolean, you're basically saying, I want all columns that are of type boolean. So this is kind of the syntactic sugar that we had before with the regex. Makes it easy to select many columns at once. If you want to do big transformations, that can be really helpful. And then we just do sum horizontal. Sum horizontal is just what it thinks, what you think it does. It takes a bunch of columns in one row and sums them row-wise. And again, you sort this and you return it. The little bit more spicy one is the car volume, which is obviously something that you worry about every day. What's the volume of my car? I need to know. And here we have to join, we have to group, we have to aggregate, and then we have to write an expression for this volume. One tricky part that we put in there for you to find a little bit of a landmine is that the counterpoint to use small data types that we had before is if you use small data types and you then do calculations that don't fit into those data types, you can have problems. So here we had these, I think we saved them as uint8s or uint16s, I don't remember, actually. Probably uint16s. And if we just multiply the length, the width, and the height, because our cars are boxes, we would just basically get an integer overflow, and we get sad. So you have to make sure that you use the data type before here. So I think the TLDR is, if you do this at home, maybe over-optimizing the data type size is also not what you want, because you don't want to have to think about this all the time. So just really don't optimize in ways that then makes it hard for you to do your job There's a trade-off there and only you can tell where the trade-off is for you Alright cool, but then I think we're ready to run the report. We do this here. We have the school function we can call a report to string and The values are whatever. This is not an analytics challenge They kind of look sensible their accounts their models. There are safety scores, whatever It's mostly about the way to getting there here, not really about the outcomes. But I think what's really interesting about the car volume, then, is that we notice that it's actually kind of screwed. Because these volumes are now in cubic meters, and you can see that, like, depending on the car age here, like, eight cubic meters sounds kind of reasonable if you think, like, two by two by two, that seems like a car that could exist. One cubic meter, that's a weird car, and, like, this is not a car. This is, like, a toy car or something. So something really went wrong in our data pipeline. We screwed up. And this is, I think, really realistic, because typically this is kind of like how people work, is that you discover something, and then you have to go searching for it. And if we now try to track this down, and we just go and select these values here and sort by the volume, you can find that, actually, some of these cars just have funky height, width, and length values. And if you think about it for a little bit longer, you might conclude that maybe these guys down here millimeters, because 1.6 meters is kind of a reasonable thing for a car to be in one dimension. And these things are maybe centimeters, because 130 centimeters, again, that's actually a little bit on the small side, but okay, the data is made up. But 133 millimeters is definitely not your car. And that is definitely something that we want to fix in our processing. We're going to do that in a second. Just remember this part. We're going to get back to it. The thing that we also said that I just want to show you quickly as an optional task, no worries if you didn't get to it, is you can also run this pipeline here and just make everything lazy. So we can do the same thing as before. We just slap lazy on it. We run it. This run pipeline function does everything for us. We can show the graph. Unfortunately, I can't because it's terrible to present. Do that in your own notebook. The idea is very much the same as what we saw before by looking at optimized true, optimized false. You can see what Polaris optimises away, and then you can time this stuff and run the lazy one, you run the eager one, and you can see that the lazy one is actually much faster here because Polaris gets to skip a bunch of stuff that it would otherwise have to do. Don't take these numbers as, like, meaning much because they will depend, like, the absolute values of these numbers, like, it looks like a factor 2, it's not always going to be a factor 2, depends what queries you do, depends how big your queries are, what your data is, and so on. But generally, the idea kind of holds. So you will often get free performance at the cost of some debug ability. All right. I think that's it for me. And we're going to go back to Olli.

Speaker 2 [58:30]

Andreas already mentioned in the discussion just now that, well, you get performance for free with lazy frames, but it does make debugging harder, and we generally recommend develop using eager mode, and then switch to lazy mode for processing large data, but always collect major application boundaries, so you actually get to be able to debug it in production as well. All right. So we've talked a lot about polars. We now want to focus on data framely. Why do we want to talk about data framely? Well, we have a pipeline, it's working, except for this one bug with the volume, but it's not ideal yet. There are a couple of issues with this. First of all, we made assumptions about the data, but we don't know if they actually hold true. For example, we perform joins on the model, but do they actually operate on primary keys? We never checked. of all, the signatures of our methods, for example, preprocess models, they don't tell us much. So what's going in here, what's coming out, we need to read the function to see what's coming out, and we need to look at the caller to see what's actually going in. And then there is no clear way to find the origin of inputs. So if we have a query like that, which is an excerpt of our report, and we need to trace a problem with the h of column, We don't actually know, is it coming from policies, is it coming from models? It's annoying to track down, especially if you're new to a huge project and want to fix an isolated issue. So to fix all of these issues, we built DataFrame.ly, and we want to motivate you to also use DataFrame.ly. So what can DataFrame.ly do? First of all, DataFrame.ly is really nice for documentation. The core concept of DataFrame.ly are schemas, and they allow for a structured definition of your data and its properties. Using that schema, you can then validate your data. So you can validate your implicit assumptions at runtime and your pipeline fails if those assumptions aren't met. Then you can use those schemas to define type annotations, much like you do in traditional type annotations to improve the legibility of function signatures. And last but not least, DataFrame is really nice for unit testing and creating dummy data, which we'll see at the very end. So first of all, what are schemas, actually? How do we define our data documentation? Schemas are about documenting all columns, their types, and assumptions about their values. So an example schema for the pets that we've had on the slides might be the following. We have three columns, name, kind, and weight. We have data types for them, and we also have type constraints. So for example, the name should be a primary key, the weight should have a maximum of 100. We can also have column descriptions for all of our columns that show up when we hover over these columns in the IDE, which is very nice. And on top of these constraints on individual columns, we can also define cross-column constraints using additional functions in our schema, where in this case we say that cats might not be more than 9 kilograms in this case. And in order to refer to our columns, there is also some syntactic sugar where we can reference the schema.columnName.col, which is the equivalent to pl.col.kind, and we can prevent typos, we can make refactoring simpler, and allows us to see very easily where certain columns are coming from. Using those schemas, We can then perform validation, because we don't only want to document our data, we also want to make sure that our data aligns with what we have documented. So validation and filtering are two concepts of verifying the assumptions at runtime. So if we have a data frame like this, for example, where we have two dogs, we can first of all perform validation, second of all we can perform filtering. So what does validation do? It fails hard if any rule in our schema is violated. So we run path schema, dot validate, we pass the paths, and then we get a validation error because the primary key constraint that we introduced fails because we have two dogs with the same name. One thing that is very nice here, we also set cast equals true, which allows DataFrame.ly to try casting all of the input data types that we have to the types that we have defined in the schema. You don't have to do all of those dot cast operations manually. The second option that you can do is you can use filtering, which allows you to partition the data into rows passing the validation and to rows failing the validation. So if you run padschema.filter with the paths, you get two outputs, the valid paths and the failure object, and the failure object allows you to inspect the failures of the rows, or inspect the rows that couldn't pass the validation. So for example, we can run failure.counts, and that gives us a dictionary that says, Well, the primary key was violated twice, and that's why certain columns or certain rows were filtered out. OK, so those are the basics about data-framely. As the next task in our notebook, we would ask you to use the data-framely schemas for raw data. I won't go through all the points. You can read through them yourselves. They're also in the to-dos in the notebook. And then we'll look at the solution in a few minutes. Just a question, because there is a package which is similar, was there a reason why you invented a new package for yourself? So when we started building DataFrame, it was in 2023, I think, Pandara didn't have support for Polus yet. So you could have obviously added support for Polus into Pandara. Our stance was we can build a much nicer API by focusing on Polars itself. So DataFrame is Polars native. It only supports Polars, but it supports Polars really, really well, and I think that's the main reason why we think it's a better choice for Polars than Pandora.

Speaker 1 [68:08]

And we're going to start looking at some solutions. Again, if you weren't able to complete all of this in time, don't worry. There's a solution data for every branch. Everything's in there. Just to give you a feeling for where we want to start from, I think the status that you started with already had a bunch of, basically, starting points. In the end, it's always going to look kind of like this. We tried to organize our scheme as well. And we have schemas for raw data, for example. A lot of this stuff is already filled out. The idea is that, OK, we're going to save you all the casting, but somewhere the information has to be for what are the columns and what are the data types. So a bunch of the stuff that we previously did manually and pre-processing implicitly now goes to this one place where you can look it up, where it's explicit, where you're going to find it again. And if you ever wonder, what is the schema of my data frame? You know where to look, and you know what to do about it. first thing that you did was you took this raw model schema and you just tried to validate the models with that, and that failed. That failed because it says primary key true here. We can actually try that out. I already checked out the solution, so maybe it's not. Yes, now it's reloading. Fantastic. We tried to validate the raw models, and this crashes because there is a primary key validation. And if we look at this, this is really similar to the example that we had with the pets, if we do filter instead of validate, it allows us to separate the good from the bad, the failure counts, and I'll say, okay, there's 20 lines where the primary key is validated. The point being that if you're in a productive pipeline and you have customers to serve or whatever, you can keep going with these good models and you can just let your pipeline still run through but get rid of the failures and treat them in a good way. You could log them. You could put them in your metrics. You can do whatever you do in writing production services with them. Or you can just look at them interactively to try to figure out what's wrong with your data and how you fix it. We obviously do some very scientific study here. We see that, oops, we just have duplicate rows for these models. But apart from that, they're all the same. We can validate that by trying to run models.unique. so that drops all, like if we have two duplicate columns, it just keeps one, and that kind of convinces us that not just are the model keys the same, but it could still be that the other rows are different, and then you have to do some investigation on which one you want to keep, but actually all the other ones are the same, so it's safe to just drop them, and we're going to do that in our preprocessing, and then everything should be fine. So what I did here is basically just get rid of the primary key, and then we're safe. In our preprocessing, though, we would like to keep this stuff as a primary key. So that's the idea of the multiple pipeline stages. Raw data is shitty. You can do anything about it. Just accept it. Then do a first step where you preprocess it. And after it is preprocessed, it should be safe for usage. And that means that it's our job in preprocessing to make this primary key constraint work. Because if models doesn't have a good primary key, we're going to be sad for the rest of our lives and the rest of our pipeline. So we're going to fix that. We're going to set it true to here. and then everything's good. So in the pre-processing, this kind of looks like this. The signatures kind of remain the same, except that we have cooler data classes now, because they actually have useful type hints. So before, you remember this just said, like, PL data frame or whatever, so you have no idea what's in these columns. And we now have our schema in here. We can click on this, and we can see, OK, what are the columns? You can also hover over this stuff to see where it's coming from and so on. And that's already a good first step to kind of documenting what's in these data classes. You can also see that you now understand how this is different from this, because it kind of looked the same before, but now it has different schemas, so everything's kind of a little bit more clear. In the preprocessing, then, we also add the type hints everywhere, so it becomes much easier to see what this thing does by just looking at the schemas. I don't really need to read anything here. If I know this schema and I know this schema, I can kind of start making some very good guesses about what function does. And that will help me a lot. So in the end, if you remember the policies, the pre-processed policies from before, it was actually a little bit longer. We get rid of all the casting. DataFrame does that for us. We just have to do the strip prefix stuff before, because obviously that is kind of in our user space. DataFrame doesn't know about that. We still have to do this. And then we just can send this DataFrame to validate. And that's all. If you don't know the syntax, a dot pipe on a data frame, again, a really cool feature, just allows you to pass a callable in there. And Polaris will return you the result of applying the callable to your data frame. So it's just really useful if you want to stack five functions on top of a data frame. You can just keep typing pipe, x pipe, or whatever. And you can just, it's a little more reasonable and nicer to work with. For models, kind of the same story. We still have to do our domain knowledge stuff that we have to inject here. But in the end, if you remember, there was this much casting there. That stuff is all gone. The only thing that's added now is that we have to fix this millimeter centimeter bug. So here, we can very scientifically just multiply everything by 10. That's less than 1,000, because we know that's OK. We looked at the data. It's safe. We do that for length, width, and height. And then everything's a millimeter, and everything's great. For the prep model schema, we can also then make sure that we never screw this up again by, oops, where are we? If we search for width, we can also write in min 1000 here. So we're not ever going to have any rows pass our validation where we forgot to make this adjustment. And finally, if we want to be super-duper sure, We can write a rule where we add some constraints on these volumes. The cool thing here is that this allows us to basically pull together multiple columns. So you can see we have to calculate the volume again here. We do this by multiplying length, width, and height. And in this case, because this is a class method, we can just use cls.length. And this is actually the column from above. So we can do all the operations with it that we want. And we basically return everything that's valid as true. And so we return everything that's valid that's between like 1 and 20 cubic meters, because otherwise it's probably a really weird car. Again, kind of made up domain knowledge here. But that's kind of the gist of it. And whoops, if we go back to our notebook, everything else kind of keeps looking the same. So if we, whoops. Where are we? So if we keep wanting to, yeah, sorry, I have to scroll past these really huge things here. If we just want to keep running our pipeline, everything just kind of still works the same. So we can build our report here, and actually we still need to do the preprocessing first. Yep. Pre-processing. And then we can do a report. And ideally, this report should now not have the millimeter bug anymore. Yay! Okay. It's all like reasonably sized cars now. And we can see that they have become bigger over time. I'll leave the interpretation of that to the reader. And I think, otherwise, that has us kind of set for being able to use DataFramD for these DataFrames. I think, yeah, there was a question there. I have that exact problem you're solving there. It's pretty amazing. I have one more requirement. How good is it with dynamic columns? Let's say you have a specified set of columns, and then some. Or you have some set of columns which confirm to a certain naming scheme, like is A, is B, is C, and you don't know A, B, C, whether there's T, E, F. Yeah. I don't think we have anything for that currently. People have mentioned it before.

Speaker 2 [76:54]

Yeah, there has been an issue in our repo for quite some time.

Speaker 1 [76:54]

Yeah.

Speaker 2 [77:00]

If you're interested in that feature, feel free to react or comment on that issue. I don't think we personally had the need for it so far because we really like having this clearly defined schema where there's no surprises in the columns. Yeah, so it doesn't exist yet, but we are aware that this is somewhat of a limitation.

Speaker 1 [77:27]

I think we can talk about this a little bit more later. I think you can do that. In my mind, that loses a lot of the upside of being able to click on it and being able to look up stuff. Happy to discuss after. Let's do that then. Thanks for the question. No. But we'll be here after. Where is he? He was there. There. Not there. Here. Like, here.

Speaker 2 [78:05]

All right. Before we continue, one quick note. We kept using validate in our report pipeline because we fixed the bug in the raw data. Obviously, in production, you might encounter raw data you haven't seen before, so it's also valid to just keep filtering in production. Just as a side note. So Andreas already showed some type annotations, and I briefly want to talk about, why this is so nice. So we can define, for example, a second result schema for our use case now, and then when we define a function, we can define it in a way that's very easy to reason about, because we now have these type annotations, we define exactly what an input is supposed to look like, it has to have the pet schema, and the output it has to have the pad count schema. So I don't have to look at the caller anymore, I don't have to look at the body of the function anymore to know what this function needs and what it is doing. And also what's easy for me to understand is also easy to understand for an AI agent. So as more and more code is written by AI agents, it makes it really easy for these agents to understand what the function is doing and how to use it correctly. The nice thing about the type hints is if you use type checking, which you probably should, it helps you a lot. First of all, it ensures that only validated data can be passed as input. You can't pass data frames with another schema. You can't pass plain polars data frames because they must have a schema, so you have to validate. If you return something here, again, you can't just return a data frame. You have to run validation. checking ensures that you do not cheat, because the type checker is very integral in this entire thing, because those dyLazyFrames are a pure typing construct, so if you run type at runtime or do an isInstance check, those are still standard Polar's lazy frames, so this is just something that exists for the type checker and for the reader of the code, So that makes it very efficient and very, very useful nonetheless.

Speaker 1 [80:19]

I think we skipped that part. We skipped that part. Sorry, I already showed the solution for that because we're in a little bit of a hurry. The idea was that you would work on the pre-processing in the same way that you worked on the raw data. We already looked at the solution. Yeah. Feel free to look at it in your own time also.

Speaker 2 [80:37]

Do we continue with this then?

Speaker 1 [80:38]

I think in the light of time, we have, I think, nine minutes left, right? It's 45, yeah. I think we're also going to probably skip this part. This is a little bit more of the same. The idea is that before we had these two tasks where you would basically amend schemas that we already gave you. The task here is to write a simple schema for yourself. I think that's valuable just to see that it's really easy to do yourself. I think we're going to skip that here because we're kind of a little bit out of time. So I would propose that we just kind of keep going here.

Speaker 2 [81:11]

All right, then as for the last thing that's very useful with data framing is that it helps you a lot to write unit tests And that brings us to the last part ensuring the correctness of our pipeline So our pipeline is already much better than before but still something something is missing So our code validates at runtime, but ideally we don't want to fail at runtime. We want to catch issues early So we want to test our data processing logic the same way as traditional traditional software. So we want to write unit tests. What's often the case with testing data pipelines is that creating small dummy data is tedious, especially if you have wide data frames. But in many cases, your data processing logic is supposed to operate on wide data frames, but you only really care about a few columns. So how do we solve that? We solved that with DataFrame.ly, which allows us to easily generate synthetic data that adheres to a schema. For example, what we can do, we can create stubs, we can create empty on a schema that just gives us an empty data frame with all the right columns and their data types. We can also generate random data, so we can do schema.sample and then just the number of rows. It generates random data that adheres to our schema. If you have very complex rules, then this might take a few more milliseconds because DataFrame.ly runs validation or filtering internally and then resamples if anything violates the constraints that you have defined. But what's very or the most useful thing for unit tests is that DataFrame.ly provides a a way to provide value overrides for certain columns. So in this case, we sample from our model schema that we have defined, and we define overrides for the fuel type column. So we say that the fuel type column should contain diesel, petrol, and petrol, and then if we call sample, we get a data frame with three rows because we defined three overrides, and 40 columns. So data frame, you just make sure that everything else is filled in, and the fuel type column is exactly what you have defined here. And now, df is a dataframe with the right schema, and you can use it to pass into your functions. All right. So, to give you some hands-on experience with this, we have the last task. We have the last task in our notebook. Again, in the a show of time. We will just present a solution.

Speaker 1 [83:58]

Just present a solution. I think we're just going to show the solution. Again, it's more of a proof of concept type thing where I want to put this idea in your minds. Please try it out at home. I think it also is fun after a while. The idea is that we have a test report function here in our tests directory. So if you saw the outline of the repository, there's a test directory here for any good software project. And you can run pixie run test to run the test. It passes. amazing. And the idea is that we just want to test, or we want to write a test for our car volume by age query. If you remember that thing was like maybe like, I don't know, five, six lines, seven maybe. So it's like possible to screw it up by implementing it a little bit wrong. And you just want to make sure that it's easy for you to test that. And what we need to do for that is we obviously need these data frames, as already hinted at, oftentimes you don't need most of the columns. You have to operate on like two columns, three columns out of a thing that maybe has 40. And so what we can do is we can basically generate a simple test case where we have two models, model M1, M2. We just inject some overrides here that we need in order for the volume calculation to make sense, but we just leave out everything else that we don't care about. Same for the policies. We're going to say we have one policy for each car. We put them in different age brackets so they show up at different parts in our report. And again, we don't really care about the other stuff, so we're just going to leave that out. And what we can then do is we can write an expected data frame. I think that's typically a good practice to write down beforehand exactly what you want to do and put it in a data frame. It makes debugging much easier afterwards if the test fails for some reason. And so we expect two rows, two H brackets from basically minus infinity to 10, 10 to 20. And each of the models is in these rows, and the volume we can calculate by hand. So we just multiply out the values and we are sure that this is correct. And then in the end, the actual test, like execution of our code is pretty straightforward. We can feed these things in here because we already validated them. They're the right schema. They have to go through. If they don't go through, we wrote a bug in our code and we are guaranteed that that is true. And finally, we can assert that these two data frames are equal and that works out. So apparently our code correctly does the volume check. And importantly, we didn't have to re-implement a bunch of stuff. We just basically checked for individual values. If you have edge cases that you can think of, this would be a great place to put them. And I think that's really all we had in terms of execution. I hope that you were able to do at least most of these things. I realize that some of the time was maybe not enough if you already had to get started on Polar a little bit. But feel free to do these things at home. And if you have any other questions, also to shoot them to us offline on GitHub or LinkedIn, and you need support or anything. I don't know. Do we actually have more slides?

Speaker 2 [86:53]

more slides? Yes, I think we have one more. Ah, key takeaways. Key takeaways.

Speaker 1 [86:56]

That makes sense. All right, let's do that.

Speaker 2 [86:59]

So, we talked about Polars and DataFrame.ly. We want to encourage you to leverage Polars to write performant and readable queries. If you talk to people who have used Polars, the common sentiment is, I came for the speed but stayed for the syntax, so queries are really nice to read. Maybe not as important today when AI agents are doing a lot of the work, but still very, very nice when you read code yourself. And then use DataFrame.ly to document your data, make all assumptions that you have about your data explicit. It makes your life a lot easier down the road. Always run validation at major application boundaries to ensure that at these application boundaries your data is exactly like you expected. And then take advantage of sampling to write unit tests. Makes your code a lot better. All right. That's it from our side. I think we maybe have time for one or two questions. Otherwise, as Andrea said, please catch us and reach out.

Speaker 1 [88:09]

All right, okay, I see there's maybe something on on the inner webs

Speaker 2 [88:24]

Does anybody have a question so I can discuss them?

Speaker 1 [88:25]

There's one over there. There's one here.

Speaker 2 [88:26]

I'm going to post all the questions on the Discord so they can answer in the FADM channel.

Speaker 1 [88:42]

experience with my pie

Speaker 2 [88:43]

IPy and data.

Speaker 1 [88:44]

and data framely because I

Speaker 2 [88:46]

because I had the problem

Speaker 1 [88:46]

I had the problem with Pandera and MyPi.

Speaker 2 [88:48]

and MyPy, that it didn't really work together in the type

Speaker 1 [88:50]

in a type-by-annotation way. Have you had any experience with it? Oh yeah, that's like 80% of our robustness in all of our lives comes from the fact that we rely on MyPy to tell us when we pass stupid stuff into our functions. So this should work really well. All of this, the thing that Olli said about how type-checking guarantees that you don't pass invalid data, we enforce that with MyPy in our lives.

Speaker 2 [89:17]

There was one more question over here It is run it is fully random by default you can pass in a generator that fixes the seed But ideally your your test should only operate on data that you that you explicitly define But yeah That is true, it does happen to us as well. To make it reproducible, you would have to pass on this generator with a fixed seed, but that would make it reproducible, yes.