Power up your Polars code with Polars extention

Speaker 1 [00:09]

Thank you for coming. Again, like if you already know me or have been to my talk or workshop, you know everything goes. So I'm very flexible here, especially for workshops. I want it to be more interactive. And if you have questions at any time, raise your hand if you are here with me, I will come to you, you know, or if you have a question that you want to just like shout and I can repeat for the remote audience. If you are remote, please use the Slido. we would have volunteer here to direct your questions. So hopefully we can still answer your questions. So I think for those of you who may tune in a bit late, unfortunately the projector is somehow not working. It's the first workshop, so we are kind of testing it right now. So, or maybe it's my computer that doesn't like it. I don't know. But anyway, I have put the workshop material link. So it's a GitHub repo on the description this session so if you go to the conference schedule click on this session you should be able to see the link to the GitHub repo somewhere in the description right so before we start I mean before I let you start working just a few explanation of why we are doing this why we want to know about Polar's plugin so who has used some like custom function when they are processing a data frame yes so the thing is we well we use Polus so how many of you use Polus yeah so we use Polus because we love it being fast it's written in Rust by the way we also need custom function because sometimes the default function provided by the framework, which is Polus, doesn't fit our needs because we may have some niche use case that, like, I am manipulating the data in a way that nobody will ever do the same. So it's not useful for the Polus maintainers to include that. So you can write your own custom function. But the problem now we have is that if you're writing a custom function in Python and use a lambda function or whatever to apply to your polars data frame, it may slow a little bit of it down because Python is slow. And also because like polars need to do some conversion when it's, you know, changing the Rust thing into Python thing and the Python thing into Rust thing. So to avoid that, what we could do is to write the custom function as a polar expression in Rust. So that's how the Polus plugin come in, it's like a very good tool that if you are capable and comfortable writing in Rust, that will give you a way to hopefully get your custom function running with Polus faster. So I have also included some pre-file check and what you need to know before coming to the workshop and what you need to install before coming to the workshop. I hope most of you have Rust installed in your computer already. That's the first step. If you don't have Rust, if you can't compile Rust code, I'm afraid you may not be able to follow the material today. You may have to do that first. And then we would use Rust, we would use PyO3, which I will explain a little bit later what what it is, to create something that can work in Python, and then we'll test it in Python. So we are using Python 3.13, you know, kind of new, in the workshop. You can do, it should be compatible from 3.9 upwards, but if you see something weird, please let me know. I will come and have a look. Also we use virtual environment, make your pick. I use UV now because, you know, it's, again, like, new and shiny and fast. I love it. So, yeah, so, you know, again, make your choice, but something may be a little bit different from the instructions in the workshop. If you are using Windows, the lucky ones that are using Windows, lucky ones, you will have to install some extra stuff. Because, you know, yeah, like, you have to have some default, like, C++ build tool and some kind of, like, make sure the file works so you can compile your Rust code. Unfortunately, I am not a Windows user. I, you know, graduated from Windows a long time ago. So, I will try my best to help you, but not guarantee, like, that I can solve your problems. I'm not a Windows expert. Hopefully, there's a Windows expert here that they can help you, hopefully. So, also, because today we are writing Rust, we will use PyO3. If you want to learn more about Rust and learn PyO3, I have also put down some resources there in the repo. You can go there and check it out. So today, you don't... You can just follow along. You don't necessarily need to be able to write Rust yourself. But in the long run, if you want to do something like customize, you know, customize your function, of course, you have to learn Rust. So those are the materials for you. And then after that, again, set up instructions, very simple, things that create a virtual environment, install Maturin. Maturin is a tool that is in the PyO3 kind of ecosystem. So if you use PyO3, then I think 99.9999% you use Maturin as well. Technically, you don't have to, but you would make your life much more difficult if you don't. So, instead of Maturin, Maturin can be pip install, of course, Polus, because we want to test it out, so we'll use Polus to test it, of course. I've also marked down the versions that is assumed in this workshop here. Again, it could be compatible with other versions, but, you know, if something weird is going on then maybe, you know, matching the versions could solve your problem. So unfortunately, again, like, I would love to do it together with you, but unfortunately the screen is not working. So yeah, I will just explain the first exercise, what we do. So I think before we jump on to write some Rust code and, you know, get excited about it. We will just look at how the workflow should work because I know like all of you are mostly Python developer or data scientist or like someone use Python a lot. But Russ may be a little bit, you know, not so familiar with it. So how we got to do is, again, we would use Maturin a lot. So Maturin can also help you to start the project structure. It's a little bit unique because it's not like a standard Rust, even a standard Rust project. It needs to have some extra kind of configuration. So I think if you do any, like, start any Pyre 3 project, the best way to start is to do Maturin init. So if you type Maturin init, which is, again, if you're following the, along with your own, know reading the material you'll see that like you know step one start a project with build settings so if you type Maturin init you will see that like a lot of files and you know all these like structures of the whole directory is setting up for you we have some make we'll have to make some adjustment because the Maturin default is not designed to you to let you use it with Polus, right? So first of all is that we have to add something in your cargo.tomo, so what cargo.tomo is, is basically like your PyProject.tomo but for Rust. So yeah, so again like if you follow along, if you see the code there that I provided, you see we have to tell the project, like you know basically Cargo and Maturin that I am also including Polars in my project here because I'm building a Polar plug-in, so Polars need to be included, PyOvry usually is already there because Maturin is designed for PyOvry, so PyOvry will be there. And I think we need PyOvry Polars, so another dependency PyOvry Polars, so that is letting you use some, basically it's letting you write these extensions because, so okay, what is Pyrofree? Pyrofree is, if you haven't heard about it, it's a library that lets you compile your Rust code into a Python library. So the funny thing about this, like, polar extension, it's kind of like a Python library that is a third-party plugin for your Polus, right? So Pyofree Polus basically is doing a similar thing as Pyofree, but they added... So it's kind of like extra stuff that they provided specifically if you're developing for Polus. So we need that, in short. And then the last one is... I don't know how the community pronounces it, S-E-R-D, S-E-R-D, I don't know. not even English but yeah we need that because we are using some functionality that it's provide and I forgot what it is I will tell you when I remember okay so also we yeah so yes questions There is a link, I know it's not very obvious because I just added like now. So, if you scroll down through all the instructions for setting it up. Workshop materials, thanks. Yes, thank you. Thank you. Yeah, let me know like I mean like you you I was like Oh, why don't you ask the question earlier because you know, I feel so sad that you are like, oh, I'm so lost for You know like 10 minutes and I want to help you. Okay, so Yeah, go to the link. It's just the readme file that I'm talking about here. So now we are at step 2 Define. Oh, no, but just above step 2 define expression logic in rust So we're talking about that section there. So those okay. So before the question I was talking about those are the requirements The requiring libraries for rust so that's on the rust side, but we also need something on the Python side, right again We're gonna test Test our project When you know when when we build the extension will test if it works if it works with polis correctly like intended so So in your pyproject.tomo, which is on the Python side of things, you have to also add the requirements there, maturin, because, again, like, we are using maturin a lot. Without maturin, we can't do anything. And also, polis. Again, I have marked down the version, yeah, there's, like, some version requirement there is mainly because, you know, for older version of things, it may work a little bit differently, so I mark it there just in case. And then after that, like pyproject.tomo, we have also added one settings there. So it's like for Maturin, so that setting is mainly for Maturin. So we say that the module name is... So in the readme, it says Polar's Plugin 101. Actually it doesn't need to be that. It's just that, like, the name of your plugin. The important thing is .__internal. It's internal because if you don't do it that way, Maturin will just build a library that you can just use it alone. But we, so this library that we are building is an extension, right, it's a Polus extension. So it need to work with Polus, and it need to do some kind of like, you will see in a bit later that we need to do some kind of customization on the Python side make sure that it works with Polus in Python. So that's why we mark it as internal, mastering we still compiled it so it's something that have a Python interface, we can still use it in Python, but we'll make sure that it works with Polus in a bit in Python code. But don't worry too much about it, just know that it's something special for the Polus extension. So now, step two, define expression logic in Rust. So this part is quite straightforward because it's just, you know, if you have do PyO3 before, that should be not, you know, it should not be something new to you except a little bit of things. But if you're new to Rust, welcome. I would, you know, just quickly go over. So use, the keyword use is kind of like import in Python. You can, you know, without that, you know, the compiler will say that, oh, you haven't, you know, you haven't imported. You haven't, you know, linked it up. Like, why are you saying something that I don't understand? And then, you know, function fn is like df in Python. And then, of course, there's all these curly brackets. So that's, you know, syntactic syntax in Rust is different than Python, but it should be quite similar, similar enough for you to understand what's going on. Also, there will be, you will see the Mule and that a lot, again, very unique Rust things. It won't appear in Python. If you want to learn more, there are materials about PyO3 that you can learn about, or Rust that you can learn about. So the first, so again, like I won't go too deep into the Rust code, like what does it mean in Rust because, you know, I have another workshop that's, like, you know, we would dive deeper into, like, writing, you know, Pile3 and Rust, but this is not that workshop, so I would just assume that you're comfortable if I don't, you know, explain everything. So yeah, in the first function, so capitalizeString, we would take two arguments, so that would be a value, which is a string, and an output, which is a multiple string. Again, like, very Rust thing, I'm not going to explain that. So after that, we just say, like, hey, if it's not, you know, empty, just capitalize the first character, you know, this is a very straightforward thing to do, again, like, if you're familiar with Rust, it should be, you know, kind of easy to understand, but But even if not, it should be similar to Python enough that you kind of, you can get the feeling of what it's doing. So after that, you see that we have some, you know, the hash, so it's like, oh, it's not a comment. I mean, I know in Python it's a comment, but in Rust, that means that I'm having a Marko here. Again, like, I'm not going to explain Marko. Now just imagine it's kind of, you know, this line is kind of like a decorator thing in Rust. So it's a hash, and then you see there's like Polar's expression. So Polar's underscore EXPR, and then there are some parameters inside. So that thing, if you have done Power Free before, you would never see it, because obviously it's from the Polar side of the thing. It just means that you can use it. So for example, here we see output type equals to string. We are using it to define what would be the output type of the polar series. Here we're just taking a string and then the output would be a string. So that's why output type is string. But without it, then we'll come into some issue because polars will not know what is the type of that specific polar series, right? So we need that. So that's the unique thing in this workshop compared to a PyO3 workshop. And then, yeah, so like in there we'll do the actual thing that like we'll call the capitalized string function that we have built before and just, you know, say that, hey, we have like a string chunk. So basically you can imagine it's like a polar series that got input, right? So again, like, if you want to know more, you can look into the documentation and see like what are these things, but basically it means that it's outputting a series that is the type is the string. Yeah, so far is it okay or you get very lost when you look at the rust code? Yes Zero so Is it the first column here is meant in this in this instruction? That's that's 914. Yes It's mainly because yeah so it's because input can be multiple columns so this is just like unwrapping it yeah yeah it's okay okay so okay the question apply into a string amortized that's a rust functionality in the string object or okay apply into a string motorized dest is actually provided by the Polus extension, like it's provided by PyO3 Polus. It's from Polus. Yeah, like Polus, we have the usePolus prelude and style, right? So that's from there, it's from Polus because, so when you unwrap the output, so inside is a special type of things as defined by polars. So yeah, if you want to see like what that is, it will be appearing in the polars documentation there. Yeah. Yeah, so basically it will take, you can see that it's taking the function that we defined before and then it will just like apply that to CA because CA is the series. So it's not just a string. So it's a special type of thing that is, you know, defined by polars. Yeah. Okay. Any questions? It's very confusing. Okay. I hope you find it interesting. I find it interesting. So... Yeah. So after that, we would... Okay. So we have written that. So that is the expression.rs. So if you have some experience writing Power Free or any Rust library, that you know that you have to have a lib.rs, right? Because it's kind of like main in Python. So the compiler will know that the lib.rs is basically the entry point when it's looking at what's going on and how to put things together. So expression.rs, which is the script that I just described, is basically like I don't want to write everything in lib.rs. I want to write in a separate file. So you write the expression.rs. This just makes things more neat. Technically, you don't have to have expression.rs. You can just put all of those functions inside lib.rs. But when your extension grows bigger, it will get messy very quickly. So this is the recommended way to do things. Yeah. So, yes. So there's already a lib.rs in this folder. Should I override it with the new content? Yes, override it with the new content. Because when you do maturing init, it will give you a lib.rs. But that one is just an example when you just do you know compiled a Python library So this is not going to work with polis that one so replace it with the new one Yeah, so the new one you see the first line is using mod expressions So that is kind of like importing that expression to RS again Like you know, that's kind of like, you know, Python will only have import but you know in rust It's a little bit difference. But anyway, so you can imagine this import for now now, not to confuse yourself. And then later you see like you have PyModule, well, because it's a standard for PyO3. So PyO3 will recognize that and pack it into a Python module. And then you see that like now we have underscore internal. Because we have named it something something dot internal, so like we have internal. And then you see that, like, in the module, we'll add... So now it's not adding anything. It's basically just adding the version right now here. Yeah. So it's just, like... So it's adding version, which is, like... It's just saying, like, what version we're using right now. So now it's not actually doing anything in this script. So we haven't added yet here. Also, we have the global underscore allocator. So that is also unique to Polus. Yeah, so it's, you know, we, because, like, we don't add the method directly like we do. And if, for example, if you have done PowerFrame before, right, we will edit there. So it's like, why m.add is not, like, we are not adding the capitalized that we have defined in the expression. Why we are not doing that? Because Polus works a little bit differently. So that's why we have to use the global allocator there to do the allocation. So yeah, so that's why it's defined there. So we are not like, yeah, we are not adding it directly in the module. Like we are not doing m.add, capitalize there. It will come back afterward. You will see where it goes after. So that's the Rust side of things. So it's like Rust code going through and through. But like I said before, because we use .internal, so we still have to do something on the Python inside to make it work, otherwise it won't work. So now we create a underscore init.py and then inside it we will put those like code inside like when we have the plugin so it will be put. So if you follow the instructions it will be put, the init.py will be put within your plugin because, you know, that's where when you import your extension, that's where it will get processed. So now we go to Python code. In the Python code, we would have very standard thing like the path and the type checking. That's not Polar, so I won't explain that. So here you see that we have. So Polus also provides these things for you to plug and play, right? So there's also the polus.plugins. You can import the register plugin functions that provide a functionality for you to put your Polus plugin into Polus and work with Polus. So here we would define another capitalized. This time it appears in Python. So now it will become a Python function that you could use or actually it's a polar, technically it's a polar expression because it will return a polar expression. So it would provide you the polar expression that you will be using. So it's there. You also have to define the plugin path, but I don't think you would change it very often. It's just like the parents of the file because, you know, you're putting it in init.py and And of course, the path is the parents of the file, so yeah, just make sure you have that line. So you see that in there, so in the Python, dfcapitalize, you see that like we are returning a polar expression. So here, we are returning the capitalize that we have defined in Rust. So you see in Rust, we didn't add it as a module for PyO3 to add to the extension, instead we added to an allocator. And here is actually where the allocator will be able to find the capitalized and make it into a polar expression that when you call it from polar side of things, it will see the capitalized function and use it. Does it make sense? I know it's very convoluted. Any questions? Yes. Where did we introduce the expressions actually in the libdut-RS? I mean, did we publish this, capitalize somewhere in this libdut-RS? Shouldn't we do this somewhere? So you can imagine now it's becoming a polar expression. So what this Python function does is to create... So basically, this is like a wrapper, wrapping the Rust capitalized, and then it's wrap up as a polar expression. So imagine PyO3 is doing this behind the scene, right? PyO3 is wrapping a function. So if you add the hash, py function, power phrase wrapping it as a Python function that you can call from your Python library, right? Here you are just doing it yourself. And then, like, oh, can you wrap it into a polar expression so polars can call it and use it? That actually, like, to call it capitalized. By inserting this mod expressions in the lib.rs, that was enough to... Yeah, that function take care of it. That's provided by Polus. And Polus will take all these things and then it will... Because we have added it to the allocator, right? So probably it will find it in the allocator and then find, oh, that's the capitalized that you have. That's what I mean. Where did we add it to the allocator? In the Rust code. If you scroll back up in the REST code, you see like global allocator, static allocator. So that we created the allocator there. Yeah, but we didn't add anything to this allocator. We haven't added anything. Why does it? I think it would do a search on the capitalized. The name needs to be matching. Okay. Okay. Yeah. But again, like that's my speculation. I haven't looked into what that function actually do. I may be wrong, but that's my speculation. So, oh, good for all the gauntlets. Now, we will build and test the plugin. Finally, we will see if it works. So, if you do mastering develop, again, like if you have used PyO3 before, you should be familiar with that. So you do maturing develop, and then it will create the plug-in for you, supposedly, if everything works. If you see any, like, compilation error and stuff, then, like, please let me know. Something wrong with Rust. We'll debug it. Yes? Yeah. So I figured on dependencies, we're missing Flourish arrow. Yeah. Yeah. Oh. We're missing PyArrows. Yeah. In python or in rust? In rust. In rust? Oh, that may be something new. Just like from the dependencies, You see polars equals so. Yeah, okay. So just checking the code, right, on the cargo. Below polars, you see the dependencies polars equals Star. We just need to add below that Polar's arrow. That may be something new. For me, before, i don't need it. Again, it works on my computer. Again, edit. Let me double check. Sorry, i pressed the wrong thing. Let me see. Just one more thing. Okay. Yes, exactly. We also need to add a module Called by module methods. With that it compiles. Are you using the same version that I'm using? Probably not. Yeah, like, again, like, I have checked it last week that, Like, I'm using the version like that, and it just worked, And that's new to me. But, I mean, like, you're going On an adventure, why not just, like, follow along and add those and see if it works yeah yeah okay other than that I was just adding that in theory polars does not depend on arrow or pi arrow so I just commented on that yeah so let me confuse yeah it's it's it's like I mean that's new to me because IT WORKS LAST WEEK WITHOUT THAT, AND IF YOU USE THE SAME VERSION TECHNICALLY, I THINK IT SHOULD WORK. NO, IT DOESN'T. HELLO. I RUN THE COMMANDS TO CREATE THE PACKAGE. And it generates a couple of, you know, a target folder and also a file called internal.evi3.so, something like that. Well, it creates some files. My question is that these files need to be also committed and published on a repository or because maybe it could generate more files and more files and how do you manage that? Right now, I will only have a init.py in there. So the file you mentioned, right? And let's go internal.abi3.so. yeah that is actually what has been compiled okay so it it will sit in your for me the name is Polus plugin 101 right Polus plugin 101 inside there should be the init file the Python file that we just talked about and that internal file right yep so so anything that is under the Polus plugin 101 so Those two files, that is actually your extension. So if you want to give it to someone else to use, you can send it to them, you can put it on GitHub release on GitHub, or you can put it on PyPI. So that's actually where your plugin is. That's your plugin. Okay. Yeah, so you can keep it away. You can keep it away from this GitHub repo because this repo is for building it, right? But if you want to release it that's that's your release. Okay. Yeah. All right. Thank you. Yeah any more questions For some strange reason it doesn't create this Compiled file in the folder. It doesn't create then You would have a compilation error, right? No, it's the combination is just fine Double-check your cargo.tomo to see if the settings are correct Okay, so some it doesn't create that file is the name matching is the name of the library matching the folder name So so the font the name like Polar's tutorial of icon 25 uh-huh should be the same as okay it is not the same but it doesn't create the folder on its own so okay so there's a few things that need to match so first of all so your Tom so your cargo.tomo file it should say the name of the package it could be my package right let's assume it's my package so if you Go to pyproject.toml under the two dot maturing setting the module name Should be my package dot underscore internal so those two need to match when Those two match oh and also the project name need to be like my package as well In pyproject.toml so those three matches then it should have the folder name as my package when you compiled it so it would be like under my package you would have to underscore internal dot yeah yeah it's tricky you have to match the names because otherwise it won't found it okay I will try that thank you yeah I can come and have a look later I think now we have okay we have like reach almost 45 minutes do we want a break and then you can just try it out yourself right now and I can also walk around and do troubleshooting does it work for everyone okay let's take a short break you can try you know you so you have done much rain develop and now you can just like test the plug-in yourself because I assume you can write Python code and polar so I will let you test it yourself and I will just walk around if you have questions anyone want me to come over and have a look So for the remote attendee, so after walking around, we have some issues. Because people have issues with the versions. So one way of making sure the versions work is to download the cargo.log from the repo. Then it will help because sometimes, you know, there's some versions that doesn't like the other version. And, you know, we have all been there. So you know, I mean, like for Python, we have requirements.txt. But for Rust, it will be cargo.log. So if you get the cargo.log from the repo, so if you have, like, version issues, just get the cargo.log from the repo. And it seems like the projector is working. Let me try. For some reason. Yeah. So let's see. Let's see. Let's see. Let's see if it works. Yay! It works, I think. Woohoo! Finally. Good. Then the next speaker have a projector to use. Okay. So, right. Let's go back to where we were. I'm sorry about that. Let's go back to README. And why is it so small? Why is it so small? What happened? What happened to you? Okay. Let me open it again. Right. Okay. Yeah. Okay. Oh, by the way, if you are curious about what I'm using right now, this is not PyCharm. This is Rust Rover. But in Rust Rover, you can install Python plug-in, so you can also run Python there. Yay. If you want to know more about it, talk to us in the booth. I'm not going to say it here. So yeah. OK, so where were we? OK, so hopefully most of you go through all the gauntlet and have already registered and run it, and then try in PolarSan it works. Now I'm lost. Wait. OK, I think we are. OK, we have go through this. We have go through this. Step four, the heading is a bit wrong. I will fix it afterwards. But congratulations, you have created your first Polar plug-in I hope you test it out and it works so now we can go through there like you know now it's kind of working so we kind of we can do more about it we don't want to capitalize all the time we want to do something else so now we are working with numbers well a lot of us work with numbers in data frames I think that's quite you know standard the example here is very simple just just converting Celsius to Fahrenheit is a formula you can find online of course I don't remember it by heart so can we created that right can we created a expression just converting the Celsius to Fahrenheit so again if you so I know some of you are more like you I already know that your confidence and have done it like power free or rust before so maybe you can do it yourself right you don't have to follow along you can just you know oh you know go to all the things that we already have create another function changing you know adding the formula there the you know calculate one thing to pay attention is the output type need to match so far the output type output types we have used is string if we are dealing with like temperatures we want well I mean you can use integer, but Fahrenheit will have, you know, decimals a lot of times. So maybe use a float because, you know, you don't have to worry about changing it back to integer after the calculation of, like, division, whatever. So yeah, like, for those of you who are new to Rust, you can't just use float. You have to decide whether it's a flow 64, a flow 32, or flow 12, no, 16, or flow 16. So yeah, if in doubt, just use flow 64. It may not be the most, you know, memory-saving one, but, you know, we're just doing an exercise So, just use flow 64, yeah, so, again, I don't want to, like, again, explain it in so much detail because I want you to try it yourself. So, again, like, if you are confident enough, don't look at it, try to, you know, just follow the formula and make one yourself and, you know, make it work and then try it out in So, maybe I will give you a little bit of time to try it out yourself. Yeah, if you are stuck, then you can just copy and paste the code and read it. So, that's just, you know, we are stepping outside of our comfort zone and trying different, like, input and output type thing. So, I will give you, like, two or three minutes. Again, like, ask questions at any time. I'm happy to take questions and then after that we will have another exercise to find the larger value so that's a little bit difference I will go over the difference in a bit when you're done with the Fahrenheit you know temperature conversion thingy. There's a spelling mistake. In the Rust function name, the age is missing in Fahrenheit. And the... A PR is welcome. Yeah. Thank you. I just wanted to point that out. Yeah. Thank you. Oh, by the way, I'm terrible with spelling. So if you found anything wrong, a PR is welcome. Do you want me to grade one? If you want to. It's up to you. I'm not saying you have to. if you have time you know you can do it you don't have to do it now you can do it after or like you know you got like a bit you know the talk you know maybe you go later it's boring and then you want to do something else you can do this okay so after that conversion we are going to do the so we are going to do something a little bit different so for the next exercise getting the larger value we would have two inputs right so because we need to compare two values we will have two columns instead of just one column so how we can do it is again like remember we have the input square bracket zero that was a very good question to like kind of lead to this place because input is actually multiple columns so you may ask like why do polars know that I'm taking multiple columns well we would we would define it in the Python side when we you know create a polar expression but on the right side of things it's still like a you know it's still input there and then when you have multiple columns the first column it will be the you know the first one in there the second one will be the second one, so we just use the square bracket to get them back. So that's kind of answer the question right now here, like you can see, input 0, input 1. Okay, so go back to the question of how polars know that this polar expression is taking two columns instead of one. It's here, it's in the Python side, we define it here, args, we define what is going to be the input for the polar expression so you see that we have expression one expression two so we actually take two columns from polars or series like column what do you call them but anyway so here take two inputs basically here it defines it it will take two inputs and then they would be packaged together and when it go to the right side of things it will be this type it will be like n series type is it Too small is too small for me, so I will zoom in. So this is actually a sequence of series. So it's multiple. So that's why we have to unpack it here with the inputs is actually multiple, so we have to unpack it. Does that make sense? Questions? All good? So if you're good, then challenge yourself and take a multiple, like you can now take yeah, extra challenge. Now you can make the largest expression and take an arbitrary amount of inputs. Can you take arbitrary amount instead of just two? we hmm yeah also I have a possible solution there your solution may not be exactly the same but that is a suggestion if you are like oh I need some hints you can have a look there so I'll give you some time to challenge yourself, I won't spoil it for you just yet. So now you understand how to take multiple input, can you take arbitrary amount of input? Yeah, I'll take the silence as challenge accepted, so I will just pause here for a I have a bit of a conceptual question. So, Polar's has a max between columns function sort of built in, right? So the question is where are the limitations in what exists in Polar's already and when do we really see speed up? If I'm deciding here's what I want to do on my columns, when should I decide, okay, it's worth the investment to sit down and write the Rust code versus try to build something out of, you know, Polar's existing operations or Python primitives, et cetera. Right. Rule of thumb, if it's provided by Polar, use it. Because that would be, you can't, if you can bid it, you probably want to contribute to Polar or become a maintainer of Polar. So if Polar give it to you out of the box, use it. So the next thing is whether you want to write a Python function, use dot apply. So you define the custom thing on the Python side or go through all this thing. I found it interesting, but you would be like, I don't have time for that. I'm working. So again, like I, if I were you, right, you're working, you just want it to work, do it in Python first. If you do it in Python, you're not happy with the speed because if you put it in production, your manager say like, hey, this is, you know, this is not good, this is too slow, we need it faster. You know, come up with something that is faster. this could be your solution so I know not all of you may come into this situation very often usually it only happened when you are handling a large amount of data or you need to have it done really quick because you are doing something real-time then this is a tool for you but again like if you're just both of typing something it may not worth your time to like you know write Rust code and other stuff right so I think if you use something very often maybe maybe it worth the time like if you use it all the time and then if you just like go through this once and then it will just make everything you in your team faster then yeah maybe it's worth again it's the trade-off between the developers time and the execution time so you make the choice yeah I just performed a little experiments just right now with this whole capitalization thing so with an rust internal function that we wrote ourselves and with a Python given function, it's called string.title, and I start to be faster with the Rust function consistently with one million string samples. If it's below that, if it's 100,000 string samples, it's roughly equal in speed. But it starts to become substantially faster with one million samples and more. I don't know if that helps, but just random experiment right now. Yes, I think that's a very good Good point that like maybe you should do some you know profiling like what you just did do some experiment see Like where it's going to like how how large is large right this you have to switch over so again, I trust your Developer skills that you will be able to you know, make some good choices Good Do we want to look at the solution or you have already seen it and it's not exciting anymore Okay, let's have a quick look Oh, it doesn't work because it's well, it's not on github is here Okay So wait, wait, I'll put it largest Yeah, that just is. Oh actually is not as coming because I forgot I read this a long time ago. So I was like, okay Okay. Let's see if I can zoom in. I can't zoom in here. Okay. So I will just tell you what I did. So on the rough side of things, I just, you know, for outputs, you know, instead of calling 0, 1, 2, I can just loop it over, right? So I can just process them one by one. And then, again, like, inside the loop, I think I don't have to explain the logic. You probably already have an idea, like, hold the largest and then replace it. And on the Python side of things, when you take an arbitrary amount, so instead of, remember like we put expression one, expression two, and then put it in square bracket, right? So actually, args basically just take a list. So I just don't unpack and pack it again. I just say like, args, just take it, you know, and then Russ will take care of it. So again, like you can have a look yourself if you want to know. So if you have figured it out yourself, congratulations, you are very smart and smarter than me. So good. Right. So next. So we have already, you know, done the numeric calculation. We can take multiple inputs. What about different data types, right? So again, if you are developing for, let's say, the whole team, because this function It's used a lot, but it's not always the same type. For example, it could be an integer, it could be a float, so I want to create something that even I put in an integer or put in a float, it will still work, because the logic could be the same, like finding the larger number or the largest number, you know, it should be the same, right? The logic, like integer or float, you know? So why not just have one and then the whole team can use it? So here in this exercise, we will try to do that, right? How do we handle multiple data types so it will work on different data types? There needs to be a little bit of work here because you can see, okay, so you can see that instead of having a static like here, output type equals to in64, we have to write a function in Rust to determine what output type is, what input type it should be or output type it should be because if, like, okay, unless you want it to always output flows, I think if you, like, you know, if you're working in the integer kind of space, you want to output the integer and it come back, then you'll probably, yeah, you want it to be dynamic so to make it dynamic you need to have a rust function to determine it and then put it as output type function instead of output type so it's not static so it's like dynamic so it's determined so how can you determine it well like for example here if the inputs are integer I wanted to output as integer I don't want to change the type so yeah I would just like okay it's the same as the input field so if the first so yeah this is again again like the first column if you have multiple columns then you know the second doesn't count here but you can also make some decision yourself to like oh if the first one and the second one is integer then it's integer if not then it's you can also do that but here is just a simple example you can improve it yourself so yeah I just cloned the type of the first one so if it's an int 32 it be inch 32 if it's like in 64 we're in 64 blah blah so it would be just the same if it's a string it would probably be a string but then like the whole thing would break down because I am assuming that is is a number anyway so the code now so we have we can dynamically determine the output type what about the actual execution of the logic right so it's a bit complicated It's complicated because in so yeah, you can see that it's a bit complicated. So for each type, we will have a different chunk of code to handle it because the type that we're using in Rust are different, and Rust is not like Python. You can just, you know, don't type it if you don't feel like it. You have to type everything. So that's why we have to handle each different type in a different code block here. But luckily, in Rust, you can use the match syntax. Now we can do it in Python too, but match is used a lot in Rust for this reason. So yeah, again, I'm just matching the outputs, the first column. You can also write some logic that what happens if the second one doesn't match. You can say like, oh, maybe an error will come out or you can make some logic like decision there. So yeah, it works like this. And of course, if you have like, you know, the second input, like, I mean, the input second one is not matching the first one, it will break down here. You'll probably have an error somewhere. I should have written the error somewhere. But yeah, it would just break down here for now. And also, oh, you see that, like, you know, in Rust, it's so complicated because there's a different function for flow and for integer. So that's why it looks a little bit different between the integer and the flow here. It's not just the type is different. Yeah. So if anything that doesn't match this, then it would just stop. And it would just break down. just say like give the error message like it doesn't work you need to match you know so if you pass in a string basically it would just go through this branch and say like you know polar stop something's wrong like polar spell like something's wrong just you know yeah give an error message and stop this is what is going to happen again like you can expand the logic you can also like maybe have other way of handling it this is just an example yeah so if you want to do the extra challenge right now I will give you a few minutes to digest this code and also if you want to the extra challenge can you also convert the largest in that kind of you know powerful way that I can take multiple multiple data types. So what time does this workshop ends? Quarter past, okay we have like 15 minutes. So maybe we have time to look a little bit of the advanced again like this material are prepared is a lot so we won't be able to come I'm not aiming to cover all of them today I would just go as far as we could and then the rest you can do it at home and then you know you can contact me if you have questions or you know whatever so but we have 15 minutes so yeah ask questions again like take a few moments to digest the last exercise and then we'll move on after Amen. Okay, so let's look at string, because string, if you are familiar with Rust, string is weird in Rust. So it's extra weird if you're not just having a string, but like, for example, you have a column of string type, it would be super weird. So let's have a look into this. What we're doing here, we are now basically combining the string together with a space. So that's what we do here. I know it's very silly, but the point is we learn how to work with string. So you will see that, yeah, oh, we need the whole thing here. Why we need to iter and then... Well, it's because, again, like, here S is a column of input, so the column of strings. So here you can see that we have to go through all of the strings inside and then do something there. So yeah, so that's the main difference here. Also, you have to collect it and put it back into a string chunk. So yeah. And then, of course, you have to also do some cloning because otherwise the lifetime is not long enough for that string to survive this code chunk. So yeah, it looks super complicated only because there are a lot of limitations in Rust about working with strings and like all this lifetime thing and borrows. So I think it's worth putting an example here. So if you are working with strings, then you have something you can start with. Yeah, so the clone is there because, you know, the lifetime thing and then also we have to to do the ether because we have to handle them one by one and then, you know, do this logic here. Does it make sense to you? Any questions so far? Take a moment, deep breath. Yeah. Any questions? No? Okay. I know you're tired. I understand. It's at the end. We only have, like, 15 minutes left. But we'll just go over it again. You can look at it at home, try the challenges, and, yeah, so the string is more or less the same. It's, you know, on the Python side of things, it's just, like, but what we are doing here Is to make element wise force, because, yeah, because what happened before is that we would work on them one by one, oh, sorry, I know why it's like super, super weird here, because what we are doing here is to, I wish I could show you, can I show you? So okay, so what it does is that, for example, I have a column of ABCDEFJHRJK, right? So this function, what it will do is that it will make it A, B, C, D, E, F, G, X, Y, J, K into A, A, B, A, B, C, A, B, C, D, E. Do you understand? So or can I, can I, I should run the code, which is like, let's see, like, I should have something here. Let's see. Okay. I know it's super small, but like I would see if I can I can zoom it in afterwards Okay So yeah, so it's something like this. Let's see if I can make it bigger Can I make it bigger? No, I like yeah, but come on plus doesn't work here. But what you can see is like Like, so I have the name John Smith, and then the next is John Smith and Mary Johnson. So I add, so okay, so like I'm not just adding these two together, but I'm also adding the previous one, right? So that's why the code there, that's what the code there is trying to do. So if I close this one and go back there, this is what it's trying to do. It's scanning it from top to bottom. So that's why it needs to do this. So it needs to handle each one by one and then add it there. So it's like... And that's why it's element wise need to be forced. Because now I am... So this Rust function, this Rust expression is processing the whole column before we are doing element wise. If we recall, so it's element wise is true, right? So far, all our things is, this is true. So when you see this, when you see the, where's the rusting? Let me look at one of these. So when you see this, this is actually handle them like element wise, so each element. Now I want a whole column. So that's why, I don't know why we jumped to that, did I skip something? I don't know. Okay. what happened? Okay so yeah so this is processing the whole column all together in this Rust function instead of processing just the cell. Is it clearer now? No. So yeah the inputs here, input zero here, this one is a whole column. The The inputs there, the inputs here, the inputs up there here, what is the equivalent? So the input here, this input, is just one. Yeah. How to switch between the two is actually on the Python side here. Is element wise true or false? Okay. So that's the difference here. That's why it's so complicated, but you may need that for some reason. So yeah, that's what this is doing. Again, like, have another look again. We have 10 minutes, so I will go over this one. So this one is, like, what if I want to take a user input, right? So in the example before, you see that we have the space. So it's like A space B space C. I link it up like that. But what if I want to be like the name one, right, so I want to have a new line, so each name is in a new line, or I want to be a comma or whatever. I want to take a user input. So this exercise is saying, like, how I can take a user input, and now you see that, like, why we need to import this library, because we are trying to use the zero lines here that But yeah, do I have time to explain this? So at the right, again, like, if you're not familiar with Rust, this, in Rust, there are something that, like, the compiler will check where the, okay, so let's talk about in Python analogy. In Python, imagine in Python, when you run the Python code, there's a checker to check whether that method exists, right? So it's kind of like that in Rust. So the compiler will check whether it has that define for it. If it hasn't, then things break down. So this is basically saying, like, oh, just derive that for me so Rust will check that there's a deserialize in this object so it won't break down, basically. Again, like, look at the Rust book if you're not sure. I don't think I have enough time to explain that, so sorry about that. But again, like, yeah, look, there's a Rust thing. So right, so now what we are doing is we will take an extra keyword argument, which we have defined here. So what we are doing is that, like, we would have a user define, no, you would take an input from the user, so the, so it's called separation, okay? So or maybe it's like, let's do it backward, let's see, like, when we are using it here, When we are using it here, we can define the argument, did I write it here? Where did I write it? Maybe I didn't put it here, so sorry about that. So yeah, what that does, I'm so concerned about the time. So what that does is that it will take an extra keyword argument here. So you see in the Python side, I would say like, oh, not just take the argument, which Which is like the columns in your data frame. But you can also have user input to give an extra argument there called SEP. So you can SEP equals to, let's say, a tab. Then it was separated by tab because we have it added here. So the way to add it is that we have to have a struct that have the deserialized derived. Yeah, yeah, is it confusing or questions? Yeah, this is the last thing, so I just want to wrap it up, but I know I do a very bad job explaining it. Five minutes left. Five minutes left, so I want you to ask questions. So basically what this does is you can have an input, for example, I really want to show you how it works, but I need to see my old code, which I kind of forgot, okay, multiple Columns hmm okay where I oh here okay yeah so you can add the separate like Here so you can like this is the I know it's super small okay I spell it out for You so inside here I have the the I use the column string which is the the one I I defined there, which is the polar expression, my custom polar expression, I would take the names, which is like the names column, and I can define a separator, which is a comma and a space. So yeah. So then this is, this work because I have defined it in my extension that I can take A user input, an extra user input there, so it doesn't need to be a column. Because it's silly, right? You'd be like, oh, separation, and then comma, comma, comma, comma, so you can just input it like that. Yeah. I hope it makes sense. And five minutes left, questions? Or I can, like, do a promotion, because on my repo, you already have the link to. If you go to my GitHub, you will see there's other pile-free workshops. If you're not familiar with pile-free, you can also do this workshop yourself. Again, available on my GitHub. You can go there and do the pile-free workshops. And I can give you a sneak peek of what it is. It's a new project. No, open. Sorry, I should not touch the mic. Okay, sorry about that. One sec. Recent project. So, I have, like, PyO3 101 and PyO3 102, I think 101 is probably what you want to do if you want to, like, you know, understand the polar extension a little bit better. So it's a similar format, again, it's on my GitHub, and, yeah, so it's like this. There's lots of exercise, but this will have more explanation of what, like, if you're not familiar with Rust, not familiar with PyO3, this is a very good point to start. I know today may be a little bit more challenging for you because you have no experience for that. But then, like, don't look at that. So yeah, like, yeah, if you, like, if you're not familiar with Pyre 3, then this is a good place to start because then it would just, like, explain a little bit more. It will also explain the Rust kind of thing a little bit more. So if you're not familiar with it, I think this is a very good place to start. And then after that, you go back to your Polar extension workshop and you would have a better understanding of why we are doing it that way, right? Rust is so weird. It's like, yeah, but then you fall in love with it and, you know, think that Rust is not that weird. And I love the quirkiness of Rust. Yeah. If no questions, then I will just let you go and enjoy the lunch. well Chuck, thank you very much