Rediscovering single-node processing: When does it make sense to move from Spark to Polars?

Speaker 1 [00:00]

for this conference in general. I hope you enjoyed it all around. So, I want to introduce you, like, a nice speaker. He's Jonas, he's from InnoVEX, he's a data engineer and also a coder by heart. And he wants to present us his talk about rediscovering single-node processing, when does it, like, with the question, when does it make sense to move from Spark to Polar? It's like an important question, also like questions I'm also quite interested to be answered. So can we get a signal to the screen?

Speaker 2 [00:33]

to the screen. Yeah, I blacked it out. Super. Okay. So, yeah, welcome. Happy to see so many people here in the last talk of the conference. So, yeah, I hope to make a good impression for the conference itself. So, let's imagine a situation. You just wrote some amazing Spark code. It would be really cool. You want to run it, test it out, but the cluster is down because, yeah, of course, it's not running. So, you'd click on start. You go make yourself a coffee, chat with your colleagues, maybe with your cat if you're in home office, come back to your computer and completely forgot what you were doing. So maybe you know this situation, I do, and one of these days I thought, yeah, maybe there's a way not to wait, that I don't have to wait for the Spark cluster. Maybe there's something new, something different. And I discovered Polars, and the idea that everything runs on one machine, not on a cluster, and does not need to run all the time. Quick aside, quick advertising block, why can I do this talk? Because I'm at Innovex, we had like six talks at this conference, and we are hiring, so So if you also want to make conference talks here and meet amazing colleagues, talk to us. But now back to topic. Let's talk a bit about an introduction about Spark and Polars, a quick comparison of the two. So dry facts, Spark has been released in 2012. It's maintained by the Apache Software Foundation. It is a distributed query engine written in Scala, so it has the JVM behind it. It is ideal for horizontal scaling, so if your job does not run, just add more nodes to the cluster. It runs in a lazy execution model, so it collects all the instructions, and only in the end when you give it a command to write it out, it optimizes the collected instructions and then runs them. And yeah, it is a server-client model. So it has a server that's always running or not, and a client that connects to it and submits jobs. Polars is rather new. It's been released in 2020. It's maintained by Polars BV, which is a company. It is single-node. It's written in Rust. And yeah, it's ideal for vertical scaling. So it's designed to make use of all the memory and all the CPU costs that your machine has, but no other machines. It supports both an eager and a lazy API, so you can choose if you want your commands to be run immediately or, again, only at the end. And, yeah, it is part of your code. It starts and ends with your script. A few notes about both. Both are released under a free license, so Spark under the Apache license, Polar's under MIT. They both provide a data frame interface, and they both have a columnar memory layout to enable aggregations on big data. So let's compare them. For this talk, I used the NYC taxi data set, which is just a good way to get a few hundred million rows. I want to make a few comparisons execution speed, runtime cost, development experience, analysis debugging, deployment integration and finally the big question when and how would you switch from Spark to Polars so let's start with the execution speed, I would like to note I will not do any benchmarks, I will not show any numbers because I was not sure if I would write both perfect Spark and Polars code to make a comparison sensible. And even then, if I did, then why would I expect that everybody who uses them writes perfect code in both? So I think those numbers are nice for websites, but I want to go over my subjective experience. And that was particularly in parallel processing, Spark is still much faster. So if you just have a job that does maybe row-wise processing Spark just could distribute the work across its nodes and have an advantage there. For everything else, Polars is faster. So especially if you have joins, join your data within itself, it felt a lot faster or it was a lot faster. But I have to say the performance was sometimes a bit inconsistent. It is rather young. I want to highlight one bit of code, the SyncDelta function, which is the lazy evaluated function, which writes your data frame into Delta. In my experience, this was significantly slower than using a collect, so execute the lazy data frame, and then write with the eager method. This was slower. I think it's mostly because the SyncDelta was new. It was introduced to the code, or it was released somewhere between when I wrote the proposal and when I started working on the talk. So at least we have it now. It's cool. Spark, on the other hand, you probably have seen code something like this. You do some kind of repartition and your code is faster, but you don't know why it is faster with this exact number. So Spark is a bit difficult here. Then, of course, about the costs, yeah, the cluster startup costs, especially for small tasks, when the cluster startup is a significant part of the runtime, that makes some of the costs. And at least in my experience, the compute size in total is not much smaller for Polar. So I found, at least for my task, so I did some feature engineering on the NYC data set, the compute size in the end was roughly the same. So I just used one machine with, let's say, 64 gigabytes of memory instead of eight worker nodes with eight gigabytes each. Let's move over to the development experience. so very subjective. Polars is fun. It feels very modern, it has a hierarchical API, so many similar commands like string commands or datetime commands are grouped into more or less modules that makes it really easy to read especially. I don't know, there's a law of computing, code is read 10 times as much as it's written or so, so yeah reading experience is important. On the other hand, I have to say, and as the Polar's community says, come for the speed, stay for the API. I tend to agree. So the speed is really nice, but when being faster at writing code is also a significant factor. On the other hand, Spark, I mean, it's established. Everybody knows Spark. No matter what problem you have, somebody has had it before and written a blog post about it. So you can really well find solutions for almost all of your problems and especially in this year, this also means a lot of LLM training data. So wipe coding with Spark is easier because there's so much more training data that the LLMs can use. Let's compare two bits of code. So this is a feature processing written in Spark. I wanted to get the taxi trips aggregated by the hour, so I did a window function by one hour. Then I added some aggregations like the trip count, the trip distance, the amount of passengers, the mean trip duration. In the end, I needed to tidy up a bit, so reintroduced the pickup date time and finally dropped an intermediate variable. In Polars, the code would look like this. Polars has this group by dynamic for I think mostly for time-based aggregations. I say group this by every hour and don't mix up data from different vendors or different pickup or drop-off locations. Then there's the aggregations like I use the length of the group, give it an alias. I sum up the trip distance as well as the total amount. This is particular for polars, so I give multiple column names into the sum function and it just sums up the columns each separately and just keeps their name. So you see in this bug example, after the sum I needed to print an alias so that the column name stays the same. And finally, I subtract the date times, calculate the mean, cast it to an integer in microseconds, and convert it to seconds. So now the code has run, or is running. You want to know how fast is it, and if it did not work, what went wrong. I found the error messages, unsurprisingly less noisy in Polars, mostly because Spark often has this long stack of JVM communication in its stack trace, so it's sometimes hard to find the actual error. of a DataFrame explain, so for the lazy evaluation in Polars. And this, I don't have to say much about it. If you've seen one and understand Spark DataFrame explain, you can read to the Polars explain and understand it just as well. They're nice. But now let's go to something that big advantage of Spark, in my opinion, the Spark UI. I would say, yeah, it's as complex and great as Spark itself. So you have your worker nodes, you have the submitted jobs, you have your stages. Those three are not relevant for ProLogs, of course. But it has this post-hoc performance analysis. So afterwards, especially if you persist your Spark logs, you can see how long your jobs took, how long, what was slow. And ProLogs just feels quite limited in comparison. So, Spark SQL query reviews might look like this. Every blue box corresponds to one Spark operation, or every light blue. Okay, the contrast is a bit bad. Every small box is a one Spark command, and every big blue box is one stage. And now you can go into this, you see for every command how long it took, how much data it read, how much data was shuffled across the Executor nodes, how much the cache was used, and so on. And you can go really deep into this and optimize your code really in a very detailed way. Let's contrast this with Polars. So again, a bit of Polars code as an example. I wanted to introduce lag features, so by one hour, one day, one week, and one year. So I wrote a for loop, joined the data to itself just with an offset. There you can see the .dt offset by as a hint when reading, yeah, this is a datetime function. I want to have an offset on the datetime column. And add a suffix to the data. And then I run the show graph command. So you don't need to remember this code particularly, just that I had this loop. And in the graph it looks like this. There's a bit of preparation and then there are those four joins up above. You don't need to read this. The basic thing is it joins by itself and you can see the operations but not how long they took. For this, Spark has a profile function which evaluates the lazy data frame, returns the result and the runtime, and then you get something like this. So there's an optimization step, there's a width column for the lag column, then there's the join, and then there's returning the data frame or writing the data frame. The problem is now there's one join, but I I had a for loop with four joins. So which one of these four joins was the slow one? I don't know. And unfortunately, I cannot read it from this graph. So this is still something to be improved until then. I'm a bit confused by this. It's not that as helpful as it could be. But now let's say the code is fast. You are happy with how it runs. So let's look at the integration. Spark, yeah, I mean again, Spark has been used by everyone, it has integrations for pretty much everything there are multiple data warehousing platforms that have their own implementation of Spark or they use Spark, so basically with Spark you can easily connect pretty much everything, so catalog servers, other databases various file formats and of course Snowpark or Databricks Spark as proprietary implementations of the API. Polars, on the other hand, feels a bit limited. So it can read from the Unity catalog. It cannot yet write into a table registered in a Unity catalog. It can read and write Delta. I think I heard this morning or yesterday in a talk that Iceberg support is ongoing. It's not yet finished, but yeah, it's just a bit less. And then, yeah, of course, for Spark, you need to have your compute cluster. You need to run the cluster all the time. Polars, you just probably have it in a container as part of your code. So you have your container image that just has Polars installed, and your code just uses it. And afterwards, it's shut down again. Then Spark, if you have a data warehouse solution, then probably you use the fully integrated scheduler in there. But if not, maybe Airflow or some other kind of scheduler. Polars does not yet have data warehouse integration. So you just need to use your own scheduler, bring your own scheduler, which is not bad. it's just how it is. And with this comparison of the two, let's go into the big question, why, when, and how should I now use Polars over Spark? Let's start with the when and why. So I think if you have worked with Polars before, you probably heard it, if your data per processing step roughly fits into the memory of your machine. So if it's in the range of gigabytes, not terabytes for most applications, then Polars can be faster. Then in my experience, at least if the processing takes place in a serialized or random way, so there's interdependence between the rows in your data, then you don't need to shuffle between the executor nodes, which makes Spark slow. So if your data interdependencies look like the one on the left, then probably you're better off with Polars. If there are no interdependencies, you can just run all the rows in parallel through your processor, then Spark might still be faster. And as we just had with the integration, if a strong integration is not your priority right now. So if you, for example, need your data catalog updated all the time with the current result of your jobs, then with Polar, it is a lot more work. With Spark, you just load the corresponding plug-in for data hub or whatever, and Spark takes care of it for you. then yeah consider the switching cost so in the age of llms it's not that bad but still consider your development time and the cost of running two resources in parallel so i think that's xkcd 1205 read it later on how much time you shave off how often do you run the task How much time do you want to spend on it? Multiply those numbers with your hourly wage and the cost of running them, and you get an estimation of whether it's worth it to switch. And at least in my experience, yeah, the cost savings with Polars rather come from a faster execution time and not that much from cheaper instances. So, now, if you want to start with Polars after all this, what should you do? First of all, I would say quick and dirty projects work really well with Polars. So, especially if you use Pandas, try Polars instead. I heard from InnoVEX colleagues that they just have stopped using Pandas entirely. They have switched completely to Polars. And if you have Spark code that uses UDFs, like this one, so the old style that uses pandas, just switch it over. This polar bear tells you, yeah, use an arrow UDF. There you have a pi arrow array, and polars can just load this with a zero copy load. So you don't need to pass anything, you just put in the pi arrow. PyArrow object. And if you just want to try something, if you want to try out Polars, maybe even with a production size job, why not run it on a single node cluster? So if you have your Spark cluster, for example, in Databricks, then create a single node cluster, give it a lot of memory give it many processing cores add a pip install or whatever, install polars to the init script and use polars there, why not? I mean there you still have the cluster start time but at least you can run polars afterwards and yeah, why not try an LLM of your choice to translate the code between the two. So I would say Polars to Spark might work better, again, because there's so much more training data for Spark available in the internet than compared to Polars. But I also tried some larger feature engineering or data transformation jobs, put it into an LLM, tell me, hey, this is PySpark code, translate it to Polars for me, and the code worked mostly. So, of course, sometimes there are bugs still, but all in all, it worked quite well. So, I come to my conclusion. Spark is still the major framework. It is well established. It has a lot of documentation and supporting material. If you have your data warehouse platform, then Spark is there, Spark works, and you can just use it. Polos, on the other hand, it's new, it's fun, it's really fast, but in my experience, what I found is the integration in business environments is unfortunately still lacking. But, yeah, both are in active development. I mean, Spark has been in development for much longer. It brings cool updates, performance upgrades, new features. In Polars, there are still a lot of new things coming. For example, as I said, the Iceberg support that might come soon. So, yeah, continue watching them. Should you change? Yeah, it depends. I hope that I have given you a bit of my opinion, and from there you can form your own opinion whether you should change it. At least, no matter if you switch production, if you decide to switch production right now, try it out. And form your own opinion, replace Spark, replace, not maybe replace Spark, definitely replace Pandas, in my opinion. Polars, yeah, it's fun. Try it out. And with that, thank you.

Speaker 1 [22:57]

Thank you. Thank you for your great talk, Jonas. So we have lots of questions. Please upload, audience, please upload them so that we can focus on the other questions. So the first question has two sub-questions. The first sub-question is, from small data volumes, is Polar better than Pandas? Or will it make no difference? I think you answered it now with the closing remarks.

Speaker 2 [23:29]

Yes, so I think, I mean, Polas is definitely faster than Pandas. If your data is small enough that it makes no difference, then in my opinion, at least Polas has a nicer API. So it, I mean, everybody who's worked with Pandas knows this setting with copy warning, I think, which shows you some important information that you don't understand and still continue to use it as before. with Polars you don't need to use .log or .ilog and know the difference between the two so it's just easier to write it's easier to work with and both for exploratory work you can just use the Eager API in Polars and run your commands on your data frame and see the result immediately and if you decide yeah, okay, this entire process looks good I can go towards production you can very easily just switch over from the Eager API to the Lazy API and just have it execute at the end, have the optimizer optimize your execution graph. And I think pretty much all the commands that work on the Eager API also exist in the Lazy API. Sometimes they are a bit different. For example, you would run scan CSV instead of load CSV or read CSV. So you would use different methods for reading and writing, but all the processing commands are pretty much the same and they work the same. So I think there's ongoing effort to really just get rid of the Eager API and just replace every Eager command with a lazy API command and then a .collect. Of course, they don't want to maintain two codebases in parallel.

Speaker 1 [25:27]

I think you also quite answer, thanks. So the next question is for Matthias, and this will be a great question. Do you consider it an anti-pattern to run polars in Databricks? Do you consider it? Maybe.

Speaker 2 [25:43]

I mean, if it works, it works. So, of course, it could be better, but I don't know if your organization is completely in Databricks. You have all your tables, you have all your data in there. Then, of course, you don't go and spin up a Kubernetes cluster somewhere else, set up the integration there and all this just so that you can run Polars outside of Databricks. So depending on the situation, it might make sense. But of course, if you can avoid the cluster startup, then it's probably better.

Speaker 1 [26:22]

Makes sense completely. So the next question is a small question with a little bit more context then. So did you try testing this with Polar's or Pike's Spark?

Speaker 2 [26:33]

Um, like,

Speaker 1 [26:33]

like the context found that unit testing conversions was much easier with polars versus spark and deciding to use DBX spark versus normal spark

Speaker 2 [26:45]

So, currently I don't know, for the second part, I don't know particular differences between the Databricks implementation of Spark and the open source, but there are differences. For my tests I used the Databricks implementation. For the first part of the question, yeah, running unit tests with Polars is much easier because you don't have to either set up a spark cluster or mock it you just run polars with maybe separate test data so yeah it is much easier to to develop unit testable code with polars

Speaker 1 [27:26]

I see, the next question, what about Ray or Dask for horizontal scaling, so what's about these tools?

Speaker 2 [27:37]

I have no experience with them, sorry. I think they are probably also good, but yeah, I wanted to talk about single node processing, not horizontal scaling.

Speaker 1 [27:50]

There are so many libraries and tools to use. They are like always, you can only have like limited view. So on the next question, did you have a chance to try Polar's cloud, distributed Polar's question mark?

Speaker 2 [28:03]

I have not. And okay, I wanted to say it during the talk, but I decided to leave it out of here. So for those who don't know, Polars Cloud is an effort by Polars to provide this really nice Polars API, but also with cluster compute. So I think their expectation is to develop this as a paid offer. So they want to host the Polars Cloud yourself, evolve into a data warehouse similar to Databricks Snowflake or so, and finance the development of open source Polars, single node Polars with it. But yeah, this would have been too much for this talk, so I left it out.

Speaker 1 [28:49]

Super. Next question. Does all of this also apply to pipeline and snowflakes? Should I use polars instead of snowflakes? And should I use polars?

Speaker 2 [29:02]

Yeah. I mean, Snowflake is a bit different because the data is not saved in a location that you can access from outside the environment.

Speaker 1 [29:02]

Yeah.

Speaker 2 [29:17]

So, I think it's harder to use Polars in there. And as far as I know, but I might be wrong, there's not yet a method to directly convert a Snowpark data frame into a Polars data frame. So, as long as this is not available and you might still need to convert a Snowflake data frame to Pandas and from Pandas to Polars, I think this will probably negate any speed advantage that you might get.

Speaker 1 [29:54]

Okay, super. I think we have time for one or maybe two questions.