Rusty Python: A Case Study

Speaker 1 [00:03]

So thank you, Max, for the lovely introduction. So as mentioned, my name is Robin. I'm a former mathematician turned software engineer. And ever since I left academia, I've been working in startups, mostly in Berlin, currently working for Tactile. We're a startup from Berlin, about 50 people. I'm principal engineer there. But I just enjoy wearing lots of different hats. That all being said, today we want to talk about Rusty Python. So for everyone that has been to the other talks, you've probably seen that there's been a lot of talks about tools that use Rust and Python together. Polars being an example, we had a couple talks yesterday. So I guess today we want to figure out what is the hype all about. So the talk will have three parts. First part is going to be an introduction into Rust. For those of you that have never seen Rust, maybe a quick show of hands, who here has worked with Rust before? All right. So quite a few people. I'm probably not the the expert in the room, I just have to point that out at this point, but cool. So this will be an introduction for everyone. The second part will be a bit of a toy example where we take a project and then we use Rust to speed it up. And then the third part I'll talk a little bit about a project that we did internally at Tactile and what we learned from doing that. Cool. So starting out with what is Rust? So Rust is a modern compiled system programming language that has been created in 2015. So it's a very new language compared to a lot of other languages that we're using. And it has a similar performance and low-level control as C and C++. There is some theoretical reason of why it could at some point actually be faster than C++. The reasoning being that it feeds more information to downstream compilation steps, LLVM that is. But in practice, it's mostly a tad slower just because less people use it. I've linked the benchmarking game if anyone is interested in the details here. And one thing that it really focuses on is it prioritizes memory safety and threat safety. And I'll show you some cool features of Rust that make this quite elegant to adhere to in comparison to languages such as C++. So let's talk about some things that are awesome about Rust. One thing that I really like, it has modern batteries included tooling, cargo, so there's one canonical way of creating projects, building projects, testing projects, documenting them and publishing them. I wish Python had something like this. I think internally at TechTel we have about three different ways of packaging Python applications, which is not great. We need some uniformity here. One thing that is maybe not as interesting for the Python community, but for people that have worked with C++ before, cross-compilation support is actually really good in Rust. So compiling on a MacBook with an AMD64 chip, for example, for targeting lambdas with maybe a different CPU is actually really straightforward. Other things that are really awesome about Rust is one of it has docs.rs. a page where whenever you publish a package, it hosts the documentation, so that's a really cool thing because it has links between the different packages, it's really easy to see what's the documentation of the package that I'm currently using. This gets us to the first special feature, all of these things are just modern language type features, but here ownership semantics is one of the things that really is different about Rust, so I got an example here, we're defining a string S1, and we're setting it to the string hello, and then we're saying let S2 be equal to S1, so at this point a few things could happen. What actually happens in Rust is that the ownership moves from S1 to S2, so S1 is then actually pointing at nothing, and the compiler will make sure that you don't accidentally use S1 after you've done something like that. So I've included a compiler arrow here. Now this at first might seem really annoying to deal with, but when it comes to not having multiple mutable references to an object, this stuff is really, really handy. And the sibling or the cousin of the ownership semantics is the so-called lifetime tracking. So what happens here is we have a variable x, which we set to five, and then we define a reference, and we refer to this X, and then we try to use the reference after X doesn't exist anymore, so X will go out of scope on the closing curly brace, and again Rust will prevent us from writing code like this, and this basically alleviates a whole class of dangling pointer issues that were with commonplace and C and C++. One thing that I personally find very interesting is it has a very expressive macro system as a compile time expression. If you have no idea what this means, you can think of like Python decorators. They are used mostly in similar circumstances. And here's a couple of examples. One of them being we define a new type message, and the message is an enum. It can either be a request or a response. And then just by putting these couple tags on top there, we define, derive, serialize and deserialize functionality, and we also make it work with this enum. So basically what this does is it writes the Rust code for turning this into JSON and then parsing it back from JSON, but also figuring out which of the two it is based on the type field. And I got one more example because I find this so cool, is the SQLX. SQLX is kind of a new take on the ORM problem, ORM problem being that ORMs are complicated beasts. And basically what this does is you give it a SQL expression and it will, during compile time, figure out if A, the SQL expression makes sense against your database, and B, it will also write all the serialization code to turn it into the object that you wanted. So in this case, into a country. So it's like crazy, crazy stuff that is possible. That being said, this is a Python conference, so let's also talk about some things that are terrible about Rust, and they definitely are. So I mean, it's a compiled language, but not only is it compiled, it's also really slow at times. So there's been a lot of work on this, and hot reloading is getting better, but it's still like you will be waiting a lot for your code to compile. I personally find Rust to be at times quite verbose, especially if you come from Python, so make sure that you'll be spending a lot of time, or I guess lines of code massaging the data to have the right types for the compiler to accept. And then lastly, my experience anyway is I've noticed that whenever I write REST code and things get a little bit more complicated and I have to actually help the compiler with the lifetime parameters, then stuff feels like solving a jigsaw puzzle at times. So it's kind of trying to convince the compiler that this is actually valid code. Cool. So let's talk a little bit about the toolings. So there's three general categories of toolings. The first one is bindings to Python. So these are basically libraries that let you bind Rust code to Python code. PyO3 is by far the most popular one, and it's also the one that we'll be using. It has a lot of cool extra add-ons, one of them, for example, being an async IO add-on that allows you to tie Python async code to Rust async code by passing futures back and forth between the two languages, so it's a really cool project. The second category is setup tools extensions, so once I have a way of linking the actual languages, there's still the question of how do I set up a built environment that then builds the Python code but also builds the Rust code and links it together, and there's multiple projects here. Maturin is the most famous one. They pride themselves as being zero configuration tooling, and that's also my experience. Like, unless you do something really crazy, most likely it's just going to work out of the box. And there's a third category, and I have no idea what to put as a category here. There's an umbrella project called PyOxidizer, and they do a lot of things. They provide tooling for shipping Python scripts as executables. They try to rewrite CPython and Rust. And they also have a way of embedding statically linking Python into Rust applications. If anyone has a good suggestion on what to put here, please tell me. Cool. So this takes us to the second part. Let's speed up some Python code. So I wanted to do something visual so that we have something to see. So basically it was either a Mandelbrot or a ray tracer. I decided to go over the ray tracing part this time. This is just the wrapping code. It uses Pygame to create a window, 800 times 600 pixels. And then what we do is we just kind of walk pixel by pixel and we calculate the color of every pixel. So if you run this code, and we can do it maybe later in the Q&A section, you see that starts all black, and then the plaque secedes, and the actual image comes out. Yeah, I think that's probably not a lot of questions. So, this is the actual logic of what we're doing here. Let's maybe focus on the sketch. We're just sending in a light ray from the screen, and then we see if we hit a ... So, our scene consists of a sphere and a light, and the only thing we do is we send this light ray, and then we see if we intersect the sphere, and if we do, then we calculate the angle towards the light sphere, and based on that, we color the point. The important piece here is that we want to keep the logic in Python because we like the expressiveness that Python has, but there's three classes in here that we use. It's vec, line, and sphere, and these are three Python classes that I've created for this, and the idea is that we want to replace those by Rust and see if this gives us any speedup. So, here are the implementation of vex, sphere, and line. I don't think there's a lot of interesting stuff here. We add some Dunder methods so that we can actually add vectors and multiply them, and maybe there's one interesting piece is the sphere, but it's the intersection between a line and a sphere, but it's really an exercise in quadratic formula, so read it if you're interested, but it's probably not the main focus. So, the number one thing, of course, with profiling is always make sure that we're optimizing the right thing. So, let's first figure out if, you know, what is actually the slow part in our code. And I like to use SnakeVis for this with the built-in C profile module of Python. So if you do this, if you run these commands, you get this flame graph type of graph. The only thing that you have to be really careful about here is that the proportions are not right. So the purple bar, for example, is almost the same time as the blue bar, even though they look like completely different sizes. And if you look now, you see that the ray trace color, that's the gray box up there, takes about 27.3 seconds, which given an overall runtime of 43 seconds is about three quarters of the time spent in this color function. So it's a reasonable target to optimize, I guess. To make this a little bit more concrete, you know, I put some measuring instrumentation on the color function itself, and if you do that and you plot the histogram that derives from it, you get the following image. So these are nanoseconds on the x-axis and distribution on the y-axis. So we see that on average we're spending about 11.8 milliseconds per execution of the color function. Note one thing that, you know, I put the average there and the standard deviation, but I also plotted the normal distribution, the stuff is absolutely not normally distributed as many things are in real life, just as a reminder. So let's optimize this. How do we do this? we are using Maturin. If we just run Maturin new Rust, Rust is the name of the library that we are giving here, we get asked what kind of bindings we want to use, so we just say pi-03. Once this is done, we can see there is a new directory of Rust, and in there we have a pyproject-toml, a lib.rs file, and a cargo-toml. I am assuming everyone is familiar with what a pyproject-toml is. lib.rs is basically where the Rust code will go, And Cargo.toml is basically the equivalent of the PyProject.toml, but on Rust side. So, it will include stuff like the name of your library and the dependencies and licensing information and so on and so forth. If you look into the PyProject.toml, you'll see that there's a build backend maturing specified here. So, this uses a PEP extension for the build backends, I think 5.1.7, if I'm not mistaken. and this allows you to hook into other tools to build your Python project. So what this means is that you can use standard ways of building Python packages. Here I'm using the build module, and it will just work out of the box. It will then call Maturin to compile the Rust code, and you end up with a source distribution and a wheels file if you run this. So let's implement Vector. So, for Vector, we define a new struct that has three values, X, Y, and Z. All of them are F32 types, and then we implement a bunch of methods on it, and they basically are just a copy-paste of what we did on the Python side of things. So, we have dunder add to add two vectors, and dunder sub to take the difference of two. Now, I already mentioned the macro system of Rust, and here it shows how it can shine. So, we have this PyClass macro as well as PyMethods and new, and this is basically for creating all of the binding clue code that will make these objects also work with on the Python side of things. So, if I now just, after adding this code, run mature and develop, and then what happens is it will build the package and they'll install it into a virtual environment. I can then start a Rust REPL and just import Rust and start calling rust.vec123 and print it and everything. Maybe one important part here, on the bottom right, you see I defined an underscore, like a Dunder representation. This is, of course, important because otherwise, you won't get nice printouts of the vectors, but it'll be only like some kind of built-in type. Cool, so this is the implementation of Sphere And, yeah, sphere, just sphere. I don't think there's anything particularly interesting in the code. All of the code, by the way, is on GitHub, if you're interested in it. So, go take a look. And then finally, we have line. Again, not super interesting. But there's one final piece, which is this PyModule object at the end. So, in Python, every module has an initialization function, and this basically creates the initialization function for the module, and here what we do is, we basically add these three objects, or the three classes to our module, which are Vex, Sphere, and Line. Cool, so then the only thing that we have to do is we have to change the import, so rather than from Python import Vex, Sphere, and Line, we do from Rust import Vex, Sphere, and Line, and if we run the whole thing again, we see that basically the performance has changed quite a bit, so instead of taking about 11.8 milliseconds per run of color. It now takes about four milliseconds to run of color. So just to understand these graphs, I probably should have put this on the slide. The left side is the Python variant that we used before, and the right side is Rust, which is pretty impressive, considering that the actual logic is still in Python. So this is a bit of how I think about all these different solutions of speeding up Rust. So, given the time, I think we probably should skip this slide, but TLDR is like, you know, it is still like a very heavyweight thing to do, and there is probably a lot of other ways of optimizing your Python code that you should try first. Cool. So, last case, or last part is the case study. So, here I will have to tell you a very brief introduction into what do we actually do at Tactile. So we're in the decisioning space. So what does that mean? That means that our customers give decisioning logic to us, and we turn this into APIs. So you can think of decisioning logic as a graph, as on the left side. So there's little Python snippets that are connected in a graph. And then we have to turn this into an API that they can call and execute their logic for them. So this part is a bit slow, or some of our customers think it's a tad too slow. So, you know, the question was, can we maybe use Rust to optimize this? So, first, let's talk a little bit about how slow is slow. So, we're talking about, in the best case, if we have a really easy decisioning logic, we're talking about about 200 millisecond response time. And of that 200 milliseconds, about 100 milliseconds is actually spent executing the graph. The rest of it is other Lambda functions, or API gateways, or cache reads, or stuff like that. So we're looking at a ray trace snapshot here of a call that is about 200 milliseconds in total, and we spend about 100 milliseconds executing the graph. So we asked ourselves this question, the spike, can we rewrite the execution logic in Rust? And to give you some data about what does that mean, so the project that we're looking at is about, we have a lambda which is only about 870 lines of Python code, and then we have a package which is about 19K of lines of Python code that we install in there, so we are talking about 20K lines. So it is not a very big project, but it is also not a trivial project. And so the time box that we set, we do not want to spend too much time on this, because you can do this forever. The question is like three work days, one developer, how far can we get, and is this a feasible way of speeding things up? The goal is to set up proof of concept for running our execution in Rust, and then remember that there's these little Python snippets, so the question was, can we run this thing itself in Rust and call out to Python for executing the logic? Given all these constraints, we figured the only thing that makes sense is to try to implement a vertical slice, so we take one type of API call and implement that end-to-end, but we We don't try to actually have full feature completeness because that's just undoable within three work days. This is how the timeline went. The green part is basically us figuring out how to do this, like how to statically link Python into Rust and how to cross compile the whole thing to Lambda. I also think, as far as I can tell, there was no one else that has tried this before. At least there was no traces of it on the internet. If you're interested, I got some hints in the GitHub repository on how to do it. So after the first day, we got a hello world to run. We were able to run Python code within a Rust lambda, so that was a great moment. Then the second day, the purple stuff is where we actually spent time implementing our vertical slice. We unfortunately hit some sec faults at the end of the second day, and it was very hard to debug those. I don't know if anyone's ever tried to debug segfaults in lambdas, but it's really not a lot of fun. And we also never really understood where they came from. We found a way so that they don't appear, but just like fair warning here is there's probably some dangerous lurking in our solution. In any way, at the midpoint of day three, we were able to successfully call the vertical slice. We were very happy about that, and we then spent some more time trying to, like, SNMalloc is a different memory allocator from Microsoft that is supposedly a bit faster. We tried to compile it against that, but it didn't work. So, you know, we unfortunately wasted that time. So, what are the results? First off, there's some stuff that went really well. So, first off, we managed to actually get the thing to work, which is impressive, in my opinion. And the second thing is, it's actually quite a bit faster. So, we're talking about from the 100 milliseconds that we spent before, we're now down to about 40 milliseconds, which is great. But we also have to remember that it's only about half of the time is spent executing the decisioning logic. So, overall, it's only a 30 percent speedup of the response time. And here we get to the truth. Well, we have a 30 percent speedup of the response time, But we also had a significant trouble with non-pure Python dependencies. I think ORJSON didn't work at all, so we had to give up on using it. Pandas we got to work, but every Python dependency that is not pure Python is going to be a major pain for us going forward. It added a lot of complexity to our developer setup, so now instead of just having a single language we have to deal with and set up tooling for, we have to do it for multiple languages. We also had the strong suspicion that while the speedup was impressive, it most likely would have not survived if we expanded the vertical slice in the same order of magnitude. So probably looking at more of a 20% speedup once we widened the slice. And then, as I mentioned, debugging segfaults is not fun, and at this point I realized how lucky we are that we're using Python mostly at Tactile, and yeah. There's also a couple of meta points that I think one should not underestimate. One of them is that we don't have a lot of Rust experience in-house, and the other thing is from talking to other developers, it's really hard and expensive to hire Rust developers. So for now, we will not go ahead. I think we learned a lot. It was a really cool project, and yeah, I wouldn't miss the time. But given all these constraints, it probably doesn't make sense for us to continue down this path. As mentioned, I got some artifacts on GitHub who's interested. Thank you very much.

Speaker 2 [24:09]

Thank you, Robin. That was a really interesting talk. I really liked the part where you actually shared your experience of actually doing this in your organization. I think that gave me some context for maybe what I'm going to try and push to let my boss let me do for three days, maybe. But we've got quite a few questions for you, so we've only got a few minutes, so I'll ask just the top-rated ones. Okay. But look, we'll do our best here to get through as many, but if not, Robin will be around for the conference, so please do come and find them. So first of all, question for you. how is Rusty Python, how is that speed up comparable to Cython or Python with Julia, for example?

Speaker 1 [24:44]

Yeah, it's a very interesting question. I mean, I I don't have that much experience with Python for Julia I actually saw sat through the tutorial yesterday. That's my only exposure to it and So my my strong suspicion is that both? Julia and scythe and are probably have a lower performance cap So like if you really go down to the nitty-gritty details Most likely you can do more stuff with rust and C++ than you can do with Julia and scythe But, of course, you should still try these other options because they are much more lightweight and much easier to implement than what I've shown today.

Speaker 2 [25:17]

Cool. Thank you. Thank you for that. Next question comes from Anonymous, who's asking the question, what tool do you use if you need to debug Rust code? So the tools like IPDB work when you're mixing both Python and Rust?

Speaker 1 [25:30]

Yeah, I mean, honestly, most of my experience has not relied on debugging tools per se. I'm not a debugging type of developer. My, yeah. No, in a sense, there's some people that really enjoy using debuggers. I'm not. I find it more like reasoning through code and putting a couple of good print statements is always easier for me then so unfortunately i can't help you with this that's that's i mean

Speaker 2 [26:06]

I would be lying if I said I'd never just written here and then moved that line down to find where my problem was. Fair enough. It's very mathematical as an approach as well. You prove that part works and you move on. I like that. Just a couple of minutes left, so a few more questions. Is there a setting in which you prefer JIT tools like Number over Rust for speeding up Python?

Speaker 1 [26:25]

I mean, I think you should try these tools first, because they are out there, they're well supported, there's lots of documentation, and most likely it's going to be much, much faster to get it to work, and if you hit any corner cases, there's actually people to talk to rather than just no one, like we hit here. That being said, I'm definitely not a data engineer, so I don't have too much experience with the restriction of number, for example, so I know for pandas and stuff, it gets tricky if it's not a flat data frame type of number crunching job that you have, so I would say try to use these tools if you can, and if not, then you have to resort to something like this.

Speaker 2 [27:07]

Awesome, thank you. So another question we've got here, another one from anonymous, is, when is learning Rust, which they identify as being very distinct from Python, you're losing certain things like you're losing introspection and you have extra build steps, when is that worth it, would you say? When is it worth it to actually consider using Rust?

Speaker 1 [27:24]

I think it's worth to learn it full stop, just because it's a very interesting language that brings new stuff to the table. That being said, the question I think should be kind of like, when is it worth using it? And then it comes kind of back to the trade-offs we're talking to, where I think most of the stuff, like Python is just so much more expressive and so much more quick to write, I would default to using Python. Sure.

Speaker 2 [27:47]

We've just got time maybe for one more question. Unfortunately, there's some great ones coming in that I'm enjoying reading, and I think maybe, yeah, if we find Robin at some point afterwards, I'm sure you can stick around for a little bit, maybe just outside, but just one for now, just before we go. Why was Rust the choice for the speedup MVP compared to something like pure Scython?

Speaker 1 [28:06]

I don't know, just personal preference.

Speaker 2 [28:06]

Why did you? OK, I guess we've got time for one more, in that case. Quick answer. Thank you, Mustafa, for that question. Last one. When is using Rust with Python a better solution than using NumPy or something similar?

Speaker 1 [28:19]

Yeah, I think kind of goes back to the thing that I said before like you know if you can use numpy use numpy Really because it's you know it's documented. It's you know there's a lot of other people out there that use it, but if you can Before using C++ to speed it up think about using rust

Speaker 2 [28:37]

Robin, thank you so much. This is great. Please give him a big round of applause.