Free T(h)r(e)ading: A Trading Systems Journey Beyond the GIL

Speaker 1 [00:00]

Arigatou gozaimasu!

Speaker 2 [00:00]

talks.pycon.de for questions, so you can, like, during the talk, submit a question for Tim, and then in the end, I will go through them, and yes, if there's not enough time for all of them, please catch up with Tim afterwards. So, it's 20 past, welcome. And I'm very happy to introduce Tim Kreitner, and Tim is going to talk today for us about free threading, a trading system's journey beyond the GIL. Please help me welcome Tim with a warm welcome.

Speaker 1 [00:48]

So, yeah, hi. Thank you all for joining me, so many of you that the room is full. It's my first talk, so please give some leniency for a nod. So, yeah, as already said, I think the most time of doing the proposal I spend on the title to come up with that wordplay. So, can a trading algo actually benefit from true parallelism? Especially, then, using Python. Maybe a bit about me. Hi, I'm Tim. I'm a senior software engineer at Vattenfall. I build, in my day job, algorithmic trading infrastructure. You can find the stuff that I do besides that on my blog, pycrastinate.com. And it's been very dormant for the last years because I had kids and not much time, but I will do my best to keep it more updated in the future. You can also find the links to slides and more like a written up article about this on that page. So without further ado, there is a quick disclaimer, the views and opinions that I express in this presentation are mine and not those of my employer, and also because it relates at least loosely to trading, this is not financial advice. still deal with your own money. So begin with the question. With 3.13, the game was kind of gone, and we had this experimental feature, and with 3.14, we're actually allowed to use it in production as well. And then the question kind of comes up, does that actually help? And this is how I tried to find that out. So today we're first going to talk about what What was the actual research question and motivation behind it? What did I come up with as a benchmark set-up? How did I migrate from async to free-threaded Python? We'll talk a bit about what the numbers say, what the results are, and then I should, well, I tried to give you some practical implementations, and if you should continue to use it, or if you should make the switch, or, yeah. So part one. Why a training system? In my opinion, it's an ideal experimental subject. It's latency sensitive. You have multiple concurrent data streams, and it's both IO and CPU bound work. So we do kind of both. And you have very clear and quantifiable metrics, because at the end of the day, you either keep up or you don't. And then last but not least, I also know about it a bit. So that's what made it kind of good for me. So what is the real problem? When you have the global interpreter lock enabled, then the tasks really process one by one, right? It's either one or the other that can really do work. And when you come to free threaded Python, then both of them can run at the same time because they're free to do so, right? What comes up is how much does this actually help? So as with any good research question, you come up with some initial hypothesis. And that is that I would say threaded should beat async for CPU bound work. I think we can agree on that. But latency should improve significantly. And that I guess that lock contention hurts at a high tick rate. So whenever you need to process a lot and the lock needs to be released and kept a lot of times, then that should probably be not so beneficial for the threaded Python. And I assume there will be a crossover point at a certain stage. So let's come up with the experimental design. When you have a trading algo, on the one hand side you have a market data feed, and on the other side you have some order management, order routing. And the market data feed comes in into the algo, you start up at the day with a snapshot of what the market currently looks like, and then you receive updates throughout the day, all the time. You try to make a decision on that. Say I want to trade or I don't want to trade. And if you want to trade, then you convince that or you send that order via an order routing service or to the exchange. If you want more about trading algos, you can go back to last year and listen to the talk from Eugen Geist. It's a good friend and he did a very good talk on what actually goes into a trading system overall. And, yeah, so have a look at that. And since it was already last year and you were all here last year, I can assume this is known knowledge. Today, to make it easier to abstract this away, I want to look at one part of the pipeline architecture. So we have this market data server and we have a client pipeline. And they communicate via a 0MQ pop-up socket. Why 0MQ and not Kafka as Eugen did in his talk? Because 0MQ library in Python is free-threaded, accepted. So that's pretty much why I went for this. And it's always in three stages. So you have a receiver where stuff comes in from the socket. Then you put that onto a queue. You process that. So you process the market update. And then And you have a signal evaluator that tries to make sense and come up with a decision. And the evaluator really only needs the latest state. So when the evaluation takes longer than the market data processor, you don't want it to use stale data and run through everything and do bad decisions. You want it to have the latest data. And to come up with, like, some kind of a benchmark for this, I said, okay, the market data processor, I'm not sure how long that's going to take, and the signal evaluator, I'm I'm also not sure how long that's going to take, so I kind of need to make that configurable and do some random work to simulate what I would actually be doing. Important in here is that market data processing is strictly sequential. So when you receive Delta updates, you need to process one before two and two before three. Otherwise you end up with a bad order book and you cannot really do a correct order keeping. So out of that come two implementations, so one async and one threaded, and you might ask why not use multiprocessing, because, I mean, we save up a lot of data, and then we need to process that really fast, and then continue about doing this, but again, we need to process it sequentially, so it doesn't matter how many cores you have, you still need to process one before two, and then it doesn't matter really if you need it. You can just do it on one core. So for the async part, you have three coroutines that symbolize these three stages. And the book ownership is really shared. There is no lock needed because you have the global interpreter lock. And the state transfer is done via like a latest state slot. The sync cost is virtually zero. On the other side, the book ownership is passed. So the processor owns it exclusively. And you only pass a final snapshot that you produce out of that book whenever you do an update so that the evaluator can take it. And the thing is that you need to have a condition to notify or wait for a real update for the evaluator to do some work. So how would you do workload simulation? Initially I thought it's easy, I'll do a sleep. And then I'll just wait. And then I looked at the benchmark and I was like, this is not really right. Because Python is actually smarter than me, and when it sleeps, it releases the global interpreter lock so that others can actually do some work. So I needed to come up with a workload runner that does some random work that keeps the GIL enabled, or that keeps the lock for the thread enabled, so that I actually simulate something where I really need to do some work. I came up with a quick sweep matrix, so some processing times to update the book, and then some evaluation times that it takes me to come up with a trading decision. And then going forward, I actually switched the control variables to say, okay, we have about 200 ticks per second, because that is, when you look at the processing time, 5 milliseconds. Five milliseconds times 200 times, that's exactly one second. So when you look at the message rate, we should exactly be able to process this, given that that we only do this. I did 60 seconds run just because I wanted to also get done because I did this at night. And we have the same zero MQ endpoint for both clients. The workload gets calibrated for every benchmark, so when you run on a different CPU, you might get different results, right? And for the queue, I said, okay, not more than 10,000 updates need to be pulled up because otherwise you're not keeping up. And then per tick, I recorded than end-to-end latency. I split up in receive to process and process to evaluate so I can see where I spend more time. How many evaluations I actually skip, so when the evaluator runs longer than the actual book update takes, you naturally skip updates and only evaluate every third update or something like this. And then, in the end, I looked at the throughput, So how many messages can I actually update? So how did the migration work? Initially the async architecture I defined as my baseline because that's what we had before. We have a process runner that blocks the event loop. And then I do a greedy drain. So I don't release the lock until I'm done with all the updates. And that kind of leads me to this blocking problem where in an ideal way when we have fast, only IO-bound work, then we just hand it around and every process is small enough to be done immediately before the other messages are returned from the IO socket. But in reality, when we have CPU-bound work, the processor keeps the lock for longer, and then stuff starts to pull up. So, coming to the threading. We get the queue. This is sync now, because we're no longer in an async world, and I update the book, and then I notify that I have updated the book. That's pretty much all there is to it. For thread safety, the state slot somehow needs to be synchronized now. The other stuff is a queue, and the order book is owned by the order manager, or the processor, really. The 0MQ sockets are thread safe, but they're kept in a single thread anyway, so it doesn't really matter, again, if they're able to deal with a multi-threaded environment. And to be honest, the refactor was kind of mechanical. You switch one for the other, and then you see that it works and that you don't leak anywhere or something else. Coming to the results. So, Threaded evaluates more tick updates, but to be honest, at half a millisecond processing time or one millisecond processing time or even two milliseconds, it's not that far behind, to be honest. Obviously, starting to go for the evaluation delay, then yes, you're going to process or you're going to evaluate less messages, right? But only when you switch to exactly that boundary condition, as I said before, where you have five milliseconds of processing time, then you really start to notice when async is falling behind. because it just cannot keep up with the amount of messages that you need to process. When you look at the end-to-end latency, then you see a similar picture. Async keeps up as long as you're able to process the first stage. And as soon as that is gone, then pretty much all hats are off. And when you look at why that happens, then you see that on the top row for the async part the actual evaluation delay that stays the same but as soon as the messages are coming too quick for it to process when it all needs to happen in that time slot the actual delay from processing message updates is starting to block the evaluation and more and more the longer you take to actually evaluate the backup is well if I run it for longer then we will start to see droppings. And in contrast, in the threaded part, you see that it's pretty much uniform, right? You see that the best part to see it is in the five millisecond processing time, because you see a constant block at the bottom for how long it takes me to process a single update. And then you see a perfect distribution, as you see above, for how long it takes me to evaluate something. And when we then look at the queue, then exactly what we assumed before happens. The threaded version is perfectly able to keep up any of the rounding errors I would put up to some of the processing not being perfectly timed to that five milliseconds or one millisecond or whatever. But the async client really starts to build up a queue, maybe small or slow as long as the processing delay is just half a millisecond. But if you need long to evaluate, then also that it just doesn't mathematically work out. So when we come to practical implementations on this, I'm much faster than I thought I would be. I would say stay with async and await. As long as you do not need to justify making the switch to free threading, stay with what you know, and only switch when CPU-bound work is really breaking the physical limitations of how many events you need to process. Or you really need those pipelines to overlap. Then that is the other case, I would say, to switch. The benchmark that I did is also kind of a guide for where you want to aim for. The code for the benchmark will be all in GitHub, so you can run it for yourself and try it out. But it's also kind of the thing that you can aim for and say, okay, I know how many messages I'm going to play, or I'm going to receive, and I know how much time I'm allowed to take for this and for the evaluation, and then I can just aim for my code to be this fast. So you can also flip it around the other way, and like the guys from DeepML did, that you can just say, okay, I maybe need to rewrite a slow part into Rust and make it faster and stay on the asyncio version. What helped me to migrate? Well, one thing that really helps is when you already have an understanding of how threads work, so maybe from another programming language or from reading a book, then the switch becomes quite mechanical. You have an async IO queue, and you switch that to the regular queue. And you have an async IO event, and you switch to a threading event. And the await calls just become blocking calls, and coroutines become threads. Where you need to spend some time is to think about where do I actually have shared mutable state which I need to lock or which I need to not lock, or how can I come to a lock-free implementation of the same problem? Maybe going a bit further, what does this mean for Python's future? For CPU bound work, in like a streaming pipeline, this is extremely efficient. you can make your pipeline much more resilient because you're going away from needing to add the processing and evaluation time together and you really come out with a max of that or that. And the Python ecosystem is adapting. As we just heard a talk like two hours ago, I think, about how the ecosystem is evolving and Python FreeThreaded is, if not next version, then maybe the version after that is going to become the default and more and more libraries will support it. I made heavy use of that free-threaded Python website where they track the adoptance of libraries and say if it works in testing, if it works in CICD, and people have reported issues or not. And the amount of libraries that support it is growing every week. And in my opinion, not seeing threading as a real alternative for developing Python, then you just cut yourself short of a very important paradigm that you can use to make your code better or actually work at all. So at the end, maybe come back to the hypothesis. Threads beat async for CPU-bound work. I think, yeah, we confirmed that. The latency improved significantly. Yes and no, but when we hit the boundary, it improved significantly. The lock-on tension hurts at high tick rates. I actually never had that because at the end, the implementation that I came up with was lock-free, so I never really had that problem. And there's a crossover point. And yes and no, as I said before, you can also use that benchmark and reverse it the other way around and say, I just really want to stay with async because that's what I know and I need to make my code faster. So there comes a point, but you can also circumvent it without doing it. And, with that, I'm done. Much faster than I thought. I hope I didn't talk too fast for you guys. And I'm open for questions.

Speaker 2 [19:42]

All right, thank you so much, Tim.

Speaker 1 [19:45]

Also, maybe the first question, because I heard that quite often during the conference already, why not use another programming language entirely? Maybe two things. First, this is PyCon, and the second part is you have people that develop, like, code that are data scientists or analysts or that want to productionalise very quickly, and know Python, and then it just makes sense to speak their language.

Speaker 2 [20:18]

Thank you. And we got also the first question from the audience and the question is, you said there is no need to migrate from async to threading, but what would your suggestion be for a greenfield project?

Speaker 1 [20:37]

It depends on what you want to do with it. So if you really have this model of needing to process something sequentially and having like, you're not sure how many messages are going to hit you in the initial wave and you do something like, I don't know, have an IoT network and a lot of updates come in and you want to do something about that, then I would say just start off with Reddit right away. But if you just want to come up with a quick prototype and want to see if it works, and you haven't really worked much with threads, then maybe stick to what you know and stick with your guns. Or keyboards at that.

Speaker 2 [21:21]

Awesome. Next one. How many threads did you use? What speedup overhead did you observe?

Speaker 1 [21:29]

As I said, for me I used three threads because I have three coroutines, so more threads didn't really make much sense. But obviously the whole algo, this is really just a small part, right? This is updating the market data and evaluating something, but you would also need to implement the other side, which then again are threads that you would need to come up with. And the speedup you can see when we go back. If the evaluation delay is the delay, but when we look at the, when we take longer to process every update, then the overall delay improvement from async to threaded on the tail end is something like 150 percent. So it's the difference between being able to keep up and not doing the work. So it's, yeah, if you would try to do the same thing and do it in async, then you would need to become faster because otherwise, or think of, I don't know, simplifying your messages and processing less.

Speaker 2 [22:50]

Thank you. Oh, so I'm trying to keep up with the order. Please bear with me I'm doing this for the first time this time Do you think the future of Python is three threaded free threaded?

Speaker 1 [23:06]

Well, from what I heard in the last talk, I would dare to say yes. But I think a lot of people are just going to stick with the global interpreter lock because they don't need their analysis to be fast or they don't use this to process this much data in this kind of way. And then they don't want to worry about shared memory or any problems that come with that. then maybe you'd still like to have the global interpreter log. So I think it will be here as a tool, but it's not the silver bullet that will solve all your problems.

Speaker 2 [23:46]

Thank you. Next one. Could there not be use cases where a combination of a-thing and threading could make sense?

Speaker 1 [23:57]

Yeah, I think so. Because sometimes you then, on the other side, where you talk to the exchange and you actually expect an answer to come back, that is really what an async thing would do. You await the answer back from that thing, right? But then you have... I've actually never experimented with doing threads and async. But maybe someone of you has and knows more about it.

Speaker 2 [24:29]

If so, you know who to talk to afterwards. So next one. Have you evaluated free threading web frameworks? Can you recommend one?

Speaker 1 [24:40]

No, I have not that's a very straightforward thing to say. No, I have not But I'm sure if you google it there someone already has done the work I think it's a much more popular niche in Python than trading

Speaker 2 [24:58]

Then there is room for the next talk on web frameworks. So next one, what are you trading and does your trading algorithm work? was thinking this.

Speaker 1 [25:17]

Well, as I said in the beginning, this is not, like, what I do at work.

Speaker 2 [25:18]

Well,

Speaker 1 [25:26]

My day job is to build the infrastructure for an algorithm. So I write the exchange connection and I write market data processing and I write order routing and position keeping and risk layers and all of these different things, but I do not develop the logic of what actually trades on the exchange. So everything else, pretty much. So yeah, and I really don't want to go into profitability and how that works of the algos of my employer.

Speaker 2 [25:54]

Fair enough, and this, actually I need to keep up with, okay, five more minutes, we do have five more minutes, so if there are any also spontaneous questions, I'm also up for just giving the microphone, if you want to, and otherwise, I give a few more seconds and a refresh well if not then there will be the chance to talk to Tim now afterwards and I do it one more time okay yes do you wanna oh I love it

Speaker 1 [26:40]

Question in the room.

Speaker 2 [26:42]

I hope this is okay for the remote attendees.

Speaker 1 [26:48]

Sorry.

Speaker 2 [26:50]

So you were doing

Speaker 1 [26:51]

You were doing the numbers on CPU-bound workloads, but not really a lot of I.O. Have you seen any numbers where it's mostly I.O. or where you're really mixing the CPU?

Speaker 2 [27:03]

Mixing the CPU and the IO.

Speaker 1 [27:04]

in the I.O.? Yeah, I ran the same benchmark also for when you technically overload the threaded process, so still staying at five milliseconds to process the stuff, but hitting it with 400 or 1,000 messages per second. And then the socket is still able to keep up. That is really not the bottleneck in this. The thread reading the stuff from the socket is... I mean, the 0MQ library is pretty much straight C bindings. There's no problem in keeping up. But then you obviously see the queue backing up just due to physical limitations that you're too slow. That's all I can say on this. And then you see a similar effect as on the async side. You have a lot of data coming in, but as long as you do an efficient way to read it in, then you're good. Sorry to keep us here. Yeah, sure. So if I understand, as you mentioned, this is a look-free implementation you have here. Just in case there's a shared state in the shared amount of threads, what kind of recommendation of libraries or atomic save or concurrent save data structures do you recommend? What I did for what I tried as well is just to put a threading lock on the update and lock the order book while I'm doing the update and then read it really quick and acquire the lock. That was really not a big problem also and just doing it a lock. But if you then have multiple things reading or leading into one processing and then you can do stuff like a ring buffer but this is really not something I would implement in Python. So, yeah, I think if you're thinking this far, then maybe switch to something else.

Speaker 2 [29:08]

All right, and now please help me give Tim for his first talk a massive applause