Navigating the limitations of Python’s concurrency model in web services

Speaker 1 [00:04]

Tarek Mehrez Great, thank you so much for joining me today. My name is Tarek Mehrez, and today I'm going to talk about Python's concurrency model in the web. It doesn't really come as a surprise, I think, when we think that Python is not the fastest in terms of execution time, and I think last talk we've also seen some tricks on how we can make sure that it runs much faster in a single core, and throughout this talk I'd like to expand a bit on this topic. So how can we even speed up things a bit more by having some sort of concurrency? explore a bit of parallelism, but I think concurrency is the main aspect here. Now, one main assumption, obviously, your favorite service will run on some platforms such as Kubernetes or some serverless in your cloud provider, and they kind of do the horizontal scaling for you. And that's not really what we're interested in here. I'm kind of talking about the basic compute unit which is running in Python. How can we speed this even further within your container, within your process? So, if there's one thing I would love to get out of this talk it's what are the options out there so if we're exploring different possibilities again for deploying web services in the cloud on the web whatever what are the options so that we can speed up our deployment we can make sure that the service is much more resilient and by resilient i mean we can take more traffic without worrying about reaching limitations in terms of worker nodes in terms of cpus and so on so this is sort of what we're going to explore today but first a bit about myself my name is Tarek or Tarek if you're not an Arabic speaker originally I'm from Egypt and currently based in Amsterdam for the past six years and I work for a company called Klarna so Klarna is arguably one of the biggest fintechs in Europe and the world we're mostly associated with the buy now pay later aspect so if you're shopping with your favorite e-commerce website probably you've seen Klarna as a payment method but we're doing all sorts of other things such as like banking we have cards in different markets and we're operating across the globe we also have engineering hubs all over europe so if you're interested feel free to reach out to me if you'd like to join but yeah now we can go back to python and one disclaimer this is not necessarily related to the work we do at klarna it was sort of my personal experiences throughout the years with how do i kind of navigate the issues i'm seeing when it comes to deploying python services in the web so what are the options what would we would like to explore and i think these are sort of the three main aspects that comes to mind so multi-processing you achieve parallelism by having multiple processes whether through unicorn or just try manually by saying multi import multi-processing and you spawn multiple processes so this is one aspect or one way of doing things g event i think is also very interesting in how it achieves concurrency and i will kind of just have a tiny dip into g event and how it achieves concurrency within web frameworks and specifically goony corn and then last but not least async io and i think this is the most interesting mention because recently when it got into the standard lib a lot of new technologies or frameworks have emerged that supports this natively and this is what i would like to explore as well so basically we'll be exploring or comparing these three in the web aspect so again this is mainly about the comparison but it's not really about deep diving into one of these so we'll just as i mentioned talk a bit about the event loop how async io works um how do you kind of change your code from sequential synchronous execution into async execution so that you can achieve this sort of concurrency but we're not really deep diving into any of them and i think it's also kind of ridiculous to say this framework is amazing this framework is bad depends on what works for you all right before that i'd like to tell you a very quick story about my very first python service that i wrote so i think six years ago a data scientist that was working for me on the same team came in and said we have this machine learning model i would like to deploy this behind an api so have slash predict i give you some strings you give me back some entities so basic nlp basic information extraction so naturally i use flask because it's a very simple way of doing things you just have your dependencies in there you annotate your function with the endpoint and you're good to go right and it's very simple to deploy as well or run not deploy because it's as simple as well flask run and you have a dev server running and we're good to go what is the issue here it's a single thread single process server that doesn't work for the very simple reason that your worker or the process and i think this is the baseline that we start from is blocked it is waiting for this request specifically it's cpu intensive since it's a machine learning model written in tiano i think tensorflow was still in beta back then so this thing took like three seconds to execute imagine sending 10 requests at a time. So, most probably they would just time out. So, this was the issue. And we need to do better in terms of that. So, how can I put this thing, how can I change this dev server into some better server, web server that I can put in production behind a container? And then Goony Corn, oh, yeah, we can do better. And then Goony Corn came as a natural decision. So, this is where we start with the first option, which is multiprocessing. What does multiprocessing achieve, you still have Flask as a web framework, still synchronous code, nothing changes in your code, but instead of deploying this behind the Flask dev server, you can use Gunicorn. What is Gunicorn? Gunicorn simply is a Python web server that provides some capabilities which people can use in production to make sure that your service is actually resilient. In that case, we have a pre-forked server. Pre-fork means we have pre-forked processes that can take in your requests so in that case if I say I need 10 requests or 10 processes that would listen on this port and taken requests so essentially what happens here guni corn tries to achieve this by spawning 10 processes and I'm talking OS processes which are obviously expensive so that's a downside and then you say okay then I can take much more than one request and that doesn't really run on its own just fine we need something else So it needs something like NGINX. Why? Because at the end of the day, Goonycorn is still blocking. So even if I have 10 processes instead of one, I'm still waiting for work to be done before I can take in the new request. And that's kind of a bottleneck. That's an issue. And NGINX usually provides more or less an asynchronous way to deal with this. So NGINX can take multiple requests at a time, send it over to Goonycorn so that you can handle slow clients. So that I don't know if you have one of the clients would wait for like two minutes You don't block the entire service. You can still do more stuff. But this is what nginx provides the takeaway from this slide is synchronous Processes on going you corn are still a bottleneck and that's still an issue. So we did slightly better But it's not great. It's not good enough So how many requests can we take at this point definitely more than one because we have multiple processes nginx does some buffering for us so with a specific factor we can assume that it's as many as the number of processes we have and some people are actually most people including myself at the beginning I thought okay this is good enough if I have ten processes working I can just use Kubernetes scale horizontally have three pods or whatever replicas running and I'm good to go how many do I need anyway how many requests do I need anyway well you will be surprised if that is synchronous and that's running on a bigger system waiting for even more api's that they're blocking on them, this kind of doesn't work. So the whole thing when it comes to scale becomes really an issue. A deeper look into GoonyCoin, and I think this is mainly a visual attempt to describe what I've been talking about so far. So as I mentioned, we have one master and workers. The master has one job only, which is to make sure that your desired amount of workers are up and running. So if any of them are killed, you kind of restart the process and if you use gunicorn before you would see something like starting unicorn starting processes blah blah blah blah and then you listen on the port and the network interface and hence the term pre-fork because you do this before the service starts there are other models they're not really getting into that you can fork while the service is running so maybe one per process that would be very expensive or maybe have a pool which you fork every time you have a specific amount of requests so that you can handle spikes so there are different models or different ways of handling this and the workers get the http request directly one thing which is not necessarily part of this slide is the whiskey interface wsgi and that's something i would like to touch upon very quick maybe this is even a better slide so we need an interface between http and python python at the end of the day is a callable you call your function but you don't really pass http info python was not aware of that so the whiskey is essentially an interface a mapping between the unicorn which is the web server into your callable which is um the flask callable in this case or it could be can be django can be whatever web framework you would like to use and again whiskey is synchronous because everything in this process or in this flow is synchronous and keep this in mind this can be will be useful afterwards all right so as i mentioned the workers are synchronous and we'd like to do better so goony corn has an amazing feature that not all workers have to be synchronous you can actually have async workers so how do i have async workers if my code is synchronous there's no async await there's nothing no import async io just my code but then goony corn will have this async aspect how does this work that's something that we will explore right now but before then we'd like just to have a very quick intro on what async actually means All right. So async execution. You have a Python function. You turn this into a coroutine, and we'll talk about this now. So different async frameworks turn just def, do something, takes parameters. That's your function, right? The moment you put async behind, it becomes a coroutine. At least that's how async IO does it. Some other, like gevent, for example, they call greenlets, and it's essentially the same concept. Coroutines get scheduled as a task and then there's an event loop literally a while through running if you check the source code that checks which tasks or coroutines are now ready for execution and these get executed why aren't all of them ready for execution because i might be waiting for an api or i'm writing something on disk or i'm waiting for a database connection to be done like a request from the database while this coroutines or essentially function doing this thing other functions can also execute, and that's basically what the event loop is doing. So, it's sort of a tiny scheduler within the Python process. And again, all of this is still single process, single thread. And that's how we're achieving concurrency here. So, ideally, we would love to take this model into the web, into the API that we're currently building. So, again, make it even more visual, assuming that these are the threads or the coroutines. They're not actually threads. The coroutines and the event loop is choosing who to execute next based on which coroutines are ready. Now, there's the notion of cooperative multitasking. Sometimes in some models or some implementations of not asyncio, but I think others would say, I'm done. I executed long enough. I will yield or give up the execution flow for another coroutine to take on and do stuff. And that's the cooperative aspect. I'm not entirely sure if AsyncIO can do that, but G-Event, for example, can definitely do that. So halfway through the function, you can say, give the execution flow to someone else. I'm expecting this to take so long. Most likely, if you're implementing an API, you don't really care about all of this. But it's still good to know in case your implementation is CPU intensive or you're doing something which I owe that takes a long time. All right, back to G-Event. So now we know what the Async execution is about. thought, can we take gevent as it is or as a framework to do this within the web? Very quick background, gevent was a really nice attempt to implement all of this within Python before asyncio was part of the standard library. And Goni Cohen thought, well, I can take this as part of my workers. You are still write flask, you are still write Django, but I will try to replace the sequential, the blocking flow or execution into an asynchronous one using gevent, and we will get to that in a bit. So yeah, as a unicorn worker, that's a very important task. All right, so each worker, remember the 10 workers or the end workers that we had? Before they were synchronous. Now they're no longer synchronous. Now they're async by having the gevent servers in place. And it has its own event loop implementation, runs on libEV. I believe libEV is also the implementation behind the Node.js event loop. So it's not really, now as part of the standard library, the Python event loop, this is different. That's a different one. And now we've achieved the non-blocking aspect. So it's exactly the same slide, right? Nothing has changed. The only thing that changed is the Goonycorn worker as I mentioned before. How does this happen? Something called monkey patching. So Goonycorn says, gives gevent the capability of changing the Python functions that you have in your Flask or Django code. And during runtime, it tries to make them async, basically, by monkey patching the execution. Now, that's a really nice idea, but it's also quite opaque. If I'm changing the runtime from sync to async without me changing the code, things can definitely go south. So from a usability perspective, and there's literally a blog post called monkey patch and pray, because you hope that this thing will just work. I do remember then that on Flask, it actually did work on the machine learning service. So I said, OK, I need five workers of type gEvent. And it did have this async thing work. I tried this with Django. That didn't work because some of the ORMs and the database drivers just did not like that. And debugging was so hard. So it's sort of a pseudo async way to do things. Now, this does not mean that gEvent is bad. you can still write your entire piece of code in gevent using their own functions, and that works just fine. The part which gets a bit hairy is including this in your Flask or Django implementation. But now we can have much more incoming requests than the workers. Now we're no longer bound by 10 requests in parallel, because this is what the synchronous worker can do, which is an achievement, I would say. That's good. But we can still do better. So as I mentioned again, gevent monkey patches the synchronous code to achieve this async execution with the downside that it just won't always work. All right. What do we have so far? Fully sync, what I call pseudo-async, with the gevent worker. Cool. We can still do better. Async IO. So finally, we have AsyncIO. And essentially, after lots of peps and lots of ways trying to put this as part of the standard lib, AsyncIO was part of the standard lib, great. And then new kind of web technologies emerged out of this. We thought, OK, instead of just Flask and Django and their legs, we can have now async web frameworks such as Starlet and FastAPI. Instead of the WSGI, which was the synchronous mapper from HTTP to the Python callable, as we called it. So this mapping to the web framework, we can now have ASCII, which is the asynchronous way of doing this. And this also enables things like long polling and web sockets and so on in a much more efficient way, which is really nice. So for the rest of the presentation, I'd like to explore a bit what asyncio actually added. But let's go back to how we defined the async execution. So again, same exact idea that functions become coroutines. But in that case, instead of using gevent with the monkey patching, we're just typing async. So before the definition of the function, you say async dev do stuff. And then it becomes a coroutine. And then what happens? This coroutine becomes a part of the event loop, which is something we're interested in. Now, something to keep in mind. If you have blocking code, not everything can be async, right? will always be this dependency that has to be blocking that you cannot just execute this in an async manner your code will be as low as this blocking part so it's not entirely magic right so and this is also something to keep in mind so if this thing exists if this blocking code exists there's not nothing to be done about that maybe you can offload this to some other parts of the system do it offline whatever but keep in mind that this doesn't speed up this part this part remains as is and yeah as i mentioned there are multiple implementations of the event loop and there's one already which is part of the python standard lib why do we care because now instead of relying on external frameworks such as g event we can now have this as part of the actual um python code or python standard lib and this means that you can add more dependencies you can rely on more stuff without worrying about will the monkey patch work? Can I unit test this properly? Is there something that might go just break without us knowing about? And this is something which is really interesting. And again, no need for monkey patching, so that's also good. So what do we have so far? As a quick comparison. We have Flask, Django, and FastAPI or Starlet. It could be much, much more stuff. It doesn't have to be only these frameworks, of course. Instead of Goonycorn, there is now UVcorn, which is also a web server, also written in Python, and now it sort of supports the ASCII in a native manner. Whether you need NGINX or not, that is an open question, because one might argue, well, I'm already async. Do I need this slow client handling? Maybe that's not necessary, but it also supports stuff like SSL termination and so on. So if you're really putting this as part of a bigger system, you might still need NGINX. Now, there's another plot twist that Gonyucorn does support UVCorn workers. So remember, they're GEvent workers. They're also UVCorn workers, so that you can start multiple workers, as we said before. So we have n workers running, but then they're all natively async. They're all running the UVCorn processes. The downside of this is you don't really share the event loop. So now you have 10 parallel event loops running, which is maybe something you're interested in. If they're not sharing a state and you're this behind an API, maybe that's great. Some comparison that I wanted to do. This was also part of me trying to ease it out for myself. If I'm choosing which way to go, what might be an interesting or a good setup to go with? With the multiprocessing, as we said before, it's purely synchronous. You're writing synchronous code. There's no async, there's no await, there's no mapping, whatever. synchronous code. G event, you're not writing synchronous code, but the execution is somewhat async if it does work. And async IO is inherently async in both the actual code that you write. So if someone is not familiar with the keywords and sees the code, I'm like, I have no idea what this is about. And the execution is inherently async. In terms of debugging, I think it was a bit unfair to say that async IO is easy to debug because there will be a lot of things that might happen with database connections, with Redis connections, with unit testing, that you realize, okay, this is not really as easy as I thought. At least it's not opaque, because there's no, again, the monkey patching. At least I can see what's happening. I can see where the async bit is, but I wouldn't go as far as saying that it's easy to debug. And in the latter two options, we actually achieved concurrency. And what does that mean? It means that we can send much more requests than a single process and a single threat can handle. And I think this does matter in a scalable system. Are these the only options or frameworks? Absolutely not. There are multiple, tons of them. There's Tornado, there's Twisted, there's Trio. I think if you go through, again, the history of how this evolved, you realize that there are multiple ways of doing the same exact thing. The only difference is how mainstream is it? So if you're stuck, can you actually go in there and try to debug things and find support or not? And the other aspect is, what sort of special use case do you have? And talking about the web, I think that's pretty much a standard use case. So going with the more mainstream options is probably an easier way to be able to find this sort of support. I think recently Tornado, because I also mentioned that I will talk about Tornado, I think recently Tornado does support the async await, So the standard lib functionality, instead of before then it was using decorators to transform these functions into coroutines. And again, different framework, same exact abstract concept. Function, coroutine, event loop. And now it also supports web. I think it always did, actually. Web requests. You can have an API written in Tornado. People seem to be walking or moving towards toilet and fast API for different reasons. but yeah if you feel like this is a better use case for yourself then please go for it one last plot twist we kind of excluded flask and django completely from the async implementation that is no longer the case because in new versions they are supporting async natively i think with django there's a limitation that you cannot go to the orm layer so you can do this on a more in cpu and on the api layer but then when it once you go to the orm probably this doesn't work anymore so i'm curious to see how this will evolve if because if that actually works end to end and then you have asgi instead of whiskey you have something like uv corn running on gunicorn then there's a very big chance that the competition against starlet and fast api for example would be quite interesting or quite tough off. Flask is also taking off. Flask is trying to keep up with this. I think in 2.0, async is currently supported. This entire comparison saying that this is synchronous and this is async, I think maybe in less than a year, probably be obsolete already. That would be interesting to see. That's it. I hope this was helpful. Thank you for your attention.

Speaker 2 [23:37]

So, thank you very much for your talk. I found it personally very interesting. So, since we have a lot of time left, we'll be able to go over a lot of questions. But first, let's go over the slide-out questions. So, the first question is, do you have any tips for error handling in async code?

Speaker 1 [24:07]

What kind of air handling? That's, I think, but probably not sure the person is here.

Speaker 2 [24:11]

Well, in general, right?

Speaker 1 [24:14]

So the good thing about fast API for example that it shields you from a lot of things so they kind of wrap Responses so whether it's serialization issue whether it's I don't know issue with the actual execution. It does give you a better Understanding of errors something that I kind of went through and it was quite annoying doing something like celery so trying to attach study on the same event loop, for example, the moment you go beyond the simple use case things can go quite wild and And I wouldn't have a specific recommendation per se, but try to keep things as transparent as possible. But yeah, I think that's quite a generic answer as well.

Speaker 2 [24:52]

All right, so the next question is, did you also run in blocking requests if you use AsyncIO with background tasks and middleware? Yes. If so, how did you solve it?

Speaker 1 [25:04]

fast api has a beautiful thing called background jobs so you can just say run this in the background and then it gets attached to the event loop and then you're no longer blocking which is amazing or you can spawn a salary if you're using salary you can spawn a salary job or Yeah, that does that for you.

Speaker 2 [25:21]

All right, then the next question is, what could be the future? GIL-free multi-threaded asynchial loops, or?

Speaker 1 [25:29]

I don't think I have anything to say about that.

Speaker 2 [25:35]

All right, then let's move on to the next question. So the next question is, do you have any recommendations for wrapping CPU bound work in an async function?

Speaker 1 [25:45]

i wish i did um i think background jobs is good you can do this as a background job i think the good thing about salary is that you take it completely out of this process and you put it in a completely different process so that's always good it depends on how real time you want to be because if it's that cpu intensive and you want to be as real time probably python is not even a good choice in the first place and not obviously as a language you can do better but also you can use something like i don't know apache link to do this intensive computation for you. So I wouldn't really rely on Python, to be honest, if Celery is not a good enough job.

Speaker 2 [26:20]

all right let's have another question so have you used any async orms which ones and how was the experience against for example django orm or sql coming

Speaker 1 [26:30]

So there is a nice wrapper called Databases that wraps SQL Alchemy, and you can use this to write async code with your ORM. That worked just fine. The only issue was testing it. So with testing, we used the sync one, but in the actual service, we used the async one to have predictability basically on the execution. I think that worked fine. It was sort of a hack, but I think it worked fine.

Speaker 2 [26:58]

All right, let's do another one. So have you ran stress tests or benchmarks? What use cases profited and by how much?

Speaker 1 [27:06]

Yes, I did a lot. This was the main comparison between FastAPI and Django. The use case was we had a customer-facing API talking to another API, and the first one was a Django. If all of the workers were busy talking to the other API, no one was talking to the customer. The customer-facing API was blocked, essentially. This caused lots of problems. Yes, we did lots of load testing comparing both so the sync and async the async was much slower and the timeouts are slower but at least it did not fail and this was good enough for our use case the the problem with django was you just had you didn't have availability to just return back an error code telling you like i have no workers to to serve you so yes it will be much resilient but also a bit slower and then you rely on cpu for example a latency time to scale horizontally but at least you're still available

Speaker 2 [28:03]

All right, so let's give some more time for the Slido people to write their questions. So does the audience have anything they want to ask? Well, then, let's move on to the next slide of questions. Well, I mean, a lot of people, they just type their questions in Slido, right? So how do you deal with async code getting ugly or unreadable really quickly?

Speaker 1 [28:32]

It depends on how deep you get I think into async IO if you're using again a framework You don't really tap into the event loop and scheduling tasks and timing out tasks and so on So that's fine The code is not really ugly if you want to tap into that it will get ugly and then it's up to you to make It pretty But I think for the basic web use case, it doesn't really it's not really that bad. I Think there's a question over there. All right

Speaker 2 [28:58]

All right, sure.

Speaker 3 [29:02]

Hi, great talk. Thank you. You mentioned the background tasks that you can give a request to work in the background while you have answered something. Is it also possible to have the, if you have more than one background job, as increasingly running at the same time, that they share memory so that these background jobs have knowledge of each other?

Speaker 1 [29:29]

I would say doing this within the Unicorn or the web server processes is probably not a good idea unless you have an external. So it depends if it's a data structure within Python itself, probably it will get a bit messy. If you're using an external storage system, I don't know, something like Redis, then probably you can do some sort of coordination between both tasks. But you cannot assume that they will run on the same process.

Speaker 3 [29:53]

Okay, thank you because the question was for inside Python if you give it to the outside, of course You can share memory because I mean if

Speaker 1 [29:59]

because i mean if they're in the same process they can share some sort of global state that's perfectly fine as long as you're not changing it if you're changing it

Speaker 3 [30:07]

So the background tasks don't die after one is finished and they could just use Total Globals or FastAPI then?

Speaker 1 [30:20]

Short answer. I don't know if I get to Speculate probably it will have its own timeout on the event loop itself So if a task gets scheduled it will have a timeout so that it doesn't keep I don't know if it takes six hours Probably that's not a good idea So if it times out and it's not done yet You will not be notified and that's why salary is good because salary has a proper like airflow. For example, you have a proper Execution time what succeeded what failed how long did it take and so on?

Speaker 2 [30:48]

All right. Any more hands? Questions in the audience? All right. Then let's move to the slide of questions. So another one is, how long does it take to calculate prediction? If longer than a few seconds, how would you overcome this problem?

Speaker 1 [31:04]

by having a callback maybe that's an option so you have an endpoint saying predict and then you do nothing and then some endpoint either have a callback that gives you back the prediction or you wait a bit and then consume i don't know depends on your system of course but separating it into multiple calls is probably an easier way to go or just don't use deep learning

Speaker 2 [31:29]

All right, then we move on to this question. So, which number of UVA corn workers for G-unicorn would you recommend? One per CPU core or...

Speaker 1 [31:40]

I think there's an equation in the documentation, like workers times two plus one, something like that. I have no idea what the science behind it is, but I think it's also sort of an art. You cannot really come up with one answer for this question, because if you have two CPUs and 10 workers, they will compete for the CPUs and you end up slowing down your service. Actually, it's not a good idea. So doing so much more, much more workers than CPUs is definitely a terrible idea. so you just need to keep tuning things until finding something that works for you

Speaker 2 [32:13]

All right, then another question is, do you have a comparison of speed between a sync app and its asynchronous version?

Speaker 1 [32:22]

but I'm sure if you check any framework documentation they'll tell you we're the fastest. I think you need to try it out I guess.

Speaker 2 [32:34]

All right, so another one is, what would I need additionally to get a node web sockets-like setup?

Speaker 1 [32:41]

Say again, sorry.

Speaker 2 [32:42]

what would i need additionally to get a node web sockets like setup to get a web

Speaker 1 [32:43]

What? to get a web socket set up in django i don't know i've only done this in fast api and it's as simple as you define your own endpoint the same way you do in flask you just annotate the function but instead of having it as an http endpoint you tell it that this is a web socket and then i think you need to send and receive messages it's quite simple actually if you yeah it's it's documented in the fast api again i'm assuming fast api other frameworks i don't know

Speaker 2 [33:18]

All right, so right now, there are no more Slido questions. However, if anyone wants to continue typing, I can ask a question personally. Please. So you mentioned a lot of types of speed-up methods using, for example, gevent and async.io, stuff like that. Personally, which one did you find the most hard or most difficult to learn by documentation or just trying to get it to work, stuff like that.

Speaker 1 [33:51]

this is a very personal um opinion but i think the django rest framework i just couldn't reason about it i don't know why just didn't make it with me um fast api i do like to have the ability to do things instead of having a very specific convention that i need to follow and that's probably why people drop tribute on rails at some point and just fill in the gaps and it works um so fast api made the most sense because it was sort of very close to flask but at the same time it was async and much faster so in terms of learning i think finding one template that you can follow will make things very super easy and it wasn't really an issue about async execution maybe only with testing the very first blocker that i had okay how to actually unit test this and it took me a while to figure it out but once you figure it out you can just copy the same rap template but with django i assume the convention itself just didn't play out well

Speaker 2 [34:47]

All right, so do we have any questions in the audience? Any other hands? Well then, in this case, thank you very much for your talk. Thank you.