Django-Q2: Async Tasks Made Simple

Speaker 1 [00:05]

and thanks for joining to begin with just a small admin thing if you have any questions please ask in talks.pycon.de and we can do questions in the end now we can welcome Mohin and give him a big round of applause for his time

Speaker 2 [00:27]

thank you for being here today we'll talk about how can we create a synchronous task in our jungle application without adding any extra infrastructure but before allow me to introduce myself my name is moin utin i'm a devops engineer and tenant race consultant at partec i spent most of my days working with infrastructure observability and automation with Python and Django has always been my hobby. Who is ParTech? We are an Italian technology company expert in software and infrastructure. We are active in the open source scene for more than 20 years now. We are trusted partners of many tech companies like Red Heart, Dynatrace, Google and many more and yes we have also a tech magazine called you so if you are curious please feel free to check it out so this is our agenda we'll talk about starting with a concrete problem we'll do a quick overview of async and q fundamental just to get everyone on the same page and then i'll I will introduce to you Django Q2, how can it help us with a live demo. A little disclaimer, Django Q2 is a fork of the original project of Django Q, which has been installed since 2021, but luckily the open source community picked it up and updated several dependencies and bug fixes. Alright, to make this concrete, I have built an application in Django called Mactamia. simple desk booking application kind of internal tool that many companies might use the workflow is really simple the user login pick the date pick their office and book their workstation at the end the user receive a confirmation mail and the application also can send a reminder mail booking before the booking date nothing exotic here the typical application Django that you would implement the part of booking and sending mail can be done in two ways synchronous and asynchronous way in the synchronous way after the booking is saved to the database the app will try to reach your mail server and the user might wait a couple of seconds depends on the loads of your mail server only then your user will receive a confirmation page on the other hand the asynchronous task asynchronous way of doing after the booking is set the email sending task will be added to the queue to be processed later in the background and the user will get the instant confirmation page both ways works depend on your use case but here is the problem what happens if our mail server goes down so let's go to check directly on our application so my application is running on docker container i have defined three services for now i have database mail server and our django application as you can see we have three container running and listening on different ports for now uh our book book view is using synchronous way of sending mail so let's get to check out this is our application on the left i have mactavia and on the right i have our mail server so let me log in with the most sophisticated credential in the world which are admin admin all right let's me book a desk an office all right and here it is our uh click sending mail so far so good let's break our mail server then so what we will do we'll override email host in order to simulate the timeout process so let's get back to our application let's book another date all right as you can see our application is stacked because it is waiting for the mail servers to get a reply this is a disaster for the user experience because the user is waiting for something they don't need to wait for and obviously we have an error and the user has lost also the confirmation email data now let's switch to the synchronous way sorry asynchronous way so I'll comment out this one okay so our application our main server is still down so let's make another test as you can see the difference is remarkable because you get instant confirmation page and the user never noticed that your email server is down this is the core idea separate what can be done later all right so this is what we have seen so far in the synchronous way the user received an error when the mail server is down and probably we lost also our confirmation and depends on the logic you implement and the asynchronous way the user gets successful response quickly and our task is added to the task queue. Alright, how Django Q2 can help us with this situation? Django Q2 is a multiprocessing task queue built for Django that means it is really easy to install and integrate into our Django project. It is really infrastructure-like that means you can use your existing database to save your list of tasks and it has beautiful admin panel where you can see every task status of your every task. I like also the native scheduling part that it has so that means you can create cron jobs directly on your Django application and you don't need to depend on external provider. It also support multiple workers and multiple queues, we'll see it later, and it supports also many type of brokers such as Redis, MongoDB, Amazon SQS, etc. Okay, this is actually the setup that you need to add, you just put the Django queue in your installed apps, since we will be using our database as a broker, so we will add this configuration in our queue cluster configuration. And then as usual you have to execute migrate to create tables on your database. On the right you can see the only thing that changed after when you use Django queue. You just wrap up the sendmail function into the async task. Your function signature does not change. You are just telling to Django queue run this later in background. Before we jump into the demo let's get a quick overview of at high level the queue architecture. So we have three components producer, broker and worker. Producer which who creates the task in our case our Django application Mactavia. Broker where the task links are saved in our case it will be our DjangoDB and there will be a worker process which runs outside of the of the server which features and executes our pending task from the broker in Django context it's called queue cluster process now we'll see how it works all right all right so we are in a situation where we have booked our desk and we haven't received an email so let's go to check what is happening in the in the back end of our application uh just one note this dashboard is not built in i've just made it for the demo purpose but you can already see we have one task in the cute list you can check it directly from the beautiful argument panel that built in on with jagoo queue so if If you go there, we can see the task that we have submitted before we were booking the desk. All right, so now we need the worker process to execute this task. Let's go back to our Docker Compose. So we have already defined a task service. As you can see, it uses the same image of our application, Because since they share the same code base, you don't need to build another image. You can just build one image for both services. But in this case, the command to run will be python manage.pyqcluster. So let's set the replicas to 1. And docker compose app to start our container. So as you can see, it is creating another container. We can see also here. All right. Let's see its logs. All right, it already started to process our booking mail, but remember, our mail server is still down, so we expect task fails. All right, as you can see, the task has been failed, so let's get to check on our admin side. In the queue task, we have no more task because it has been executed. it has been moved to the failed task directly from the admin panel we can see also the reason why it has failed now let's get back to our application we'll we'll fix our email server so i'll just comment out this one we have to restart our task the worker task in order to get the right configuration all right so it is being restarted now it will not retire automatically because in my configuration of queue cluster i have set max attempts to one but you can set more attempts if you want based on your use case but for the demo purpose i've set it one because i want to show you that how easy you can resubmit the task directly from the admin panel so here is our task i just select it resubmit to the queue and here is it so you see no data is lost we have reset correctly our email confirmation mail and our application becomes much more resilient all right if you remember i have also mentioned that our application sent a reminder email before the booking date this has been done by the built-in cron function that Django Q has as you can see this is this job which use our send reminder function this is executed every day and the next run is tomorrow but the good part is we can handle everything from the admin panel so i just want to run this task now what i have to do i will set today now and save this is very practical for if you want to test anything directly from admin it's really handy and also if you if you want to pass if you if your function accepts any arguments you can also pass directly here and it will be executed with the arguments that you are passing so in the right you can say we have already resigned all the booking reminder for the email that has booking in our application so really really really really simple uh with django queue all right let's get back to our slide so this is what we have seen so far you can also create a scheduling job directly using the the schedule function from django queue it accept also the cron expression but you need to install the creator prep package also it has a cool feature like catch up so in case your cluster was down on restart it will try to schedule to to catch up your your jobs all right now our application has background jobs working fine cron job working fine we want to add more firepower because we are having too much task to execute. Django Coup supports multiple worker instance. It is really easy to scale horizontally. Let me show you how. So in our Docker Compose, we have one replica. We'll just set to three replicas. And again, Docker Compose app to create new containers. As you can see, we have tasks 3 and 2. Now let me add a bit of task to our application to see how it leverages the multiple workers. Alright, it is being added some tasks. Obviously, you can see it has been processed by different workers, but we will see soon also once it's finished okay 1200 tasks have been added all right it has also finished let's see how many tasks each worker have processed so we can have this grab this is windows so don't hit me all right we have as you can see it's a it's a fair leverage on different task workers because task 3 has processed 446 and task 1 350. So I think it's pretty fine. If we go to our email server, we can see it has also sent all those mail that we have submitted to our application. Really, really easy. All right. So all tasks are not the same. In our application we can have no slow task like sending mail and we can have also long task like generating report having one queue means your long task can block the slow task which is we want to avoid the solution is workload separation and Django queue luckily supports multiple multiple queued names so we can create a a default queue cluster where we can set all these short tasks. And we can create a long queue cluster where we have all those long executing tasks. To implement this, also really, really simple, what we have to do is add on the existing configuration the dictionary alt cluster with the name of the queue that we have to create, in this case, long, and all the configuration that we to override from the default settings in this case we are putting a long time out because our long task needs more time to execute you can root the task directly on the call so if you see on the right in this task i am telling send this task directly to the cluster run so the worker long process will pick and execute it and if you don't mention any cluster parameters in your async task it will just speak by the default cluster obviously we have to execute another cluster process but the only differences will be we have to put this environment variable which has to match the name of the name of the queue that we have defined in our alt cluster really really simple let me show you all right okay before I start up another container let me show you why it is important to add the right name of the of the queue for example now I'm adding 10 tasks to our long task queue and if we go to application and see in the dashboard you can see there is 10 tasks which are in the queue and are not being processed by the default queue cluster so the workload separation is very safe only when we will start our long queue cluster so let's begin all right I'll set replicas to 1 and as you can see I am starting with this environment variable and the command to execute is still the same all right let me save docker compose up again let me throw some logs task long as soon as i've started our long queue worker we can see that it has been processed all these tasks that were impending in fact now our queue is there is zero and it has been in the succeeded status as you can see here so very very very very easy with time you want to move your loads of queue list to another broker with django queue it's also really simple all you have to do is replace the orm configuration to a radius configuration and that's it your task function remains unchanged, and the only thing it changes is automatically handed by a queue cluster. One thing to note, obviously this does not migrate your existing task to the new broker, so you have to do it manually, sadly, but otherwise it is really very simple. All right, to conclude, Django queue is not the most powerful task queue, but it is really the simplest way to start the reason you want to start with Django Q2 is that you want to use all your existing infrastructure like database it is Django native so you will use vram admin and management commands all things that you are familiar it has a beautiful built-in monitoring so you don't need to create nothing from the from the sketch but it has it has also some limitation if obviously if you if you are expecting to have millions of tasks per minute you might consider to move salary and also the the task has no real-time progress so you will never know what is happening you will know only at the end actually if it's failed or succeeded all right this is all for me you can read more about on django q docs you can find everything code on my github aladdin 97 python talk and if you have any question or you just want to chat please email me or on linkedin send a quick message allow me 30 seconds more please this is my very first conference talk so i want to thank python for having me and obviously part tech for supporting me to come here uh allow me to improve myself so there is a qr code which you can scan and or you can just type tinyurl.com it will help me to improve myself so thank you very much if there is any question

Speaker 1 [20:27]

Thanks a lot. So we have two questions right now. First is, what are the advantages compared to Django built-in background tasks available since Django 6.0?

Speaker 2 [20:38]

I was expecting the question. So basically, with the new version, Django gives you the abstraction layer of adding the task, but you will still need a background to process them. So I don't think it is still mature to add your stack yet. Django Q2 has already everything built in, so it is much easier.

Speaker 1 [21:03]

Next question is, can I use Django Q2?

Speaker 2 [21:08]

No, it's only for Django ecosystem, so you have to use only inside of Django.

Speaker 1 [21:17]

How does this compare to celery?

Speaker 2 [21:22]

you don't need any extra infrastructure because if you want to put salary in your stack, you must have to add Redis or Ramit MQ With Django Q, you just use your Postgres database It is important from a DevOps perspective because Whenever you add any kind of stack in your application, it might seem cool But when it comes a day of maintenance, it's not so very cool

Speaker 1 [21:51]

I think that's it. Thank you so much.