Deploying Synchronous and Asynchronous Django Applications for Hobby Projects

Speaker 1 [00:08]

Thanks for being on my talk, Deploying Synchronous and Asynchronous Django Applications for Hobby and Small Projects. That being the highlight. I'll do a small round of intro. I'm Elhin. I've been a developer for over a decade, and fair share of trying to build stuff and not really working exactly. I tinker with a lot of projects, most are incomplete, but still, it's fun, right? I work at Kraken Technologies now. Kraken is one of the sponsors, and we have a booth down. There's also a quiz at 4.45 near the Kraken booth, so please come by. If you want to see the projects that I've tinkered with and those are incomplete, this is my GitHub handle. So let's start with the why. So we've all been there. We like building hobby projects. It's exciting, it's creative, and often just starts as a simple experiment. But once it's locally built, how do we actually take it from there so that others can actually use it? So this talk, I would like to do a combination of deployment on cloud, on-prem, a bit of async Django, and also deploying both synchronous and asynchronous Django together. Coffee, yeah. I know I'm the one stopping you from all having a deserved coffee break. Raise your hand if you have ever waited for coffee because the machine was too busy. Let's build a Django app. So let's build a Django app to solve this. And I'm going to turn this into a case study for running Sync and AsyncDjango together. Let me show you how I over-engineered this solution. I call it the CoffeeQ app. So, normally you have coffee, you have people waiting in queue, and, yeah, it takes some time, right? So, let's say, let's flip the problem a little bit. Let's bring in a server where the coffee machine basically communicates to the server saying the brew is over. And then you have users who are like just listening, using a mobile application or on a tab, just looking at notifications, right? And when it's ready, the server sends the notification to the users. And you can also scale it up. You can have multiple coffee machines and sending stuff to a server and then notifying. Taking this as an example, let's say we built a sync app to control the coffee machines. or just get information from the coffee machine, and you have a sync side of things where you actually notify the users. Let's say SSE, for instance, is a very lightweight notification system that we can actually use. Taking this as an example, let's walk through how things are. Oops, I think I forgot the agenda, but let's go to the agenda. So we're going to talk about deploying Django applications, deployment strategies for sync and async. using Camel. We'll also demo a little bit. I don't like live demos, but I'll try to make the experience as close as possible. We'll talk a little bit about the deployment landscape, some stuff around self-hosting. So deploying Django applications. Pretty straightforward on the sync side of things because we've been deploying RISC-E apps for a long time. And there are some common servers that you can already use. So, Unicorn, the most popular choice. It's very common and it's great for production deployments. YSGI, it's more configurable, but a bit complex. You can use advanced features like caching and stuff like that. Mod with YSGI used to be a thing. I don't think it's much common now. Going to ASGI servers, there are multiple of those, Daphne being the first one. I'm talking specifically about Django, So when Django was having the whole WebSockets thing, Daphne used to be a thing where you can actually use it for using Django channels. It's also built on Twisted, which was, I think, pre-Async, if my memory serves me correct. There's UVCon, which I think everyone is kind of known now, especially because of fast API deployments, and also we can use that for Django Async stuff. And there's also HyperCon, which is a very interesting new server which also supports HTTP2. A fun fact is, HypoCon is also used by Anthony Shaw for experimenting some of his sub-interpreter stuff. So, coming back to Sync and Async, right? Why use both? In general, web applications are not purely Sync or purely Async. It's always a combination of both of these things. And Sync is simple and reliable. Async unlocks the scalability for long-running tasks and external endpoints. So mostly use it for slow external endpoints, like long-running requests. WebSockets, again, collaboration software, bidirectional communication, you use that. And also server-sent events, you use that for one-directional communication where the server actually just talks to the client. It's also very lightweight. It runs on HTTP. The protocol doesn't change. There's a lot of good stuff that you can actually look at it. And running both together, there are some fundamental components. And this is mostly I'm just focusing on the small projects, not on the big, larger project bits. But when you run sync and async together, it's important that you share the core system, like databases, sessions, and auth, for instance. It's easier if they align. They give the user a seamless experience, so there's no multiple logging in and accidental logouts and stuff like that. And there's also security through consistency, where you basically have the same set of secrets, same tokens for both sync and async. Again, this is for small and hobby projects, not for larger systems. And there are two core approaches that you can actually take, one being the part-based where you just have one server or load balancer and say slash sync for all the sync routes or slash async, whatever you want to name it, or the domain-based approach. The path-based approach has multiple stuff in it, so same domain. So it's easier. It's just different paths. No cost issues, no cookie issues. DNS is somewhat simpler. It's the easy button. There's a bit of corn when you have both of them together. It's a little hard to scale. And domain gives us that. Domain is basically just separating it. In this case, I'm just using a subdomain. So let's say the CoffeeQ Sync app and the CoffeeQ Notifier, two different domains. It's isolated, and you can actually also have a clear separation. The cons are, so this is a bit Django-specific, but in general, there's the CSRF and the session cookie domain that you have to always worry about when you're kind of sharing the system in subdomains. And you can see the dot over there at some times can be used as a small hack, so you can use multiple subdomains. So now we kind of looked at the things of how to deploy them, and the deployment landscape kind of looks like this. It's ever-changing, but I'm just trying to explain the current state. So there are VPS providers like Lenovo Digital Ocean, Hetzner. Heroku kind of changed the game, I would say, long back when they extracted away all your infrastructure problems and then gave you a service where you can actually deploy. And there were also free credits. Docker came in and just standardized the whole app deployment in general. And there's also the whole platform as a service growth. So render, fly IO, railway, edge functions, , cloud fair workers, and also nowadays, database as a service. So they just take away everything. You just have to call an endpoint, and that's pretty much it. And orchestration tools like Kubernetes also has kind of made things a little more standard. So Docker Swarm and things like Nomad are also orchestration tools that you can actually use. And then you ask this question, what do you want to choose? Because you just want to deploy this coffee app, and you have, I don't know, so many services. So it can be really difficult to choose. So let's take first principles and look at components that are easier to understand. So what do you really need? You need predictable environments. You need scalability. You need zero downtime updates. And you don't need vendor logins. And you need low ops overhead. And Kubernetes is good for a lot of things. But it's not something that you want to deploy your coffee app on, right? And you want Heroku simplicity, but with control and portability, portability being a key word over here. So I'm going to introduce you to Kamal, which kind of tries to solve this problem. So what is Camel? Camel is a fresh take on deployment. It's designed for today's apps. It's containers without Kubernetes complexity. It automates the tedious parts, like pushing updates, like managing containers, rolling back if something breaks. It's like Heroku, but on your own servers or Cloud VMs. It doesn't make that distinction. And I'll come to it, why is it like that? And the key features being zero downtime deployments. In the modern era, we expect this. This is a given. You need zero downtime deployments. And Camel uses this as like it has health checks and rolling updates. So it ensures your app stays alive. It's basically Docker. So it leverages Docker container under the hood so your app runs consistently everywhere without needing deep Docker expertise. You could deploy to staging production of even 10 servers at once with a simple config. And I'll come to the config later. And there are no more convoluted scripts. It's predominantly a simple YAML. It might take some time to get it right, but it's pretty straightforward. And your deployment workflow now becomes a little more easier. So it's like you have a Dockerized app. You define your deployment settings in deploy YAML. You deploy to servers. And this is the important thing. Camel actually uses SSH to connect to your servers. And it pushes Docker images and just manages the containers in your server. So it doesn't really matter if it is on cloud or on-prem. It just has to get an IP address and basically use SSH to do the other things. It also gives you zero downtime deployments, as I said, because it switches the traffic to the new container easily. It's all under the hood. It's pretty much stable commands that it runs. And it also rolls back. So that's something in the modern era you actually need. Let me show you how a config file looks like. So I'm covering the config file, so I'll stand here. It's predominantly like this, and I'll take it apart one by one. Oops, sorry. So can you see the highlighted portion? Yeah, OK, all right. So from top to bottom, right? So you have the name of the service. You have image, you have registry, and you have builder. Name of the service is used to determine most of the things, like what prefix should all the containers have. The image is basically the image name that you store in the registry. By default, just supports Docker Hub out of the box. So if you have username and password, if you can provide that, or token, for instance, then you basically can just start pushing containers to the registry. And the builder is also useful, because you can actually give build time arguments to it. This is an important thing about servers. So that's where you define your IP address. And that could be anything. It could be a cloud instance. It could be an on-prem. I don't know. It could even be a Raspberry Pi. And you might not be able to see it, But below, there's also an SSH configuration where you just say, hey, this is my SSH key. So just uses that to log in. So host is also an important concept. So it's not like you just need one IP to deploy stuff. With host, you can actually have multiple of that. So you can keep giving 10 servers, come and make sure that it SSH to each of these machines and deploys them. Our roles is another way of tagging servers. So for instance, if your server is like you want a Redis server or a DB server, you could just say, hey, this is the tag and this is the IP. You could also just deploy everything on the same server. It doesn't matter. As far as you've given these declarative options, Camel takes care of the other stuff. And the most important stuff is proxy. So proxy in Camel is kind of handled by another tool in the Camel ecosystem. It's called Camel Proxy. And what it does is it does zero downtime deployments for app, route, and it routes the traffic to the specific app. That's all handled by the proxy. It does hell checks. And the version 2 of Camel is also pretty interesting because it comes with let's encrypt SSL configuration. So you don't even have to worry about that. You have to need, you need a domain, you need to point it to a particular server, and then it just gives you an HTTPS version of your CoffeeQ app. Environment variables, that's also, like any modern stack, so you could, I'm not sure if you can actually see this, but below you could see they're just clear, defined environment variables that you can give, and then on top you could see environment variables that are dynamic. And the syntax is a bit odd because it's Ruby at the end of the day. But you could see how you can actually dynamically just put it in n variables and then just use that as an IP address. So you can actually keep moving your servers but don't change anything in the configuration file. Kamal also has a very good way of giving secrets because you could just, again, you don't commit plain secrets, you basically derive secrets in Caml. And this is basically done by another of Caml's functionality of secrets files. And this is also very, very flexible, in the sense you could have it in your .env, you could fetch it from one password, last password, or even Bitwarden. And you could also write scripts where you can just say, hey, fetch the secret from my obscure secret manager, and there it is. Its installation is also pretty straightforward, I mean prerequisites, of course you need Docker because Camel just relies completely on Docker. You also need Ruby, I know it's a Django talk, but still, you need Ruby to install Camel. I just open SSH in general, SSH is mostly there in most of the machine, and installation is just pure simple, it's just gem install Camel. When you install Ruby, I think it's better to install it separately from your system Ruby, because otherwise there are issues with it. I mean, you can just verify installation. Now I'm going to switch a little bit and talk about config files. So Kamal suggests, in theory, just a single config file. I would like to propose a little bit of separation over there. There's a config file for accessories, config file for sync app, config file for an async app. And I'll show you in a screencast why does it make it a little interesting. But before I move into this, accessories, a small word on accessories. Accessories are just basically Redis and DB kind of accessories, like the ones that are not your app but supports your app. So let me see if I can screencast. Run my screencast. Yeah, so, okay, it's a bit dark, isn't it? Oops, yeah. So this is, like, just an accessory file. As you can just see, it's just taking a DMV image, like Docker. It's basically just looking at a Docker file. You can actually make sense out of it. There's no magic to it. And this is a sync app. And I know it's a bit dark, but this side, you can actually see the envars. And these things are very specific to Django, because you're making both sync and async Django work together. This is just the async stuff. And I'll just walk through how deployments can happen. right? So I'm just storing all my stuff in .env. You don't have to do that. You can also do a different secrets manager. So basically exporting the environment variable. And this is where you bootstrap the server. So what happens is you just point to a server and say, bootstrap a new server. Come and install all the necessary things it needs, like Docker and other things. It also makes sure it's able to connect to it. And then you can just go ahead and start deploying your accessories, like for instance, ReadySendDB. And this is your deployment app. So over here, you can just see, you can actually read through it. It's very intuitive that you understand the build is happening. This is the build step. This is the place where you're actually pushing it. Oh, it's a bit fast, but I should not. So this is the place where you're actually pushing it. There's a network that's created. So it's very verbose. You can actually understand. And it's easier to debug to. And what it does is it also does a health check before actually making the endpoint alive. So this is just for the async, the same thing. And since it's your Docker build, in general, it's just very straightforward. It's also caching stuff. Now, the same workflow, you can run it on your local machine. But you could also run it in a pipeline, because it doesn't really matter. OK, let me get back to my, yeah. Oh, that's a screen. So self-hosting. Now, self-hosting means managing your own server instead of relying on cloud providers. I do take a small exception to it in the sense I would still say providers like Hetzner, which give you compute for a decent amount of money and predictable costing, is kind of still a bit of self-hosting, not entirely in that sense, but you're still controlling it. So what self-hosting is predominantly, I mean, going back to first principles, it just basically It gives you more control and flexibility. And control is a big draw, I would say. Cost, also, because there's less unpredictability. You know you're basically just having one server, whether it's your home server or cloud server. You're not basically bitten by bills for your hobby project, or smaller ones. And privacy, it's also important, Because for privacy-focused projects, self-hosting just removes all the reliance on external providers. I don't want to go off-topic, but with the current environment, there's also an element that we are looking towards the future where we might need more control of our data. But there are challenges. It's not set in product. Forget it. You are the sysadmin. Security is entirely your responsibility, like firewall and updates and backups. So you could actually use Camel to do some of these steps. For instance, you could execute scripts with it. But it's still on your control. It doesn't give that out of the box. But it still makes sense in some environments, like running multiple services in a controlled environment, hobbies, of course. Need for flexibility also. I mean, with Kubernetes, for instance, you could actually move things around in the sense if you have a bunch of Helm charts and Kubernetes interfaces are kind of similar across the cloud providers. But there are things like IAM which are still not portable. And so you can just basically deploy on Chrome or on cloud consistently. The workflow is much, much simpler, and it's also flexible, so you can change things around. So don't stop experimenting and create stuff that no one needs. All your ideas don't need to be a million-dollar startup. It's fascinating to build, and it never gets boring. Thank you. Thanks for the interesting talk. Let's take a look if there are questions on Slido. I don't find anything there. So any questions from the audience? Usually, I found that whenever I pull the Docker images, it gives me a rate limit because I'm not a premium one. So what I did is I stored those images in the ECR, and I want to pull those whether that integration is possible. Yeah. I think it gives you a lot of these integrations. Any more questions? Did you end up actually using and setting up the coffee and scheduling the app? Not yet. That's one of my incomplete projects. Thank you. Okay. Let's speak again. Thank you.