Python Hates Being PID 1: Writing Container-Aware Code for Kubernetes

Speaker 1 [00:01]

Good morning everyone. Please, yeah, have a seat. We welcome you to and welcome coverage.

Speaker 2 [00:15]

Yeah, that's right. That's right.

Speaker 1 [00:16]

to our talk, which is Python hates being PID, grading container-aware code for Kubernetes. Kavish is here from, he works in SAP in Berlin. So yeah, I probably ask you for an applause so we can start.

Speaker 2 [00:41]

Thank you. Thank you so much. We'll just jump right into it. Already, as we mentioned, I've been working for SAP for a long time, mostly in enterprise software. And currently, I work for their AI core team. And I know me, I'm a Python expert. So I'll do my best to just explain what I have learned over the past few years. A quick show of hands, how many of you are familiar with Docker? Almost everyone. Perfect. And how many of you are familiar with Kubernetes? Perfect. So we just skip all those slides. Jumping into it, how does Python behave outside a container? So we have a very simple script here. Python is going to start, print its PID, and then it's going to sleep forever. And what we will do is we will run this Python script in a terminal. It tells us its PID. And then we open another terminal and send it a sick term or a terminate signal with the PID. How long do you think this process will take to exit? A few milliseconds, right? That's exactly correct. About eight milliseconds. So very fast. How does Python behave inside a container? So we now take the exact same script, we package it up in a Docker file, and we have the exact same script that's going to print our PID and then sleep forever. We Docker build our container. We docker run it, and now we see that the PID has changed. This time we are going to send it a docker container stop. It's the same as a sick term, a terminate signal. How long do you think this one is going to take to stop? Any guesses? 10 milliseconds? 50? 100? One second. Do I have a one second? All right, one second one, one second two. Do we have a five second? Five second what? have a 10 second. All right. Correct answer right there. It takes more than 10 seconds to exit. Sir, you don't need this talk. You can leave. Why does that happen? We're going to take a quick look at all of that, but it's related to the talk title, which is Python's behavior as PID1. We'll just have a quick look at what containers are. Since you're all experienced with Docker, we know that containers are not virtual machines. They're not separate hardware running inside your hardware. All of it is an illusion that the Linux kernel makes your process to isolate it. It does it using two things, namespaces and control groups. Namespaces isolate your process by giving it its own file system, its own process tree, its own networks, yada, yada. And control groups decide how much resources will be allocated to this specific process. And then once you package all of these things together, you have these processes that look like they're running in a container, but they've just been isolated very smartly. How do they look like from the host? The same example that we ran now that took 10 seconds to die, you would see from within the container that the PID of your Python script is one. However, you can inspect your container and see that it also has a PID on the host, and this looks much normal. So your host is seeing all these processes. It's just within the container that your process has been isolated. If you want to go deeper, there's a wonderful talk by Larry Garfield called The Container is a Lie, fantastic, just check it out. And then there's another talk tomorrow by Alexander Zatsev, sorry, I'm messing your name up, demystifying containers. I don't know the exact content of his talk, but I'm hoping that it's gonna go much deeper into this. Now, why does all of this happen? In a Linux kernel, you have default handlers for almost all signals. For example, the SIGTERM, or terminate signal, has its own default handler, and what it does is, if a process does not have a handler defined for this signal, then the kernel uses its own default handler. And for terminate to terminate, It's going to gracefully ask the process, hey, can you please shut down? On the other hand, we have SIGKILL signal, which all of us know directly shoots the process in the head. And what does that do? It leaves a lot of mess behind. You have child processes. You have threads. You have DB connections, none of which were gracefully closed. You can also check this out in the Python terminal. If you import signal, which is an inbuilt library, If you look for the default signal handler for SIGINT, interrupt, it is defined. You have default underscore int underscore handler, which just goes ahead and raises a keyboard interrupt. You will see that for SIGCHILD, it's SIGDFL. What that means is Python is not going to handle the signal. Hey, kernel, can you try your own default handler? SIGPIPE is SIGIGNORE. That means Python says, I'm not going to do anything about it. Let the kernel decide. And then finally, SIGSTOP, SIGKILL, and sigterm, all three of them in Python, they just respond none, which means there's no handler. Ideally, that means that if your Python process is running somewhere in the user space and it gets the terminate signal, the kernel will decide whether to run the default sigterm handler or not. However, PID1 is special. PID1 is the init process that the Linux kernel starts. And PID1 is sort of the parent process of all processes in your user space. So every single process that you're running is either a child or a grandchild or a grand-grandchild or a grand-grand-grandchild and so on of PID-1. If PID-1 dies, every single one of these child processes also has to die because they cannot exist without a parent process, which is why PID-1 is special. It's immune. You cannot just kill it, which means if you send a sick term, a terminate signal to PID-1 in Python, it will simply ignore it. Actually, Python, of course, ignores it itself because we saw in the previous slide that it does not have a default. But the kernel also ignores it because it knows that this process is running as pid1, and so it should not be terminated immediately. Now, what happened in that 10-second example? We ran our Docker container. Our Python process is happily running. Now we sent docker container stop, which is the terminate signal, to our Python process. Python ignores it. Kernel ignores it. Everybody ignores it. Nobody cares. Docker then goes ahead and waits 10 seconds to check, is this container still running? Is this Python script still running? And after 10 seconds, if it's still running, this time you get shot in the face. This time it's a SIG kill. And as opposed to SIGTERM or all other signals, you cannot handle a SIG kill. If you receive a SIG kill, you immediately die. The kernel just kicks you out. So that is why it takes 10 seconds. Finally, P81 is killed. The moment P81, your Python script, is killed, Any other child processes are also immediately killed by the kernel because they cannot exist without PID1. All right, what are the duties of a PID1 process? Mainly two, one is you handle signals, and the second is you reap your children or you take care of your children. There are two kinds of ways you can mess up both of these. The first one is if you receive a terminate signal and you don't clean up resources properly as we saw in the Simba meme, you might leave a lot of mess behind. The second way is zombie reaping. What happens is, every time you create a child process, it is the parent's process' responsibility to make sure this child process exits cleanly. There are two kinds of ways your child process might not exit cleanly. One is called a dead process, a dead child process, and the other is a zombie process. The difference is very simple. If I, as a parent process, start a child process, my child process is still running, and now it finished, and I, as a parent, am alive, this process is completed. I still have to go ahead and reap it. However, a zombie process is a process that is completed, it's done, but it was never reaped. It was never removed from the process table, it's still holding a PID. Why is that a problem? Why are zombie processes a problem? PID is an integer, and you have a very finite number of PIDs in your kernel namespace. Now, it's not a small number, it's quite a large number. It runs in thousands. However, when you're running really large enterprise-grade applications, your Python applications are forking left and right, creating sub-processes. You have multiple replicas running on the same node. And it's a very likely outcome that you would run out of PIDs if you don't take care of your child processes, if you don't reap them. What I mean by reaping is just a wait system call. When you make a wait system call on a child process, you go ahead and read its return code. And then you remove it from the process table, and then the PID is released. There's a very good chance that you would run out of PIDs if you have too many zombie processes. And debugging that is very, very tricky. Imagine you have a system where all the process IDs have been taken. You want to debug it, so you run psaux. Guess what? psaux also needs a PID, and the kernel simply refuses to run this process. So even debugging something like this gets very, very tricky. If this happens inside a container, you're still fine because you can do something from outside. But if this is your host machine, you have to reboot. And if it's somewhere overseas, you have to call someone. Hey, can you press the power button? It gets really messy. One quick example here. We have the same Python process that sleeps infinitely. And this time, it's going to create a sub-process. It's going to run sleep2 with an ampersand, so in the background, going to run it in a shell. This time, again, we run it in a container. It starts as P81. And now we're going to run psaux on this container to see what process is running inside. Can you guess how many zombies you're going to find here? So we ran sleep two. Five, all right. Lower. Two? Yes, that's the correct answer. Why two, though? So you ran sleep. This is how it goes. Now, I, as Python, I'm running SP81. And we ran process open for sleep2 in a shell, which means the first thing that Python will do is it's going to run sh as its first child. Now, sh goes ahead and forks and creates sleep2. Sleep2 starts running. But meanwhile, sh is already done. So it kills it. It's gone. It exits. The moment sh exits, sleep is now parentless, because sleep was started by sh. And what the kernel does is, the moment a parent doesn't exist for a process, it just assigns its parent as pid1. So now suddenly, sleep, which was a child of sh, now becomes a child of Python app.py. So now sh is dead. Nobody reaped it. Two seconds later, sleep is dead. Nobody reaps it, and you have two zombies. This is a simple example, but this can escalate very quickly. What are the solutions? So this is such a standard problem that you might have seen a lot of standard solutions already. Anybody heard of TINI? TINI? Yes. So TINI is just the opposite of INIT. It's a very fancy wordplay. What it does is it takes care of those two important responsibilities of PID1. One is it will make sure it handles all signals, including sick term. So if it receives a sick term, it will make sure it calls wait on all its child processes. And the second is it will make sure that it reaps all the zombies. So the same thing. It calls wait on all the child processes. You have some options there. You have TINI. You have DUMBINATE. However, even after these solutions exist, best practices would be if you're running Python within a container, always handle your signals because there is no other process on the system that knows what resources you own, what you're gonna do with them, how many DB connections you're open, whether you wanna flush your cache to disk, whatever. Only you know it, so if you write your own handler, you can deal with that much, much better. All right, I'm gonna skip this slide, Kubernetes in 60 seconds, because I think most of us know, But just in short, if you have a Python application running in a container, Zoom, and you want to orchestrate it, you want to deploy it to the cloud, you want to spread it out over multiple geographies, one node running in the US, one node running in India, you want to scale them up, suddenly you go viral and you have 5 million users hitting your website, you want all the scaling to happen automatically, Kubernetes does that for you. But the scope of this talk is looking at how is your Python application constrained within a container. So we already spoke about namespaces. Namespaces isolate the PID. So if your Python process is PID 1, it's going to face certain problems. Now let's look at control groups, C groups. C groups decide how much CPU and memory you get as a process. In Kubernetes, you do this with YAML. All Kubernetes developers are YAML developers. They write YAML files. And at the bottom, you see limits CPU 500M, memory 256 MI. What that means is, this specific container running your Docker image is only going to get these many resources. We will talk a bit more about what that 500m means, but for memory, 256mi is straightforward. Now, let's do another thing. I am running a Python command that prints os.cpu count. And I have a M1 Max machine, so it has eight cores to performance cores, so total 10. Seems fine. I will now run the same thing within a Docker container, but the CPUs are restrained to 0.5. So this is happening with cgroups. This container is only going to get 0.5 CPU. We'll talk a bit more about what 0.5 is later. How many CPUs do you think this process is seeing from within the container? Exactly. It's 10. Why is that? It's very tricky to figure out what the host is doing from within the container. And this is by design. You don't want the container to have too much information. You don't want somebody who's renting a room from you to know where you keep the gold in your house. So the containers are intentionally restricted that way. However, there are some other languages that have solved this problem, figuring out how many resources exactly this container has been given. I think Go does it. I think all JVM languages do it, but Python is still figuring it out. What exactly happens when you assign a specific restricted amount of CPU to a container. So in this case, assume the CPU you gave to your container was 100 M. There are two things that are in play here. One is something called a CFS quota, and one is something called a CFS period. CFS stands for Completely Fair Scheduler. It's a Linux kernel thingy. It is what decides which process gets how much CPU in a specific period. This is how we can look at it. The image is actually quite helpful. Every period of CPU run, in this case 100 milliseconds, is used as a quantum for deciding how much time a process gets to run on it. So in this case, if I give 100 m to my container, all the processes within that container can only run for 10 milliseconds in every 100 millisecond batch, Which means, in a 100 millisecond batch, if my container has already, if all the processes in my container have already used up the entire 10 milliseconds, they're done. The CFS schedule will no longer schedule them for this period, which means the remaining 90 milliseconds, they can't do anything. This is called CPU throttling, and if you've seen any dashboards with Python applications running on them, you will see this very frequently. This is what it looks like. I'm sorry. Yeah, this is what it looks like. Every few milliseconds, you will see a burst of CPU activity, and then your process runs out of its quota, and then the kernel immediately throttles it. It then waits for the quota to reset, and then it again gets a burst of activity. It runs for a few milliseconds. It runs out of quota. Again, it's throttled. Let's take a good example here. So we have two examples. One is a single Python process with two threads running, and the second is two separate Python And we'll see how exactly this throttling happens. Assume you have a single Python process with two threads, and you have given this specific container in which this process is running a quota of 500 M. What that means is, for every 100 milliseconds, this container and all the processes in it will only be allowed to run for 50 milliseconds. What does that look like? Now, thread A starts running. It immediately grabs hold GIL, global interpreter lock, And GIL makes sure that you can only run one thread at a time. That means thread B can no longer run, because it does not hold the GIL. Thread A runs for 50 milliseconds, completely exhausts the quota. And now, even though thread B could run, it cannot for two reasons. One is it does not have GIL. And two, the quota is already exhausted. There is nothing the CFS can do for any other process in this container to run. So now everybody has to wait for the remaining 50 milliseconds until the entire quota is reset. And after those 100 milliseconds, thread A releases the GIL. This takes about five milliseconds. Thread B grabs it, and now thread B starts running. So you can see a lot of wasted CPU power happening here. In fact, the previous slide that I showed you with all the throttling, you will notice that the CPU usage never goes above 2%. But it's the C group CPU limit that you've put on your container that's causing all the throttling. So you might have a lot of resources, but if you don't configure them properly, your Python application will starve. Let's look at multiprocessing. So now you have two Python processes. Both of them are now independent, which means GL is no longer in play. So both of them have their own global interpreter. They don't mess with each other. Process A and process B both start running in parallel. Both run for 25 milliseconds. 25 plus 25, 50, the code is exhausted. And now both of them are kicked out. Contact switch happens, both are kicked out by CFS. And now for the remaining 75 milliseconds, nobody can run. This is even more severe throttling. I personally have written code where I do multiprocessing, create workers, as many OS CPU accounts as possible. I have 10 CPUs. I created 10 processes. Imagine what happened here. I have 10 processes running. In an ideal scenario, all of them start running at the same time. I have only 50 milliseconds, which means all of them probably only get, on an average, 5 milliseconds. And then for the rest of the 95 milliseconds, all of them are throttled. So extreme wastage of CPU. We've already seen this slide. I'll go ahead. This was the CPU side. Now we will look at the memory side. The CPU side is a bit forgiving. If you try to overuse it, you will get throttled. If you try to underuse it, fine, you're underusing it, but memory is much more dangerous. You've got a teeny tiny bit above the memory allocated to you, you're immediately sick, killed. Pandas is very, very famously memory hungry, just keeps loading a lot of stuff in memory. But here's an example. I am running docker run again, and this time with cgroups, I'm restricting it to 50M, 50M of memory, 50 megabytes. Now, I use psutil and then just print the total memory allocated to me. You will notice it's huge. It's 8 GB. That's how I've set up my Docker desktop. However, if I run Python again and try to allocate anything above 50 MB, this container immediately dies. The kernel immediately sick kills it. How do I know? If I just print the exit code, it's 137. If I also run Docker PS, I see exit code 137. 137 is 128 plus 9. 128 is the base offset. That means it was killed by a signal. And 9 is sick kill. I'm reading it out because I learned this just a few days back. So that's that 137. And in Kubernetes, 137 is 99.99% out of memory killed. How do we fix this? How do we correct this? From within a container, how do I figure out how much CPU do I have? What is it restricted to? How much memory do I have? Unfortunately, no standard way. We already spoke about it. It's quite tricky. And every container runtime does it differently. Docker does it differently. Containerd does it differently. I think they both do the same. But every other container runtime, it will do it a bit differently. So it's very difficult to standardize this. Each runtime container orchestration may have its own conventions. But as long as you are using Docker or Containerd, you can do something sneaky. you can go into these cgroup files and directly read them to see how much quota was allocated to you. And these are those paths, specifically for Docker and Container D. sysf is cgroup cbmax. It'll give you two values, what is your quota and what is the period. So for example, 50 millisecond quota, 400 millisecond period. And you also have memorymax and memorycurrent. Memorymax will give you how much memory your container is restricted to, and memorycurrent is how much memory your container is currently using. And then you can use it to do something very fancy. Imagine you have a process pool executor and you want to spawn processes with it. We already spoke about the case where if I use the value of OS CPU count, I might immediately spawn 10 processes and then all of them will get starved. But you can do something better here. What you can do is you read the CPU max file. Now you know the quota. Now you know the period. You can do some basic math. By the way, there's a lot of blogs that talk about worker mathematics and they tell you more fancy ways and fancy formulas to come up with ideal number of workers. But here, all I do is I divide the quota with the period, and then I max it with one, so I have at least one. And then I only create that many workers. So in the previous case, I would create 10. All of them would be starved and throttled. In the second case, I just create one. And now I know that this single worker will get the entire CPU allocated to it and will not be throttled. Just an example. What can you do with the memory? So in Kubernetes, you have these health endpoints, where Kubernetes keeps pinging you at a certain duration to check whether your pod is still alive. And if your pod is not alive, it gets killed and restarted. What you can do with your health endpoint is something we have also tried to do. You read your current memory, and then you calculate something called a memory pressure, just a fraction, a fraction of how much memory am I using out of the total memory allocated to me. And if the memory pressure exceeds a certain threshold, then you can start doing something about it. Because OOM kill is unwindable. The moment Kubernetes kills your pod with an OOM kill, your pod immediately dies. There's nothing you can do about it. You cannot release your resources. You cannot close your connections. No flushing, nothing. So a good way that you can take some control back by keeping an eye on how much memory pressure your application is currently experiencing. For an example, let's say you have an Ellaru cache. You keep putting stuff in the Ellaru cache, even though it's capped at some limit. But you can keep an eye on your memory pressure. And if your memory pressure, say, goes about 85, you just clear your cache. And now you can be certain that your pod will not be OM killed. So that was it. Just a quick readiness checklist. First thing is, if you're writing doc files, don't give Python PID 1. Use TINI. Use DUMINIT. Anything else that can do a better job than Python. Signal handling. Always register sick term because it gives you an opportunity to clean up your resources instead of just leaving them around on the OS. Zombie reaping, if you're creating sub-processes, call the wait system call on them so that the PID that was held by it is released. Write some code if possible so that your Python app is CPU and memory aware so that you have more control on exactly how it is running within your container. And then finally, observability metrics are very important. Imagine you have a dashboard that tells you how much your CPU is throttling. The moment you see those peaks, you know exactly what's happening, And then it's very easy for you to diagnose the issue. So that was it. And we are open for questions. Thank you. Thank you.

Speaker 1 [24:40]

Thank you so much for the talk. That was enlightening for me. I would like to ask you to question Karish.

Speaker 2 [24:51]

No hard questions, please.

Speaker 1 [24:52]

Using, yeah, using the talks, you know? Are you aware of? So when you go in and talks.pycon.de, you'll find this talk, and we encourage you to ask questions, not only for this one, but for everyone, using the platform in such a way that we can also share the questions with the ones that are remote so yeah I'm right here waiting for yeah okay we have our first questions how do you keep track of all the things you need to clean up in your sick term handler

Speaker 2 [25:45]

It depends on you, what your application is doing. For example, if it's a DB intensive application, you know you have some connection pool that is open, and instead of just leaving it to the kernel to get rid of it, leaving it to the database to get rid of it, what usually happens is, if you have a lot of DB connections open, and your Python process does not close them and simply exits, now it's up to the database to decide how long is it going to keep this connection open before it closes. And every database, RDBMS, every database system will have a quota of how many open connections can have. So this has happened to us before where we noticed that we were exhausting our DB connection quota limit and suddenly then our applications would stop working. What was happening was we were simply not closing them. So it depends on the application, what exactly are you doing and then you can decide what exactly to clean up on sick term.

Speaker 1 [26:34]

Thank you You said container runtime have different conventions The group files are run runtime agnostic because they are implemented by the kernel, right?

Speaker 2 [26:52]

Yes. Yes. Oh, that's the question. That's correct. C group files are specific to a kernel. However, if you don't know exactly which C group. C group has C group v1. We have C group v2. Also, your container runtime might decide that it wants to do things differently. So that's the only reason we said that it depends on exactly what runtime it is. To be safe, I said that this is how it works in container D, because that's what we work with. But I have no idea how another container runtimes do it. Also, if all of them had the exact same convention, this would be a solved problem. So I'm not sure what's happening there. There's a huge CPython thread on why it's not working and what exactly is going wrong. But yeah, feel free to let me know if you find. Maybe we come up with a library that solves this problem.

Speaker 1 [27:42]

thank you the next questions I'm not sure if I can read correctly what about this is Python specific and better solved by other languages how do they do it

Speaker 2 [27:55]

Good question. I tried to look it up, but I did not understand it that well. What I know is for Golang, Uber wrote something called GoMaxProx. And what GoMaxProx does is, I don't want to give incorrect information, but it somehow, during runtime, injects this information into your Go code as system environment variables. And then you can just read those environment variables. So we could do something similar. For example, if I know that I'm defining my Kubernetes pod to have only 50 M CPU, then I also create an environment variable saying max CPU available as 50 M, and then from my Python app, I can easily read it. But then this would be too tedious to manually deal with. So I'm assuming that's how Go does it, but I don't know.

Speaker 1 [28:45]

Thank you. Import man-ruthless containers, I believe PID1 is not used by default.

Speaker 2 [28:56]

In Podman. Podman, rootless, okay.

Speaker 1 [28:59]

Yeah, would this not help us avoid the sign language issues?

Speaker 2 [29:03]

issues. Yes, not only Portman, but even in Docker now, because this is such a frequent problem, Docker run gives you an extra flag called double dash init, and then it will just do this automatically for you. But this just solves the PID1 problem. You're still not solving the problem where you still have to deal with open resources, cleanup of open resources on sick term. So if you're not PID1, you will get a sick term signal, and if you don't do anything about it, you will gracefully die, but you still don't get a chance to clean things up. So yes, It deals with the PID1 problem, but you still have another one to solve, which you have to solve manually.

Speaker 1 [29:36]

thank you so much so we might have one minute more for another questions or there another question otherwise yeah we can give me a second yeah so we appreciate your your talk thank you so much courage thank you

Speaker 2 [29:57]

Thank you so much for joining. Have a good day, Ed.