Demystifying Containers with Python: Building a Minimal Engine from Scratch

Speaker 1 [00:00]

the people who are coming arrived yesterday or yesterday late or today so the qa session will be in the talks pike on the e-talks we will not do the live qa session so all the questions please give in the the talks okay and we are also very happy if you after talks you share in the social media, and tag our speakers, and PyCom, that's really welcome. Okay. Okay. That's good. Okay. Let's start. So today we have a very interesting topic. is demystifying containers with Python, building a minimal engine from scratch. Let's give a warm welcome to Alessandro.

Speaker 2 [01:14]

Hello, people. We'll be talking about containers and how they work, but first let me start with telling you about my journey with Python. I studied particle physics, and back then at university, we didn't use much Python. We used C++ and sometimes even Fortran. I was introduced to Python at my first job as an R&D engineer, where I was analyzing data in Python and writing data filtering algorithms for laser gyro compasses that we were building. Then I did my PhD in particle physics in dark matter searches, and this picture is from those times. It's taken 2,000 meters underground in the deepest lab in the world, and behind me is a fridge that can go down to temperatures 100 times lower than the temperature in space. So it's pretty cool, quite literally. And there we used Python everywhere. We used it for data analysis, for modeling, for controlling some of the equipment in the lab. And it is there when I started doing software development, writing the data analysis framework for our experiment. And since 2025, software development with Python has become my full-time job. I'm now at Blue Yonder where I work with machine learning infrastructure for demand forecasting. Transitioning to IT from academia, I realized that there are a lot of computer science concepts that I've been using in academia, but I didn't really understand how they work. And containers is one of those. So I used them, and my mental image of containers was like, well, it's sort of like a lightweight virtual machine, but it's really not a virtual machine. What does it even mean, right? Believe it or not, this mental image was enough for me. I guess I was worried about how charges propagate in my silicon crystals and didn't worry much about how containers work. But now, when I don't need to worry about charges anymore, it is time for me to learn how containers work. And I thought what is the best way of doing it is, of course, to implement it in in its simplest form in Python. That's what they're going to do. So what is a container, and why is it not a virtual machine? What is the difference? The difference is that the container is running processes on the same host kernel as all the rest of your processes. And let's prove it. I have a Linux virtual machine here, and I have a Docker installed. And let's run a container and run a top command in it. And now let's see what we see from the container. We see just two processes, the top and the PS that we just started. And they have PIDs of 1 and 8. But now let's see what we see from the host. We see the very same top process, but it is running as PID 5504. So what's going on here? Why does this process think it's a PID 1 when it's actually not? And why don't we see all of the other processes when we look at it from the container? The answer is Linux namespaces. What is that? Let's just go to the main page and read this together. A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. One use of namespaces is to implement containers. Oh, it looks like we are on the right course already. So if you continue reading that main page, you will see that there are eight types of namespaces, each of them isolating their own resource. And resource here has a more general, like it's using a more general sense. For example, a host name is a resource, and you can have a UTS namespace that isolates the host name, meaning that if you create a UTS namespace and change the host name in that namespace, it will only be changed in the namespace. It will not be changed in the host machine. OK, and there is a PID namespace to isolate process IDs, and that's what we've just seen. There is a mount namespace to isolate file system mounts. There is a network to isolate IPs and ports and network interfaces. There is a user to isolate user and group IDs, IPC for inter-process communication, cgroups we are going to talk about a bit later in this talk, and there is a time namespace to isolate the system blocks. To create a namespace, you use an unshare command. And it goes like this, unshare, then you specify the types of namespaces that you want to create. You can create multiple with the same command. And then there is a fork option to start the process as a child in that namespace. And then you can specify the commands that you want to run in the namespace, and if you don't specify it, you are going to be running a bash shell by default. Let's try this out. Let's do... We will need sudo for that. Let's do unshare pid, all right. Let's now see our process ID. It's 1. We know it's not 1 for sure because init is 1, right? Let's run ps command. All right. What's going on here? We still see all of the processes, and we still see our shell as 5.5.6.0. Okay, what's happening here is when you do PS, you're actually reading the proc file system, and the proc file system is even when we are in this namespace, proc file system is, let's go to this one, it is still the proc file system of the host. We want to mount the new proc file system, but for that we don't want to affect our host by that. So for that we need to create a mount namespace. Let's do that, all right. So we still see everything. Let's mount the proc file system. So it's going to be type proc, and the device name doesn't matter, and we'll mount it right on top of the proc file system of the host, because we are in a different time space, so we are not going to affect anything. Let's go and see if it worked. Okay, now we see just processes 1 and 14, and now it works. And if I go back to host, and you'll see that my proc file system is still the same. So now it looks a lot like what we've just seen with the Docker container. There is one problem, though, if I go ahead and touch a file here, right, I will see it right here. That's not what we want in the container. So we isolated the mount namespace, we have our new mounts, but we did not isolate the whole file system. We can do that with the command. So this command, if I run it, for example, in home vagrant, this would change the apparent root of the file system to that directory, and it would run a command in that root jail. So let's see if it works. Okay. We will need sudo again. And let's go to home vagrant. It says binbash not found. How is that possible? Of course, binbash is there, but it's not in my chroot directory, right? Because we are doing the chroot, we are changing the root file directory, and then we cannot see our binaries anymore. So if we want to chroot to this place, we want our binaries, and the binaries need their libraries. So what we need is we need the image. What is an image? Let's review the Linux architecture real quick. So we have a machine with an architecture, and first of all, we have a kernel running there, and the kernel talks to the hardware, manages the processes, manages the memory, and the kernel exposes a system called interface, and that's called a kernel space. And then we have our C libraries that are talking to the kernels of the interface. We have the core utilities, like bash, ls, and all stuff like that. And we have all of the user applications, configurations, all of this stuff. All of it is sitting in our root file system. What we can do is we can package the libraries and the binaries and put them somewhere else on the file system. We can even put them into a tar file and upload them to internet internet, and this is exactly what an image is. And you can have as many of those as you want sitting in different places in your file system, and they can even use libraries from different Linux distributions. And the secret sauce here is that the kernel interface is very stable, right? So you are going to have different libraries using the same kernel, no problem. So you could now into this place. So if you put your binaries and libraries into some place on your machine, you can truth to this and you are going to have the binaries that you need. Okay so what we need, we need a namespace, we need an image, and we need to truth into this image. Let's implement all of this in Python, starting with the image. So we are going to straight up hard code an Alpine image, that's a minimal Linux distributions that you can get. It has binaries and the core utils. So we'll just implement a pull command that would download this and unpack it somewhere on our file system, and then we would expose the path to that place where we unpacked our binaries and libraries. And now the heart of our container engine is going to be container.py module. Here we implement a run command that gets the path to the image. And then runs the unshare command, creating the mount and PID namespaces, the UTS namespace here as well. And then it starts the Python executable again, and it starts the very same module. As we defined here, this would start a child process in the namespace. So this is now in the new namespace, isolated from the rest of our host. And here, we can change the host name, set our prompt so we could see our host name, and then which route into the image, and mount the proc system, and execute the command, whatever command you want to execute in that container. And let's implement a simple CLI with a pull command to pull the image, the hard-coded image, and then a run command that would run this container and run the specified command in it. Let's go ahead and try that. So we called it pycont. We will need sudo to do all of that. So I pulled it already. Let's run it. All right, we are now in the container. You see the host name changed. Let's see our process ID. All right, looking good. Let's look at it from the host. There we go, we have it running on the host still, of course. And we can, for example, let's see release, okay, this is Alpine Linux because we are not on the host anymore as far as the file system is concerned. So looks like a container, right? There is a problem. Let's start another one. All right. And let's do this. Let's create a file. And, of course, file is right here, because we are in the image file system. So whatever we create here is going to appear visible for any other container that we start from this image. We could copy the image when we start the container, right, and delete it when we are done. That would be very inefficient. There is a better way, and this better way is the overlayFS, overlay file system. That's another feature of Linux. What it lets you do, it lets you combine two directories and create a view of them, and the lower directory in this lingo is going to be read-only. That's exactly what we want. We want to define our image directory as read-only. Then we create a completely empty one for our container, and that's going to be called an APA in the overlay FS lingo, and then we mount the file system, and this creates a view of the union of these two directories. If we want to create a file in a container, we would create it here, and it would be actually created in the APA directory. If we want to edit the file here, what would happen is copy and write, so the file from the lower file system would be copied into the upper one, and the edit would apply there. So that's cool. What if we want to delete a file from here? So we cannot delete it from here. So if you delete it from the merge view, what would happen, a special white-out file would be created in the upper one, and then this view would not show you this file like it's never existed. And you can do the same with the directory, that would be an empty directory with a special attribute saying that it's a white-out directory, meaning that it should act as a directory never existed here, all right? So after we are done, we can unmount it, and then the upper directory would remain, and when you mount it again, you would again see the very same view, without ever touching the image. Okay, so how we do it, you do the mount, minus T, like the type of the file system overlay, and then you specify your lower upper directory, there is also a work directory, and this is like a temporary directory to do all the changes atomically, right, and yeah, you merge it, you specify the merge path. Okay, so let's put it all into our engine. We define a new module, create a path where our container directories are going to live, define a setup where we create the upper work merged directories, and then we run the mount command in here, and we implement the clean up to delete all of this afterwards. Let's now put it into the heart of our engine. So what we do here, the change here is instead of using the image, we are now using, we are now calling the overlay setup and using the merged directory to truth into. And here we have to change the, add the container ID because now different containers have different paths. All right. Let's see if it works. Okay. Let's exit all the containers. here, what are we going to do, we run the very same thing, okay, nothing here, I don't I want less, I want touch, okay? All right. It's working. Let's exit. Okay. Looks like we now have the isolation. There is another issue we need to address, and it's a noisy neighbor issue our containers can affect each other and the hosts in another way they can consume all the resources available on your host they can consume all the memory consume your cpu or another interesting way they can start creating new processes in an infinite loop and consume exceed the maximum process number on your host to deal with this problem you need to implement and limits of how much resources each of these processes is allowed to use. And you do it using the cgroups feature. So this feature allows you to create groups of processes and assign limits on the resources that these processes can use. The way you interact with this feature in Linux is via pseudo file system, cgroupfs. So you are not doing any syscalls manually. you are just interacting with the file system, and the magic happens behind the scene. So to create a new C group, you would just create a directory in this FSC group, right? If we do it, we just create an empty directory, and the kernel just creates this view. There is a lot of things in here. One is C group controllers, and here you have the resources that you want to limit. So you see all of them here, CPU, memory, all of the stuff. So there is a memory max, memory swap, PID max. So if you want to limit the number of processes this group is allowed to use, you would just write a number into this file, and that would work. After you do this, still nothing happens. You need to add the processes that you want to use this group via the cgroup-prods. So you just write your PID of the processes that you want. And it's going to be here. And all of the children of that process are also going to appear here. And cgroups is also hierarchical. So you can create subdirectories within the cgroup. And to control which controllers the children are going to use, you use the subtree control file. So you write something like plus memory in here, and then if you create a subgroup, then in that subgroup, only the memory controller is going to be used. Oh, thank you. All right, let's put it into our engine. So we create the new cgroups module. We write a setup which would create the new cgroup. Depending on which resources we want to limit, We would just write plus and the controller into that file, and then we would write all the maximum amount of memory that we allow these processes to use into the files. Then we implement the enter method that just puts the PID of your process into the C group's procs, and then some cleanups method to delete all this afterwards. So going to our container module, We are now adding all these arguments to limit the resources. The new thing here is that before starting the namespaces, we are setting up the cgroup. And yeah, now we have to provide a lot of new arguments in there, so there is some boilerplate. And we are adding into the finally clause all the cleanup. And in the child that is running in the namespace, we are entering the cgroup. We are putting the PID of the child into the C group. Right. Okay. And, of course, we need to add all the arguments to our OCLI. Let's see. This is going to be demo three. All right. Okay. How are we going to test it? Let's do let's allow only two processes in this container. And now let's just start another bash from here, and let's try running anything here. Nothing works because both processes are used up. If I exit one of them, then everything works. All right. So now we have control of how much resources our container can use. Another issue we need to talk about is that in Docker, all of the processes are running as root. And this is because true namespaces overlay all of these privileges, right? So when you install Docker, you actually get this warning message saying that, well, everything is running as root, right? So if you escape your namespace somehow, or escape the prison of the container, you have the root privileges on the host. So that's a security nightmare, and this happened before, there was an exploit with this. So what you can use instead, you can create a new user namespace, and you can create a provide this argument that would map the UID of your user, non-root user on the host, to UID 0 in the namespace. And in that namespace, you would be root. But outside of it, you would only have the privileges of that user that started the container. So yeah, to implement it, it's pretty easy. We just say that, okay, if we are running not as root, root, then we create this new user namespace. Okay. And also, there is a caveat that cgroups always needs root privileges, so if you're not root, you cannot create your cgroup, so we are not going to do it. And let's try it out. And now let's run it as non-root. Okay. But let's look at it from here, from the host, and from the container. Container still thinks it's root, but the host knows that it's not. So that's a pretty cool thing that Docker doesn't do by default. You have to specify an option for that, but some other containers do it. All right. So the other caveat is that unprivileged user namespaces are disabled on some Linux distributions. And if you want this feature, you have to go ahead and enable it manually when you install some of the containerization engines that happens for you during installation. All right, so let's wrap it up. There are many things that we didn't cover. We only looked at the basics. And the things that we didn't touch at all is networking. So that's another namespace. and you need to create virtual Ethernet ports and connect them. So we didn't do that. Also, we didn't talk at all about OCI, so Open Container Initiatives, that it's a specification for images. If you follow this, you can download any image from the internet, and then you would know how to unpack it, because there are layers in the image implemented by overlayFS. We didn't do that. Also we didn't mention that trued jail can be escaped, so actually there is a slightly different mechanism used in production-grade containers, so it's not trued. It's a pivot route, I think, but we used a simple one and it's not very safe. What we did cover, however, is that processes in a container, they run on your host, and this is done by the namespaces feature of Linux. You can do chroot to get a new root file system. And you can use overlayFS to have a read-only image and to copy changes on the right. Of course, cgroups to limit the resources. We talked about rootless containers. And all of this can be implemented in Python with no dependencies, no daemon, no magic, just pure Linux commands. And most importantly, the takeaway from me was that if you want to understand something, you could just implement it in its simplest form in Python, and you will have fun, and I guess for me personally, you would feel like now you know how it works. Thank you.

Speaker 1 [27:27]

Thank you so much, Alexander, and we have several questions, so two similar questions That's the talk about the containers work on Linux But how about on Mac or Mac OS or Windows they don't have the namespace and C Groups doesn't exist. All right. Yeah

Speaker 2 [27:48]

When you install Docker on your Mac, you are actually installing a Linux virtual machine and you are running the container in the Linux virtual machine running on Mac. So the containers that we use are Linux containers and they use Linux features.

Speaker 1 [28:07]

Thank you. The second question is, you mentioned there was an exploit that allowed escaping the Docker prison. How does it work? And is 4VM more secure?

Speaker 2 [28:25]

So how it works exactly, I don't know, but what happened is that due to a bug in the kernel, I guess, your process escaped the namespace, and then since it was always a root on host, it got root privileges on your host. and whether VM are more secure or not I guess they will also escape from a virtual machine it's just an exploit of a bug you're never safe from a bug like that but you can what you can do is you can allow a process to have the least the smallest set of privileges that it needs and this is what the rootless container lets you do So, it's never root.

Speaker 1 [29:16]

Thank you. We still have many more questions. We still have two minutes for the questions. Do rootless containers run into issues running certain syscops? Say that again? Do rootless containers run into issues running certain syscops?

Speaker 2 [29:37]

Okay, that's pretty interesting. So when you enter the namespace, in that namespace, you are still root. You are still doing all the syscalls as a root. You would ask, okay, but why can't we now escape and be still root? Well, no. When you escape, you are not root anymore. That's the point. You are only root in that namespace. With the containers that you run as root, run as sudo, you are actual root on the host. So you are not only root in the namespace. So that's the difference.

Speaker 1 [30:13]

Okay, thank you. We can take one last question and maybe the rest questions, maybe you can answer on our side, you can meet Alex. So last question, how do you limit resources of rootless containers if C groups require root?

Speaker 2 [30:34]

Good question. I did not manage to do it. So I actually don't know if Podman allows to do it. Oh, okay, no, okay, yeah, I know how it's done. Because when you install Docker or Podman, you have a daemon that is running as a root. So what happens is daemon creates a cgroup for you, but then it starts the process in the container as non-root. So to do this, you can do it, but you do it in a separate process.

Speaker 1 [31:05]

Thank you so much, Alex, and I learned so much. And, yeah. Thank you.