6 Years of Docker: The Good, the Bad and Python Packaging

Speaker 1 [00:04]

Yeah, welcome everyone. So when the schedule came out, it was like, oh, that sounds like two completely identical talks. So maybe we have to distribute somehow. And yes, indeed, I also wanted to give a short introduction into what Docker is. And gladly I can skip that over because there was a very good introduction before. So I can concentrate on what I'm interested in most and that is ranting. So I will concentrate on what's the bad thing maybe about still 2019 Docker and especially Python. So, who read my abstract maybe already knows, maybe some others do not know, it is a little known fact that actually Docker was first shown to public at a PyCon, so exactly at the same conference like this here, but not in Germany, in the US, and it wasn't a keynote or something, it was a lightning talk. so who joined the lightning talks yesterday so go there i mean sometimes there's awesome stuff presented i mean that was somehow a revolution so this year solomon hikes and he showed in five minutes a new thing and he said oh it's something with containers and um yeah so he yeah he did this live demo and there he showed okay docker run i mean it looks exactly the same like now six years later so it was quite impressive impressive at that time so people were like boom what's happening there what can we do now the world has changed remarkably in the exact same lightning talk one minute later he already threw out the first garbage into a container so he misspelled hello world and I think this should should go to into a museum because I mean that's that's kind of awesome right and it's printed so many times so i think that's that's really great so uh a very short introduction but i will only concentrate on on the things that are relevant to understand the the all all the stuff uh behind it so we already heard in the talk before docker is not the same thing as a virtual machine so here if we want to depict what a virtual machines what a virtual machine is it's really an emulation or simulation of a whole new computer inside a computer so it's a virtual machine that's why the name is like this so uh we really virtualize things like cpu memory disk and there's a hypervisor doing all the things so in the end it's really like we heard already it's a it's a new operating system emulated by a computer. Docker is something completely different. So in the basics, it's something like, I mean, people who are familiar with Linux, they knew things like change root a long time before. So in principle Linux is just a file system. So if you add a USB drive into your computer, a file is created somewhere slash dev and stuff. So you can have a new operating system just by copying all your files and move it to somewhere else here into a subtree. And then you can do a command doing sub change root. And now you have a new operating system. But it's just new files. I mean, the same kernel is still running. And then Docker, just as we already heard, is just adding some stuff around it. So, things like cgroups for isolating resources like memory or CPU or networking stuff and namespaces to separate the processes so they cannot really see each other and things like this. So, more or less, it just adds up such that it feels like a virtual machine, a complete new operating system, but it is not. It is the same operating system, it just uses a different file system there. so I like this picture a lot so you can think of containers a little bit like this so there's the stick in the middle which is the kernel and each container is just like a ring so they share all the same container whereas in a virtual machine you would have many sticks one stick for each new virtual machine now there's a problem so I told you that docker is not a virtual machine but docker is not a virtual machine on a Linux system most of you I guess will not have Linux on who has Linux on on his local okay that many so roughly a third maybe so two-thirds have a machine like Windows or or a Mac and indeed I I mean, at least at each and every single Docker container is running in a virtual machine if you're using something like Mac or Windows. So that's a crazy situation. You build up a virtual machine, and in the virtual machine, you install Docker, and then you have these many containers inside the virtual machine. And, of course, this creates problems. So, for example, I would concentrate on Mac because I have Mac. but in principle it's the same on Windows so Docker on Mac, if you just install Docker for Mac then there is a special VM, it's created just for the purpose of hosting these Docker containers and at least in my experience, so most problems if you have any, is the bottleneck is the mounted file system somehow, so somehow file system access from your host machine to your docker containers so it's not about the cpu usage of these virtual machines virtual machines are really efficient these days so it's not about oh my virtual machine is eating up all my cpu most of the time if you look into it it's just the file system somehow the bottleneck and this is why the you can improve the situation a bit so if everything is up and running and you have no problems then be happy there is no problem for you but i mean for for example in our company we have most of the developers have mac and there is not a day where there is no complaint about anything concerning docker so one remark is just in the docker program on the mac you can decide which parts of the file system which folders are mounted into the virtual machine so not into each volume in the docker container but for the whole virtual machine and normally it's something like users so everything is mounted into the virtual machine so if you experience some problems it might be a good idea to reduce it to really just the part of the file system which you're actually trying to use inside the docker containers for example the folders where you where your source files are and then also as use as little volumes so if if you want to attach a volume into a docker container then also be sparse there so don't just mount your whole file system into your docker container for convenience because then it's there and you can access everything just mount the sub part what you're using inside the docker container then there are some i mean this goes it really goes into into details and it goes so far that i haven't really understood it so there are two types of flags you can attach to to more volumes that you mount inside the docker container so there is this thing called delegated and the thing called cached and more or less the recommendation says something like delegated has the lowest consistency guarantees so there is some eventual consistent. So you create a file somewhere in the container and only maybe a second later it pops up on your host machine. Whereas if you don't use any of these flags, it's really synced. But of course, I mean, if you do this syncing, that becomes a bottleneck. I mean, if you write many, many files, it will become a problem. And then cached, it has some stronger guarantee, so somehow it means the container lags behind. So if you create something on the host file, it is only a second later in your container. But there's also a completely different approach than mounting actually your file system inside the container. There's, for example, Docker Sync or rsync. So this means you just sync your files with a command. So, I mean, most of you might know rsync, for example. So you can just say, okay, so if there's a file newly created here, just sync it to the other side. Or if there is a change, just sync it over there. so um now there is the next crazy first step so we we are here at the python conference so we want to dig deep into into the python part of the whole thing so the first thing people will be like what is if somebody um will install a virtual env inside a inside a docker container people are like why would you do it I mean virtual and means I want to isolate my dependencies somehow but if I have a docker container already it's isolated already so I can install whatever I want there so why should I install a virtual ENV in the first place but it's as always things are not that easy so I would say it's indeed it is better to use virtual ENV in inside a docker container because i mean even if it's a container it is a full linux running there and linux so depending on the distribution it will use some python for its system things that it's that it is doing and so for example the python versions are somehow coupled to the distribution so maybe you know debian buster it will ship python 3.7 i mean it will also ship python 2 because some dependencies still have Python 2 but so there is some I mean it's not a strict coupling but if you would for example delete the Python 3 7 district version in inside a container which is running Buster there might be a problem so maybe you don't realize it immediately but maybe some point later you will figure out oh something breaks down so it's easy to break this up and and to to yeah to break it so just never figure with the system python and also not with the system python dependencies so of course if there is a program and it installs dependencies python dependencies you should also not mess up with them so it's it's it's in any case a better idea to use your own virtual env even if you're inside a container then um yeah so that's the point so try to isolate also your dependencies from the system dependencies there is an alternative to virtual and we already heard it as well so there is conda it is quite interesting so it's more or less a package manager plus some distributions so there is a repository let's say and some environment manager which would be the alternative to virtual and so you can somehow create multiple virtual which are then isolated to some other python installations the the interesting thing is that it is also keeping track of all the non-python dependencies so for example things like numpy scipy and that's also why people really like to use conda because if you use conda mostly you don't have these problems we we heard before like there's all of a sudden things blow up so if you use alpine there might be some crazy stuff happening you might be better off to use conda inside alpine i never tried it but i guess if you if you encounter such problems try it with with conda but i mean the last point here admittedly it feels even more crazy because you're shipping all the binary static files bring it into your container which already is more or less a container for the all these files already so this is really some inception thing so if you think about it for example i mean a problem would be something like you you have there is a there's a severe vulnerability in in open ssl for example i mean you patch your your docker container but still it might be that conda ships uh partially one of these libraries as a static binary in their packages so i don't know if it's the case but these things could happen so you think you're running a safe thing but in conda it's packaged up and it's running an old thing inside there okay then it comes back to if you're running on on a non-linux box uh please please i i think it's it might be even the same on on linux i don't know i haven't tried it um but i i think it's not the same problem so for example you if you build this virtual environment don't put it into the same directory where you mount your source file so it is about local development maybe i forgot to to tell you so the main focus here is i want to to to talk about the part where you say i want to develop my application in a docker container on my laptop so it's not about shipping it into production so of course we want to leverage their containers as well but for this talk i want just to so if i want to do local development for example you can put it somewhere like temp event or something it will speed up a lot I made it in brackets so these are of course things which might be cargo cult so you know what cargo cult is so you do things because you did it always and people said to you like the druids said it speeds up your local docker thingy so I mean you would have to try it out sometimes things get fixed upstream and with the next version there is no difference anymore and you can also do the virtual environment into your into your mounted volume so i don't know but still it helped me sometimes and still i mean it helps a lot so if you if you try it in your in a mounted volume it's really really slow and there is a drawback Of course, I mean, we already saw a Docker file before, so yeah, this command, so docker run, depending on where you install your dependencies, so where you fill up your virtual environment with all your dependencies, it might be that if a new Docker container is created, you need to recreate it. I mean not if you build it into your thing but if you would do it afterwards because a new container then the virtual amp is just gone so this would be better if it's on your mounted thing because the new container would come up and you take the old volume and put it in again then so there Florian Jette finally arrived so he gave me some tips for example this is something which might speed up so if you for example have to reinstall dependencies a lot because you figure out and you want to reinstall or you have you have a setup like this that you want to reinstall all the dependencies with every docker run then this can be quite slow because normally pip does caching in your user directory so it only downloads once all the stuff so if if you'd already downloaded pandas once it doesn't download it again it just says oh it's in the cache so a trick can be mount your pip cache into the container so explicitly make a new volume and say okay please take the pip cache and put it into my container of course there is also dragons because normally those wheels can be compiled for a specific architecture like debian or whatever so uh it depending on how good pip is in figuring out that it's the wrong architecture it might be safe but still i mean you should be careful for example if you have a debian and a redhead or whatever or maybe maybe different python versions i mean this is encoded in the wheel so there should be no problem but still you should think about what you're doing here yeah it is a bit tricky to get it working i mean if we have time then i can show you how to do it afterwards yes so I mean some permissions are of course then a bit I mean but that's a that's obviously a docker problem so when I tried to figure that out I was like really I mean yeah it's so it's it's the question who owns the volume yes okay and as I said before so depending on where you install your your virtual environment things are changing so one tip might be so create your virtual event virtual environment already in the docker build so once you build your image already include the virtual environment into your image that way every time you start a container the image is already there but the drawback is of course if you want to change something in your requirements you have rebuild the image which takes some time i mean depending on what you're doing there it it's okay and a pro tip here um we we saw both versions in the talk before uh so in in his first docker file uh it was uh turned around so the the trick here is uh the those run commands they are cached so docker will figure out oh there's nothing changed until this run command here so i can just use the layers before so docker is more or less like a pancake stack so puts layer on top on the end on top with each run command so if you do not change a file before a run command the run command will just be taken from the cache so just make sure that the code that you are changing is some somewhere at the last line because I mean the code you will change all the time and if you rebuild your image it will just use the cache up to the last line here until the add SRV I don't know why it's SRV maybe it should be SRC so that's the tip so in his talk in the actual code he showed it was in the correct order so everything is good then let's go into the real thing so putting the snake into a box so there is a first question we have to answer ourselves sorry for the picture i did it myself i searched for one egg in a box but didn't find a picture so i had to do it myself but it looks looks really really bad there is a question so we already saw before you can either use docker or you can use docker compose normally people would say if i only have one container why in the why in the earth on earth should i use docker compose for it because docker compose is for orchestrating many containers like a redis as we heard and the postgres together with a web server or something like this if you only use one container for your unit tests everyone would say why should i use docker compose for that so it can make sense and the the thing is it's just the the the the api of the things so docker compose is a command line client that means you always have to type in those commands in the command line and you need to figure out the commands and if you want to put For example, in Git, you would need to write some wrapper bash script, which then puts in all the Docker commands you want to, for example, for mounting a volume. And that is where Docker Compose is really, really convenient, because it has a very declarative syntax with a Docker Compose YAML. So it's a YAML file. I mean, you can, you like it or not. I mean, I'm okay with YAML. so you just declaratively declare what you want to have in your docker container like which volumes which users and so on and it's a file so you immediately can just put it into git so you have a versioned environment so that's a real plus so it's command line options in git if you want so that's why it makes sense to use docker compose who knows why this picture might be here at this point over there for docker compose exactly so we also heard already so docker compose actually really is written in python indeed I contributed back then it was called fig but now it's called docker compose but you can look into the project it's quite fun because you can read it if you're a python developer it's different than go So, yeah, if you want to figure out what's going on there, it's easy. So one thing we all know, if we develop Python, we always do proper packaging. So we will never just execute a script or something. I mean, we always have proper packages and we need to install these packages. And for local development, we do an editable install, right? Anyone against it? No, that's like this. but there is a problem so this is the this is a threat of one of the issues with pip install editable there are some quirks with it so mostly it will create a directory in your local root of your source of your package egg info and it puts in some stuff which in the end also will be in the wheel that you build so it needs somehow the metadata data of your package and it always pip will always create it in the root of your package. You cannot say please put it somewhere else. It's just the documentation of pip says that's the advantage over Python setup develop because it's so easy or something like this. Now the problem is we want to mount our host file system into the docker file system. Now if we do for example our pip install minus e in the image, you cannot mount at image time, at image build time the file system so things happen like you install it but afterwards you have a fresh, you have your fresh host file system in the docker container and again and this created directory egg info just vanished, so you have broken editable install in your docker container and actually you think there must be a nice solution but if you read that up i mean this is some it's not completely about docker it's just how do we go on with editable install and what is this and do we still use setup tools and whatever so it is a problem um oh two minutes that's short okay so i mean yeah so there is there are there are some some tricks to overcome and maybe we just jump to the last one so there is a this python setup develop and you can say please put this egg info over there but then you have to set the python path manually so in stack overflow or no that's a that's a blog post so there it's a nice explanation how you can do it and then it says it turns out this works perfectly the only catch being we need to update the python pass to deal with our fuckery so yeah but that's one of the best solutions you can think of so don't use pip and that's somehow strange 2019 the best solution to use docker is to drop pip i mean that feels somehow strange but it is like this we can maybe skip this because i could not reproduce it it might be that with python 3.7 so i'm not an expert of such things but we want to mount our our file system into the docker container but there are these python cache files you know them it's the pi yc in python 2 or the pi cache folders so python gets the byte code gets generated and it gets put it gets put right next to the code you cannot choose where to put it it's just where the code is so you the first thing you will realize is once you execute something in your docker container when your source file is mounted inside you all of the sudden have new directories on your mac with things like linux inside because i mean it's the same file system mounted so um and before it was actual problem because for example sometimes people need to do things in both python 2.7 and python 3 so but there's only one source directory so those files will get mixed up at least for the python 2.7 i think there's things like this happen and also if it's a different architecture like different kernel version things like this but as I said so I wasn't able to really reproduce it so the error you will get is something like this so it will hint you to please remove the pycache file yeah so I mean it might break your thing so So if you ever encounter such a thing, try to remove it. I mean, you can Google for yourself. But for me, I have an alias more or less for this thing. Or in our Slack channel, I think there is a bot. So anytime anyone asks a question which contains under under PyCache, it just answers with type this into your file and into your thing. It just deletes all of these things. I mean, Python does not have any tooling for doing this. It just creates the things, but it will never delete them. It's sad somehow, but it is like this. So this is the best you can do if you want to mess with this. So in Slack, you can create it just easily. You just say keyword and message. So it's not a real bot, but it's very effective. It helped like 10 people in a month. Do I have one minute? Okay, so this picture, a snowflake, this is what I realize all the time. So we heard already Docker is there to remove the snowflake problem. So it runs on my machine, but on your machine it doesn't. So we invented Docker for exactly this reason. What we are doing when we are doing local development with Docker, it happens. we build up this hugely complex docker compose file which will mess up with all the things like where is my pip cache, things like this. So we build a snowflake again and it might be actually quite dangerous because then we think we have the same in production but it isn't because we overwrite things here and we use it differently there. So normally it should be the case that really we use the same image in production and for local development. But as you saw, there are so many quirks, we cannot just do it. So please be aware of that. If your Docker Compose file is like two pages long and you mess around with everything, then you might have an actual problem. And one last remark is, so for example, if you go to a new library on Docker, on GitHub, and the first thing would be something, okay, I want to try it out or I want to run the tests. so the first contact to any python library might be something like a docker compose file or a docker file at least i guess you you have seen that a lot so there is a new package you look into it and at least the first thing i do is i search for the docker file because then i can just try it and say okay that's fine but i mean we now saw we have to build in all these quirks so So there is a chance that people have to maybe set some Python path things and so on. And that's just ugly. So I think we should work on that. And I mean, the entrance to what we build and show on GitHub and present there should be something like this. So it should be easy just to docker up and everything is running and everything is nice without hundreds of quirks. So the last, so more or less the summary. So I think the most important thing is awareness. So at least for me, it was like, okay, I must be the most stupid developer in the world when I try to correctly dockerize my Python package, like with the thing with the editable install and so on. I was like, it cannot be that the status is like this. So I must be the culprit here. So I think it's not. So there are many, many quirks, and you will figure many of them and maybe already have figured them out. So you are not alone. Probably it's not you. Probably it's just the situation. So contribute or maybe share it also. Write blog posts. So if you search for all these things, you find many blog posts, and they really in detail explain what the thing is. Then I think the then of Python should work here as well. so there should be just one obvious way to dockerize a Python application but at the moment it's not. So I have coded something so I have made a short template for let's say the tiniest properly Python package with as low cargo cold and quirks as possible but still you have the lowest barrier to entry so you just have to set two environment variables that's the user of your local computer for messing around with all the permission stuff and that's it. So you can go there and have a look and that's it. So hello Wobblet.