Docker and Python - A Match made in Heaven

Speaker 1 [00:02]

yeah thank you for the nice introduction this is basically an introduction talk to docker and also to testing and shipping Python applications with docker so you are right here if you may be newbie to docker or haven't used it only moderately but maybe you won't learn anything if you are very experienced docker user already but that's it this doesn't mean that you have to go you can sit through my talk and watch the most advanced docker talk after mine as well but beware this is a disclaimer this is an introductory talk yeah so the examples I will show are of course not a real application they are basically slide where Python hello world applications which will be shipped and tested with stock also please don't expect any real big applications here as well and also although this is an introduction due to time I will have to leave out some features of docker as well and a 40 minute introduction talk cannot be a substitute for a full course or a book on docker as well yeah if you have questions i think we have time at the end yeah a few quick words about mine i was already introduced i'm a technical lead for data science at rosen i was before a theoretical physicist here are some contact details if you want to reach me and discuss about anything like docker or python or other programming languages and yeah a few quick words about the company so rosen is a service provider for the energy industry and does asset care we're doing a lot of machine learning but i will keep it short we are a sponsor if you're interested in what we are doing please come outside and visit our booths so no much talk about the company we are working in predictive and preventive maintenance so what is my motivation for giving this talk basically so there are as you can see in the picture already a lot of solutions for dependency management and deployment of python you know them already like pip or conda from anaconda and you can freely choose any of them and they are all good basically and yeah i personally like anaconda a lot and of course like in the picture you can also easily mess up your local environment if you have a lot of them installed at the same time and you don't know what you are doing and docker is basically not a replacement for a package manager like pip or anaconda it's nearly yeah help if you want to do something else or achieve something else maybe you need a point where you need docker is for example if you have a complex application which you want to run or ship maybe not only on your system but on other people's computers and you include not only python packages but for example some other strange libraries you dearly need or big binary blobs of other stuff which is not so easily achievable with the standard solutions for package management yeah and what we usually also do is because docker is not a replacement for package managing we run like pip or conda within docker so docker is not meant as a replacement i'm repeating myself here but this is a gist also of my talk so another motivation for docker i really like this meme a lot because usually in software development we have this works on my machine anti-pattern this happens a lot it can happen as well with Python and a usual answer is put in a Docker container and it will work on any machine. This is basically true because Docker can feel like you're shipping your machine to another computer or having several local copies of your machine available with strangely different configurations. But Docker is also not meant to be an excuse for a messy environment. Everybody knows that that it's really easy to quickly mess up your local environment on your local machine and then you can say yeah okay if i do it in a container i can keep it messy and it won't hurt my my development setup but we should work in a clean way so don't use docker as an excuse to work messy and although docker feels like shipping your machine there are subtle differences between docker and a virtual machine or your bare metal machine and these differences are really important to understand and to keep in mind but in fact the huge advantage of docker is reproducibility of your environment you can easily write down a recipe to build a piece of software and have it run anywhere but do it cleanly yeah so this picture i everybody who has already worked with docker or had a look in the documentation for docker already knows this picture i took it straight from the documentation of docker it's like an explanation what is the difference between a virtual machine and containers and I like this picture a lot and it's easy to understand so in both cases you have on the bottom some infrastructure maybe your local laptop you want some want to run something on you have an operating system which runs on your local hardware and then the differences starts in a virtual machine you have an hypervisor which runs a full complete guest OS like a new Linux with a different kernel, a BSD, a Windows, whatever have you and on that you have your applications and libraries which you run. Docker works a little bit else. You don't have an hypervisor, you have a Docker daemon or a Docker engine and basically your applications run on the same Linux kernel or Windows kernel which is on your host system and they are only isolated from each other by some other mechanisms. So there is no complete guest OS. And both approaches have, of course, their pros and cons. Maybe you need a complete different OS than you have on your local machine for your application, then you might need virtual machines. But if you only want isolation, maybe Docker is your friend because running a complete guest OS has performance implications. And of course, maybe you need also virtual machines for security reasons because you can escape containers. Basically, there are some exploits known or there were some exploits before and both systems can of course be automated creation of containers is automated by docker and some alternatives creations of VMs is automated by products from VMware or from HashiCorp like Terraform and Vagrant so you are free to choose what fits your purpose in this talk we will talk about containers but this doesn't mean that containers are inherently better than VMs in any case. They have both their own pros and cons. So how does Docker work? So on the top of the slide is a famous quote I really like. It's from Jesse Frisell. It's containers are not a first-class citizen of the Linux kernel. And I added the second part there are a combination of namespaces and cgroups. This means that containers are not something which is built inside of Linux which you can just call and say I'm here on my Linux machine on my freshly installed Ubuntu and I just say please create a container for me and run some kind of software. Containers are basically a concept introduced by third-party software like Docker which relies on functions in the kernel like cgroups and namespaces. Cgroups are there for resource isolation so in containers you maybe want to run the software and want to constrain its resources maybe my application should only get I don't know one CPU maybe only one CPU and 256 megabytes of RAM for example and the other one should get three CPUs and a gigabyte of RAM this you can achieve with c-groups this also implies to IO limits or networking limits and the other part namespaces is basically the isolation part so you want to have these processes see different kind of resources from each other and from basically also separated from your host system so for example you want to say you want to have a different process tree for all your containers so usually in Linux you have one process tree and everything is a child of init or systemd for example but in containers you want to isolate that as well so that inside the container you don't know that you part of your host system and there are namespaces for different things so there's a mount namespace this means that you can have different file systems in your containers and that they don't know the file system often of a different container or the host process tree which I talked about they can have their own network devices they can have their own host and domain names and their own user IDs the only thing I sometimes miss as a downside there are no namespace for time so you cannot isolate time for example could be important could be unimportant to you I don't know so I said containers are basically coming from third-party software so what software we can use to create containers so this talk is about docker so I will be talking about docker but I will also of course mention some alternatives docker is like Python in my experience the batteries included solution you get a good CLI which helps you and there's a lot of people stuff available for it so if you just want to start this is a good starting point but the downside of docker at least in my experience is it it comes with a demon which needs to be root which can be a bad thing for example in a corporate environment this is my experience so if you go to IT and say I want to install some software which runs a demon which is root they are sometimes a little bit skeptical because being a docker user gives you implicitly administrator privileges at least implicitly there are alternatives like potman which is gaining some steam at the moment which is basically a one-to-one replacement for docker but doesn't need to be root which is also good in the last years there was where other famous projects like rocket RKT but this has been archived by the cloud native computing foundation I think one or two months ago so it's basically a dead project so you don't have to look it up again and there is LXC Linux containers which gives you good RPS for creating containers from programming language like Go or Python or Ruby and of course as I mentioned in the beginning you can do the same for VMs I mentioned already vagrant for example for automating VMs if you want do that. There are also some projects which are like on the boundary of VMs and containers. I don't know if you know Firecracker from Amazon, which is basically the idea to have micro VMs which are as fast as containers but more secure. And Kata goes into the same direction. But again we were talking about Docker here. But just keep in mind that there are alternatives which might fit your purpose better. I don't know, I cannot decide for you what you want to use so let's start with the basic terminology of docker and docker there at least for me there are three important vocabularies there's a docker file the docker image and the docker container and often this vocabulary gets mixed up somehow so what is a docker file the docker file is a description or as i would put it the build recipe for what you want to build you put into it the description of libraries and software you want to have in your container at the end and give it to docker to build from that an image and an image is basically layers on disk or you could say the executable stored on your file system and if you run an image it becomes a container which is then like the running process or the running application and as i said this vocabulary is often interchangeably used so a lot of people don't know when to say container and image it happens to me as well but keep in mind an image is like the binary on disk and the container is like the running instance how does it look if we make the picture a little bit bigger so in the end we have not only the images and the container there's also stuff around it we have the registry which is like a repository for pre-built images so there's an official one docker hub where you can get all the stuff you need, like Docker has batteries included, so you can find Python on there or databases or web servers or whatever have you. You can put your stuff on there if it's open source or you want to share it with the world. But you can also host your own registry on-premise in your corporation or your home if you want for your own stuff, which for example we do in our company. And what Docker basically does, if you tell it that you need an image for something via docker pull it will look in the registries you have provided for in the configuration maybe it's your company one or the official one download this image to disk and if you do docker run or docker container run it will run the image it becomes a container and then your application runs and does what it is intended to do or you can interact with it and this is basically all the basic vocabulary and this is all managed by the docker daemon daemon running as root on your system which is controlled by the CLI which talks to the daemon or it has also a REST API which you can talk to or for some program languages there are also software development kits for interacting with the Docker daemon which we will see here. There's a Python SDK which I will be using for testing. There are Docker images for Python of course. You could build your own if you wanted to you could start from scratch with the like an image for a bare Linux distribution and install Python on that but you don't need to there's an official Python image in every version which you can think of to dot X 3 dot X whatever you need also continuum IO provides images for anaconda and mini conda which you can use so you don't have to build it yourself on what you should basically do or what I basically do is I start either from Python or from miniconda install all the stuff I need and then I'm ready to do what I want so how would you build your own image and then this is the part I'm a little bit scared about because and it involves a demo so let's see how that works I tested it last night so it should work but let's see so here is an screenshot screenshot of how a docker file would look like so in the beginning I said a docker file is a description of the build recipe. So now we do need to know what do we put in there? We usually start with a from statement and the from statement says okay We want to start from a container and put additional stuff in it or modify from an image and you want to put additional stuff In it or modify it in some way and this is achieved with a from statement here And here I say I want to start from a Python container the colon notation says You can attach a tag to the image name, and usually the tag is used for version numbers, for example. So here I say I want to start from Python version 3.7, so the official container. And now what do I want to do with it? So maybe I have a script I want to put in it. The next statement is a run statement, and with run I can execute shell commands during build time in the image. So here I run mkdir and I create a directory called source where I want to put my Python script in. With the next step, with copy, I copy myscript.py, which is on my file system next to my Docker file. I can copy it inside of the image during the build in the source directory. So I have it available in my image and can run it later there. Now with workdir I can change the directory during build time to the source directory because I want to start the script from there afterwards. With run I can execute another shell command and here I use pip to install numpy for example because maybe I need it in my script and in the end it's like the most important command it's the command command it tells during the build process the image what should be executed when we run the image as a container afterwards so when I start the image as a container I want to run Python myscript pi this is achieved by the command command looks like this is a little bit strange syntax you can write it a little bit else as well but it's basically you put into the square brackets the commands and as strings and separated by a comma where white spaces would be and this we can do also live and I will now see if it works or not is it somehow readable okay nice good because I don't see So here I have a Dockerfile which I just mentioned. It's basically the same. It starts from Python 3.7. I copy in a requirements.txt where I have all my files in. I run pip install on the requirements.txt. I'll copy in a main.py which is my main Python script I use another command expose which is like a documentation command it tells for documentation purposes that I want to export port 80 from my container to the host system afterwards and in the end my command is running a web server which is executes some commands which are specified in main.py which is a basic REST API which we can look at here so it's a very very small script it uses fast API to create a very fake REST API which just returns something you put in again unmodified and the requirements txt would only be fast api and now we can build it if you want. So to build a docker file I use docker build minus T and then the name I want to give to the image could be my RP for example and then they say dot and dot specifies the local folder because docker build wants to know where the docker file is it's not so good now I'm in the wrong directory and now it runs the build it's usually not that fast but I've done everything before because I don't trust conference Wi-Fi so it's basically using everything from cache which would which is already there usually it would start downloading Python 3.7 and pip would also take a little while but it's already there and cached so it's pretty much fast and now I can run the application if I want this be docker container run I say dash dash RM to delete the container when it's finished minus D to write on the background minus P to define port forwarding so I want port 80 on the host map to port 80 in the container so that I can curl my API I give the container a name blah is maybe a good name very descriptive and then I run my RP so this works you see this cryptic hash which is basically the ID which you can use later or you can use the name and most commands so therefore I defied a name if you don't define a name it gets chosen randomly from a list of adjectives and names of scientists which is also pretty funny and now I can curl it localhost 80 and now if I curl it I get my response back so it basically works and now I don't need it anymore so I say docker container container stop and I called it blah and now it's gone so pretty easy and part of the demo is finished yeah but what would you do if you would like to run multiple containers at the same time so like in this very fakie slide where example you have a rest API which just returns an int you give in this is normally not your real application usually you get your data somewhere maybe from a database or you would store some information somewhere and maybe you have like a second application which provides data to your first application or you want to run a database in a container or a Redis cache where you want to cache some values you can also do that in a container and of course you could manage multiple containers just via the CLI you could say docker container run first thing and then the second thing and the third thing at the fourth thing but this gets tedious relatively fast and also it gets little tired some because you have to remember names and define networks so that containers can talk to each other but docker can luckily automate everything for you using a tool from docker which is called docker compose and compose just does this under the hood what I just told you so it's usually you define another build recipe but not for one container but for a multi container application and it runs that it sets up some networks connect correctly so that the applications can talk to each other and there's some other funny things which I will show you and this is basically then run on your local machine so this is not an orchestrator so if you want to have it run on different machines at the same time or on a cluster you would need something like kubernetes or docker swarm docker compose only runs it on your local machine and I don't think I have time to talk about orchestrators because this is like one or two talks on its own here we are back so we'll be looking at docker compose now and docker compose has its own format for build recipe it's called a docker compose yaml so it's a yaml file hooray we all love yaml or I don't at least and what you define there is basically what containers do you want to run and what is their configuration you start with a version so this is a version of the docker compose API you want to use then And where come the services? These are basically the containers you want to run. You give them names. The first one is called web. This is my API. I specify a build command. This means in the folder API, there is a Docker file and this should be built and run as a web container. I want to map port 80 to port 80 so that I can call it again from my local host. But this time I also need another image. This is Redis. I want to run Redis container. This is a pre-built image I want to get from Docker Hub. So I use Redis colon Alpine. Here the tag is used to specify the distribution on which the Redis was built with Alpine Linux here. And now I can just call docker-compose up and it will be run. Already? I'm talking slowly, sorry. And yeah, maybe I skip this part because it takes a little bit longer. So, to come to the second part of my talk, which is cut short, I think, testing with Docker. So, of course, you have a lot of code in your containers. You want to test it somehow. And of course, you can run unit tests inside of the same container or different containers when you build your application. But what is with the rest of the stuff? So if a Docker file, you have configurations, you have basically some bash scripts, which are run as a command of the container, and you have maybe several containers you put together. So this is all in need of some kind of integration or end-to-end testing, maybe. So how would you do that? Of course, you could maybe write a real big bash script which tests out every combination you can think of. And this is also a viable solution. But maybe there are other solutions, maybe also solutions in Python you can use. so maybe we start with a solution which is just using bash I don't know if you know it but bash has also a unit or a testing framework called bets and this is relatively easy to set up to test your stuff because yeah you're already using the docker CLI to start containers and then if you use like a testing framework from bash it's pretty easy because you have setup code which is just maybe building and running your container then you have your test which for example could run curl against an endpoint and then you test is the exit code okay and then your test has executed correctly and all the tests are green and you are happy but this is a little bit tedious because now you have if you want to like test the output of your json's you have to escape all the quotes and stuff like that and also the product project is not so actively maintained i looked it up I think the last commit is from 2018 so maybe you want to use it maybe not but there's a better way or there are two better ways I like really real much so there's a docker SDK for Python so instead of using the CLI you can talk to the docker daemon from Python you can achieve this by pip install docker then you have the SDK you can import docker and then you can basically start and run containers build images stop them kill them do whatever you want and usually you can use this in testing to write test fixtures so I will not execute it I will just show it for time reasons so if you have this installed you can just call import docker and then you can for example in pi test write a fixture which just say okay I need to talk to my docker daemon this is done via this initialization command here then you can say okay I want to build an image where the docker file is in the local directory call it my RP good sorry then you can run the container also via a command in Python define the pop mapping and some additional configuration and then in a like in a typical test fixture and pi test you yield then you go to your test where you buy a request maybe talk to your API if this has worked out then you go back here and you call something I call tear down container but you basically do is stop and remove the containers and you are done No, it's not. Yeah, you could replicate what Docker Compose is doing with that, but it doesn't come with built-in Compose functionality. So you can, there's no, like, the Docker SDK doesn't have something where you can just import the YAML and run everything at once. So here what you have to do is by hand in Python code, which you can do, but it could also be tedious if you have a lot of containers, which you want to plug together. What do you mean? There is basically an alternative to that, which you can also use. So I just asked if you can use Docker Compose together with that one. You cannot use it directly together with that one. You would have to replicate what's in the YAML file. But if you want to use a Docker Compose YAML, there is a plug-in for PyTest called PyTest Docker Compose, which automatically generates PyTest fixtures based on what you've written in your Docker Compose YAML and takes care of Docker Compose up and down for you. And this is, I think, basically what you want. And I would prefer that if you want to test more than one or two containers at the same time and you already have a Docker Compose YAML. and this is also an actively maintained project which is also good and you don't have to write the fixtures yourself which is also good and then there's also a third project which you can use, it's called C-worthy it also tries to automatically generate test fixtures for you or you need to help it a little bit to generate test fixtures for PyTest it defines classes for your containers and inside you define like patterns like when is the container finally started, what kind of errors you expect and then you write the tests but for your containers you have to write your own classes and there are only some pre-built classes available like for Postgres, SQL or Django so maybe if you use these two this is an easy startup for what everybody else needs to write their own classes and it has no compose support and is still alpha level software So what I personally do at work, I use either the Docker Compose PyTest plugin or for small things, I write my own fixtures with the Docker SDK and PyTest. Is there something similar to the Docker Compose testing using the PyTest plugin for Kubernetes or Docker Swarm? So for Kubernetes, I cannot know because we're currently not using Kubernetes. For Docker Swarm, I'm not aware. What we usually do is we replicate what we want to deploy on a swarm on our local machine, use in Compose and test it there and afterwards deploy it to swarm. But maybe there is something around I'm just not aware of. Could be, I don't know. So, and let's come to my final slide. So what are use cases for Docker and Python? Basically, you can deploy a lot of stuff like REST APIs, for example, web applications. You can also have an advantage with Docker if you want to deploy something which has dependencies like databases or other stuff, which you can also put in a container. you can distribute scientific workloads which might require some strange library put into it which you don't want to have installed on your local system you can also put like training of machine learning models and this is also something we have done you can do continuous integration with docker so even if you don't want to use docker to deploy your application you can have like use docker to perform your testing and what basically at least for me is a main advantage of docker is a reproducible environment across different machines if you have an image build it is the same image on all the machines you can run it on and this is really nice and there are i think two things which you should not do with docker which are for me like pretty clear stuff but some people confuse it anyway so don't use docker for security reasons because the isolation is not so good like with other stuff you can get so there are for good reasons there are VMs and sandboxes and jails so if you want to isolate applications for security reasons docker might not be the best choice for you and again don't use docker as an excuse for a messy environment because like you say okay my build process is complicated and messy and I don't like it so I put it into into a container this is also not like a good idea yeah this would be basically the end I don't know if I have still minute left then just let me show here there are some resources so I said the talk cannot be a replacement for a full course on docker so there's a Udemy course docker mastery from Brad Fisher a docker captain and I really really like that course and then I can recommended if you want to learn more about docker. There's also docker labs with a lot of more sophisticated examples. There's a documentation on the docker SDK. There's a github for the bash unit testing system bats and I also included the documentation for cworthy and pytest docker compose and i think that's it yeah thank you very much so if anybody have has questions i think it's the right time to ask thank you so i'm not a docker expert like probably on the other side of the spectrum But I remember when I had to set it up on Windows, I had to disable some stuff that I never even heard of, like Hyper-V, I think it was called. So the question is that if I do something on Docker, package it up, and send it to somebody, do I have to expect that person to install Docker on their machine? Or is there a way to run it without having to go through all the installation steps? so basically if you want to distribute a docker image the other user who needs to use it does need to have docker installed in some way it can be painful on windows especially in corporate environments like i don't know very crazy virus scanners or isolation environments can hurt you there and there's a subtle difference between docker on windows and docker for windows as well so it can be difficult but yeah basically basic answer is the other person needs to have docker installed so thank you for the doc i have a question about the size of the image yeah so i also try a little bit i'm really not an expert but what i try to use python 3.7 then i try to use the slim and then the alpine and when i try to install requirement then crash on the alpine because i was missing some c thing i expect nothing like that do you have any kind of advice on that because alpine is what will make the image lower like the smallest and so it's better for deployment but then we have to debug it to ourselves that's terrible so yeah so my experience with alpine linux are also mixed and what you experience is basically yeah alpine is small and this is correct but alpine uses a different libc than most other linux distributions and this can hurt really a lot even if you try pip install something and it isn't built against the libc which is used on alpine then you maybe experience something like you have yeah and my usual way is i try something like you as well i start with normal as a normal like miniconda or python 3. something and then i see does it work very well with slim does it work was alpine again but in the end in at least in the environment i work in it doesn't hurt if an image is like 500 megabytes of size or something like that um so this is my first ever introduction to docker and i would be interested to hear more about what sort of things you can isolate so you were saying for example you cannot isolate time what kind of things would you like to put in the containers and isolate to share with your colleagues and with other machines so what we usually do with docker and what we try to do with it is like for example there are different stuff like so one thing could be like this fakie example I showed I have like a web server I want to run and I have like a big machine where I run I don't know, 100 of these on. Another example, like I said in my last slide, you have scientific workloads like simulations or training machine learning models, and you want to distribute it on a cluster or something, then it's easy to put that into a container because you want to deploy it to some other computer or to a cluster, and you don't have to install all the dependent libraries there. Take the example of an FEM simulation. It might need some kind of open source library, which is hard to compile and stuff like that so you do it one time for the image and then you can distribute your calculations freely you only have to for example change the configuration of what you want to calculate can you just quickly elaborate on how to on a good way to set up your own registry on premises which you mentioned earlier now I think there is a github repo you can fork with a good description on it so i didn't do it myself on the company it was our it department which did it but i think they didn't experience much problems so at the moment we are running two registries in our company one like for permanent deployment of images and one for continuous integration where like if you build an image and just for testing purposes you push it there and this is then automatically deleted after the continuous integration job has finished or after 24 hours and from what i was told this was pretty easy to set up i think you can google it pretty easily um so my level was quite on par with the introduction and that's about it And now I learned today that I can also use Python to deploy an image in the end and container. Except for testing using PyTest, is there another application or use case where you would use rather Python to set up, let's say in backend, API, whatever? Because so far I only would have used Docker without Python to set it up, but maybe you can elaborate on that. when to use Python for the deployment when it's not only about testing? So it depends. Maybe you have an environment where you like, for example, it's hard to say. I mainly use it for testing. I know that, for example, Docker Compose is written in Python using this SDK. And maybe if you want something like Docker Compose, but slightly different for your special use case, you could use it. Or if you have some other thing, maybe you don't want to write Docker files, but or you want to write docker files using python and then automatically build them with the sdk and run them but yeah it's for me it's mainly testing but i think there are other use cases as well are there from your experience any um caveats from running databases in a container well there are some caveats so you have to like Yeah, most people don't run their databases in containers. I did use it here for the example application. I think it's still a discussion in the community if it makes sense to run the database in a container or not, since there are sometimes performance issues and also other stuff. You need to create volumes to store your data and stuff like that. And it's not so easy to, for example, deploy a database on a swarm where you don't have control, where the volume is stored, for example, if you have different machines but that's still an open discussion and I'm not very opinionated in that You mentioned about security concern with the Docker can you please explain a little bit more detail? For example there was, I don't know I think one or two years ago a very famous exploit, Dirty Cow, where you could escape a container and become root on the host system for example this would be for example a security concern though and if you can escape from a container to the host then and become root there or somehow privileged there then you own the system and this is something which you should want to prevent basically and there are and docker is not built with security as its highest target basically so they are taking care of security of course but it's not like docker was not built for a security purpose therefore there are other things and therefore sometimes you can start your docker container as no root or you can start But the Docker container is in unprivileged mode, right? Yeah. To work around that. I think the Docker daemon can become unprivileged. I think this was in the last release somewhere. And there are alternatives like Potman, which are completely unprivileged. And then we have less of those problems, basically. All right. I think this is it. Thank you very much for your questions. and thank you very much for your amazing introduction it was super interesting let's give an applause to our speaker