The state of DevOps for Python projects

Speaker 1 [00:03]

Good afternoon, everyone. I'm Tobias. I work for a small Berlin-based startup called Adzmi. And I want to talk a little bit about the state of DevOps for Python projects or how a small team of Python developers uses DevOps tools and practices to code better. And I thought I'd start with a little joke. DevOps, again, this is a topic that we've heard about a lot over the last years. It kind of feels like the hype is over, but just looking at this room and looking at all the people that are waiting outside feels like there's still quite some interest. I want to maybe start with a little show of hands. Who knows what a CI pipeline is? That's all of you, almost all of you. So I guess, you know, I'll try to keep the basic introductory topics, keep those slides very short, and focus a little bit more on the little more advanced stuff. All in all, DevOps has been around a little while, so I'm not going to reveal any groundbreaking, bleeding-edge things here or something. This is a look at solid practices that we've put into practice and to sort of give you guys an idea of how to take these tools that have matured over the last few years and build something solid with it. Focusing a little bit on the things that we do at Altami that are maybe a little bit different from what other teams do. So DevOps, if you ask two people, you're probably going to get three answers on what entails DevOps, what really is, what does that term really mean. For me, it's a combination of two things. There's these concepts and guiding principles on how to work better, on how to collaborate better, how to do team integration better. And then there's, of course, a huge stack of tools that allow us to really put these practices into, or these ideas into practice and put more efficiency into our workday. In the end, it's all about putting code into production. And you'll see in this talk, I've sort of tried to streamline it a little bit and also split it into this half that talks about concepts and ideas and then also about the tools. Yesterday evening when I was going through it, last time I found out that I was at about 45 minutes altogether, so I cut a bunch of slides and now we're going to be mostly talking about the tools part, the practical part, which I think might be more interesting for you anyways if you all know what DevOps, at least the idea or the concept is. So we'll fly through these and then get to the practical section. And for that I came up with a little story, I guess, Try to come up with a way of how to you know, just not go through all the tools we use but kind of make it Illustrative a little bit and what we're going to do is follow the journey of a commit as it's being written by one of our developers and goes all the way through our system through the Various stages the various deployment pipelines and deployment stages all the way into our product. So that's the main part of the talk. I Want to briefly talk about who we are and why I'm standing here in the first place what we're doing Alts.me, a small Berlin-based startup in the industrial and air space. We're only two, two and a half, three, three years old. And what we do is work with cement and concrete producers to optimize their production processes. Doesn't sound like the most interesting task or field in the world, but it's a very important one. Cement and concrete as building materials are responsible for eight to nine percent of global CO2 emissions. And saving every little bit of that is very important. to our goal of saving the planet, I guess. And these industries being hundreds of years old, some of them, and the processes also being hundreds of years old, they're not very much up to date with digitalization or today's techniques that we can use to optimize their processes. So a lot of manual work is being done. And what we do is use machine learning, really do put machine learning into practice to help them optimize and learn more about the way they produce these materials and how they can make it better, thereby saving them money, of course, saving resources, and hopefully also doing our little bit to save the planet. So our big goal is to prevent 100 megatons of CO2 from entering the atmosphere by 2030. We're on quite a good track to achieving that, having saved 50,000 tons of CO2 just last year with our first initial customers, and we're currently onboarding more. So that's something that I feel is worth working for. The team size, it's actually on the next slide, thanks for the question. We started out with just three, as a startup does, with just three Python engineers, everything's on Python. In the beginning we had the front end in Python even, there's a tool called Dash, maybe some of you know it, that allows you to code a front end in all Python. We've grown to 10 engineers now, then there's these two little firefighter construction worker emojis down there. That's one of them is me, the other one is a colleague, the people supporting the work of the Python people with DevOps infrastructure questions, et cetera. So requirements. What do we have? One of the things, of course, that we want to do, like every other team does, very fast iteration. We want to be able to put code from our laptops into production as quickly as possible within minutes or hours rather than and I've worked at places I've worked previously it would you would have bi-weekly release trains so every two weeks there would be a release and then you if you wanted your code live you had to wait until that next release train came by and that's of course something that we can't have as a startup of course as little manual intervention as possible so you know every we're all great developers and we're all smart but you know everything every time we do something manually there's a chance of that breaking and every little step that we can prevent or that we can automate, we reduce the risk of that step breaking. So of course in this whole process we want to do as little as we can manually. The whole system we want to be very flexible, of course being a young team with constantly changing requirements, constantly changing situations, we have to be able to adapt to that really quickly and not be focused on a very strict process. should be easy to understand for everybody so that everybody really knows what's happening to their code, how that's being deployed, and not just be one black box that you throw some code in and magically it appears on the other side. Then finally, and this is maybe a little bit of a special one for us, we sometimes work on features that span the entire system, so everything from the front end to the back end components, to the machine learning components, to the database layout even. We want to have the possibility to work on that and modify that without interrupting somebody else's work. What many teams, and I'm sure virtually all of you do, is to have not just one production environment where all the code goes, but to have copies of that production environment. They might be called staging, they might be called testing, or QA, something like that. These are static environments where you push code before it's ready to go live to see these These are the changes that we're about to take live, run some tests on them. That didn't quite cut it for us, so we went one step further with that. We'll get to that in a little while. Concepts and credos. This is the theoretical part, and I'll try to rush through that so we can get to the juicy bits. What is DevOps? What are the ideas behind that? And I did a little bit of research and preparation for this. There is a talk from 2009 by two Flickr engineers or Flickr engineering managers who were talking about this whole idea of bringing the development teams and the operations teams closer together. In traditional setups in tech companies, these would be separate teams. They would be the developers who would do the coding. They would be done with the coding, tests would be run, throw the code over to the operations people who would then put it onto the production service and make sure the code actually runs and monitor it. And sometime around there, 2009, 2008, this whole idea of let's merge these two teams and get them to work much more closely together. This whole idea came up and was given the name DevOps. So that's sort of where that comes from, merging these two principles in the team. Some ideas and techniques were older than that. So Jenkins, which might be familiar to some of you, an integration server, has been around a lot longer than that and these integration servers or deployment pipeline enabling tools are quite essential for any DevOps workflow. So these are the guiding principles that I could come up with.

Speaker 2 [08:41]

Thank you.

Speaker 1 [08:42]

And again, there's not one single definition of what DevOps is. So this is more of a of a shared idea of working better together, but I think what's pretty central to the whole process is developers work across the entire app lifecycle. So from actually coding the writing the code to writing the tests and really putting that into production and then also monitoring that the applications and the code that they write runs well on production. going all the way back to writing fixes and writing new features. So this whole life cycle, as it's often called, is now no longer split apart into separate teams, but rather belongs to one team or belongs to one function. That, of course, entails a shared end-to-end responsibility for the entire code, for the entire system, not just a little bit anymore. And then some of the principles automate all the things, and I think especially with the DevOps approach, That's very present to really be able to run all the things, everything that's there for a deployment, have that run automatically. Constant collaboration, open communication is something that shouldn't just be there for DevOps teams, but really for everybody else as well. But I want to put a focus here on that as well. It's always an important one. And one principle that we found very interesting, we try to follow as explicit over implicit. In our work days, when we, you know, especially as we, you know, as we gain more experience, there's so many things we start to learn that we sort of take for granted because we know them. But that's not always the case that everybody else knows them, not even from our own teams. So we've kind of taken on in our team this principle, explicit over implicit. Whenever you think somebody else might know something about this, don't assume that, but rather make it explicit. And that always helps. So much for the theory. Let's look at the journey of a commit from a puzzled developer here all the way into the release party all the way in the bottom and we'll follow this little commit along a series of stops through our whole system. And I'll be rushing a little bit. Apologies already for that. There'll be a bunch of concepts and some tools that I'll show along the way and we won't really have time to go into detail on any of these, unfortunately, being a 25-minute talk. If you're interested in any of this stuff, also the code that I'm just going to rush across the screen here, please come talk to me afterwards And I'm super happy to to elaborate on that and also to share the things that I know and that we've built as a team Of course always interested in helping other teams helping other people grow All right, where does everything start of course in vs code the best idea out there no discussion So vs code is what I personally use most of our team uses uses because that other one, I forget the name, we shall not speak of it. Of course, there's code being written, Python code in this case, and then that's being committed. I personally love using the terminal. Many of our developers also use PyCharm. That's the name. See, I actually forgot it. PyCharm. PyCharm has a great Git integration as well. This is all very standard, and you all know how Git works, what that is, so I don't need to go into detail. little feature or one little thing that we do differently here if you look at the branch at this and I'm hoping you guys can maybe read this in the back we're committing our we're pushing our commit onto a branch and this branch has a little suffix there it gets a plus sign and then it gets an identifier and that's going to become important later on so the next stop of our little commit is GitLab and GitLab is the most central tool for probably the most central tool for our work it's if not the best was certainly the most cost-effective tool that we're using. It adds to me up until a few weeks ago we were on the free plan because the free plan really offered us everything that we needed for our work as a professional company with tons of products. We recently switched to the paid plan just because we were feeling bad that we were getting so much value from this great tool and not paying anything for it. So now we're proud supporters of GitLab. Right, so GitLab is where everything happens. All our repositories live there. You can see we're quite busy creating merge requests. There's, I think, over 30 repositories now after three years. And we use GitLab for, obviously, repos, all the continuous integration, continuous delivery, integration, coordination. The CI runners are actually running on our private hardware. But GitLab also has these shared runners that you can even get for free, also part of the free plan. And that's great if you're starting out a project or if you're not a professional company. If you're just getting started then it's it's really nice that there's already this Yeah, these pipelines can be run as well There's this other tool also starts with git and then has another three letters as it has a similar workflow, and I think it's called Yeah, it's called something there as well this whole functionality So github of course I'm talking about and they are sort of the same thing in different colors both do the job Next job next stop for our little commit as a merge request. And the merge request is the tool or the vehicle that we use for all our coordination if code needs to go from a feature branch into the main branch, which is called main for us. There are a few limitations that we put on when a merge request is allowed to merge. We do require that at least one developer, one other developer, does a review of it and approves that review, of course. And then, of course, the automated tests need to be passing. Right, and the first thing that happens when a merge request is open is that a CI, a continuous integration pipeline is triggered. And this is the most, I would argue from my perspective, the most central tool, the core tool that powers any of these DevOps or modern development workflows is the pipeline, which contains all the automated tests. It contains much of the container things that might be happening. It contains most, ideally, all of the deployment steps. So really, the developers will be writing code, pushing that to GitLab, and then afterwards be done with actually manually doing things. Might not always be the case, and we'll get to a slide in a little while about the manual things that we still do. So C-iPad Bans starting, and C-iPad Bans for us look a little bit like this, nothing out of the ordinary here as well. This was actually taken from one of the libraries we run, but they look similar for applications as well. First thing we do is build. We'll see that in a little while what happens here. Then there is a bunch of test jobs, there is linting, safety is being run, tests are being run, type check is being run, and finally there's the deployments being run. And in GitLab, this is all set up with a YAML file. This is actually taken from one of our repositories. We have a custom Docker image that we use as the basis for our entire build process that's not required, and this image doesn't really contain all that much, just a few tools that we need for this. And I'm super happy to go into detail with this, but let's look into these stages. So the first thing that happens in the pipeline is we're building a Docker image. So every single commit that's pushed to our GitLab is being turned into a Docker image. There's two reasons for this. The first is that our entire deployment runs on Docker entirely. So absolutely everything is pushed into production as a Docker container. And the reason why we're first building this container and then running the tests within the container is quite simply that if we run the tests in the container, we know we're guaranteed that the environment in which these tests are run is exactly the environment that we need. If we run the tests on the GitLab runners, on the machines directly, there might be other dependencies installed, there might be some libraries that are there or that aren't there. And if we run them in the container, we know this is exactly what we need. And then also we're pushing that exact container that we just ran the tests on, we're pushing that container onto production. And we also know that now the code that we tested that everything that we know everything is fine with is the exact same code that goes into production. So that's one of the principles that we follow. One of the things that happens when we're building a container is to run pipenv. And I think this was already the point of discussion and at least another talk that I saw which package manager to use. I guess the Python community has been plagued with that question for a while. We did the big contest last year between PIP Poetry and PIPenv. PIPenv came out on top. One thing that convinced us is that it has the seal of approval from the packaging authority. I don't know if that's actually worth a lot, but at least it looks like it's good. And PIPenv is quite feature-complete, has safety integrated, has great support for automatic virtual environments, which is fun, so these other ones will do the job as well. And then we run, of course, like everybody here, hopefully also does, run code checks and tests, PyTest for testing, MyPy for type checking, which we just started recently, but it's a great addition to the toolkit. And then we use Black. I don't know, anybody else using Black? How, yeah, that's roughly half of you. And for everybody who isn't using it yet, it's fantastic. It takes away so much of the little quarrels that you can have in a team when there's multiple people working together. Black just says, this is how we do it, and there's no more discussion around it. And it takes a little while to get used to that type of talk, I guess, that Black gives you. But once you accept it, you don't have to worry about this anymore. So then we get to deployment. All the tests are green now. Our little commit passed all the tests, all the linting. And we're at a deployment stage now. And this is our setup. It's very also not, you know, nothing out of the ordinary here. We have a testing stage. We have a staging environment and a production environment. Production, obviously, the live environment that our customers use. Staging is something semi-stable, I guess, the next thing that will be released into production. And then testing is something we can just play around with and destroy if we need to. And in fact, every commit that we build and that's built into a Docker container, that's then the tests are run on it. And if it passes a test, it gets deployed onto the testing environment. And that means that this testing environment is constantly being overwritten. and if there's multiple people working on different branches, they are all deployed there. That's intentional, so this environment is more for us to test our deployment procedure, and we'll see how we work around this problem of constantly overriding one another's code. Everything runs on AWS ECS. ECS is the Elastic Container Service, also a great tool to get containers running in production quite quickly and without having too much to worry about. In fact, let's talk quickly about our infrastructure setup. Really won't go into detail here because that covers at least another 25-minute talk. Everything's on AWS. We said this very early on. We want to keep things uniform and very simple to handle, very simple to grasp. We're not going to do anything fancy here. Everything's on AWS, and everything's in Terraform. So every little resource that we're using in AWS, we have actually written as code in Terraform. And that allows us to do some really interesting things. For example, if we need to commission new resources in AWS, that also runs through Terraform. So all our Terraform code lives on GitLab. And if we want to make changes to that, we can also do that through a GitLab merge request that's being deployed automatically through these GitLab runners. So here's the big feature. And I spent some time on this slide, and I hope it actually... Oh, yeah. So I was told when I showed this to my colleagues, you don't have enough style in your presentation. So dynamic environments is something that we build internally to solve the problem of constantly overriding one another's code. So if I'm working on a feature that might take a few days or a few weeks even to complete, if I'm releasing that or if that's being pushed on the testing environment, somebody else's code is going to override that quickly. So what we can do is create a dynamic environment that actually does an entire copy of the whole system, does a new deployment to a clean environment space, does a copy of the database and of S3. S3 is AWS's file and data storage. So I now have a complete copy of the whole system that I can mess around with, that I can delete, I can modify the database without ever impacting anybody else's work. And we've set up our GitLab so that when it sees a branch with this little suffix down here, it will recognize, okay, this branch needs to be deployed to one of these dynamic environments, and that's exactly what happens. So it takes that branch, doesn't deploy it to testing, but it deploys it to the dynamic environments, and so this is a really tight integration into our development workflows. I can now just have my little branch, commit that, push to GitLab, and it automatically gets deployed onto my custom-built dynamic environment. And this is all built, again, with Terraform, so Terraform is really quite the helper here. And it's not a whole bunch of magic. This is a small Python script that does all this. And I'm also, again, very happy to share that if you guys would like. All right. So we're not actually deploying to testing but to myenv. There's very little time left so I'll rush a little bit. Our little commit now lives in its own dynamic environment. Now a friendly developer came along and says this looks great. Merge request accepted. domain and when a merge domain happens the whole CI pipeline runs again and this time deploys everything to staging which is sort of our acceptance area where the acceptance tests are run where our product managers and the business people can go and see hey this is this looks great I want to ship that into production then now comes the slide that I'm not proud of the release preparation which is the last manual part so I guess you know one thing to take away it's a marathon right so of course there's this idea of this perfect automated and fully fully autonomous release process to my knowledge at least I don't personally know any team who has that if you do have that congratulations we don't quite have that yet there are a few steps before something can actually go into production our system you know there's some some movement in the company to get rid of those as well so what we do to prepare for a release we keep a change log of all the stuff that happens for any for any repository for any project and that's written manually so that's that's written manually we also manually increase the version and then we push that back onto the into the repo and add a git tag and this git tag signals to GitLab hey this is now a release ready or release candidate and we would like to push that into onto the production server and that's exactly what happens so the CI pipeline runs again does all the tests and pushes that container image into production this time right And the actual these actual deployment steps I didn't really get into in very detail But they're not all that much work because everything's on ECS. The docker image is already there So there's just one little call to a little tool that tells the ECS the Amazon ECS service Hey, please update the service and rather than running the old image now run the new image and that's already that So this this whole cloud infrastructure also plays quite the integral part our little commit now lives in production but of course that's not the end of the story as a true and and and a proper devops team of course we're also taking care of monitoring our systems and making sure that everything works as intended so we mostly use these four tools to make sure everything runs amazon cloud watch is a centralized logging service so all the logs that occur anywhere go into there and can be viewed amazon cloud watch i don't know maybe You guys have played with it not a great interface, but it sort of gets the job done Then there's kafana for metrics. We use that both for system level metrics and also for user metrics So we can see How much data is coming in how many clients are currently using the platform etc Sentry is a very great tool to just find errors or follow through on them being solved and then of course slack That's our main communication tool as probably for many of you as well And we believe in you know the alerts and the work should happen where it happens anyways, so we push notifications into slack Automatically when something goes wrong, and I think that's the last side exactly cool. Thanks very much

Speaker 2 [25:25]

Wonderful timing. Thank you for your talk. Barely.

Speaker 1 [25:27]

Really?

Speaker 2 [25:29]

Then we go to the questions first one. How did your pipeline grow historically as in which steps did you? Make mandatory first and prioritized which ones came later would you do something different? today, how

Speaker 1 [25:43]

How did our pipeline evolve over time? Yeah. To be very honest, that's kind of how it started. There wasn't very much evolution. So when we started off, you know, the people building this pipeline came from different backgrounds. So at the company where I'm currently at, this is how we started the pipeline, also with this whole idea of running everything in Docker. Docker is really quite the core concept in our work, and also this idea of building the Docker image first and then running the tests within there. There's a bunch of tools that came. So, MyPy, for example, we just started using six months ago or so. Before that, we were just relying on, you know, duck typing. And that saved our asses at least once or twice already. Sorry for the language. Black also came in a little bit later in the beginning, I guess, when you're just trying to get something on the air, you don't care about your code formatting or so. But then once two, three, four, five more developers joined the project, everybody with their own ideas of where to put the braces and the conons, really, really handy to have a very rigid idea of how that's looking. I would do it the same way again, to answer that halfway. I would do it exactly that way. I think it's a great pipeline.

Speaker 2 [26:49]

okay uh next one how do you optimize ci runtime do you have any caching in the pipeline yes

Speaker 1 [26:54]

Yes, yeah, that's that's been a big topic We optimize it by using docker build caches quite a lot So when you see this docker build stage at the very beginning building a docker image ideally doesn't take all that long You saw the the docker file there. It's not very long It's just a few dozen lines of codes of code, but most of the time nothing much will be changed except the last little line there Yeah, I don't I don't want to go into too much detail now But docker build caching is something that can speed this up greatly And I can greatly recommend that as well if you want to learn about this do please talk to me

Speaker 2 [27:28]

Okay, next one, why not use review apps in GitLab to deploy feature branches?

Speaker 1 [27:34]

Review apps is something that we did look into a little bit, but to our understanding, that's more concerned with the front-end aspect. So if you're building an app that has a front-end like ours does, you would often have, you know, deploy this one little part, like create a second environment of that so people can see, hey, this button is now green instead of red. But what we need is really a copy of the entire system. We need the database and the database layout to be modifiable. We need all the machine learning to be, you know, we want to be able to mess with all that. So it's not just one little change that we make, but sometimes there's a feature that permeates through the whole system. And so these review apps didn't really help. What we do do is to use this environment feature of GitLab that's just called environments that lets us see all these different dynamic environments that we have. So we can go onto GitLab and see, hey, developer A created an environment called this, and I can actually go onto the front end of that. but we need it a little more than just being able to sort of spin up multiple front ends

Speaker 2 [28:36]

Okay, next one. Are you using Dash in production? So is your product based on Dash? What is your experience with that?

Speaker 1 [28:43]

Yes, and no

Speaker 2 [28:44]

Yes.

Speaker 1 [28:46]

The first iteration of it was using dash exclusively and the experience was if you have somebody who can dig into it It's great, but it's quite difficult, you know to understand if People aren't willing or aren't able to invest a lot of time into really understanding it It did the job for us quite well, but we're actually replacing it currently with something In written and react. So this is a new front that we're that we're building

Speaker 2 [29:13]

Okay, next one, how do you manage configuration between stages, that is, secrets, connection strings, and so on?

Speaker 1 [29:20]

AWS Secrets Manager is quite simply the tool. So AWS is a, most of you will probably know it or have heard of it or work with it, and they have a solution for everything. And quite often I have to say it's a giant corporation, but what they do, they do well in this AWS space. So the Secrets Manager is a hosted solution for storing these secrets encrypted and accessing them on demand. So we're using that for all the secrets management like SQL access strings, passwords, API tokens, etc.

Speaker 2 [29:56]

Okay, then I'll look for another one. Apart from automating your release nodes, are there any other improvements or challenges that you see?

Speaker 1 [30:09]

That's a good question.

Speaker 2 [30:13]

Puffer company, I guess.

Speaker 1 [30:14]

Yeah, well, one thing that we are doing is moving away from ECS. So when I was talking about ECS here, that was sort of the status quo. We're moving to Kubernetes as a company, mostly because, and I don't know if he's still here, but hat tip to the previous presenter, Yanis. We wanted to use Argo Workflows, and Argo Workflows is the one orchestration platform that only runs on Kubernetes. So we kind of said, hey, everybody's talking about that anyways, let's try it out. And it turns out Kubernetes is a tool quite well done, also really complex, nothing that I would recommend to somebody or a team just starting out or if you're just running your little Python thing and need to get something running, do not touch Kubernetes. But once you're a bigger team or a company or a university team that has a bit of time to spare, it's something worth looking at. There was something? No? Okay. Yeah, so Kubernetes is something else we're looking into.

Speaker 2 [31:06]

into yeah then very last question how do you deal with cleaning up dynamic environments

Speaker 1 [31:07]

Yeah.

Speaker 2 [31:14]

yeah there's like release deployed resources after i'm asked yes so

Speaker 1 [31:18]

So this little script that creates these environments. It's a little Python CLI tool that developers run That has a create and a destroy command to create command as you might have guessed creates the environment and destroy command destroys it it has happened in the past that people forget to destroy their dynamic environments when they're done with it and The solution to this is me going through them once a month and looking whether there's something to be cleaned up Right. So that's very very manual and not great with GitLab review apps or this environments feature there's something that you can build that when an environment gets deleted it can automatically launch a job from your CI pipeline and that's something that we're also looking into automating this whole process of dynamic environments more especially moving to Kubernetes we want to move all this over there as well and that would be a great addition as well but that's in the process and I can tell you a little bit about it if you're interested. Not all that much though.

Speaker 2 [32:18]

Okay, I give one hour short because we have a break now. Why do you need dynamic environments in the first place? Isn't local personal development environments enough?

Speaker 1 [32:29]

Possibly, yes, but if you're running ML jobs that can take, I don't know, sometimes a few days. We do hyperparameter optimization, for example, and I'm really not an expert on this stuff. I only know that these jobs can run 12, 24, 36 hours, and you really don't want that running locally on your machine. You want to make changes to this algorithm, and you want to deploy it on the production. You want to go home for the weekend. You want to come back on Monday and then see what your outcome's worth. You don't want somebody else to overwrite your code, and you also don't want to leave your laptop running. These things do get hot if you've ever actually, I don't know, did a compilation on yourself. So, yeah, we really wanted to also recreate the actual setting in the cloud because the configuration on your laptop is going to be 100% different from your configuration on your production system. So the closer you get with your development environments to what you actually have in production, the better.

Speaker 2 [33:17]

Okay, then, thank you for your talk. I believe you will be around for...