Streamlining Python Development: A Guide to a Modern Project Setup

Speaker 1 [00:05]

Thank you, and welcome to my talk, Streamlining Python Development, a Guide to a Modern Project Setup. So, a few words about myself. So, my name is Florian, I'm a mathematician, and I've worked with Python for, have been working for more than 20 years. And for more than 10 years, I've also contributed and developed own open source packages. So the topic of how to actually come up with a good Python project, how to make the development as efficient as possible, has been a topic that accompanied me for quite some while. And as you all know, the Python community is moving really fast. So this is how this idea came about to give a talk about this. I should also mention my employer who makes all this possible. So we are Innovex, we are an IT project center, and if you want to know about us, check us out on our website or talk to one of the many attendees here. We are at least 25 people of my colleagues here on stage. Not on stage, but okay. So and I also want to mention that we are hiring at the moment, but let's get started. So, a short agenda, I want to actually cover in the introduction that what actually makes up a good project set up, so how do we define good, and how can we also maybe measure this. Then the question is also how do we achieve, of course, this goal, and in the larger part of the talk I'm then going to present like one streamlined project set up, of course this is all very opinionated, and we're going to talk about the PyProject, TOML, like all the toolchain one can use in a project, like Hatch, Ruff, MyPy, PyTest, and so others, all those little helpers for a project, and in the end, I'm going to conclude with an overview. So let's get started. Let's get running with the introduction. what actually makes up a streamlined Python project. So first and foremost, the most important thing is that you develop efficiently, right? So we all have some project going on. We all have deadlines. We all have certain tasks at hand. And you are developing efficiently if you can spend most of your time actually focusing on the task at hand and not fighting with virtual ENVs or your tool chain or with bugs that pop up the whole time. And you really can, if you ask yourself at the end of the day how much time you have really spent on solving your problems, solving your actual task, then the higher the percentage, the better, of course. and an efficient Python project helps you to focus as much time. Then we are, in many cases, not working alone. So a good Python project should also foster easy collaboration within a team. So help you to not step on each other's toes and to avoid any complications. And in the end, of course, you want to seamlessly build and deploy whatever you build in your Python project as some software artifact on PyPy or within your company. So how do we achieve those goals? So first of all, it's a lot about conventions. You surely know this also clean code principle that you should favor conventions over configurations. and this starts with a project structure that is as standardized as possible and luckily kind of the Python community over the years converged towards one standard project structure I'm going to present. Then also code formatting and linting. There are many, many tools out there and it's definitely important to kind of stick with one tool and with one set of conventions in the end. Also, documentation is a really important part for a Python project, sometimes a part which is not valued enough, I think. So having a clear readme directly at the root of your project that helps people to understand, okay, what is this project actually about? How can I get running? What are the first steps to get it installed? Besides conventions, another also really important thing is automation. So we should try to optimize as much as possible to avoid human errors and, again, to actually focus on solving the task at hand. Things like dependency and environment management, I think this is one of the real big problems Python over many, many years, that this can still lead to a lot of trouble, then building and publishing should be really easy, also the versioning, especially if you have many requirements or if other projects need your project, then you could, for instance, go with semantic versioning that I'm going to mention, and testing, linting, and also type checking can be completely optimized and whatever helps you to find errors before you even run your code is, of course, really helpful. In the end, all those requirements should be easy to use and should be easy to use for everyone in your team. Just mentioned semantic versioning is really a minor thing, but important to understand So we all know those version numbers with the three parts, the major, the minor, and the patch. And semantic versioning, like broken down, is really only to say if you are introducing any breaking changes, then you increase the major versions, new features, you increase the minor versions, and the last patch level is only for bug fix and hot fixes. And it avoids you in your project to be aware that other projects are using this. So if you, for instance, define your requirements, you can have those special specifiers like tilde equals 2.21, which basically says that you want this library in a version smaller or equal 2.21, but less than the next breaking version, which should be three. Poetry has something similar, but tilde equal is at least defined as one of the PEP standards. And also if you are developing something within your company, you should make use of this to avoid, yeah, dependency, struggling with the dependencies for others. I should also mention that in this talk, I'm not going to talk about what's the best package management tool. So actually my colleague gave a really great talk last year here about all the different tools and in the Python ecosystem we really love to have a lot of choice, which can also be quite cumbersome, and so in this talk I'm going to focus on Hatch, which I really like, so my personal preference. And you see in this Venn diagram that it's not covering all the features you might want, but over the last year, I mean, Python ecosystem evolves quite quickly with a special plugin for locking your dependencies. It also covers package management, and you can also install Python versions with it. So it has all the features you need. So now coming finally to what's a streamlined project set up, I already mentioned Hatch. And as I said, it's a Python project manager built by Ofec Lev. And it allows you to what you would expect to also easily build and publish packages so you don't need extra tools like Twine or whatever. all included it has a really nice robust environment management where you in a declarative way define your environments in pi project toml and hatch makes sure that it's always reflecting the state of what's defined in pipe project toml so no more it works on my machine but not on yours and did you install with pip install this or that. So Hatch takes care of this. It also comes with custom scripts. I'll come to this in an example. It has easy Python management, so basically replacing pyenv if you're using this. If you're an Anaconda user, then you're using Anaconda, but at least for people in the pip world. And you can also do the just introduced semantic versioning based on git tags, which is also common error, at least I found that it's a common error that people tag something but forget to raise it in the source code and so on, and this is in a black in allows you to pull those version numbers into your Python project based on git tags, and yeah, it also has a sophisticated testing environment with various environments, So testing things and basically replacing the tool talks if you're using it. So it really unifies a lot of things. And here I want to show what a typical, let's say, Python project in general, but especially also for Hatch looks like. I just named it PyCon.de. I don't think you see my pointer. A docs has a docs folder, as you surely all know, where you have your documentation, your references about your project. Nowadays, I would definitely recommend to use mkdocs, which caught up with Sphinx and also has a nice configuration. It also gets a lot less in your way when configuring it compared to Sphinx. Then we have the typical source layout with the actual package PyCon DE underscore demo and Y source. So in most modern project directory structures, you see this source because it helps you avoiding certain errors where Python would import the current directory. and this doesn't work here with the source so you need to really install this in an editable mode with pip install minus e or let's say Hatch does all this for you but it really avoids some common errors. Then you have the test folder of course with your unit tests and editor config which is also a nice convention because it defines for a lot of IDEs and editors like how long, how many white spaces a tab should be, and so on. So you can, within your team, lay down in here your conventions. Yeah, git ignore, a pre-commit config. I'm going to come to pre-commit an author file where you specify who worked on this and maybe who's responsible right now for the project. A change log is really helping a lot for other people who want to see, okay, so maybe I've been using this project for a year, and so what happened since then, since version 1.5. For open source packages, it's also really important to have a contributing file where you tell how and which Git workflow you maybe use and how to contribute and how to help. Then the next one would be a license. I've already mentioned the mk-tox. Then the pyproject.toml, like the real control center nowadays in any Python project. So no more setup.py, no more setup.config. So in the modern way, it's only having a pyproject.toml. And I also mentioned a README file, which should be really the introduction towards your project. And I see in a lot of projects where people, especially like internal projects, where people don't really care that much about README, but it can help a lot, especially if you've got new team members and people looking for something to have to spend some time having a nice README. So as I mentioned, PyProjectOMLs nowadays like the most important configuration file where you define some build systems. So in this case, in our case, it's hatchling. But yeah, this is what Hatch is using. I think also Rye is using hatchling. So this basically tells only what software is helping me to build this package. More important from a user perspective is all the metadata. So if you're uploading it to PyPy or some internal artifact store that you have the description, a link to the readme, which is also displayed, and also certain classifiers and the abstract dependencies of your project. And here I hard-coded the version number. Here you would then normally say that just derive it from the git tags. And so this is only a small part of a really long, long PyProject toml. What's really nice is that nowadays all the tools started to use PyProject tomls for configuration. So you also have your PyTest, your MyPy, your Ruff, your coverage configuration, all in PyProject, which makes it also easier than having a lot of different files for this. And I'll come to those tools. So as I mentioned, what is cool about Hatch is that you have environments. Like for instance here, I defined a test environment and the environments inherit from some default environment and additionally adds the coverage and PyTest. And now I can define also scripts for those environments because I'm one of the people who always forgets like, okay, what flag do I need here in this command? So I'm always having a hard time remembering all those long lines. But with scripts, I can now just say, like, if I want to run my PyTest with coverage, then in the end, it's going to just execute this. And this even works recursively. So non-cough is the coverage, minus minus non-cough and so on. And this is also nice because if you're working in a team and someone changes anything in here, then you know the next time people are running the command, then they will use already the new flags. So this helps a lot with making things consistent. And how would you run this now? So you would just say Hatch run test. So because it's the test environment, colon coverage. and it would really internally just open the virtual env, would make sure that the virtual env that is there is reflected by those dependencies, so it's consistent with those dependencies, otherwise it would update it and would execute a command. So you're not really activating any environment anymore. It's really much hidden and you definitely don't fight as much anymore with the different environments. I've also mentioned Ruff before. How many of you know Ruff already? Okay, there's already a lot. For those who don't know, you know there's Plagg, iSword, Flake8, Autoflake, all those nice little tools, but that can also be quite hard to set up consistently so that they don't bite each other. and Ruff now replaces all those tools also with Ruff format, especially plaque. And yeah, I would just like recommend to anyone in a Python project, use Ruff and it makes it also way more efficient than handling all those different tools before and it has tons of plugins. Then, MyPy is another ingredient for me for an efficient Python project because it uses type hinting in Python, also like a feature that has been around for quite some time, but I don't see it in so many internal projects actually already being used and checked by MyPy, so that you add compile type information and find errors already before running the source code, and additionally, it also helps you to document your code and enhancing the developer experience. So just a small example, like let's say I want to calculate the Fibonacci numbers, and I say n is an integer, and it returns an integer, so first of all, this documents my code also really nicely, and then I introduced here like a really stupid mistake that I return the result as a string, and if I would now run, for instance, hatch run lint typing, and typing is again the shortcut for some lengthy mypy call, then I get directly the result that I'm returning something which is not expected and I find the error before even writing a unit test or something for it. So alternative would be unit tests and it's still good to have unit tests, so MyPy makes me way more efficient and also that most IDEs like Visual Studio Code tell you this even when you're developing the codes directly. So I already mentioned unit testing here. PyTest is still the go-to tool, I would say, and has been a de facto standard for many, many years. It has powerful feature sets like Fixture, so it really is good to know PyTest really well because it helps you to efficiently set up your unit tests and also the more advanced features are definitely worth learning. It has a lot of useful plugins. So what I always use is PyTest coverage because I want to see if my code is getting better or better or if anyone in a team is maybe introducing new code which is not unit tested. PyTest recording for mocking external services is something I found out quite recently, but I really like. And PyTest sugar just makes the output a little bit easier on the eyes if you compare to the default one. I also mentioned Tox before. As I said, Tox helps you to test your code with different Python versions, with maybe different additional dependencies that you require in your code, and Hatch basically replaces Tox, which is also nice, where it's all integrated, and you can make use of this to test different combinations of your code under different environments. Precommit is also a really helpful tool and should be there in every Python project. You can avoid a lot of human errors because you can enforce when someone is committing, you are running some basic checks, like in this case trimming, trailing white spaces, doing linting, formatting, you can even run MyPy and so on, and only if those tests pass, then it's committed, and I think it's also really nice to introduce this actually when you're working alone, but also especially in a team. In the end, I want to mention automation with CI-CD for automatic and reusable testing, then not on your machine but in the cloud, which is then also way more scalable. You can also do things like package and publish your stuff based on Git tags, so a lot of automation can be done. And here, think about first what kind of branching strategy you want to use in your team, like GitHub Flow or another one. But it's important to agree on something. And you can also, as I said, automatically deploy and build your documentation. So coming to the conclusion. I think we really covered it a lot in those 30 minutes. So for me, a streamlined Python project, a modern-style Python project, has a PyProject terminal where everything is configured, has a standardized folder structure, like I showed, with a source layout and a really useful readme as a first intro into this project. Easy package management with one of the newer package managers and there I really like Hatch and can only recommend this. And then automated quality assurance with the tools I presented like rough PyTest, pre-commit MyPy and some CI, CD like GitHub or GitLab. And proper documentations and also you can automatically build references with mkdocs and in the end you should still remember that the most important things are actually that you optimize as much as possible and that conventions are key for efficient development yeah i couldn't help myself but put this into a little cookie cutter template so if you now think, okay, he talked a lot, but what is the actual, I want to see it. You can just use cookie cutter to use this template. I call it the Hatchler to try it out, what I just talked about. I'd love to hear your feedback. I want to say thank you especially to the maintainer and creator of Hatch for some awesome work in his spare time. Michael, who helped me with the slides. And yeah, with this, I want to say thank you. And let's hear some questions.

Speaker 2 [25:37]

so much. It was a really great session and I hope you all have learned so many new things. Now we have some questions. First question is, have you used UV as a PIP replacement and how would it fit into your infrastructure?

Speaker 1 [25:53]

Yeah, so UV is this new tool which was also adapted by Astrill, by this company, I think. So basically PIP on Rust. And I haven't used it now. I think it's going to definitely replace maybe PIP one day and going to be part of a lot of other tools internally. It's not in this, so as far as I know, Hatch doesn't make use of it right now, but I see no reason why it couldn't just call internally uv instead of pip. But the answer is I think at the current state not, but let's see what comes in the future.

Speaker 2 [26:45]

What content goes into the readme and what into docs?

Speaker 1 [26:50]

That's a good question. For me, the README is one or two sentences about, okay, this project is about solving this and this problem, and you need it for that. Something really, really easy, then maybe the first steps, if you like, that people can just copy and paste, maybe pip install it there and there, and then maybe a link to already the larger docs, so keep the readme markdown or mostly markdown file, keep it short, concise, and always updated. Maybe also a link to how can I contribute or which other persons that I should contact, but nothing really lengthy, but it should be always up to date. No one wants to read the README, where you try to get started and you see, okay, it's completely outdated, it doesn't work anymore, and everything larger, like a lot of examples and the reference and other use cases, you should rather use mkdocs for and have then a proper site serving this.

Speaker 2 [28:15]

Okay, so the next question is, how does Hatch compare to Rai in regard to feature completeness and maturity?

Speaker 1 [28:22]

Compared to rye in...

Speaker 2 [28:25]

Right. Hatch compared to right. In regard to feature completeness and maturity.

Speaker 1 [28:32]

Yes, so let me switch back. Okay, that's the problem when you have too many of those. Yeah, so Rai is in the middle and Hatch, meanwhile, too. So I would say I haven't used Rai, actually, but I looked it up, like the features. I would say they are comparable. Rai is built on Rust, so it surely is faster, faster, but actually speed was never that much of an issue when working with the package manager. So I like more that it helps me with my development style. As far as I know, Rai doesn't have this nice feature of having really managing many environments like Hatch does. But yeah, if the one who asked the question, I would love to have a discussion about this later on what your experience with Rai is.

Speaker 2 [29:32]

Unfortunately, the time has ended, but we have lots of many questions. So I would like to tell you that you all are free to ask questions in person with Florian whenever you want in the conference. And here I would like to end the session. Thank you so much for listening.