Python Dependency Management

Speaker 1 [00:03]

Yeah, thanks. Welcome, everybody. So, I'm Patrick. I'm a software engineer. I'm working for Blionda. Maybe you have seen our booth outside. What we are doing is we do machine learning for retail. So, for example, if you go into a grocery store and want to buy a banana, we make sure that there actually is a banana you can buy so that the supermarket is not out of stock. And on the other hand, we optimize that at the end of the day, there are not too many leftovers, so that there's no waste or not that much. And at Bluionda, we do almost everything with Python. But today, I don't want to talk about machine learning or retail. I want to talk about Python dependency management and this whole packaging and dependency management topic in Python is quite an emotional topic, I would say. So, for example, a few months ago there was quite an emotional discussion on Reddit where authors or users of different tools were fighting which one should be the one recommended tool and so on. And I was actually a bit surprised about that because I couldn't understand why people are getting so angry because of something like this. So personally, I think dependency management in Python and packaging is not perfect, but works good enough, at least good enough for me so that I can focus on the real problems, like predicting bananas, for example. but what are we actually talking about today at first I want to go about a few general things and what are actually the problems and then I will pick a few of the current tools there are lots of tools actually by now I will pick a few and just show what they can what are their features but I won't make a comparison in the way, okay, tool A has this feature, but tool B doesn't. So, because that's kind of negative for the one tool, but maybe tool B doesn't even try to solve this particular problem and so on. Okay, so let's start. So, let's imagine we have some piece of code, this Python module, banana predictions, and now we want to make this available to others because others could also benefit from our great banana predictions. So how do we provide the code to others? You could upload the script on some website so that users can download this, but the usual way is you create a distribution. So, you do that via setup tools or distutils. We have to set up pi. Here's an example, pretty simple, just a setup function with a name. You put in a version and here we say, okay, we have just one module which should be packaged, kind of. So, and with the sdisk command, we then can create a source distribution. and that's just an archive with our code and some additional meta files. We could also use a create wheels instead of source distributions because source distributions have the problem that it sometimes, as it can take pretty long when you install them, then you have code that has to be compiled and wheels kind of solve the problem. So, how can users now find your code? There's the PyPI, the Python Packaging Index, there you can upload your packages and others can find your package and then download it or install it. So how do you upload it? There's the upload command. The problem with that is the upload command doesn't use HTTPS, so what happened then? Another tool was created, Twine, to solve this particular problem. So, now, if you want to, if your project grows, you might want to reuse other libraries you find on PyPI, how do you do that? There's setup tools with install requires, so you can just put in a list of the libraries you want to install, and yeah. So, but then how do you install packages you find on PyPI, there's yet another tool called pip. Before it was easy to install, but that had quite some issues, so now the standard way is pip. And next problem is when you have multiple projects which benefit from the banana predictions library and when you need multiple versions for whatever reasons of that library, then you kind of need dependency isolation. And how is that achieved? Yeah, there's yet another tool you have to use and that's virtualenv. So with virtualenv you can create an isolated Python home basically. So the next thing is when you have developed something, you have, you know the requirements, you have installed and the versions with, yeah, so with this particular versions your project really works. And if you move that to another machine or some other developer wants to use that, you kind of want to have the same version so that the other developer can start with a working project. So you want reproducible environments. So there is kind of this requirements.txt file. So if you have installed packages with pip, then you can do a pip freeze, then you get a list of all the packages you have installed with the exact versions, and you can dump that in a requirements.txt, and someone else can install that. Okay, so now what do we have? So we have seen we have multiple tools for the whole workflow. We had PIP, virtual env, Twine, and so on. Another minor issue is that PIP itself doesn't have real dependency resolution, so if you have a bigger project with lots of requirements, then it might be that the dependencies cannot be resolved because multiple of your dependencies depend on the same dependency with different versions and so on. Another problem is that for those reproducible environments, you actually need multiple files. So if you want to deploy something into production, you don't want to have your test packages or dev packages or whatever. So what is usually done then, you create in addition to a requirements.txt file a def requirements.txt file or a test requirements.txt file, whatever. Another problem is kind of the inconsistent usage of requirements.txt files because it's really the left side here what you want with all the dependencies with the specified version. So the left is basically what you need. But on the right, is that what you actually want? So you just want SciPy. You don't really care that SciPy depends on NumPy and so on. It's even more important if you make a distinction between applications and libraries, because if you develop a library that is used by others and you have SciPy as a dependency, And then you say your library needs exactly SciPy 1.1. Then the other library that wants to use your library cannot use another version of SciPy. That's kind of a problem. But for applications where you know, okay, that's one blob you push into production, that's something that runs like a web service, there you want to have all those pinned requirements. Okay, so there's a new proposal, kind of. There's a specification, that's the pip file specification. And the pip file is a TOML file, it's human readable. There you would specify your dependencies, your dev dependencies. And the pip file log is an additional file, which is machine generated, it's a JSON file, And there you have all the packages with the exact versions you need to reproduce your environment. So another problem or limitation can be that if you need a more complex build, then extending disk utils or setup tools is not that easy or not that nice. It's just not a nice developer experience. So to solve that, and you cannot say, okay, I just don't want to use these two tools or set up tools, I bring my own. But that will be possible with those two PEPs. So there's one PEP that defines a build system independent format for source trees, and another one where you can specify minimum build system requirements, and that will manifest itself in this PyProject.toml file. And so that's also a .toml file and on the bottom you can see you have this build system section and can say it needs, here it's the Poetry build system in version greater 0.12 and you give them the entry point where the implementation of the build system is. Okay, so lots of problems, there are proposals how to solve one of them and what kind of happened over the years is lots of people made attempts to solve those issues and bring a better workflow where you don't need lots of tools but you have one and so on And if you look at the current landscape, it's, yeah, there are quite a lot. I didn't even know half of them before the talk, actually. So there's pip tools, there's bento, there's poetry, something flit, hedge, conda, pipinf. There are other approaches like pans, basal. Actually, if you are interested in PANS and PECS, monorapers, there's a talk I recommend. It's tomorrow by Stefan Erb, a colleague. So you should really check that out if you are interested in dependency management and so on. And so lots of tools. Every tool is the best, of course. so I don't have the time to present all of them I also don't have the possibility to go too much into detail I picked those four pip tools, poetry, Hatch and pipenv those are also the two tools which were the reasons for this emotional discussion a few months ago so let's start with pip tools It's the most simple one actually. It's just two command line tools, pip compile and pip sync. And what it does actually is it kind of improves this pinning workflow so you can define one requirements in file is the convention where you specify the dependencies without version constraints. and pip compile will then generate out of those requirements.txt file. And yeah, it also has this pipsync command, which synchronizes your virtualenv with an updated requirements.txt file. For example, if you update the requirements.txt file, you call pipsync, then your virtualenv will get exactly those dependencies. So dependencies that are not needed anymore will be deleted and so on. How does this look like? So you don't need this requirements in file actually, you can just use a set of pi with the install requires list and if you call pip compile, you get this requirements.txt and what's nice is you will even see where dependencies of dependencies come from. So here we see that for example, urllib3 was pulled in via requests. There's also the possibility to use pips hash functionality. So you can use not minus minus generate hashes, then you get more security so that you can really be sure the package that is downloaded and installed is really that I used before. Yeah, if you want to upgrade dependencies, you can use pip-compile-minus-minus-upgrade. Like I already said, pip-tools improves this whole pinning workflow. And yeah, that is pretty simple. And it brings better dependency resolution than just pip. So then there's pipenv, pipenv does a little bit more. It's a wrapper around pip and virtualenv so that you don't need those tools, don't have to use those tools by hand anymore. It has also lots of convenience features. So if you install a package, for example, and you don't have a virtualenv yet, virtualenv yet, then it will create one for you, so you don't have to do that manually. If you want to install a project that specifies certain Python version, so let's say, the needed Python version is 3.7, and you don't have Python 3.7, but you have pyenv installed, then pipenv will install Python 3.7 via pyenv for you. and pipenv implements this pipfile specification in addition to setup.py, and pipfile says actually it targets applications, not libraries, so pipfile and pipfile-log you would use for when you create an application and when you want to develop a library, you would stick to setup.py and define your requirements and so on there. That's how a PIP file looks like. Like I already said, it's a TOML file and you have this separate sections for main packages and dev packages so you don't need multiple requirements TXT files anymore and this optional section where you can specify which Python version you want to use. So, the log file here, yeah, there's nothing really special. It's a JSON file, it's lots of metadata, and yeah, on the bottom you can see the dependencies with the exact versions, and pipenv uses hashes by default. And yeah, the list would just go on. What's nice when you are developing your project and then you install a new package and it's not already in pip file, then pipenv adds this for you. So that's also something that's automated away and it regenerates the log file. Same if you uninstall something. and yeah as there are actually many more features of pipenv so if you think that looks nice then check it out and the next one is poetry so that's a little bit different of what it wants to achieve actually, so it's really managing the dependencies, also building the packages and publishing them. It also is a wrap around pip and virtualenv, so it also has this automatic virtualenv creation. Instead of just pip file, it uses the pyproject.toml we saw earlier, and it generates a poetry log file as a log file. So, yeah, and it claims it has a better dependency resolution than other tools, but I don't know, maybe. And it can also scaffold a project for you. So if you want to start, you can just use the new command and then you get a basic Python structure and you see here the pyproject.toml. If you add your dependencies, it also updates the pyproject.toml file and the log files and so on, it's the same as in pipenv. It has a version command so that you can bump the version of your project. You can give them arguments, do you want to bump the major version, patch version, and so on. When you use the build command, it creates source distribution and the real file for you. Then you can use the publish command to upload it to PyPI or some internal Python index. And yeah, that's how the log file looks like. So it's also a TOML and you can see here we have the requests dependency in version 2.20 and then below you have all the dependencies and requests specified. And if we would, yeah, if the whole file would fit on the slide and we would see the same sections for the other packages, so ULF3, for example, would also be main package there with a pin version and you would see the dependencies urllib3 has. So that's kind of recursive. Okay, that was poetry. The next one is Hatch and, okay so Hatch tries to simplify the usual development tasks it does that by wrapping even more tools so it wraps pip, virtualenv, twine, pytest and so on but it doesn't use the new files, so neither pip file nor pyproject.toml. It uses plain setup.py and the requirements.txt, but it also has this scaffolding feature to create a project pretty fast. When you use Hatch install, you can specify a package or if you are in a project, then it would just install all the requirements that are specified in setup.py, for example, and it also creates a virtual end for you if there's none already. Yeah, you can execute tests with Hatch, so it would execute PyTest. You can also use the coverage flag, so you get a coverage report too. And it's actually working pretty good, so I tried that with a private project. I just cloned it, executed Hatch test, and yeah, it picked up the test requirements TXT file, installed everything, and could execute the test, so it's pretty nice. It's like Poetry version. Hatch has a grow command, So to bump the versions, you can also specify major, patch, minor, and so on. It also has a build command to create source distributions and wheels. To upload to PyPI, it has hatch release. And yeah, those were the tools, like I said. All those tools have actually more features. I just don't have enough time for everything or to go too deep into detail, check it out. I think all the tools are nice and solve problems. The question is just, does it solve your problem? So if you ask yourself now, which tool should you use? I really don't know because yeah, It depends, like so often in software engineering. But what you can do is you can ask yourself some questions. What is it you try to achieve? Are you building a library? Are you creating an application? Or what is even more important, in my opinion, or I recognized it when I played around with the tools, I am pretty biased by our, because of our Blue Yonder infrastructure and so on, and so I was, sometimes when I saw the features those tools bring, I, at first I didn't even understand why you would need this, just because I am used to the tooling we have at Blue Yonder and how the infrastructure looks like, how we deploy to production and so on, So you should not forget something like that either. And I think that's already it. So yeah, but I, the takeaway message is really there's not one tool that solves everything. It doesn't make sense to create a shitstorm or yeah, a flame war, whatever. Thank you. And so we are also hiring, so if you are interested to hear more about Blionda, just come by to our booth. And I would actually prefer no questions, but I would be happy to talk with you. So if you want to discuss more, then just find me. I'm here the whole week. I'm probably at our booth at some point. Yeah, thanks.