An unbiased evaluation of environment management and packaging tools

Speaker 1 [00:02]

Yeah, hi everyone. Welcome to this talk about an unbiased evaluation of environment management and packaging tools. I'm really excited to be here with you today. And yeah, before we jump in a few words about myself, I'm a machine learning engineer at Innovex. And Innovex is also a diversity sponsor today. So if you want to learn more about the company, come and check out our booth. In my free time, I like to work on personal projects, which I usually share either on GitHub where my handle is SotronenEyes, or on my personal web page where I have a blog. And two of the most popular repositories, which might be interesting for you as well, are Machine Learning Basics, which implements fundamental machine learning algorithms like support vector machines, Perceptron, decision trees, and so on in plain Python. So really, we're the focus on understanding the algorithms rather than performance. And another one is Magical Universe, which introduces Python features like decorators, iterators, and so on, using a world of magic. So if you like Harry Potter like I do, then you will definitely enjoy this project. Okay, let's jump right in. First of all, regarding the goal of this talk, I think Python packaging is quite messy. And if you get into Python, understanding which tool to use when, which tool can do what, which tool is better than the other, is really, really hard. And lots of talks exist on the topic or tutorials, but I couldn't find really an evaluation of the different tools, what they can do, which tool to use when, what they are good for, without having this personal preference and they're like, yeah, we use this in our project or I had problems installing this, so I don't like it anymore. and this is what I want to do with this talk. I really want to give you an unbiased evaluation of the available tools. And what we will do is that we will look at a categorization first. Oh yeah, by the way, there is also a PDF version of the slides online on the page of the talk if you want to follow along. And we will look at each category in turn, first at a short definition and motivation, and then at the single-purpose tools. and in the end we will also look at the multi-purpose tools. So regarding the categorization I identified five main tasks that are relevant for packaging and environment management. These are Python version management, package management, environment management which is usually concerned with virtual environments but doesn't have to be and we will talk about this later as well and package building and publishing and as you can see here we have lots of tools for the different tasks and some can do several tasks some are single purpose and you will see this Venn diagram a lot so don't worry if you can take it in right now and as I mentioned it's also on the slides and the PDF version so let's maybe look at each category very shortly so you get a first idea. Okay, let's say Python released a new version and it has some new cool features that you want to try out. Then you would want to be able to install different Python versions, switch between them easily to try out these features. And that's what Python version management is about. You can see here there's one single purpose tool we will look at, PyEnv, but also some that also fulfill other tasks like conda and next category is package management you should all be familiar with that since if you work on a project you usually have dependencies you want to use other projects and the code and you would have to install packages usually with pip but there are also other tools and yeah so this is another category we'll look at again you see here we have some tools like poetry that can also do package management, but a single-purpose tool is pip. Then if you work on projects, you might have several personal projects or work projects, and projects usually have dependencies, and sometimes a project, two projects have the same dependency but different versions, and this can become very messy. So then you usually would use a virtual environment or an environment if you use one of the other tools, and this is what environment management is about. Again, we have lots of tools that can do that, but in particular, the single-purpose tools would be vent and virtual, which we will look at, but then also some multi-purpose tools. And yeah, let's say now you have your project, you want to share it with other developers or with the community, so you want to create a package yourself, then you would first have to build the package. We will talk about what this means later on as well and you have lots of tools you can choose from if you want to build your package and in the end then you might want to publish it to PyPI or another index then you would have to do package publishing and again lots of tools. So you can see here that it can be really hard to get a grip on those tools. So this was just maybe as a short introduction of the different categories and now we will walk through each of them individually. So first up is Python version management. A tool that can do that is able to install different Python versions or lets you install them and lets you switch between them easily. And why would we want to do that in the first place? And there are several reasons. For example, different projects might require different versions of Python or a single project supports several Python versions that you would want to test and, yeah, or you might just want to check out the newest Python version or an older one and which features it has to offer. Python has a single purpose tool which can do that, which allows you to install a many different Python versions, which is pyenv, quite a popular tool. Oh, sorry. And pyenv is really easy to use. So, the most important commands would be pyenv install with a certain version. Now, this example, 3.10.4, but you also, yeah, it also lets you easily switch between the versions. You can either activate a version just in the current shell session Using pie and shell and the version number you can set it locally for the current directory Directory using pie and local and the version or you set it for your user account using pie and global So and pie and is the single-purpose tool power for Python version management Okay, next up is Environment management usually virtual environments and what does that mean a tool that can do environment? environment management lets you create and manage environments which are usually virtual why do we want virtual environments I already talked about this yeah usually projects have specific requirements so they depend on other packages and usually or often projects require different versions of a package and this can cause dependency conflicts and but yeah you can also get problems if you just use pip install with a package name since this places the package with your system-wide Python installation which can cause some problems like system pollution. Some of you might say now but there's a user flag now with pip minus minus user install or pip install minus minus user and this definitely handles some of the problems but as a beginner, some people might not be aware of that option. So you just should keep in mind there is definitely a reason to use virtual environments, lots of reasons to do so. Python has two single-purpose tools for this, which are venf and virtualenv, which you can see here, but also some multi-purpose tools, but we will look at them later. So, venf and virtualenv. VENV is built in into Python and it's a simple tool, it has one purpose namely creating and managing virtual environments and the most important commands are creating a virtual environment, activating and deactivating it, creating is easy, you just call Python minus M which is the flag to call a module directly with VENV and the name of your environment and then you can activate it by calling source on the activate script and deactivating it by just typing deactivate. But as I mentioned, venv is a simple tool. So it does not have a lot of features. But that's why we have virtualenv, which tries to improve venv. So it's extendable, it's faster, but it also has some other advantages over venv. But you have to install it. So it does not come with Python. And the most important commands are similar to venv, a bit simpler is creating a virtual environment where you just call virtualenv with the environment name. But again, activating, deactivating is the same as before. Okay, so we have looked at two categories already, Python version management and environment management. So before we jump into packaging, I want to make sure that we are all on the same page Regarding the two most important files, so, yeah, packaging has come a long way, and until PEP, I think, 518, we used setup.py files to create packages, and then usually setup tools as a build tool, and in PEP 518, the usage of a pyproject.toml file was introduced, so right now, if you create a package, you will need to create a pyproject.toml file, and this file then replaces this old setup.py which was used in the past and pyproject.toml contains all kinds of settings you can have project settings like the name I have an example file here from the pandas library I don't think you can read it very well but yeah you have the name you can have urls the dependencies but you can also define scripts so it's these the single file where you would put all your project set up and dependencies and names and so on. So now we can get into packaging. First of all, what is packaging or package management? It's about downloading and installing libraries and their dependencies. And why would we want to use packages in the first place? Again, there are multiple reasons for that. usually packages or what is nice about packages is that they allow you to create a hierarchy of modules and to access modules easily so you can import a package very easily using the dot syntax like from I don't have a good example now like from pandas dot something import something sorry about that yeah so this makes it easy to use the code but packages also make it easy to share code with fellow developers and if you use your PyProject.toml file to set all the dependencies of the package, other developers can just install the package and they don't have to worry about installing all the dependencies individually, they just get the package code, install it and can use your package or your project. The standard package manager for Python is pip, so it's the single purpose tool which which you see up there in our Venn diagram. Again, we have the other multipurpose tools like pipenv and conda or pdmpoetry pyflow, which we will look at later. But you're probably all familiar with pip. Pip is shipped with Python. It is the standard package manager and it allows you to install packages either from PyPI but also from other indexes. And the main command is very easy. it's just pip install with package name. Of course, there are lots of flags, but we won't get into details here. Okay, so let's say now, hmm, not working. Let's say now you want to create your own package. You have code that you want to put into this nice little box and share it with the community. then you first have to create, or you first have to build your package. This means that you have to create distribution files. Distribution files are usually a wheel file, dot wheel, and table files, which are dot tar dot gz. One is the source and one is the build distribution. I won't go into details here, but it's interesting to know about if you want to learn more about packaging. There are lots of resources on the internet. Okay, so Python has lots of tools to do the building step. The single purpose tool, which I already mentioned, is setup tools, which has been around for a long time, but there are also Flit, Ensconce, Maturin, Pyflow, Poetry, Hatch, so lots of other tools. But as the single purpose tool, let's look at setup tools. It was developed as a build backend already in 2004 as an enhancement for disk utils, which was used at the time. You can easily use setup tools by just specifying it in the build system section of your PyProject.toml file. You can see an example here. And since setup tools is a build backend, not a build frontend, you would also in the end have to use usually the build tool to build the package and you can simply then call Python minus m build. Usually you would, most of the people I know would use a multipurpose tool to do this rather than doing it manually like this but you can do it in theory. Okay now you have your package, you have your distribution files and now you want to maybe put them on PyPI or another index, maybe some private index, that's the step of package publishing. So a tool that allows you to do package publishing lets you publish the package to PyPI or another index. In our Venn diagram, we see that Twine is the single-purpose tool for publishing a package, but there are also the other ones like Flit, PDM, PyFlow, Hatch, so on. Twine is the official PyPI tool to upload packages. It also works with other indexes than PyPI and it's a single purpose tool. It's just concerned with uploading or publishing packages and what you would do is that you upload the distribution folder. So the folder where you have your wheel and your tarball files using the command twine upload and then the folder name and with the files. So now we can get into packaging or the multipurpose tools, but before we go there, I want to do another short recap, namely on log files, which you might not know. So in your pyproject.toml file, you usually define abstract dependencies. So that means that you just list the dependencies without version numbers or with version ranges, but you don't pin them to exact dependencies. what is done in a log file. So a log file contains concrete dependencies. And again, here's an example from Poetry. And if you can read it or maybe look at it later on, you would see that it really contains the exact versions of all dependencies and sub-dependencies of a package. So this allows you to do reproducible builds and really put everyone on the same page regarding developing a project. So this is a log file. Sorry, I always have to switch between those two. Okay, so regarding our multi-purpose tools, we will start with PIP-ENV, which sits here in our Venn diagram, so it can do environment management and package management. PIP-ENV has been around for a long time. So as the name says, it combines PIP and virtual ENV, And it introduces two additional files, a pip file, which is also a TOML file, similar to pyproject.toml, and a log file, which we just saw, so you know what it's about now, which is intended to replace requirements.txt, which was used or maybe still used in some projects. and yeah, pip file is used to define your project dependencies similar to pyproject.toml so you would put the name there, the dependencies and so on and the log file then allows you to do deterministic builds so really pins down every version of every dependency. Now you see the most important commands here with pipenv you can install a package using pipenv install you can run a script using pipenv run but you could also activate a virtual environment using pipenv shell so when you use pipenv you don't have to manually create virtual environments anymore the tool is doing that for you using the dependencies that you specify in pipfile and then there is conda of course conda can do everything pipenv can do so environment management, package management and in addition also python version management Conda is a general purpose package management system so it's not limited to Python packages and special about Conda is that it uses its own index for packages so it does not pull them from PyPI. Conda is a massive tool so it has a lot of features and I won't go into more detail here since again you will find multiple resources to learn more about Conda. Now we can look at our packaging tools and since I promised an unbiased evaluation of them, I came up with a list of features that I think are important regarding packaging tools and we will look at each tool now and these features. So, does the tool allow you to manage dependencies? Does it resolve and lock dependencies? So, does it also create a log file? Does it have a clean flow for building and publishing a package? Does it allow you to use plugins? And does it support the two main, or there are lots of PEPs on packaging, but I think these two are important since the projects differ. So, one is if the project supports PEP 660, which is about installing packages in editable mode. So PIP, for example, allows you to do that. And what that means is that if you're developing a package and you want your changes to be like you install the package and then you work on the package and add new features and you want those changes to be directly available in your environment, then you would install the package in editable mode. So it's quite an important feature to have if you are developing something. And PEP 621 is about how you specify your project metadata in PyProject.toml. So it sets a standard way on how to do that. That's nice to have, but not like a development feature. Okay, so let's start with the first tool, which is Flit. So Flit, as you can see in our Venn diagram, has a very specific use case. It's only about building and publishing packages. This is just in text again what we just saw in our Venn diagram. It does not care about managing your Python versions, about managing packages or environments. It's just about building and publishing a package. Flit is special since it has a very specific use case. First of all, it's just for pure Python packages. so if you have any extension code like in C++ then you could not use flit since flit is just for pure Python packages so those that would not need the build step and it only cares about packaging and that's also reflected in our feature table so it has a clean flow for building and publishing a package and supports the two peps however it does not allow you to use plugins It doesn't care about resolving or locking dependencies or managing them, since it's just really for the publishing and building of a package. But it's very easy to use. So you can use flit init, which would create a new pyproject.toml file for you. And if you then want to build and publish your code, then you would use flit publish, which is doing both steps for you. And if you have dependencies, you would have to add them manually to pyproject.toml. So we will see now in a bit that there are some tools which have commands to do that. Okay, next up is Poetry, which sits here in our Venn diagram. Quite small, the name. But you can see that it can do everything except for Python version management. Here's in text form once again. So, yeah, it fulfills all of these tasks. if we look at our feature list then we can see that it does not support PEP 621 which is interesting that has been an open GitHub issue for that for about one and a half years I think but all the other main things are supported like it allows you to use plugins, it has a locking functionality and resolves your dependencies dependencies for you and allows you to manage dependencies, which is also reflected in the main command. So you can create a directory structure, so it would create an empty test directory, your pyproject.toml file, but also the dunda init files and so on for you using poetry new and the project name. You can also create only the pyproject.toml file interactively using poetry init, which would prompt you to provide the basic information about your package like the name, the description and so on. And you can use poetry install to install a package from a PyProject.toml file. And what is happening here is that poetry looks at the PyProject.toml file at all the dependencies, it resolves all the dependencies and it also installs them for you. And package dependency management or package management is also implemented, so there is the command poetry add with a package name, which would install the package, but also write the package into your PyProject.toml file for you, so you don't have to do that manually. And yeah, there's a nice command to look at all your dependencies in a tree format using poetry show with a tree flag. running code is also easy you can either activate your virtual environment which is created for you using poetry shell so then you have a shell session with the virtual environment activated or you can run scripts using poetry run I mentioned that earlier that in pyproject.toml you can also define scripts for your project so those you could run with poetry run but you can also run everything else like poetry run and then python script name for running your Python script. Maybe good to know is that if you use Poetry Shell and your virtual environment is activated there, then you would not need Poetry Run anymore. Then you could just call Python and your script. And regarding the log file, so Poetry has a locking functionality, so when you install a package for the first time, Poetry will resolve all dependencies for you and write the exact versions of all those dependencies into the poetry.log file. And it's best practice or recommended to publish that log file with your project so you allow other developers or contributors to really have the exact versions of the packages and dependencies installed. And if you once created your log file, it's not updated automatically, but you can do that with the poetry update command. And, yeah, it's also very easy to build and publish a package using PoetryBuild, PoetryPublish. PoetryBuild creates the distribution files for you, and PoetryPublish would put them on PyPI if you have a PyPI user name and password. And what you will see is that, in general, for these multipurpose or packaging tools, the commands are all very similar. So you will see this build and publish command, like, in every tool. Okay. Next up is PDM. PDM is strongly inspired by PyFlow and also Poetry. That's why it sits right where Poetry sits in our diagram. And, yeah, it can do everything except for Python version management, just like Poetry. Yeah. Since it was strongly inspired, it has very similar features, but it's a rather new project. It was started in 2019, I think, so it requires Python 3.7 or higher. And it's one of the few tools that implements PEP 582, which is an alternative to using virtual environments. So it's based on so-called local packages and not on virtual environments. And, yeah, check out the PEP if you're interested. This This PEP is currently under consideration, so it hasn't been accepted or implemented widely yet. PDM supports all of the features on our list, so it supports the major PEPs, it allows you to use plug-ins, you can easily build and publish a package, it has a locking functionality and allows you also to manage your dependencies. As you can see now here, very similar commands, but PDM has a bit fewer commands than Poetry. Like there is a PDM init command, which would allow you to create your pyproject.toml file interactively, but there is no command that would create the whole directory structure for you. The install command is similar to Poetry. It would read the pyproject.toml file, resolve all dependencies, and install the packages for you. Then dependency management is easy. you can use pdm add with a package name. This would get the package from PyPI, look for the correct version that does not conflict with your other dependencies, and then put that into your pyproject.toml file, the package name. And again, you can easily look at your dependencies that you currently have using pdm list. Running code. There is no command like pdm shell, which would activate a virtual environment in your current shell. but you can always use pdm-run, which then implicitly runs your scripts within a virtual environment that has all the dependencies installed that you need. And also the log file functionality is very similar, so when you install a package for the first time, all your concrete versions, all your exactly pinned dependencies will be written to pdm.log, and you can also update that file using pdm-update. And, yeah, building, publishing, very similar. You have a PDM build command. It creates the wheel and table files and PDM publish to put them on PyPI or another index. And, yeah, last but not least, we have Hatch. Hatch allows you to do these three things. So it allows you to do environment management and building and publishing a package. you can also see that here regarding Python version management there is an open GitHub issue but it was just created a few months ago and regarding package management this is something that is being worked on and the core developer said that he wanted to add the locking interface by the US PyCon which I think was last weekend so maybe it's already now implemented so while locking interface package management basically comes down to being able to lock and resolve dependencies. And that's why it's currently not implemented but will be in the future. Now, this is also reflected in our feature list since the locking functionality is not implemented yet you cannot manage dependencies or resolve them but it will be added at some point. But all the other features structures are available and also the PEPs are supported. And yeah, now the main commands. You can create your whole directory structure using HedgeNew with a project name. There's also an interactive mode for this using the minus i flag. And you can initialize an existing project using HedgeNew and the init command. And this would also convert setup.py files into PyProject.toml files, so it's nice to use if you have an old project and you want to convert it into the new standard. Dependency management is not implemented, as I mentioned, so there is no command like hatch add, but you can easily look at your dependencies using hatch, dep, and then show, for example, as a table format. Now, if you have dependencies with hatch, you would also have to put them into your PyProject.toml file manually. running code there's a shell command which would activate a virtual environment in the current shell or you can also use head run to run all kinds of scripts or just Python commands and same as before so and there is building and publishing with this simple straightforward commands build and publish. And now one cool thing about Hatch is that it allows you to declare your environments in your PyProject.toml file. So there is a section there. I don't have time left if you want to do questions, but this is definitely a feature to check out. So you could have one environment for style checks or formatting your code where you have the dependencies like black, eyesort, flag aid which are just concerned with that and scripts defined on that. You could have one or declare one environment for building your documentation with mkdocs and I really like organized structured things so I love this about Hatch and damn it now I'm putting some bias into this presentation. So yeah this is a cool feature which is special about Hatch. And now before we get to the question, here you have an overview about the tools, and this is also an important takeaway. There is not the single best tool in Python to do packaging. So sometimes if you just want to use a virtual environment but don't create a package, then maybe vent for virtual env is the tool to use, but regarding packaging, it depends on maybe what your team is using, that you all maybe use the same tool, that you're familiar with the tool and that it's easy to use and actively developed, but this is a big discussion. There's a big discussion going on in the Python ecosystem at the moment regarding packaging since there are so many different tools and it's hard to really say this is the single best one. Yeah, sorry if you expect it now that I give you the best solution for all your problems. Okay, this is the end. Thank you very much for listening and now we have time for questions which you can ask on... I forgot the name. Yeah, Slido. Thank you.

Speaker 2 [34:18]

Thank you very much, that was a really interesting talk, I was very happy to listen to it. We got a lot of really interesting questions. The one that was the most popular is, would you recommend PDM over poetry and if so, why?

Speaker 1 [34:33]

You and your opinions. Oh, wow. Difficult questions. No, I can't really give a good answer to that. I wouldn't recommend one tool over the other. I know that some people have issues with poetry since at one point they pushed a new change and didn't warn the users before and this broke code for some people and that's why I know some people don't like poetry. but for me it always worked well. I haven't worked so much with PDM yet, so I'm maybe not the best person to answer the question if that's really a better tool.

Speaker 2 [35:16]

Thank you. The next popular question is, which of these tools has the best dogs community?

Speaker 1 [35:24]

Okay, so my personal favorite would be Hatch, since it's so actively developed, and I think the core developer is just doing all this development with a mouse, since he's disabled, and I find that really impressive, and I think it has a very cool set of features and very friendly contributors, so yeah, I really like that.

Speaker 2 [35:58]

So the next question in the line is, how would you package code for an offline installation on a customer server? That's quite a specific but interesting question.

Speaker 1 [36:12]

I think I have to think about that. So maybe the person that asked that can come to see me later, then I can think about it before I give a wrong answer.

Speaker 2 [36:25]

Thank you. The next question is, Pfeiffer was in the middle, but you didn't mention this in your presentation. Why?

Speaker 1 [36:31]

Yeah, sorry, I forgot to mention that, so I didn't have time to get into all of the tools, and PyFlow hasn't been actively developed for a long time, so that's why I decided I had all the slides ready, but then I decided to, since I had to, yeah, focus on some tools to not present it since I think active development in the Python ecosystem and especially on packaging is a really, really important feature, and working with a project that is not actively developed can be really frustrating.

Speaker 2 [37:09]

The next question, is environment management needed with Docker or package management is enough?

Speaker 1 [37:18]

Yeah, I think those are different use cases since if you think now about a beginner in programming, Docker is not the first thing that you learn about and that you use. So I think virtual environments are really important for this other use case where you would not use Dockerized or containerized applications. But then, yeah, that works also just to use Docker and your containers. Thank you.

Speaker 2 [37:51]

Thank you. There are so many questions. Yeah, we won't get to all of them You mentioned some tools are Python only. How would and that's a broad question So you may not have to answer all of it. How would the diagram change if C++, Julia, etc are considered? I'm assuming it's extensions and not the whole ecosystem and not just the whole ecosystem But if you're including extensions, would the diagram change?

Speaker 1 [38:15]

I don't think I understand the questions. So, for example, Maturin and Enscons, which you saw in the beginning, those are also one, I think, is for Rust code. Yeah, here. And only Flit is a tool that is for pure Python packages, so the other dependencies managers also work if you have extension code. Does that answer the question? Not sure if I understood the question. Otherwise, just ask me later on. We can discuss that.

Speaker 2 [38:52]

What tool would you recommend for building packages that are only to be used internally?

Speaker 1 [38:59]

Okay, so I don't think the internal thing is the important point, since it is more important that you can use the tool nicely, and I already, spoiler, that my personal tool favorite at the moment is Hatch, so I would always recommend Hatch now to people to use.

Speaker 2 [39:19]

nicely with the next question which is what is your stack?

Speaker 1 [39:25]

That depends if I'm at work. It really depends on which customer project I work with and what Is used in the corporate environment? so Yeah, that can be lots of different tools. So and of course a lot of and docker kubernetes and Well, there are so many different tools we started using Snowflake now recently which is not so popular yet but I think will definitely come. I've worked with the Databricks at customers so most of my work is at work time so it really depends on what the customers are using.

Speaker 2 [40:06]

Related to going into the opinions are there any tools you would not recommend?

Speaker 1 [40:18]

You have to, yeah, build your own opinion.

Speaker 2 [40:22]

Next question, thanks for the interesting talk. Can you summarize, give some tips on how to combine PIEMF, CONDA, and PIP?

Speaker 1 [40:45]

I know that it's possible to combine them. I have never combined them myself. I usually always use these multi-purpose tools since I find them quite convenient and nice to structure my work. And I always, like in my personal projects, I always keep it simple since it's rather tutorial-based or educational projects. But at work, we're always in big teams with many people usually. And that's why the multi-purpose tools can be very helpful. So, sorry, no. but I know that there are tutorials out there.

Speaker 2 [41:19]

There were a bunch of questions related to Conda, the main one centered around why is it not in the center, and what did it miss in the summary?

Speaker 1 [41:28]

It's not in the center since, yeah, maybe I'm wrong now, but I don't think that Conda is concerned with building and publishing packages. Is that wrong? Yeah, but now here in our Venn diagram, building and publishing would be that Conda has a command or a tool that... Okay, then it's wrong. It's definitely wrong, thanks. Thank you.

Speaker 2 [42:16]

And maybe your last question for today in the evaluation is poetry does not have a tick in the last row Metadata, but also supports PI project. What's the difference?

Speaker 1 [42:26]

Okay, so this new PEP is about how you specify the metadata in pyproject.toml. Not that it's in there, but how it's formatted, so how exactly you provide it. And the PEP came after introducing pyproject.toml and then recommended a way how you should specify it. And Poetry, of course, also contains the metadata, but it has its own way on how it's provided or specified.

Speaker 2 [42:58]

We have time for one last question, and there's one that has been asked several times in different versions, which is, which of the shell and package community rebuilds packages for new Python versions or different platforms? So there's a question about how the packages are being rebuilt.

Speaker 1 [43:16]

I'm sorry, can you repeat that?

Speaker 2 [43:17]

Yes, the question was, which of the shown package communities rebuilds packages for new Python versions or different platforms? So in other words, I think it's a question around how the package index is being kept up to date for the different...

Speaker 1 [43:36]

I would have to look it up.

Speaker 2 [43:42]

our q a section today to give another huge round of applause