Extending Python with Rust, Mojo, Cuda and C and building packages

Speaker 1 [00:07]

So, thank you all for being here. We are Wolf and Ruben from Prefix, and we're going to talk about Python as a glue language and extending it with other languages and how you can make that easy. So first, who are we? Wolf. Wolf is our CEO. He started Prefix a few years ago, but before that, he started creating Mamba. Some of you might know it as the replacement of Conda. Currently we're extending that with the Prefix company. I joined prefix from robotics because I noticed that that was one of the biggest problems that I encountered in robotics, and I love to focus on the developer experience. You can find us both on GitHub, of course, because most of what we do is open source. So Python is a glue language. Most of you probably know a few of these libraries that you see on the screen. Most of them are written in a different language than just Python. So these languages allow you to create binaries that can be installed as a Python library. And that, of course, requires those languages to be installed on your machine to be able to build these libraries and be able to use them. So some notable are, of course, NumPy, SciPy, and most talks here will probably have mentioned those libraries as well. Going further, like, okay, that creates a problem for you. And you've probably all seen something like that error before, where you're trying to build some C++ or another language, and you're missing some libraries that are on your system. You depend on some build systems that you need to install, packaging tools that are all different between these ecosystems. It's very complex to share this with your colleagues or other users that want to use your project as well. And of course, then you still need to be compatible with other libraries on your machine, so having different projects on your machine can be very hard, because there's one CMake installation or there's one version of the compiler that you have installed, but different projects need different compilers. This is all a pretty big problem if you first start off with building these byte packages. So previously, Conda was introduced and the whole Conda ecosystem to solve this problem. So it's a cross-platform, cross-language package management system, which allows you to install those compilers, which allows you to install those build systems, and then create your own packages, which then later can be run as Python libraries. Currently, Conda has a huge ecosystem of packages already on CondaForge, which is an open source package repository, which is completely free to use. So don't be afraid to test it out if you've seen any of the news items on the Anaconda distribution. The recipes that are being used are relatively simple. So some of you might know Nix or Spec to be even more data-focused. But yeah, Conda tries to keep all of that pretty simple and allows you to write your recipes very simple. And then we as Prefix came into the scene to really modernize this ecosystem. So we're trying to improve it both in speed but also in features. And yeah, the rest of this talk will also go into more depth on what these features are actually meaning for the ecosystem. So then Pixy. Our biggest project currently is Pixy, which is an open source package manager, and it allows you to do conda installations but also pipe guy installations and extends that with some more workflow options. So for those who don't know it, it's really fast. Feel free to scan the QR code. It goes through a benchmark that shows that it's like actually a lot faster than conda. I have pulled some of the data out, and I can show you not in timings because that makes no sense because it depends on your system. But here's a comparison for different environments, large to small, and how much faster Pixy is. So in this case, it's only installing cached environments, and like in the worst case, it's two times faster. In the best case, in this test, it's going to be ten times, but if you include the full setup of what a normal condo environment would need to do, Pixy can do that sometimes a hundred times faster. Feel free to try it out for yourself. Then Pixy can extend your package management experience with a little bit more help. So for instance, you often have linting, formatting, extra startup of your application, and it needs a lot of, yeah, tools to be set up. Pixy allows you to, for instance, sometimes alias it as simple as just a format command, but you can go way more in depth and basically replace what you would do with a make file. So in this example, you can do pixie run start, which is the second line of the tasks, and that would start a certain binary. But wait, you first need to build that in this example, so you run some CMake, and thus you can depend on another task. And that task then depends on configure, which is even more intense, where you can also add inputs and outputs. So pixie can also have cache tasks and all of that really cool magic stuff to manage your workflows in an easy way. And for the users and for your colleagues, it's just Pixie Run Start, and it will take care of all the runners that it needs to spin up to get your project to be run. And that makes it just a lot easier to share. Then when we started making Pixie, the really first most important thing that we were going for was reproducibility. And we do that through log files. These log files contain the absolute precise setup of your environment. The dependencies lower than this are just your glibc version, for those who know what that means. But going up, we really look at everything together, and the hashes are in there. We can see which licenses are in your environment. And sharing this through your repositories with others really recreates the environment like you just did on your machine. And this is really important if you want to have an easy experience when you go through a GitHub CI or something like that. you're like, oh, it works on CI, but it doesn't work locally for me, this problem becomes so much less of an issue when you're using these log files, because you will at least have all the same packages installed. So you don't have version incompatibilities and stuff like that. Then Pixy is written in Rust, which allows us to create a single binary. So you can just literally download it and run that file, and then you have Pixy. no more package manager needed to install Pixy for you. And especially compared to Mamba and Conda, you don't need Python anymore, which is part of why Conda is so cool that it can install Python for you. So that was not least. We also support PyPI packages, where the Conda ecosystem previously just used pip for that. We are using UV. You probably all have heard of it at this point. And that creates a really good experience when you want to combine these two ecosystems because even with UV, it's not very trivial to make sure that you're not getting duplicate packages or that UV starts overriding your conda packages. All that stuff is handled by Pixie itself. It is built into Pixie, so it's not an extra dependency that you need to download. And Pixie will just manage it for you like normally would be done with UV. And next I want to give the word to Wolf, I'm going into building packages.

Speaker 2 [07:32]

Thank you. Yeah, so why should Pixie build packages? We want to really manage the full development workflow. So when someone starts to work on a package or on some software, they should have the ability to basically start building it and then packaging it up and sending it, for example, to our server to distribute it later on. And that, for us, kind of completes the circle. So PixieBuild has been from the get-go one of our large goals that we wanted to deliver with Pixie. And also having that ability will also enable us to have mono repositories where you have, like, one workspace and then many sub packages, and Pixie will tie all of that together and build them in the right order and manage all of this. And PixieBuild is really the key ingredient to get there. And with PixyBuild, we can also have a lot of simplification and standardization because the user doesn't have to care about CMake or Cargo or the right invocations of all of these tools anymore. It will just be Pixy run start, and Pixy takes care of installing all the dependencies, building all the packages, et cetera, et cetera, in a single tool. So that's kind of the promise that we're setting out with PixyBuild. And if you would go on our website, we have it. I think we removed the in-construction label recently, but it has been sort of from the start that we wanted to make PixieBuild a reality. And PixieBuild sits on top of RattlerBuild, which is our second big project. And RattlerBuild, again, sits on top of another library that's called Rattler. Just like Pixie, Rattler is our set of Rust crates that manage the... manage conda packages, that resolve conda packages, and that can install conda packages. And so Red Lab Build is a reimagination of conda build. If you've ever built any conda packages, then you have interacted with conda build. And conda build has been around for, I think, roughly ten years. And obviously it sits at the heart of the conda ecosystem, but it has been sort of slowly decaying. There is a lot of technical depth in conda build. With Rattler build, we're really reimagining it from scratch and building it in Rust again. We have an improved recipe format and a lot of other niceties and features. And if you look at the diagram, then you will see that we are a lot faster usually than Conda build. But obviously, for most of these packages, it's dominated by, for example, compiling Rust or compiling C. But some are, like, huge improvements just because the recipe execution Itself has actually taken a lot of time in conda build Yeah, we're also very proud of the domain red lab built very easy to remember you can check it out For more documentation and read our blog post to understand more the motivation and the benefits But back to pixie where you will not have to deal with red lab it and red lab it is just under the hood What we are doing in pixie and for pixie build is we're creating build backends And we already have a number of built backends. So we have one for Python, one for Rust, one for CMake, and one for Rattler build that gives you the most flexibility because you can essentially do everything that you can do in the recipe file, you can do in the build backend. The main focus is that we want to make it standardized so you have the build backend and you have your Python project or you have your Rust project. And through the build backend, the package itself will all follow the same layouts and invocations, but at the same time, obviously, we need to give a lot of flexibility and customizability to developers so that they can achieve everything that they want. And using PixieBuild, you will basically go from the manifest through the build backend and then end up with a package, and we take care of everything in between. If you look at our idea of workspaces, you can now build a Pixie project where you have a workspace at the top with one Pixie Toma, and then you can have multiple subfolders that each have Pixie Tomas and depend on them as a source dependency using the path dependency. And we also changed in the Pixie Toma, it used to be project at the top, but now it's workspace and package. And the package basically defines also the build and host dependencies, which you might know if you have built conda packages before. Yeah. So that's roughly the idea. And then we support multi-language. It's all cached and we also have editable installs for Python. And you can use pixy build as one command as well to actually build it into a proper conda package that you can then take and upload, for example, to a channel on prefix.dev or on S3 bucket or on anaconda.org or wherever, conda packages are supported. And I think now it's time for the demo.

Speaker 1 [12:40]

Yeah, so if you want to see which packages were actually going through you can scan the QR code, but I will quickly Exit this if you have scanned it and go to my vs code example the other one No, this one actually so what you will see is that the main pixie tumble in that workspace that Basically looks the same as what wolf shows. Let me throw this a little bit more It will have those dependencies as path dependencies And this is the new thing that we've introduced, because previously it was always a version or a build string or whatever. Now you can also just do paths or get repositories. And Pixie will figure out how to get those paths, look into the Pixie Toml or PyProject.toml in there, and run the build backend for you so you don't have to install CMake for yourself. Pixie will take care of that. So if we go into some of these examples, the CPP bindings is just a C++ project with a CMake list file. And in here, you can see that we define that we use the Pixie build backend and how to build it. In this case, it only needs Nanobind and Python because it's, of course, a Python library that we're building as a binding. And then later on in the simple main file, you can use this and import it as a normal Python library. And to quickly show that, I can do Pixie run start, which is a task that runs Python validate. But before that, it ran the build for you, and it will create all these libraries for you and then installs it and runs the Python package for you. This can then also be rerun if you, for instance, change the code. So for instance, let me try that on the Rust bindings. This one. No. This one. You go to submodule. sad hello world in this case, but what if we make it a high world? We can do pixi reinstall. It will reinstall the packages, rebuild them, actually. And if we now do pixi run start, you will see that it does the high pixi world. And so it might look super simple, and the example is extremely simple, but what really just happened is that we did a full Rust build of a Python library, and this was pretty complex to get running on your machine. In theory, this machine doesn't need Rust anymore right now and can just, like, install it for you. You can try out the demo later today and ask any questions about it. Then, because we have it in the title, Mojo doesn't allow us to build a Mojo backend or binding and then run it in Python, but it's actually the other way around. So in this case, we have some Mojo code from their examples, which we can run, but we can also import that same Python file that I just ran for you. So if I do pixi run Mojo test Mojo, oh, Mojo test also works, you'll see that it doesn't work. Test of the mode.

Speaker 2 [15:53]

I think now it's a mojo-mojo.

Speaker 1 [15:56]

Mojo Mojo, ah, yeah, that was it. Thank you classic And then it actually runs both all of these bindings that have been built for you in a mojo project Which makes it so much easier to try new things out copy your repository from the internet and try it out on your local machine then Because we are running out of time. Otherwise, I will go back to the main slides so we have So, what we can also do, for instance, is build Python completely from scratch in a single file because we support all the recipes from CondaForge and the build backends. You can also build CUDA. I'm currently on my Mac. But you can also do the exact same thing for all the CUDA libraries that you would need to install. And this is just a quick way to get this running on your machine without requiring you to use both opt and pip and some Docker images because otherwise you cannot get it to work. This now can run on your host machine without any of the other hassles that you would normally require to get it started. So for the Outlook, what we're really going forward with is the actual custom build of this backend. So we did it for CMake. We did it for Rust. But there's a lot more types of packages in the wild that people really want to support. It should be really easy to let you make your own backends. For instance, in some companies, there are some extra special things around a Python package that you would want to do. Let's build our own backends, get them on the repository, and you can easily, without running your own recipes, build your own packages. Then recursive source dependencies is something that we really want to get to, because now you can depend on a path dependency, but that path dependency should also be able to depend on a path dependency, and that results in a full tree of builds that could be run in one go. of Bazel but simple, as some of you might know that one, and we want to make more backends for you. So, for instance, Ruby, Fortran, Go, you think of it, we want to support it through the Pixy workflow. Then next to what we're building for Pixy and PixyBuild, we also want to do even more improvements on the conda ecosystem, so we're going to start working on package signing to improve the security of the conda ecosystem. We're going to work on optional and conditional dependencies, which is, of course, known in the PyPI ecosystem. But we want to bring it to all the other ecosystems as well through the conda ecosystem. And yeah, you can try Pixy today and let us know what you think about it. There's also a Discord that you can join or this super simple command to install it on your machine. And otherwise, yeah, thank you for listening. And please feel free to join our community. We'll be right back.

Speaker 3 [18:59]

Thanks, Ruben. Thanks for this wonderful talk. Yeah, you are well in time. So we have a few questions. So first question, why did you build an alternative to Conda instead of contributing to Conda? The Python ecosystem needs better tools, but not even more different tools.

Speaker 1 [19:18]

I let Wolf answer that one because he started this one.

Speaker 2 [19:23]

So, it's a bit of a loaded question, I guess. There are some political reasons, but also some practical reasons. The practical reasons are Pixie follows a very different idea than conda does. We have log files from the start, conda doesn't have log files. We don't have named environments yet and maybe never will. Conda has named environments. We installed the environment in the project itself. We have a very project focused approach. And Conda has a sort of pretty different way of thinking about stuff. And so, in that way, we were also very inspired by other tools like Cargo or Poetry and not so much Conda in that sense. At the same time, we are definitely, like, contributing majorly to the Conda ecosystem. And we are more than willing and open to sort of share our core tools with Conda, essentially. And we already moved, for example, the core libraries that I talked about briefly, Rattler, We moved them under the Conda community governance, which is like an open-source governance that we are also part of. And so anyone is free to basically use them, including the Conda project, and we would love to work with them to adopt basically the same robust Rust sort of, yeah, foundations that we're building. Yeah.

Speaker 3 [20:47]

And there is a coming question which got nine upvotes. Will your pixie.log file support the newly accepted PIP 751 that standardized the log file?

Speaker 1 [20:58]

Yeah, that's a good question. We're actually looking into that as we speak, so I cannot give you a definitive answer But we're not only focusing on the Python ecosystem So while we really want to work together with it and we might like the PyProject.toml Also support that form of log file. It's not the focus for Pixie to start doing that as their main As its main main log file So while we want to work together with the Python ecosystem because it's so ingrained in the conda ecosystem we're not it's not our only focus so we need to make sure that we support all of the other things as well so if that limits it then we would divert from sharing the same log file

Speaker 3 [21:41]

Now the next question with nine approach is it possible to use pixie solely for managing pi pi packages without engaging with contact system

Speaker 1 [21:51]

If you only use it for PyPI packages, that could be really difficult because the conda environments that Pixie will install for you are kind of the basis on which you install your Python libraries. So if you, for instance, install modplotlib, you will always need a Python interpreter, and Pixie is not built in such a way that it will search for a Python interpreter on your system. It will actually always install it for you. So we focus more on isolation. where people normally would start up a docker image to get their python version we just install it locally in a virtual environment like you would do with a vn for something like that but then don't touch the global and global installed python although it's of course cached so you only install it once and it's not like we install hundreds of interpreters on your machine

Speaker 3 [22:43]

Thanks, the next question is what is the advantage of pixie against poetry or UV?

Speaker 1 [22:51]

So, that comes back to a little bit of the same answer. We do a little bit more on the first step of the Kunda ecosystem. So you will install Python, but you will also install some, like, low-level C libraries or all of the other stuff that you need to get it working. For PyCharm, you will install all the CUDA libraries and all this stuff is now getting a little bit into Py, the PyPI ecosystem, but a lot of the bases are still not there, you would normally install them through up or use a docker image for those kind of installations where pixi can just do that for you and then you can have one log file that sums it up all of it and that's basically the biggest benefit between pixi and the others like we also do what they are doing but first do the basic setup for you

Speaker 3 [23:40]

the question are coming up so the next question is are there plans to build wheels also

Speaker 1 [23:48]

Yeah, so these Python projects that I was just running are just a PyProject.toml. So if you call pixy build on them, you will build a conda package. If you call uv build or poetry build, if it's a poetry package on them, you will get the wheel out of it automatically. Right now, we're not, we're still deciding if the wheel will actually also be a separate outcome of the pixy build command. But in theory, the level of extra setup that you need is almost zero, because if you run When you rebuild in the Pixy environment, you will get the same outcome as you would normally have. So it's like there's no more layer of disadvantages there.

Speaker 3 [24:28]

Thank you. The next question coming is can pointing a path to a python package support editable mode So that I don't have to keep the rebuilding. Yes

Speaker 1 [24:39]

Yeah, that works.

Speaker 3 [24:42]

Okay, so the next question is is pixie safer than the UV regarding the financing or are we also dependent on the venture capital?

Speaker 2 [24:52]

I guess that's the question for me.

Speaker 1 [24:54]

with me. Oh, go ahead.

Speaker 2 [24:58]

So, yes, we are also a venture-backed startup, but we've also sustained a lot of our open source work, for example, through money that we got from the sovereign tech agency. That's a German institution that gives money to open source projects. And we are sort of trying to definitely make sure that the open source part of what we're doing can stay extremely, like, it's always going to be open source Pixie and everything and RattlerBuild. And yeah, through this additional sort of grant-based funding, we can sort of have a pretty good split and make sure that the interests are very well aligned.

Speaker 1 [25:39]

We're making it pretty hard to do a rug pull by ourselves, for ourselves.

Speaker 3 [25:45]

Okay, so the next question is UV also offers workspaces with different build backends for path dependencies How does Pixy stand out here?

Speaker 1 [25:54]

It is basically the same workflow except that you For like the the biggest sell here is that you could also do other things than just to build backends For Python so you could also start your NPM project in a separate folder Like of course you have obviously a lot of users that build websites and if you do all your NPM Workflow in the same monorepository pixie can definitely help you there Otherwise UV has a pretty good default setup already. It's mostly when you want to start doing CUDA and want to start extending all of your Python work, then Pixy becomes a really important player.

Speaker 3 [26:35]

Thank you, I will enter two more questions. We will have four minutes left still. Okay, if you is fine with you Yeah So the next question coming is if one uses a git repo as a package path does pixie know if the repo has been Updated upstream if so, when does it occur or in each run or at specific times?

Speaker 1 [26:55]

Yeah, so Pixy uses the log file to validate if the manifest is still up to date with log file and with the environment. So it's a three-step situation where if you don't update your manifest, it will not automatically try to update your Git repository, but you can just simply call Pixy update on that package that you have, and it will revalidate your Git repository. You can also just remove the log file and restart, but that goes a bit against the idea of having the log file. But, yeah, just a simple update command, and you will have it.

Speaker 3 [27:29]

Thank you. The next question is for the teams considering switching to UV from poetry What we would be missing in not choosing pixie

Speaker 1 [27:40]

Yeah, we expected these questions, of course. So I'm going to reiterate myself a little bit, but the power is that you can benefit from the conda ecosystem. And apart from just having more packages, it also is like a shared responsibility of the condaForge ecosystem to make these packages actually work together, where the PyPI ecosystem is a free-for-all. You can just upload whatever you want on there. CondorForge has one layer of security where it's all validated by other contributors to the CondorForge project, and then that whole project makes sure they work together. So if you have two different libraries, but they're built with different C compilers, you will get into issues on the API part. And that is another issue that you will get in the conda ecosystem.

Speaker 3 [28:28]

Thank you. The next question is very short one. Will this work within Windows?

Speaker 1 [28:35]

That is one of the biggest selling points compared to some of the other package managers that can do cross unix for instance I can do it cross-platform completely so you will not have to deal with your colleagues wanting to run on Windows

Speaker 3 [28:52]

Thanks, Luke. Thanks, Ruben. It's a wonderful and interactive talk. I would like to give a big round of applause for him.