How to mix conda and pip without causing “environmental” damage.

Speaker 1 [00:03]

to the titanium room. Before we get started, I was asked to announce or to promote the lightning talks. So if you have an idea about the talk that you want to do in five minutes, then, you know, use this half an hour to think about that idea and then go to the booth and promote or leave your submission for the talk. All right. And with that, let's introduce Mahe, who will talk about something that we're all very familiar with, environments and dependencies in Python. Give her a round of applause.

Speaker 2 [00:47]

Hello, everyone. Good afternoon. Welcome back after the lunch break. I hope everyone is sufficiently caffeinated. If I see any sleepy faces, I'm going to assume it's because of the food and not my talk. Hi, my name is Mahe. I am a software engineer at Anaconda. I'm based in Berlin, but I'm originally from India. And I've been an Anaconda maintainer for almost four years. Speaking of Conda, how many of you know about Conda? Okay. And how many of you use Conda? Mm-hmm. And how many of you maintain Conda? Okay. Maintainers! All right. So this talk is going to be about Conda. Surprise! And pip. And the differences between them. I will also talk about why mixing conda and pip is so unreliable. And at the end, I'm going to tell you about one community tool that is trying to bridge this gap between conda and pip. What this talk is not about, this talk is not about the conda versus pip debate. Because saying conda versus pip is like saying apples versus oranges. Why do I say that? You will learn throughout the talk. I chose to add a small glossary of some terms that I will be using throughout this talk, because it's better to ensure that all of us are actually talking about the same thing when we use certain words. Package management refers to the process of installing, updating, and uninstalling packages in a controlled manner. Dependency conflict is what arises when you have two packages that depend on different versions of the same package. So let's say you have a package A, and it depends on version 1 of package C. And then you have package B that also depends on package C, but version 2 of package C. Now, this is a dependency conflict if you want to have A and B installed together in one environment. A package manager is a tool that does package management. OK, now that we have that out of the way, I will talk about the evolution of Python packaging. And naturally, we start by talking about distutils, which was Python's original standard library module for building and distributing software. And if you were using distutils, you would do python setup.py install to install the software that you already downloaded on your device. Disutils, unfortunately, did not have an uninstall concept. So you could not uninstall stuff after you had installed it, because it was not tracking where it installed what. And there was no dependency management. And one other complaint that people had of Disutils is that because Disutils was a part of the standard library, its maintenance was tightly coupled with CPython release cycles. That means feature development was very slow. From the frustrations with disutils came setup tools in 2004. It was created by Philippe J. Ebby, and it was a third-party extension on the top of disutils. Along with setup tools, Philippe also created easy install. And easy install was like the first real package manager for Python. And of course, it had PyPI integration, Because by that time, by 2002, PyPI was already a thing. PyPI refers to Python Packaging Index, which is like this ginormous library of lots and lots of packages. And when you do pip install something, it comes from PyPI. So easy install could go to PyPI and fetch packages for you from there. It also had dependency management, even though it was not complete. Unfortunately, setup tools also did not have an uninstall concept. So you could not uninstall stuff even with easy install. Then came PIP in 2008. And it was created by Ian Bicking. And PIP could uninstall stuff. Yay! And of course, PIP also had PyPI integration. And when I was doing my research and looking at Reddit threads and Stack Overflow threads, I found that at that time, people were really happy about how much better the error messages were with pip. That was one of the best things about pip at that time. It had great error messages as compared to previous tools like easy install. Around the same time as pip, Ian also created virtualenv to solve the problem of conflicting dependencies by creating isolated environments. So the core problem was, Python used to install everything in the global site packages directory, which meant that if you had two packages depending on the same package but different versions, you were stuck. You remember this example, right? So because Python was installing everything in one global site packages directory, you could not have two working projects at the same time that had the same dependency. Another problem was that over time, because everything was being installed in the global site packages directory, it became a graveyard of abandoned projects. Maybe you were working on a project, you installed that project, all its dependencies in your site packages directory, the global one, and then you abandoned that project for whatever reason. And now all these packages are still lying there, right? And you go there and you see all these packages and you don't even know what belongs to what, right? And if you decide to delete something, maybe something else will break mysteriously. So virtualenv was able to solve both these problems in a very elegant way. What virtualenv did is created these isolated bubbles called virtual environments, each with its own site packages directory. So whatever you installed in one virtual environment had no effect on the other. So pip and virtualenv together solved the core problems of the packaging community at that time, and they were great together. They were conceptualized and created to work together, and they did. As pip became more and more popular, its predecessors became less popular. Easy install was eventually removed from setup tools in 2021, and disutils was removed from standard library in 2023. The tools, pip and virtualenv, are presently maintained by PyPA, which is the Python Packaging Authority. It's an organization that takes care of packaging-related matters. It's not like the government, even though it has the word authority in it. Okay, so did pip solve all the problems? It did solve a lot of the problems of a vast number of Python users, but there was one community within the Python community whose needs were not entirely fulfilled by PIP. And this was the scientific Python community. This involves people doing data science, scientific computing, numerical analysis. And these fields make use of libraries such as NumPy and SciPy. And NumPy and SciPy have really heavy C and Fortran dependencies. And because pip can only deal with Python packages, it could not manage C and Fortran dependencies. So if you were trying to install NumPy or SciPy with pip, you were in a messy situation. Because you had to take care of all these non-Python dependencies by yourself. Enter conda. That's conda entering. In 2012, Continuum Analytics, now known as Anaconda, created conda, which was a language package and environment manager, meaning it did not care what language your package was written in, whether Python or non-Python. It could manage all packages, and it was a package manager and environment manager clubbed in one. So you did not have to install a separate tool to manage your environment. Conda was doing both. Okay. So what did we do till now? Let's take a pause, okay? We went through the evolution of Python packaging, and then we reached the point where Conda was created. With this information alone, we have established that Conda was born to solve a different problem. From the beginning, Conda was trying to target one community. That was the scientific Python community. It was never trying to be better than pip. It was never trying to replace pip. was born to solve a different purpose, right? Now that we know that, I want to encourage all of you to start thinking about conda and pip in terms of ecosystems rather than tools. So conda and pip are two parallel ecosystems, not two competing tools, okay? And when you start thinking about conda and pip in terms of ecosystems, you realize that they have different objectives and different needs, and therefore, there are different tools in these two ecosystems. And when you realise that, you will understand that newer tools like Poetry and UV would fall in the PIP ecosystem, and tools like Mamba and Pixy will fall in the Conda ecosystem. Now, these ecosystems have differences in their architecture and distribution models. And I'm going to talk about both of them one by one. The first architectural difference is in package formats. Conda packages, all the packages in the Conda ecosystem have .conda or .tar.bz2 format. And all the packages in the pip ecosystem had .whl or .tar.gz format. The way metadata is structured in these two ecosystems is also different. So metadata is data about data, right? Meta, we know what meta means. So in the packaging ecosystem, metadata refers to all the extra information about a package that package managers use to decide what to do with a package, OK? In the conda ecosystem, packages have an info directory that contains these files, which have the necessary metadata for the package manager, right? In the PIP ecosystem, there is a similar directory. It's just named differently. It's called distinfo. And it also has several files that contains the metadata that PIP package manager needs to decide what to do with the package. The place where packages are hosted are called package repositories. In the conda land, packages are hosted on channels, right? You must have used the main channel, anaconda-main, also known as defaults, or the community-maintained conda4 channel, also Beoconda. So in the conda world, packages are obtained from channels. In the pip world, packages come from indices. And the most popular index is the Python packaging index, PyPI. OK, this one is a bit more complicated. So I'm going to take my time to explain this, so please pay extra attention. Also, don't get too distracted by the microwave at the corner, OK? explain why it's there. All right. At this point, presently, pip can handle non-Python dependencies, even though originally it could not. That was the whole reason conda was created. So the way pip is handling non-Python dependencies is via wheels. Wheels are compiled libraries. That is binaries. So what pip does, it bundles all the non-Python libraries, non-Python dependencies of a package within the wheel. It creates one bundle that's independent, and it shops the non-Python dependencies like that. Whereas conda, for conda, Python, non-Python, doesn't matter, everything is the same. So it's able to deal with non-Python dependencies like a regular package. So okay, imagine this. You get a wheel from PyPI. You get another wheel from PyPI. These two wheels have non-Python dependencies, and they happen to be the exact same. These are two separate packages, two separate wheels, but they have the exact same non-Python dependencies. Because wheels are independent units, if you install these two wheels on your system, you will have two copies of this library, because they have the same dependencies. So you're going to have two of them on your device. Whereas conda, if you get one conda package, another conda package, they have the same non-Python dependency, these two conda packages are going to share one common non-Python dependency. So you will not have two copies of them on your device. One good way to understand this is via an analogy. So how many of you live in a Vege, a flat share? I'm sorry about that. I hope you're doing okay. So let's imagine a Vege situation, right? There's a Wege and there are three flatmates in this flat share and all three of them have their own microwave. They each move into the apartment with their own microwave. And now in the kitchen you have three microwaves, right, taking up a lot of space in the kitchen. But they refuse to share it. They want to have their own microwave. That's the pip situation. Now imagine another Wege situation. One guy moves in, he has his own microwave, and the second guy is about to move in, they they have a conversation, and this guy says, hey, I already have a microwave, don't bring your own, just use mine, right? And then he moves in, and he's able to use the same microwave. Third roommate moves in, also uses the same microwave. So that's, I'm not trying to make fun of Pip or Vegas, just a little joke to keep you awake, okay? Okay, so that's all the structural differences between content and Pip ecosystems out of the way. Now we are going to talk about the difference in distribution models. Distribution models refers to the way packages are made available in an ecosystem. So packages on PyPI, anybody can upload packages on PyPI. And usually it's the authors themselves who upload packages to PyPI. How many of you have put up packages on PyPI? Yeah. Thank you for your service. On conda channels, for example, conda forge, let's talk about conda forge first. On conda forge, both authors and non-authors can upload packages, and usually it's non-authors because what conda channels do is downstream repackaging. So if you want to make a package available on conda forge, what you do is create an application and then CondaForge core developers they look at your application and give you a thumbs up and then you get to host a package or make a package available on CondaForge. So there is a screening process unlike PyPI and Anaconda main also known as defaults, only Anaconda engineers can put up packages on main so you and I cannot do that even though I'm an Anaconda engineer but I'm on a different team so I can't do that Okay Okay. Lots of talking. Lots and lots of talking. Now I'm going to show you two little demos to illustrate what happens when you mix conda and pip, okay? So this is a conda environment, and I already have pip installed in it. I'm going to use this pip to install a very famous package, numpy, okay? Let's see. Okay. Done. Then I'm going to use conda list to list all the packages that are in this environment right now. I see NumPy is right here, and Conda has very intelligently added a PyPI label here under channels to show that this package did not come from Conda, it came from elsewhere, something called PyPI. Great. I have NumPy here, so I'm going to try to use Conda to update this NumPy. When you try to update a package with Conda, and if it's already the latest version, conda just tells you, hey, it's the latest version. I'm going to try to update this numpy now with conda. I get an error saying that numpy doesn't exist. But wait, conda list, you just told me that numpy exists, and now you're telling me numpy doesn't exist, right? That is because conda, even though it tracks packages that it did not install, it tracks all the pip packages, but conda solver is completely blind to non-conda packages. It doesn't know what to do with them, it just doesn't look at them. And for good reasons. But this can be confusing, right? Now let's look at another demonstration. Same conda environment with pip install in it. I'm going to use conda to install another famous package. All right. OK, this is going to be slow. OK, not so slow. You see that But conda is telling me that conda will also install numpy, because numpy is a dependency of scipy. I say yes to that. Okay. Let's do conda list. And we will see that numpy is there, and scipy is there. Great. Awesome. Now, I'm going to do something dangerous. I'm going to use pip to uninstall the NumPy that I installed. Yes, do that. OK, now let's do conda list. But NumPy is still mentioned here. That is because conda doesn't know what pip did. Conda has no idea that pip went behind its back and deleted NumPy. Conda is totally unaware. So why did I install SciPy? I installed SciPy because I was working on some project. So let me work on my project. I try to import SciPy in my project. And I get this error saying that, can't do that because NumPy doesn't exist. And I'm like, huh? What? NumPy is there. You just told me that NumPy is there in my environment. You see, this can be so confusing to somebody who doesn't know what's happening. I know what's happening. I know that PIP deleted a conda package without telling conda. So if somebody doesn't know, they're going to be really, really confused. And also, maybe you ran a pip command, and then you came back two weeks later, you forgot about it. You ran conda list, you see numpy is there, you're going to be so confused. So these are really simple ways in which environments can become completely broken. And there can be more complex ways also. I just demonstrated very simple ways to break an environment. OK? It's very clear that conda and pip are not sharing their knowledge with each other. Pip deletes conda packages, doesn't tell conda about it. Then why do people continue to mix conda and pip? There are three main reasons. The first reason is that a vast majority of Python packages are available only on PyPI. Let's look at some stats. Main channel has 5,000 packages. CondaForge has 32,000 packages. And PyPI has 780,000 packages. That's a huge difference, right? So it's only inevitable that there will be some package that you need that is only available on PyPI. The second reason that people end up using pip inside their conda environment is because conda channels often lag behind version updates. So maybe conda channels already have the package you need, but they don't have the latest one. And the latest one is only available on PyPI, so you have to use pip to get that. And the third scenario in which people mix conda and pip is while doing Python tutorials. People are trying to set up a project, and they run into some issues. They are really desperate to get this project running, so they go on the internet. And they say, oh, this is happening, what do I do? And then some Stack Overflow answer says, do pip install numpy. And they try it. OK, something happens, but it doesn't fix their issue. So they go again, and they say, it didn't work, what do I do? And then it says, maybe somebody has posted a code block that says conda updates hypy. They just copy it, paste it, and something happens, but their problem is not fixed. And all of this a few more times, and they end up in a really messed up environment. OK, all this to say that we have to accept the possibility that conda users are going to need some package from pip, from PyPI at some point. And we have to accept this, and we have to take it into account when developing things, right? And there have been attempts previously to increase interoperability between conda and pip. But I'm not going to talk about all of them. I will talk about one. I'm going to talk about conda PyPI, which is the latest effort by the conda community to make mixing conda and pip better. So what you do with conda PyPI is you can install wheels as conda packages. So what conda PyPI does is it goes to PyPI, picks up a wheel, converts it into a conda package, and installs it into your environment. So for all intents and purposes, for conda, it's just a conda package, right? So it just deals with it like a conda package. And the way to do that is you install PyPI into your base environment, conda install dash and base conda PyPI. And then you do conda PyPI install and name of the PyPI package. And it's going to know that you're trying to install a PyPI package, and it's going to go to PyPI, get it, convert it, install it. OK? So there are a few other things that Condop-PyPI can do, but I'm only talking about one feature, OK? Oh, actually, I'm talking about two. The other thing that Condop-PyPI can do, does, is it adds an externally managed marker file in your environment. So what that does is, OK, you install Condop-PyPI, you learn this new thing, and you're doing Condop-PyPI install, and you're getting all these PyPI packages converted to conda, you're very happy. But you're in the habit of doing pip install all the time. So you forget. And then you end up doing pip install, blah, blah. And then you mess things up again, right? But conda-pypi doesn't let you do that. Once you have conda-pypi installed in your environment, and then you try to do pip install, it's going to scream at you. It'll say, hey, don't do that. That's not OK. Don't do that. Just use condopipi. I think that's a great guardrail that condopipi puts in your environment. This is the QR code that will take you to the condopipi GitHub repository. You can go there and check it out, install it. Know that it's in a very early stage of development, and it's experimental. So things could change. And right now, condopipi, you're starting out simple. It only supports pure Python wheels. Things will change in the future, and maybe you can contribute to conda.pypi. Maybe you can be part of the development, right? Because maybe you'll be the end user at some point. Okay, yeah, we got there, we did it. At the end, I want to say that the purpose of my talk was not to list all the differences between conda and pip, Or to say, which is better? The purpose of my talk was to encourage all of us to develop an ecosystems mindset. To start thinking about Python packaging in terms of ecosystems. Because when you start thinking in ecosystems, you become more tolerant to technical implementations. And you start thinking about context. You think, oh, but what were the needs of this ecosystem? Why did they choose to design this tool in this manner? And you become more understanding. And also, you realize which ecosystem does your use case fall into. And that makes you less frustrated and angry and more just calmer, right? So don't be angry. And think constructively. And think in terms of ecosystems because ecosystems outlast the tools that created them. Thank you very much for coming to my talk. This is the QR code to my LinkedIn. Connect with me and this will also link to my GitHub. You can check out my GitHub. You can get the slides. Blah, blah. Just get this link. Thank you.

Speaker 1 [27:27]

Thank you very, very much. I learned a lot that I didn't know. Thank you so much. We have a few questions. I'll start with this one. Especially UV is shifting how package dependencies get solved. Server side, it looks similar to Conda. Client side, it is more similar to PIP. Do you think that Conda will move into the same direction and become more like a tool in the PIP ecosystem?

Speaker 2 [27:52]

Conda will become a tool in the PIP ecosystem. I don't think so. I don't think that's what we are trying to do. Like I said at the beginning of my talk, Conda is focusing on a specific community, and it's not trying to do everything. So I think it's unlikely that Conda will try to do that.

Speaker 1 [28:17]

Here's another UV-related question. I like that Conda makes it easy to get scientific packages together where there are non-Python dependencies. Some are even only available via Conda install. But UV is continuously spreading. Can I use Conda packages in a UV environment?

Speaker 2 [28:37]

Like I said, UV falls in the pip ecosystem and the package formats that UV deals with is .whl and .tar.gz, which is like pip, similar to pip and not to Connor, so I don't think you can do that.

Speaker 1 [28:53]

How is conda pipe, how did you pronounce that?

Speaker 2 [28:57]

Kind of PyPI. PyPI, right.

Speaker 1 [28:59]

PyPI, right? Connected to conda build.

Speaker 2 [29:04]

So CondaBuild builds Conda packages using recipes, okay, and CondaPyPI is something that lets you install wheels as Conda packages. I think they're doing different things. CondaBuild is used to build Conda packages whereas CondaPyPI is used to... Yes, I know you're thinking that wheels are being converted to Conda packages, so is there a build process involved? I don't think so. There's no build process involved. Yeah, so it's not similar.

Speaker 1 [29:36]

I hope that clarified it. We have time for one more. What is the difference between Conda PyPI and Conda Smithy that used by CondaForge to create Conda packages from PyPI? PyPI.

Speaker 2 [29:52]

You get that one? Yeah, I think so. I think it's similar to the last one, maybe. Okay, I want to mention one tool.