Python with and without Pants

Speaker 1 [00:02]

Okay, thanks Patrick. I guess that's also the right spirit because the talk kind of has two topics slightly. One is about the pants build tool and the other one is about when would I actually consider using a build tool because for me when I hear the word build tool, I think about Java and 2000 kind of like if you think about programmers that try to fix the year 2000 glitch and sit in the cubicles and have eight bosses and have this massive code base. they need to work with, and they need a build tool to get it all done, because otherwise they can't compile the code and stuff like that. That's kind of what I feel when I hear the word build tool. But we are here at a Python conference, and if I think about Python, I don't think about that. I think about, I don't know, programming in the beer garden, more relaxed, maybe without pants, because people don't care, as long as you get the job done. That's kind of like more how I think about Python. And there's kind of like a difference to it. And the goal of this talk today is that I can try to explain to you that we have build tools that I would normally consider in the Java world, for instance, but they're also totally useful in the Python world. And I'll try to explain that to you. And before we dive into the content, quickly a few words about myself. I'm Stefan. I'm a software engineer working at Bleonder. So one of the guys that tried to sell bananas. Yeah, you can find my Twitter handle. I will also use that to tweet the slides and some examples. And, yeah, just come to me afterwards if you have any questions. And we now try to dive into the content. And we will actually look at two tools today. As Patrick mentioned, one is PEX and the other one is PANS. And they both have kind of like different use cases but are kind of related to each other. And I'm a user of both of them because I'm Aurora committer and Aurora is using both of these tools. Aurora as well as PEX and PANS have been developed by Twitter and so they're used daily by thousands of engineers but they're not that known in the Python community and that's kind of what I'm here for today so that you know about the tools and can consider if you want to use them or not. We'll start with PEX and if you go to the website and look up PEX, the name stands for Python Executables. There's a pep about it, at least on the fundamentals of what it does and how it works. And you can think of them like a virtual amp, but all in a single file. So like a hermetically sealed single file that you run, it bootstraps itself, and then it runs your Python code. That sounds pretty, I don't know, but I'll try to give you an example so that you have a feeling for how it works, because I think that's much better than putting something on slides. So I prepared a super simple demo. There's a folder called mypackage. In that folder, we have a done domain. We can look at that one and we see it just contains print hello world. And as many of you might know, if you have a folder with done domain, you can just run that folder and you see hello, whatever you printed in there. But Python has an interesting feature, and that is it doesn't only do that once you have it in a folder, but you can also squeeze it in a zip file. So what I can do, I can do app.zip, put in my main, and if I now run that, it will also print hello world. And you can go even further if you do something like, oops, use it in Python, so we want the shebang, tells us what to execute, my binary, and we pipe that to a file, so that begins with that, and then we go ahead and take our existing app that we zipped, and we append that there as well, and then we make all that thing executable, and then we run it, and it also prints the stuff. So, that's pretty interesting. The Python has built-in support for writing stuff in zip files, and if we add a header to it, we can just execute it, And no need for any bash script all around that, just works like that. And the whole idea of pex is built around that. So we have a zip file, put code in it, and can just execute it and it works. And to kind of prove the point, I've brought a pex file with me. It's called mathfun.pex, and it's kind of like, I can execute it, it just uses numpy to compute a random number. pretty boring, but we can look into it, so we can unzip it, and you see it contains lots of stuff, including NumPy. We also can see the contents, so again, we have the main, can you see that in the back? Alright, so we have the main file, that's the one that gets executed by Python once we call it from the outside, and yeah, the rest is a little bit of bootstrapping, because sometimes we need special cases, or at least PECS handled them for us. But more interesting, there's also a dependency folder, and it contains our dependencies. And as I mentioned, we needed NumPy for that. And if we again look in here, we see, okay, this library, this binary is using Qlik as a command line parsing, and it's using NumPy. But interestingly enough, there are two versions in there. There's NumPy 1.15 for CPython, but compiled for macOS, and one is a mini Linux wheel. So essentially we have the Linux and the Mac NumPy in the same zip file, in the same PECS file. And that means I can build this, run it locally on my Mac, but also copy it to any Linux server and run it there as well. No need for Docker or anything. I just can wrap it all in a zip file, run it here, run it on your laptop, whatever. And that's the whole idea about PECS. And if you're kind of interested in that, you might think, okay, and I'll go ahead, install me the Pax binary to build my own binaries. And you could look at the command line and think about, okay, how do I use it? How do I put in my requirements? Kind of like, how do I build my own zip file that I use for deploying or running my code? Does that work for 3.5 or just 3.5 now? That one just works for 3.5, But in general, you can use multiple Python words. We come to that in the end. I'll run out of time if I answer questions now, but we can come to that afterwards. But yeah, just to illustrate the point, what you ask is, does the binary I've just shown, does it work just for Python 3 or other versions? In this case, it worked just for Python 3. I would have to build a new one for Python 2, and I have an example in the end. but before you now go all ahead and remodel your CI pipelines to write some bash to actually build PECS files I would like to introduce you to PANS because PANS is a build system it's just like the thing I would normally imagine you would have a huge java shop but it also helps to build PECS files because it's especially good if you have different code or have code that works in different PECS files so that you can just write it once and include it in different binaries And again, if you go to the website, you can look up what Pan is doing for you, and it's a build system for Python, Java, Scala, Go, C++. It also has support for libraries such as Thrift or protocol buffers that can generate your files from your specifications. And it essentially supports all the stuff you would imagine that a build tool has. So it can bootstrap itself, it can generate code, it can do dependency resolution, for example, for Python dependencies. It can also do compilation. we don't need that really for Python because that's more something you would need in Java or Scala. It can run your tests, do some linting so that everything is formatted as it should and finally it can do the bundling which in our case means mostly build a hex file with it. And the thing is if you look at why Panels was built it was actually kind of like designed so that Twitter can use its large monorepo so they have one massive git where everything is in it all servers service or clients whatever you can use it for that but you don't have to and that's kind of why my message here you can hate the idea of monorepo or you can love it i don't really care what i care about that the tool supported it but we can also use the tool for other use cases we can come to that in a second but the thing you can remember is when you have a monorepo your tool needs to support slicing so you have this massive code base but you're now just working on this one tiny component and you want the ability to just run the test for this component and maybe other stuff that depends on it, but you don't want to compile everything all the time because you want to stay productive even though there are, I don't know, hundreds of engineers doing changes simultaneously in other areas. And that's what PANS is good at, but it's also suitable for our Python stuff. And so first of all, as I mentioned, it allows us to build PECS files. So with our own code, but also with dependencies in it, It also allows that to kind of use the tool we normally use. So for example, at least we often use PyTest to execute our tests. I think it's similar for many of you and kind of like Pants would use PyTest to run pattern tests. It also has iSort support so you can kind of sort your input and check that it's all okay. It has support for SetupPy which is commonly used to build source distributions. That's kind of in there by default. And I think one of the most important points is that it's a low barrier of entry, meaning if you now consider it would build for a company with a monorepo, that normally sounds like you now need a build farm to use the tool and set up servers and all that. But PANS is pretty simple because it's just a client. So just a local application that you kind of install, then you can use it directly. You don't need to set up anything. You can set up cache servers so that you don't have to compute all the stuff all over again, but it works locally totally fine. Just install it and use it. That's what we're going to do now. And another good thing, it's pretty extensible, so it's written in Python. And if you have a use case that's not yet supported, you can just write a little Python code and can then extend pants. And as it was a monorepo, you can even have your extensions in your own code repository. that makes it pretty flexible. Okay, for all that, it's like a lot of talking, but I think it's best that we have a look and just see how PANS works. I've built an extensive example. You can, I don't know, have a look at this URL. It's mostly all in there. You can also follow me on Twitter and I will tweet it afterwards that works as well. And I think it really makes, if you wanna have a feeling for PANS, it makes sense to play around with it a little bit. And, yeah, okay, so, how does the example work? Because I need to explain it a little bit, otherwise it might get confusing for you. So, actually, it's again a simple application that just prints hello world. Once this is done, so we have this organized source, Python, then we have hello world, our folder with our packages in it. We have a CLI application with a main.py, and it just does printing one message, Printing one string, print no word. And the formatting for the message is done in an additional module, that additional module called greeting.py, so just another library that we use in our CLI. And finally we have test, the test formatting is all okay. And this is kind of like our minimal Python example, but we need a little bit of support for pants so that it knows that we have Python code and what it actually means. So what comes on top are build files, and we'll have a look at one of these in a few seconds. So actually, normally you put one of these build files in each of your, you have flexibility where you put them, but normally you put in one build file in each Python module. So kind of one per folder, because it's kind of like pretty fine, granular description of your thing, but you could also just have one giant one at the top level. You have flexibility, but recommendation is you have one of these in most of your folders. In this case, it's pretty extreme because we just have a simple, simple main.py and nothing else, but normally you have larger modules that you know, okay. But what's actually in the build file? And here's a simple example. So what you see here is that it's a Python library. So the build file would just contain this statement here. It says what I have in my local folder is a Python library. and you give it a name, and you have a glob to say, please include all the Python file that you find in the folder, include it in here, so that belongs to this library. And last but not least, you have a list of dependencies. For example, one dependency would be, I depend on click for my command line parsing. Or you could say, I depend on other libraries in the same repository, for instance, hello world foo, hello world bar, and I depend on that to build my CLI, for instance.

Speaker 2 [13:45]

This is good, I know.

Speaker 1 [13:46]

This is like it's used as a specification kind of but it's Python code yeah and if you want you could do magic stuff in there like I don't know do anything else like contact the server and give me the name that I need there but people will come and slap you I guess yeah it's kind of like one of the design decisions that you normally write the specification but in a large repository Sometimes you need special handling, corner cases, whatever. So they give you the ability to do what you need to get your job done until it's finally integrated in the tool. Okay, so this specification is used to build a dependency graph. So, for example, my target would be one dot in this graph, and the dependencies would be other dots in the graph. And it's kind of all connected through the dependencies declared here. and then we can go ahead and use the command line to do certain things. For instance, we could say pants test and give our my target library and then we'll go ahead first place, okay, what do I need to do to run tests? I might need to compile, then execute the tests and then it looks at all the dependencies and figures out which one do I have to compile and which one do I have to test to get the whole thing done. That's kind of like the general idea and we will now jump into the console and yeah have a look at it okay first thing is you can ask pants for what it can do for you so kind of the general goal that you can use pants to to achieve what you want the list is pretty long and it even can be extended with plugins so we don't look at all of them we'll just look at a few of them. And the first one I want to show you is that you can ask pants to list everything that's in your repository. So pants list and it will just print all the kind of like all the dots in the graph we've seen. You see a bit more than a kind of like plain would be in the demo but it's a little more advanced but we look just at the subset. So the first block is kind of like external dependencies that we have. third-party requirements like NumPy, we'll kind of ignore them for now. Then we have the Hello World. This is our application I talked to you about. Then we have additional library math fun. It's in the repository with some additional information, but we'll just look at Hello World for now. And finally, we have the test zone here. And we can now use this information to tell Pant to do something for us. For example, we can tell Pant, please test, and we say, please run all Python tests. So we look at the test folder and say please run everything that's stored under test Python. And essentially Pant does the thing so it's if you look at all the different steps it will do the setup it will do maybe code generation it's not used here so it does anything does preparation for the interpreter finally runs the test and we see here it runs the test for our math library and tests for the hello world example. And that's kind of like how you would interact underneath it's using PyTest But when we started, that's how you run tests, but when we started, we started with a binary, so the kind of like thing we had in our pex file went to execute. This one is here as well, so we can ask pants to run something for us. So we get pants, we have a soft folder, we have Python, hello world, and in there we have the CLI and there's a binary called hello world and we can run that and now again pants will do it bootstrapping and we'll run the entry point the command line we have you don't see it here because it kind of hidden in the output we could either hide the output or I prepared something else to make it more readable so we have something more advanced to put spit it out so you see to actually run the command line. If you would be interested to see kind of like what dependencies we have in there, because as you remember we had the build file and the build file we said what dependencies does our binary have. And it's not like in virtualenv where you have everything in one large virtualenv, every dependency installed, but in Python, in PANS you just have the ones you've declared in your local build file plus the dependencies. So we could say please drop me into, That didn't work. Oh, no, it did work. We can just drop into Python interpreter, and now we just have the dependencies installed that we had locally defined. So we could do the same thing here. And we could also say, yeah, please go ahead and write the whole setup into builder binary. So when I go ahead, look at the build file, and it has actually created, we see here, and the dist hello folder it has created all hello world links on and we can do the same thing as before we can say hello pycon and we can again use the mode and I don't know, whatever is in there and use the binary and print it so that's kind of like the thing you normally do if you run stuff and build your binaries and all that but there's a little bit more to it because PANS knows about dependencies, so it can do more important stuff, which is figuring out what's changed. So we go ahead, go to our message, and maybe do some little thing and change it. Now we have essentially changed code. You can see that in Git it has changed something. And I now need to look at my cheat sheet because I can't remember it. and what I'm now doing with this command is I'm telling pants please look at the head of the repository so kind of get master and compare what has changed and it can tell me that this particular target so this particular point in the graph has changed so it knows okay I've changed one build file it belongs to this point in the graph but it also knows what what stuff does depend on it so i can also go ahead and say okay i know that dot has changed but what else has changed as well so you can look okay there are direct dependencies that depend on this and you can even use this then to run your tests and so you no longer have to think about which i've changed the line which test you have to run but your build tool does it for you so it figures it out and can even do stuff like i'm now in the branch and there's master i could run all but it's enough if I just run the test that would actually affect something. So because it's a totally unrelated code, why should I run the test if it hasn't had anything to do with the code I changed? And this is something Pant is offering. So jumping back to the slides, you can get pretty advanced in those build files. Here just a few short examples. On the left you see everything related to Python. We'll just have a look at two of them because the other step would take too long now, but the first important thing is we have the setupPy, so this is the part below, and this allows us to integrate with the existing ecosystem in Python. So we can say, please generate a source distribution for me. So our library's not just living in the repository, but we can say, yeah, please also export it so that it lifts, kind of like create my table so that I can upload to PyPian. This supported there, and there's one interesting thing here, the zip safe, who has seen that before in SetupPy, and do you now know what it means? Because it now means that if this thing is safe to run as a file, and if you set it to true, it works, if you set it to false, the tools like PECS have to do some kind of like bootstrapping and dump it to disk, and that's the only way it works. And we can also define binaries. This is how it looks like, though pretty similar. You have the name at the top, so you need to find the entry point, so where we wanna run. You can also define the dependencies, and interesting enough, you can define the platforms. You say you can run locally, you could also build for Mac OS or Linux, and you can also, that's a good question. I haven't tried it. I think the first thing it depends on is if you have the libraries available on PyP, like, yeah, we'll have to try it. Yeah, and that's kind of it. The main gist, that's all in the repository if you want to play around it. And I would now like to kind of move to the summary of it. So, first of all, what is Pax again? It's a single file hermetically sealed like a virtual end but in a zip file. It helps you to unify local development and deployment because everything we've seen in PANs is underneath wrapped in PECS. So if we open an interpreter or if we run tests, that's all happening in the PECS file as well. And we can also use it for deployment because suddenly we have one file that we can copy to our servers. And in some use cases it allows you even to replace Docker with WeGet because why do I need Docker if I already have an interpretive machine You can just do a VGA, download the file, and run it. In certain cases, this simplifies your setup. Of course, you need the interpreter and the machine because that's not contained in the Pax file, but still it simplifies quite a few things and makes Python behave more like Java where you have the virtual machine and just download your Java file and run it. That's a good thing. And for parents, it's like a build system. It has quite a few features. It has to work for Python and Pax. And the important question is kind of when would I recommend to use it? I think most of you won't have a monorepo, and that's totally fine. But there are the use cases when you can still use it. For example, if you have, like, a large monolith that you want to split up, split up, and say, okay, I want to pull three services out, that's actually a good use case, because, okay, let's start with PANS. I keep it in one repository for now, but I can define the different components in my graph, and I will kind of, like, define this as, I don't know, microservice one, microservice two, and you can slowly iterate and kind of like ensure they're decoupled get the code reused and then only once you're done you can say okay now i move the whole thing out of the repository because if you do it quite in the beginning you will have lots of complexity with test execution because you will find lots of places where you have cross imports and it doesn't really work and the approach with pan would be something that would probably help you to get there And finally, sometimes you have the use case that you don't just have the server, but also clients. And sometimes it makes sense to have a separate repository, but sometimes also have it all in one, because you can just have one repository to care about, one Jenkins job that runs all your tests. And this is kind of where I was also coming from with Aurora, because Aurora has two clients and two binaries that you need on each server, and this is all maintained in one repository, which make it pretty easy because you can certainly make changes all over the place without having to first update the library, then up bump that all users use the library. This becomes significantly simpler. Okay, so coming to an end, I hopefully managed to explain to you that the build tool is not something for the mega corporations, just something specifically for them, but also we as Python programmers can profit from it and maybe make your life a little bit simpler. Yeah, so thanks everyone for your attention. If you have any questions, just come to me. I will be at the booth outside afterwards. Thank you. Question? Oh yeah. Yeah, thanks for your talk.

Speaker 2 [26:23]

Yeah, thanks for your talk. That was really interesting. I wonder how Pax actually works internally. So you say it contains wheels. So does it build a virtual env when you first start it? It does.

Speaker 1 [26:35]

I can show you, essentially it is, all those are unpacked. So the PaxFile we unzipped, and it has this devs folder, and this is not just the wheel, but actually a folder, and you can kind of navigate into it. So what's actually doing is that just setting the Python path that points to it and then the interpreter knows find the modules and what to do

Speaker 2 [27:12]

Okay, so what about binary modules? How does that work? I mean, I know that there's no portable way to import extension modules like in Windows, Linux, whatever, directly from a zip file. That actually works.

Speaker 1 [27:27]

Actually, that's a good question. I don't know all the details, but you could either ask on the Slack channel. They are pretty responsive. Or we could probably poke around in that bootstrap folder because I think they're doing some edging in there. Any other questions? Okay, then. Thanks. What a great talk. Thank you.