Script, Library, or Executable: You can have it all!

Speaker 1 [00:03]

All right. So thanks, everybody, for coming. First, before I get started, I have a short story. Many moons ago, when I was taking chemistry in university, I had a professor, and it was an ADM course. And I wasn't a chemistry major, so I wasn't particularly interested in waking up at ADM. and for some reason he would give muffins out for people who answered questions and that somehow coincidentally made the lectures a little more interesting so i figure after lunch maybe muffins aren't the best approach but i'm the guy carrying notebooks around because i like to use pencil and paper at a computer conference kind of makes no sense but two questions to think about while the talk is going on um one how many ways can we launch the same code by the end of the talk um If you get it right or get close, you can have a notebook. Two, what's another way that I didn't mention that we could launch the same code? So two things to keep in mind as we get going. As you can see here, the title of the talk is script, library, or executable. You don't have to choose. You can have it all. So my name is Luke Lee. Here's my Twitter handle. That's the best way to get in touch with me. I primarily do GUI and CLI applications in Python for oil and gas. But you're not probably super interested in me. You want to hear about what I'm here to talk about. If you'll notice this link down here will have all the references to the code and all the stuff that I talk about and everything if you want to find out more. So the goal of this talk is to show one way to expand a CLI script to support a CLI interface, a library, a GUI, and an executable, all with the same type of code. i just want to stress that it's only one way because there's a ton of different ways to do this and this is just a way that has worked for me in the past and i've had success with so first off let's start with a simple cli example what we're going to do here is we're going to walk through building the wc unix clone if you aren't familiar wc is a simple unix utility that counts lines and words in a file the code isn't particularly interesting here it's just going to use it to build as an example so here is if you know your co-worker comes and says to you oh i really wish i had wc on windows here's what you might end up whipping up if you're using python 3 again the code isn't super interesting all it does is open a file take the first argument count the lines and words and print them out so there's a couple of things that They're missing from here. And again, for this one simple one-off case, this is probably just fine. But if we note there's no help, the code isn't very extensible. The reason for that is that the argument parsing and the real work are kind of mingled together. Not super problematic in code that's this small, but it could be a problem later. That also makes it more difficult for other developers to reuse that code later. And then the other caveat is that it requires Python. So if you have a user that is maybe on Windows or something and they just want to use this, you might have to have that weird conversation where you talk to them about Python 2 versus Python 3. Always a bit uncomfortable. So the first principle of being able to support all these interfaces is the separation of concerns. What this means is, I mentioned before, that the argument parsing and reading of the file were mixed together. So that makes it a little harder to work with. So in this case, we want to separate the concerns of dealing with the CLI interface and then the processing of the file. You'll hear the separation some concerns idea preached a lot with GUIs. It's probably more useful and talked about there because you might have a widget over here and then you have a model over here. And if you have those two things mingled together, then you can't switch out this widget or switch out this model. You have everything stuck together. When dealing with the CLI like we're doing here, we can just think about sys.argv as our user interface and in this case counting lines and words is our action so again it's useful to separate that so what would that look like also we're still using this very small little bit of code and i've trimmed it just so that we can fit it on a screen so maybe one way we would do this is to separate those two things out with like a cli function that will handle all of our argument parsing and then this get file info which is basically almost all the same code that we had before the difference is it's going to take an argument called file so it's not going to deal with sys.argv anymore and it's going to return some data so it's not going to print it and then we have the standard python idiom of dunder name equals dunder main here and that's just going to be the glue that ties things together parse the cli and then call the function and print that out the the other interesting thing here is that we're doing the printing in the dunder main portion all that does is keep us from printing when we're importing. So that way we're going to only print now when we run as a script. And then real quickly, the get file info, I refactored it just a little bit so that it returns a name tuple. So the API is just a little cleaner. It's easier to see. But again, nothing super different here. So what does the CLI function look like? If you've ever used arg parse, which is built into the standard library, there's not going to be anything very exciting here. A couple of reasons I'm using ArcParse is, like I said, it's built into the standard library and it will provide us with automatic help. And the other thing is it's kind of boring, which could be really nice. If I give this to most any Python developer, they're not going to freak out and spend all their time trying to figure out how the argument parsing works, because more than likely that's not the most interesting part of your application. So every now and then it's okay to pick boring technologies. That being said, there are some really nice CLI frameworks in Python. One I really like a lot is Qlik. CLI frameworks are kind of like static site generators. Once you start to learn a little bit of code, you write your own. So there's no shortage of choices out there. But in this case, we're going to stick with argparse for other reasons that we'll find out later. Again, the code, not particularly interesting, except for this args equals vars line. argparse does this really annoying thing, to me at least that when you parse the arguments it returns a namespace object which works pretty much like a name tuple but then if you pass that data around everywhere everyone knows you used arg parse because you're passing around an arg parse name tuple which i find really annoying i don't want to call a library function and it know i had an arg parse type in there so this vars arg vars function just turns that into a standard dictionary so i can talk dictionaries to everyone and no one has to know that i ever used arg parse at all not because i'm ashamed of it but that's just because I don't want to leak that implementation detail out. Okay, so that takes care of a CLI interface. Now what happens if we have a co-worker that comes by and says, hey, you know, that WC thing you wrote would be really neat. I want it in my code, and they happen to be doing Python work. You're like, okay, well, you could copy the CLI file around, or I could refactor it into a package so that you could use it in your code. So the way we would do that is we have to expand our directory structure a little bit. First, we're going to have to have a setup.py. And then I'm going to call the tool pywc because I'm not that creative, and it's pywc. And we'll have an api.py function. That function or that file, you can name anything you want. I just used api here. And we're going to put most of the code and all the real work goes in this api function. And then we have these other two strangely named files. So if you've been around Python for a long time, this won't look weird to you. but if you haven't been around Python for a while, this will look very, very weird to you. The first one is the dunder init.py. That just tells Python that this is going to be a package. Coincidentally, that's not required in Python 3 anymore, only if you're making a namespace package, but that's a conversation for a whole different talk. But in Python 2, this will be required, and it's still recommended, so we're going to do that. And then we have this dundermain.py, which we'll get into a little later. As you can think of, it has a strange name, But since it has this double underscores you that should be your clue that it probably is something built in So what does a setup that py look like? There's a million different ways to do a setup that py that again is another thing That's a whole nother talk or tutorial all on its own the setup function takes the tons of arguments There's a ton of different ways to do it so I'm just gonna not show any of those things and show the very simple thing a Name a version and then something that I find kind of useful when you want to list packages in our specific example it's not super useful but a lot of times people will list out all their packages as strings and then if you add a sub package and you forget to add it to your setup.py and you push it to pypi or something and then you're not getting all the files so one thing i always do is use this find packages function and what that does is it goes recursively into the directory you're in finds all the python packages and will make sure they're included in your distribution. Other than that, tons of options, but we're going to skip over that. So kind of the main star of the talk is this dunder main.py file. So what the heck is that? Looking at the code, not anything that we haven't really seen before. We're going to import from our API to get that get file info function. We're going to have our CLI function that's going to be exactly like it was before. And then we're going to have a main function instead of putting all the code in the dunder name equals dunder main. The main reason for this is if we ever import from dundermain.py, I don't want all this name pollution. So if someone runs dir on our module, we don't want them to see the args variable and the file info variable and all these other things. So what I'll do is I'll package that into a main function so that those names don't leak later and make our API kind of ugly and then again it's not that not that different we're just going to call that so that's kind of how our library works but then you might be thinking okay well we kind of lost the CLI interface because now everything is locked away inside of a package how do we run a package well setup.py has one way for us to do that the entry points argument so let's revisit setup.py and add a little bit of code. The only thing different here is this entry points line. And all that's doing is it has some kind of fancy dictionary that you pass it. And it has to be called console scripts. And then this next line, it has some really weird syntax. But what it says is install a console script called pywc. And when someone calls that, I want you to call pywc package, the dunder main module, and the CLI function. So it looks kind of fancy, but that's all it means. And what this is going to do is when you pip install the package now, you're going to get a PYWC executable or script probably on your path or in your virtual environment or somewhere. So that sort of takes care of the CLI interface, right? So now the library developers that wanted our code, they can pip install it, and they magically get this PYWC script. But maybe our original CLI user is going to be a little annoyed because maybe they have no idea what the heck pip is. Maybe they're not really a Python developer. They just happen to have Python installed. So telling them to say, oh, we'll install pip, now pip install, and now use this is just not going to happen, right? So how can we support them as well as the library developers? The trick to that is Python has this built-in argument called dash m. And this is a standard Python interpreter argument. What this really means is run this module as a script. So that's the trick with our dundermain.py. So what dash m does is it looks for dunder main dot py or dunder init equals dunder main for the name of the package and runs it like a script. So you can think of this dunder main dot py file as kind of the package equivalent of the dunder init equals dunder main idiom that you see in regular scripts. And just to show you that this is something that I invented, and I'm not completely lying to you, it's used all over the standard library. So here's a list of all the different things you can use. not all of these are using a dunder main dot py file some of them are using just dunder init equals dunder main but there's all sorts of really cool stuff in here i'd encourage you to like check all these out run python dash m with all these names in the dash h and see what they do um some cool ones i think they're around is sysconfig if you're ever really interested in how your python binary was compiled and what kind of options it used do python dash m sysconfig it will tell you all that information um another trick if you're a web developer is json.tool python-m json.tool give it some unformatted ugly json you'll get nicely formatted indented json out on standard out um zip file tar file you can do all these things just by installing python you can get a calendar pop up a web browser base 64 and code stuff um all sorts of stuff with this dash m if you've never seen it before it's really cool so now our package is going to behave the exact same way so if you're keeping track now we have two ways to run this package we have python-m pywc and i'm adding the dash h here because we get that for free with arc parse and then you can also use the pywc-h that was installed with the entry points for setup tools okay so that takes care of the cli interface and the library user so now we can tell our original cli user oh we'll just copy this pyc directory instead of this pyc.py file run python dash m instead of python so it's not as dramatic of a change for that user and then the library developers are happy because they can pip install it just like it's regular pypi stuff and in fact you could put it on pypi if you want the next part comes in when maybe you have a user or someone that's heard about this cool application and another department and they have no idea what Python is and they're not going to be like most of us and just sit here and look at a black screen with white letters all day. They're not going to want a console or a CLI approach. They want a GUI approach. So it might look like I have a huge typo that I should have checked way before if I'm going to be talking about GUI on the screen right now. I assure you that it's not the case because what I'm going to do is use a project called GUI, G-O-O-E-Y, to build a GUI, G-U-I. that's actually more confusing than explaining to you just how it works so typically with the GUI project just go with me with the names here how that's going to work is you have basically only two line change we import and then we use the decorator the trick is you have to put the decorator on the function that handles arg parse now remember I said that we were using arg parse for a number of reasons this is exactly one of those reasons GUI works with ArcParse. And what it does is super clever. It's kind of like you look around at the code, and it's like one of those things, I don't know if you've ever dealt with it, where you look at the code and you're like, man, I wish I would have wrote that. Really cool. What it does is it acts as a decorator, and it replaces the parseArgs function of the ArcParse library with its own code, and that makes it call back into itself. And then it looks at all the actions that you add in ArcParse. So, for example, in this one case, I have add argument action store. So it looks at all these actions on your arg parse object and maps them to an equivalent GUI type that would be useful for that. So if you had store true as the action, it's going to say, okay, that's probably a boolean. I'm going to give you a checkbox. In this case, store might mean that you're going to get a text box. It does all of this stuff behind the scenes, and you don't have to do anything. So you can get an automatic GUI with basically two lines of code. Another thing to note is it uses WX widgets, which some people may say is great some people may hate but if you're just building a little CLI script getting like getting a GUI for free is pretty sweet so here's what it would look like if I used it with absolutely no customization at all luckily there's tons of customization I'm just too lazy I'm not a graphic artist so this is what you get again so it's mapping you know the count words and count lines it's all those as booleans the store Excel does There's a text box and you get a start button. So if you want to use GUI and you only want a GUI interface, that would be all you need to do. But the whole point of this talk is to support all of these launching mechanisms at the same time. So we want to support dash dash GUI. So we want to opt in. And so to do that, we have to kind of jump through a little bit of a hoop. We can't just use the at GUI decorator because that will turn our script into a GUI only application. So again, the separation of concerns idea, like let the CLI do its thing, let the GUI do its thing, we'll keep them separate, and only reuse the stuff that makes sense. So we'll add an allow GUI option equals true here. The only reason that even exists is we don't want, since GUI introspects your arg parse, we don't want an option in the GUI itself to say, show the GUI, kind of doesn't make any sense. So that's going to hide that, and then since we don't want to use the decorator approach because we want to use an option to do it, we'll make a GUI function, and then we kind to have to simulate the way a decorator works, which is a function that takes a function. So we create a small function, bind that argument to say, don't show the GUI when I call you, and then call a GUI as if it was used as a decorator, but without the fancy at syntax. So then we basically have a GUI. Then you might think, man, I really wish I could install that GUI. Well, luckily, setup tools has a way to do this. So decor Larry to console scripts is GUI scripts. Works basically the same way. The only thing that's going to be different is this GUI scripts is going to tell PIP to install it as a windowed script. And what that really means is a fancy way for saying don't pop up a console window. So, again, we build the GUI to keep people away from having to sit here at a console. So if we launch a GUI and then we launch a black and white window that spams a whole bunch of text, it's probably going to freak a lot of people out. So we're going to hide that by using this GUI scripts. And we're going to call it PYWCG. and again same story call the pywc package the dunder main module and then the gui function so for all those keeping track two more ways dash m pywc dash dash gui or pywcg so remember pywcg was pip installed now with the console scripts and next so we have cli users happy we have the library users happy we have the gui users happy but then maybe we have another set of users that has no idea what python is or in the worst case maybe you have a manager that says oh you really should not use python you're like i can use what i want you just won't know so the way to do that we pi installer um what pi installer does is it's kind of hard to describe as anything other than just pure magic um and i'm going on like this is just a bet here but the guy that wrote pine solar i know i owe him many many beers and he's from germany so any chance he's in the room one day the answer to that is going to be yes one day um anyway so all that aside pine solar you give it a script and it performs a whole bunch of magic on that script to find out what your dependencies are, package them all up into an executable or a directory that you want, and you ship it to a user and they have absolutely no idea where it came from. Unless they want to dig around in the directories and they see stuff called like python27.so or dll or something. But again, I seriously doubt this sect of users is going to be doing that. So, the other cool thing that PyInstaller does is it supports all sorts of common stuff that people use, like PyCute, NumPy, WX, which is useful for us because we're using the GUI project. You can also make executables for Windows, Linux, and macOS. The only downside of that is you can't cross-compile. So if you want to make a Windows executable, you have to do it from a Windows machine and vice versa. And again, the huge benefit of this is no one has to know that we used Python. So that manager that said you can't use Python, maybe they won't care if they get an application that works exactly the way they want it to, and they have no idea. They get a button to double-click, everybody is good. So, how do we expand to support PyInstaller? In general, we wouldn't really have to do much of anything other than maybe rename some stuff. So the way we're going to do that is PyInstaller takes a script, and it only creates one executable from that script. So we can't really pass arguments to that script to turn on the GUI and vice versa. And I think it's a little cleaner if you just separate things out. Again, the separation of concerns idea goes deep. So we have our setup.py. Nothing's going to be changed there. Now we'll create a cli.py that's going to be out of our package and a gui.py that's also out of our package. The reason for this is then we can tell Pinesol to create a cli from the cli.py and a gui from the gui.py. And then all this code can be shared inside of dundermain.py. So we're trying to reuse as much stuff as possible. So what does CLI.py look like? Super boring, super good. So import the CLI function and call it. What do you think the GUI function looks like? Exactly the same. Changed, like, a few letters. Awesome. And how do you use PyInstaller? Pretty simple. PIP install PyInstaller. PyInstaller CLI dash dash name PYC. So what this does is PyInstaller looks at the CLI.py, finds out all of our dependencies, which in this case isn't going to be much. But in the case that you're using NumPy and all sorts of other stuff, this is huge. The fact that this works is just insane. And in the dash name argument to say we want to create an executable, we want it to be called pywc. That will be pywc.exe on Windows, and it'll just be a regular executable on Linux, and on Mac it will be inside of a dist folder. There's different options for all of these things for different OSs. So that would create the CLI.exe that we can run. Now we want to make the GUI executable, and that works almost the same way with two small caveats. We're going to name it something different, and then there's this dash W argument. That's kind of the equivalent of what we did when we said in the setup.py use GUI scripts. Remember I said that would make it not launch the console window. Well, this dash W does the same thing in PyInstaller language. So it says create a windowed application so that we have our own GUI, we can do our own logging wherever we want. Don't pop up this console window. And the second caveat is, unfortunately, misfortune for me, is that this line still doesn't quite work if you run it as it is. There's a pull request waiting in PyInstaller to package up the GUI project. It has some language files and some other stuff that don't quite get packaged right by PyInstaller. but luckily you can change that if you add a dash dash additional hooks directory and you give it one file if you look at the references based on what the link i have here i'll show you exactly how to do it it's like one file or i mean i haven't checked today who knows maybe he's not here because he uh you know merged that pull request and it works even better okay so now we've got the CLI we've got a library we've got a GUI and then we package everything up so no one has to know we're using Python I feel like that's kind of weird to say at a Python conference that you want to hide that you're using Python but whatever so the takeaways this idea of separation of concerns so the idea that we want to take our interface and separate it from our real work so in this case it was pretty trivial to do that but that in a bigger project you can see how that would give you a whole bunch of opportunities to be able to reuse the code. So if you notice, we support all these different entry points and ways to launch your application with almost no additional code. We're basically just reusing the same one, remix and repeat. So the separation of concerns idea is really, really useful, whether you're trying to make a GUI or a CLI or whatever. That's still a good practice. How do we do that? Entry points. That's That's the first way we can support a library and executables at the same time. So you pip install, and in the setup.py we had these entry points, console scripts and GUI scripts to be able to do both so that we can install both. Next, what I always think of as kind of the star of the talk is the dundermain.py. It adds support for the "-m argument to Python." In this case it served as kind of our single file entry point. We shared most of the code in dundermain and just kind of imported it in other places. The other cool thing is it's a commonly used pattern, like I showed you. The Python standard library does it all the time. And I'd encourage you, when you install a package, just run it with dash m and see what happens. Sometimes you'll be super surprised. GUI, again, looks like a typo, but this is going to make a GUI for us using WX widgets, typically with two lines of code. In our case, it was a little bit more. PyInstaller, just pure magic. When you do the PyInstaller line, you're going to see it spit so much stuff on your screen. And it's oddly gratifying to just see it's doing so much work for you. And then you can run it with log debug and look at all the stuff it's doing. It's just amazing. I can't get over it. Okay. So that brings us pretty close to the end. And I don't have beer and donuts, but there's not really a notebook emoji. So I'm going to talk to Apple about that. Does anybody have a count for how many ways you can launch it? guess how many four five three it's not prices right eight close so this is the ways that I can think of I think the count is ten okay so so ten but that was just like an elaborate troll, because there's these three other ways to do it, too. These kind of work. They kind of don't make any sense. So you can run the GUI executable with a dash dash GUI argument. So if anybody had 13, well done. These kind of work as a quirk of the way the GUI introspection works. And that's kind of like a take-home homework, if you want to figure out how that works. Does anybody have a way that we could have launched it that I didn't mention? as what as a service okay anybody else as a microservice oh all right more buzzwords more buzzwords what else we got what else we got um so one i didn't mention but i kind of scrolled past is in the dash m there's a project called or a package called zip app and what zip app does is built into the standard library and it does almost exactly this it will zip up a directory and it will just run your Python dundermain.py and you can ship it that way too. So that's kind of cool. Again, like always use the dash m and like dash h and see what's in there. Okay, anybody, any other ways you can think to launch it? How about a Docker extension? I'm just trying to think of weird. All right, that's all I have. I think we may have time for like two questions. So there's my contact info. So the link will point you to GitHub Gist, and if you want to comment on the talk or say anything, point out different ways to do it. Or if you have ideas and you have horror stories of something, dealing with supporting all these interfaces, I'd love to have a beer and rant and rave about it. Or if you have other ideas of ways to do it, I'd love to hear that too. So any questions? Correct, so the question is, now that we depend on the GUI project, do we put that in the setup.py? Yeah, I didn't show it here, but I would add that as the install requires argument. And then you would also typically have like a requirements.txt or something. But the way I've written it, and if you check out the real code, you could see that the way I've written it, the CLI doesn't depend on the GUI. So if someone pip installs it, it doesn't matter. It won't complain unless they try to use the dash dash GUI argument. Sure. I don't. It's possible there's something like that. Oh, yeah. So the question is, do I know of anything that's like a GUI for Qlik? There might be. I think the reason that GUI, one of the reasons I think GUI works is, like I mentioned, And arg parse has been in the standard library for a long time, and it's really boring and really easy to program against, whereas Qlik moves really fast. So I think if you made something like this, you might just end up chasing your tail a lot because of the way Qlik moves, but it's possible there could be something out there like that. That would be really cool, too. There's no namespace arguments in Qlik, so I'm on board. Any other questions? Okay, great. I tried it already, and it works somehow. Okay, awesome. So the answer to the previous question is there is a way to do it with Qlik. It's called Qwik. How do you spell that? Q-U-Y-C-K. Okay. Q-U-I-C-K. So, okay, exactly like you would say quick. I got G-U-I and G-O-O-E-Y, so who knows? That could have, yeah, okay, anyway, quick. That could be another way that it would work. And it sounds like it works somehow. Awesome. So that works. Great. Any other questions? Do you want to customize, like, typical things, So the question is, can you do subcommands, like the way the get CLI works, like you do get space add or get space commit or, you know, something like that. ArgParse does support that. I think it's called argument groups, and so you can have argument groups and do that. And I think Qlik does it, too, based on where you put the decorator or there's an argument to say which group it's in. And so there's definitely a way to do that, yeah. All right. Thanks, everybody, for coming. If you had an answer or you just want a notebook, they're up here.