Writing Plugin Friendly Python Applications

Speaker 1 [00:02]

So hi everyone, and before I begin, I just want to do a little bit of a survey. Can you raise your hand if you've ever used a plug-in, an add-on, or an extension for any piece of software ever? Yeah, I expect a lot of hands up. Can you raise your hand again if you've ever written a plug-in, add-on, or extension? Okay, a couple more. Alright, now here's the really hard one. Have you ever written a piece of software that accommodates plug-ins, add-ons, or extensions? Okay, okay. And now here's the final level of plugin mastery. Have you ever written a library that can be used to define plugins, add-ons, or extensions within a host program? All right. Oh, hey, cool. There we go. All right, cool. Well, that's cool. Thanks for the participation. So yeah, my name is Travis. And today, we're going to be talking about how to write plugin-friendly applications in Python. Let's go over today's agenda. So first, we're going to define what plugins are. So we have a clear kind of theoretical concept of what it is We'll also be talking about some examples of plug-in systems in the wild and then after that we're going to talk about why or why not you should write a plug-in friendly application and Then once we have decided that yes indeed we want to do that We're going to talk about how it's done, and I'm going to walk you through the creation of a project It's actually going to be a creation of a image search CLI application Cool, and then finally we are going to talk about a real world example where I talk about something from my work and how to retrofit an existing piece of code to be plugin friendly. So a little bit more about who I am. So yeah, first and foremost, I'm a Python programmer. I started writing Python back in version 2.6, so that was about 2010, 2011. So I definitely have been around for the version two days. This is actually my second time here at PyCon DE. I gave a talk last year on how to use OSM data together with Python and Postgres. And if you want to learn more about me, including my last year's talk, you can go to TravisHathaway.com. Lots of good stuff there. And then if you want to know more about the code that I've written, please check me out on GitHub. There's actually quite a few things that I've done there. All right. And then where do I work? I work at a company called Anaconda. I've been working there for just over a year now. And there I primarily work on the open source package manager, Conda. And so Conda is going to be the real world example. And so we have been working quite busy in the last year to open this application up, make it more plugin friendly. And Conda is written in Python. And so finally, who are you? Ideally, you're all Python programmers because I'm only going to be showing Python code. And then I also would be nice to have a little bit of experience building your own packages. You never have to have built a package, but as long as uploading something to PyPI doesn't sound super scary, then this is going to be good because we're going to be using Python packages as our way to distribute plugins. And then, yeah, most importantly, you want to learn more about plugin architectures in Python. So first question, what is even a plugin? Let's take a look at a textbook definition. So a plugin, also called add-on or extension, is computer software that adds new functions to a host program without altering the host program itself. And let's look at some examples of this in the wild. So if anyone's ever used Sphinx, which is the really popular Python documentation generator, If you've ever used Sphinx, for example, to add automatic documentation from doc strings or add a new theme, that's an example of a plugin or extension. If you've ever done anything on your browser, such as installing add-ons for Firefox or extensions for Chrome, those are plugins or extensions. Same thing with VS Code. Those also have a wide world of plugins that make the software much better to use. And then in the world of web, we have projects like Django, which has a ton of different extensions that you can install and download for that. And then finally, we have web platform, CMS platforms, WordPress and Drupal, and those have a very, very big ecosystem of plugins. And the thing to remember about these projects is that, I mean, maybe for the example of the Firefox, right, but a lot of these projects would be not nearly as good if they didn't have an entire ecosystem of plugins to use for it. Alright, so how exactly do plugins work? So I'm going to talk about the quote-unquote plugin mechanism, as I'm calling it. And it relies on two different things, kind of two different abstract concepts. We have the host program, and then we have the plugin program. And so the host program has a couple different responsibilities. You can kind of think of it as the development platform. It's responsible for choosing a language and perhaps providing a default set of libraries. So if we go back to our examples, WordPress, if you're writing a plugin for WordPress, you're most likely going to be using PHP. If you're writing an extension or a new plugin for Django, you're going to be using Python. And then the host program also defines something that are called hooks. And then the plugin program provides the implementation for these hooks. So what exactly are plug-in hooks, you might ask yourself. They are defined by the host program, as I said, and provide an interface that plug-ins use. And this results in an implementation that the host program can use to extend or modify its functionality. So on the right-hand side, we have a diagram that I've written, and it basically just shows the relationships between the host program creating a hook saying, hey, plug-in, this This is what you need to provide me. And then the plugin takes that hook and says, great. I'm going to put my implementation inside of it. And then the host program uses this implementation. And then other than having a system of hooks and implementations, plugins also need a way to be distributed. And so right here, I've listed three different ways that you can distribute plugins. And the top most is what I would say is the most tightly integrated. And then as we go down, it gets less so. And so on the very top, we have the host program providing an installer for you. So you don't even have to leave the application. You just basically go to a particular plugin section or extension section, and then you can search and add them from there. So VS Code and Google Chrome are great examples of that. And then if we go down, we now leave the host program, and we were relying instead on a package manager to install the plugin. And so this can be done via npm or pip or conda. And this is where, basically, you have the host program is packaged as a Python package, and then the plugin program is packaged as another Python package. And when you install the two together, the host program becomes magically aware of the plugin program, and it can begin using it. And then the last thing I wanted to list is sometimes applications can specify a folder where you simply drag and drop the contents of the plug-in code there, and then it's dynamically read, and then it's made available to the host program. And this is done in applications such as Drupal. Now that we know a lot more about how plug-ins function and how they're distributed, let's ask the question of whether or not the application itself should be plug-in friendly. Here I have several pros and cons, and we're going to go through each of them. The first pro is that plugins enable you to grow a community of developers writing plugins. Like I said before, where would a lot of those other applications be if it weren't for the huge community of people writing and extending the functionality of the host program? But on the other side of that, when you open up your program to executing third-party code, you also do involve a security risk. And basically what you have to have is a list of verified plug-ins, although I want to briefly mention that I saw a very interesting talk on WebAssembly yesterday, and I really thought it was interesting how WebAssembly could potentially help solve this problem, because WebAssembly would allow you to essentially execute plug-in code inside of a sandbox, so very interesting. Okay, so back over to the pros. So when you choose a plug-in architecture, it can make your application very highly adaptable And so not only can you accommodate the use cases that you know about today, your application could accommodate the use cases it doesn't even know about that could appear in the future. And a really easy example of this is to think about if you have an application that supports JSON and XML file types now, but let's say in the future we have some new file type that your application has to support, all you have to do is simply write another plugin to accommodate that. I know it's not that easy but in theory it could be that easy and so yeah and then moving from back to the cons so when you do have this plugin architecture you are increasing the complexity of the code specifically the host program because your host program not only has to worry about what it does it now has to worry about registering and using an entire system of plugins and so for For applications like VS Code or Firefox, this can be highly complex because we have actually a built-in plugin manager in the application. But even if you're distributing your plugins via Python packages, you still have to have code in there to register the plugin and then use them and then make the plugins configurable to use. So that's all added complexity for your program. and then back over to the pros not only is it good for building a kind of large open source community of plug-in contributors you can use this for internal projects at any small mid or large firms and yeah they're a good way to help organize teams you can have one team specifically working on the host program where multiple other teams take ownership of a single plug-in that they write to help them in their work and then back over to the cons the very last thing is that now that you have a plugin system you're forced to maintain a distribution mechanism and so this was kind of talked about in the increased complexity but yeah so for example if you want to have a host program but then you want to break out some functionality into a plugin well now you have to worry about keeping those two up to date those will then have different versions those have to be managed as different Python packages and yeah that's all just added administrative overhead. But we're going to assume that everyone has weighed the pros and cons and has thought, yeah, I'm ready to write a plugin application. This definitely fits my need. Well, good news. I'm here to show you how to do it in Python. Yeah. And so what we're going to do is we're going to be using a library called Pluggy. And so, yeah, this is a very relatively simple library, I would say, and it provides a framework for defining hooks and implementing plug-in hooks. And Pluggy lies at the core of PyTest's plug-in system, which is a very popular testing framework. And it has now reached, I think as of a little bit ago, or maybe actually I don't know how long ago, but it is now at version 1.0. And yeah, it's a really nice library with a very narrow scope of what it does. And so in our example, what we're going to use Pluggy to do is we're going to be able to define the hooks that our host program, we're going to define hooks that our plugins are going to use to implement. And then using Pluggy, what we'll also be able to do is use Python packages as a means of distribution. So we won't have to write our own internal plugin registering and installation mechanism in our application. We'll simply use Python packaging as a way to install or uninstall plugins. Cool. Okay. So now the example application. I wanted to pick something really kind of interesting to do, and because I work on Conda, I work on CLI applications, so I said to myself, hmm, what kind of application do I want to write? Well, I really like pictures of rabbits and cats and dogs, and I thought, wow, it would really cool to write a CLI application that can get me as many pictures of rabbits, cats and dogs as possible. So what we'll do is we're going to write a CLI application and the plugin hook is going to allow us to install different search backends. So we might initially write our application to use Google Image, but then we can allow plugin authors to add other search backends such as Imgur or even Unsplash. And so right now what I'll do is I'm going to walk you through defining the plugin hook and then I'm going to walk you through creating the plugin implementation. So let's and during this as I walk through we're going to first look at the host program side of things and then afterwards we're going to look at the plugin program side of things. So first with the host program and when you write an application in Pluggy you start off by importing the plugin, the plug-in module, and then in plug-in applications, there are two very important objects that you have to create, basically as singleton objects, and the first one is a hook specification marker, and then the second one is a hook implementation marker. And so the hook specification marker is used on the host program to specify which plug-in hooks you want to make available, and then the hook implementation marker is going to be used on the plugin side to say, hey, look, this is my implementation for this specific plugin hook. How do we do, and so let's look at how to use the hook specification marker. So the first thing that I do is I create a class, and this class is called app hook specs, and it functions purely here as a name space. I'm going to, when I register my hook specifications, I'll use this class later, but just know, This class isn't doing anything special, it's purely serving as a namespace. And then if we look below, we can see that we're now using our HookSpec object as a decorator to mark the first plugin hook that we have. And this plugin hook is a, basically you can think of it as just a callable, in this case it's a class method, and it has the name searchBackend. And then we're using a type hint to say, hey, this is going to return a searchBackend hook object. And now this search back and hook object is a custom data type that I'm going to talk about next. And so what we do here is we define a search back and hook object. And we do this because what we want to do is we want to hold more than just the search function, right, the function, the piece of implementation logic that goes and grabs our search results. We want to be able to store other pieces of data on top of it, such as the name. This class is a very simple class that inherits from NameTuple, and this basically means that it has a small memory footprint, it's immutable, so we don't change it, and we don't usually want to change this. And then below, what we do is we define the properties on the object. The first one is going to be the name, and this is the unique identifier for the search backend plugin, and so in our search CLI application, if we wanted to then say, hey, I'm going to do an image search, and then I want to use Google, and then Imgur, we would basically identify it by the name that we defined here. And then below that, we have this search property, and then that's going to be a callable. And so you can think of this as the function that goes out and grabs the search results from whatever search backend we want to use. Okay, so now that we've talked about how to define the hook specifications on the host program, let's move over to the plugin program. And what you're looking at here is more or less the entire implementation of this. So let's walk through this line by line. And so what we first do is from our host program, we are going to import that hook implementation marker object and then the search backend hook object. And so below what we do is we use the hook implementation marker as a decorator on this search backend function. And one thing to remember is that when we define the hook specification, we basically made it, we put it as a decorator over a method that was called search backend. Now we're putting this hook implementation over a function called search backend. The important thing is that these two names need to match up. This is how Pluggy internally says, oh, great, hey, this is the hook specification, here's the hook implementation, boom. Now we have a pair. In our hook implementation, what we do is we are returning the search backend hook object that we just talked about earlier. It's going to define a name, which would just be the plugin name for this, and then it's It's going to also define a search function. The search function here would accept a search query and then it would do whatever a search function does. You can imagine it would make an HTTP request or adding any sort of access tokens or anything like that. I left the implementation out for brevity. Then just to really drill in this whole procedure I just went through, I wanted to show a visual summary because I know some people here might be visual learners and this might kind of sink in a little bit better. But yeah, so from left to right, what it does is that we basically have a plugin that implements this search backend hook, and that hook returns what I'm calling a plugin configuration object, the search backend hook object. That then has two properties. It has a search function that we can use to retrieve results, and then it also has the name of the plugin. And then this is in turn used by the host program. So this is how we get from left, going from the plugin, all the way to the right, to the host program. And so now that we have the hook specifications defined, we have the plugin implementation there, we need to make our host program aware of the plugin. And I said earlier that, yeah, you use Python packaging, you pip install this stuff, you have these two packages, and they kind of just know that they're there. But let's figure out exactly how this works, because it's not quite magic. So this is the final step, mostly in plug-in applications, is that we want to tie everything together. So let's walk through this code line by line. So first we do this by implementing a plugin manager object, and it just takes the project name as an argument, and then what we do now is we use the plugin manager object to add the hook specs, and remember before when I talked about this app hook specs class as basically being a namespace? So that's going to take all the methods on that class and then register them as plugin hooks. And then finally what we have is we're going to call this load setup tools entry points method on the plugin manager object, and it takes the project name, which is important, because that needs to be referenced later in something called a setup tools entry point. But what is this entry point stuff you're talking about? I've never heard of that and I don't blame you because before I used Pluggy, I hadn't quite heard of this before either. But from the packaging.python.org website, they provide this lovely explanation. So entry points are mechanisms for an installed distribution to advertise components it provides to be discovered and used by other code. And so in our case, the plugin system is going to be that installed distribution and it's advertising its code to our host program. It's saying, hey, host program, I got some code over here that might interest you if you wanna come check it out. So that's how that works. And what does this actually look like in the code? Well, I showed two different examples here. We have a pyproject.tomo file, and we also have a setup.py file. And really, it's just as simple as having two nice little entries. This is the more modern way of doing it. But what we do is we have project.entrypoints.app. That's that project name from earlier. And so then we are basically assigning Google Image Search. And then with this, what we do is we're essentially just, this is like the module path to where the code lives. So our search, so the plugin module name is called app Google Image Search, and then where the code lives is a module called main, so .main. And then we can see the exact same thing in the setup.py file. So now finally we can tie everything together and actually call this plug-in hook and this is an example main function of how that might look. And so first we have the main function accepting a search query and then the search backends it might use and then below we have a, we're creating a list called search functions where we will collect all of the different search functions that we want to run. And so in this for loop, what we do is we use the plugin manager that we invoked earlier. And the plugin manager has a hook property. And then the hook property is aware of all the different plugin hooks. And so when you call search backend, it's going to return an iterable of all the different results of all the different hooks. And so for our case, right, it's going to be that search backend hook object that has those nice properties like name and search for the search function. And so when we loop through this, we basically are saying, hey, hey, search plugin, are you currently configured? Are you in this back end set thing? And then if it is, then we just add that function there. And then later on, we're going to call it. So that's a really simple example of how to use that in action. And I'm going to skip the recap, because I think we're getting a little bit low on time. But what I want to talk to you about is this was just a toy example, right? And not very useful. But as I was writing this talk, I wanted to make an actually useful project that could be used in real life. This is exactly what this project is. It's called LATS. And you can find it on my GitHub account. It has lots of documentation. It's also got, I even wrote an imgur plugin. It's got a couple built-in plugins. It's a real deal thing. And actually, I just wanted to show it to you real quick. So this is the command line application. You run it. some help messages there and when you do lats search husky hopefully we're going to find some dogs it will print out this lovely table thank you demo gods and when you click that we get a nice little dog from mgr that's pretty awesome yeah so good deal go check that code out it's um yeah it's definitely it's it was a labor of love that's for sure i just gotta Alright, so getting to the end of the talk, I said something about a real world application. Okay, what is all this conda stuff? Oh yeah, that's right. I work for a company called Anaconda where we write tools for data scientists and one of those tools is the conda package manager. And so the really cool thing about everything I just showed you and also about the way that things are set up in LATs is that it is almost exactly the same setup. It's the setup that's almost exactly the same in Conda. That's awesome. And if you want to talk about how it works in Conda later, come find me after the talk. Or you can also post in the Discord channel. I'd be more than happy to answer questions there. And if you didn't know, Conda is a package manager. We have over 30 million active users by some counts. The project is now over 10 years old. Because the project is getting older, it's basically starting to show its age. In order to help breathe new life into it, we wanted to start developing plug-in hooks that would allow people to extend the functionality. We also want to do this in order to help people get more involved in its development. For that reason, we have created currently three new plugin hooks for it, and I believe the most successful now is the solver. For those that don't know what solvers are, basically the solver is when you run conda install, it's in charge of figuring out how to find all the different packages for that particular command. This gets more complicated the more packages you have, but one success that we've seen is we've been able to incorporate the work that had been done in the Mamba project, and now that work can be accessed in Conda directly. And then we are really happy because now that we have a solver plug-in hook, this means that as developments continue in the world of solvers, our project can now adapt and be more responsive and be able to incorporate this new technology easier. Then we also have other ones for writing subcommands and creating virtual packages. So the future of this, I've been working on a couple proposals for adding a plug-in hook that would allow people to further customize networking. So this would allow you to, for example, if you wanted to grab a package from SSH or use different network transport protocols you could be able to do that with the networking plug-in and the other one is that we're currently talking about adding a pre and post command execution there's a lot of different functionality that you can add if you can interrupt the various conda commands before and after their run so that's really exciting and so yeah and I think I'm getting to the end and concluding thoughts so yeah I just want to let you know that plugins, they can be a really great way to organize your application, especially if you want to collaborate and work together with different people to build a really cool application. Like I said, it could be used for open source projects, but it can also be used internally as well. But as you do this, carefully plan the parts that you want to expose as plugin hooks. Because remember, as you write these and as people actually write plugins, which is awesome, But then you're going to have to support these hooks and you're going to have to be very mindful of how changes could potentially break those applications. So this is not something to be done lightly, but it can be a fun kind of thought experiment if you just want to go goof around and create your own Python project and add new plugin hooks like I did this year. So I had a lot of fun doing that. And then if you have an existing code, one thing I want to make sure that you do is to always document the internals because as you open up your application with various plug-in hooks and especially with a language like Python, especially if it's open source, people are going to go digging around and they're going to run into errors that they might not be able to easily solve by themselves. And so when they can go through and read your code and actually make sense of it, it's going to help them and it's going to reduce the amount of issues that you get on your issue tracker. All right. So thank you, everyone. I really appreciate the time that everyone took today to sit down. And yeah, I welcome questions if there's time for it. Thank you, Travis. We have time for a few questions. So most voted on Slido is, would you implement a default back-end, hard-coded, or already as a plugin, which is bundled together with the main program? Yeah. So I would always implement a default. And I can actually show you what this looks like in Pluggy itself. And so basically, let's take a look. So if we look at this code down here, the one thing I didn't cover about Pluggy is that it can load stuff from entry points, but then it also has this register method that you can register your own kind of internal hooks. And so for the example program I wrote, I have two different built-in things. One was for Unsplash, and the other one's a placeholder. It's just really simple, kind of more for testing. So yeah, I would always recommend doing default implementations. And when you do the default implementation, then you also give plugin authors a great kind of starting point for learning how to actually write plugins for your application. Cool. OK, I think we are just time for one. Can we define the data type structure of the result developed by the plugin via plug-in? OK, so are you meaning kind of like the type hints? I think so, like what will be returned by the URL. Yeah, yeah, yeah, so the really interesting thing about what I talked about here, let me see if I can find it. Right? So this whole step where you register a hook and you say, yeah, return that, that doesn't actually do any validation. And what I would like to do would basically be to make it more user-friendly for the plug-in developer is to actually write some sort of validation to say, hey, you gave me a plug-in hook implementation, but you didn't return this object. Can you please go back and try to return the object? plug-in doesn't actually enforce that but yeah maybe that would be a thing that we can that could be added to the project some nice could be a contribution yeah perhaps perhaps that'd be really cool so it's a great project I really like using it so thank you again for your presentation