Play Stupid Games, Win Stupid Prizes

Speaker 1 [00:03]

Okay, let's get started. This talk is called Play Stupid Games, Win Stupid Prizes. We're at PyCon D and PyData Berlin. It's Wednesday, October 9th, 2019. I'm James Powell. If you like this talk, you can follow me on Twitter at Don't Use This Code. Now, this talk is a talk about metaprogramming, which should explain for you why I chose that particular title. It's a follow-up talk to a talk I gave only a couple of days ago called Why Write Code When I Can Write Code That Writes Code. And so, in short, since we don't have a lot of time for this session, only about 30 minutes. We're going to go pretty quickly, and I'll tell you a little bit about what metaprogramming means in Python. It refers to a lot of different things, when you might use one of these approaches, and why you might do that. But for the most part, I can translate this agenda for you. Blah, blah, blah, blah, blah, blah. Then you're going to clap, and then hopefully you'll follow me on Twitter. Okay, so let's go. So generally, when people talk about metaprogramming in Python, they mean a variety of different approaches or different techniques, different mechanisms that are part of the language itself. One of the most basic mechanisms is just employing the dynamic nature of the language itself. Python's a very dynamic language. It has a very rich runtime. And within this rich runtime, you can do a lot of things which are just simply not possible in C, C++, or even Java. And this falls under the purview of what people might call metaprogramming. Additionally, oftentimes when people talk about metaprogramming approaches in Python, they mean hooking into the object construction process. This is one area where Python has added quite a few hooks to customize what it means to create an object. Furthermore, there are more hooks in Python beyond just in the object construction process. And so oftentimes, people talk about metaprogramming in terms of all the other safety valves or escape patches that have been added to the language. Additionally, they mean some of the techniques that are available in Python for constructing things directly, or even some of the techniques that are available for constructing things indirectly. Now, one of the things that I want to say, I won't talk too much about why you do or do not want to use metaprogramming Python, on why you do want to have code generating code or not, for the most part, the answer is either one, you're bored at work, or two, you want to really punish your co-workers. However, one reason that does come up every so often is people say, don't repeat yourself. And so you employ some sophisticated technique in order to avoid replicated code. I don't like this phrase because I think it often leads to very brittle or unnecessary programming techniques. And so whenever somebody says don't repeat yourself, I always ask myself, why not repeat yourself? If don't repeat yourself is called DRY, then this would be RY, except we need to kind of contract that into one word so that we don't lose the acronym. Now, one of the reasons why you would or would not repeat yourself really comes down to, I think, something very similar to a concept of normalization that you see in databases. When you talk about normalization in databases, you're typically talking about reducing an encoding size. In terms of code, this means less typing of the same boilerplate over and over and over, taking something that would take you 500 lines of code and a data writeout and doing it in less code and making it so that each incremental addition to that structure requires less incremental code. Additionally, and very importantly, when people talk about normalization, they talk about reducing update anomalies. In other words, you have some structure that's repeated a couple of times. It's repeated within some, there's some uniformity among those repetitions. And there's a fear that if you change one of them, the other ones won't change in line. And so an inadequate code review may fail to capture some update to some part of the code base, which requires an update to a corresponding part of the code base. And so you create your own structures, your own metaprogramming structures, in order to eliminate these update anomalies. I think that's the general lens through which we should look at some of these approaches and try to analyze some of the approaches. One of the things, however, that's a little bit false about that is oftentimes you don't know, you can judge whether something reduces an update anomaly or reduces the possibility of an update anomaly quite easily, but oftentimes you can't predict ahead of time what the update anomaly is going to be unless you know something about the code base. For example, if you're recoding a system that you've coded before, you can kind of guess where you need advanced structures, where you need to create your own superstructures, and where you need to create your own frameworks, where you need to create your own libraries, and where you don't. Because you've already gone through that process and you generally know how that code base will change over time. And much of your work, especially if you're building a library that serves some new purpose, some library that hasn't existed before or serves some new business purpose that has not previously existed, you may not be able to predict where those update anomalies are going to be because you're not really sure how that code base will evolve over time. And so there is a little bit of a lie there, but I think in terms of evaluating these approaches, you can look at them and say, yes, this does or does not eliminate the possibility of this anomaly. So let's take a look at some of the approaches. I told you one of the first approaches was just making use of the dynamic nature of the language itself. So we know in Python you can define a function. Here's a very simple function. It's called f. It takes two numbers or two strings or two sequences, and it adds them together. We know that we can do this in a loop if we want. This code itself is completely meaningless. It just defines f over and over and over. But it does illustrate something that's interesting to us, which is that in Python, every statement or every line of code is an executable statement. As you may know from some previous talks I've given, there are only three Python statements that do not directly generate bytecode, that do not generate executable statements, pass, global, and non-local, where the latter two only amend what the bytecode would be generated, but themselves do not have any executable meaning. So you can see that Python's very dynamic. You can do crazy things like define functions within functions or define a function within a loop. Now, it may be the case that you have two functions, f and g, and they're very slightly different. One adds two numbers, one multiplies two numbers or two sequences or a sequence and a number. And you can do some funny things like you can reassign them. So f is g and g is f. And if you run this and you look at the help text for it, the help text will actually show you that the help for g is actually still reflecting the name of f. Do I have to do a print here? Well, don't worry about that too much. But you can see that there's some details there. But other than that, it's not particularly interesting. Now, it may be the case that if you have these two functions and you want to figure out a way to unify them, you know that in Python it's a dynamic language. You can pass functions around as values. So you create a function called h, and that h could take both the values and the underlying functions, and it could apply them. And this is all fairly boilerplate, fairly standard. And so the way you'd apply this to make a metaprogramming approach is you have four functions, and the functions are all very similar looking. They all take two arguments. They all start with my, and they just perform some binary operation on those, whether it's addition, subtraction, multiplication, or division. And you want to find some way to either, one, reduce the size of this encoding, reduce the amount of typing, because if you want to add some other binary operations, you're going to have to type that same boilerplate over and over, and reduce the possibility for update anomalies. for example if you want to add some doc string here or if you want to perform some explicit coercion here you have to then apply to every single function and there's a risk that you may fail to do this uniformly so you may lead to some anomaly in your code base where some part of the code base isn't updated but another part has not been updated and so you may you may reach for some of the dynamic nature of python in order to solve this problem one of the simplest things you might do is you might say i can just write a higher a higher order function called bin op that takes two values and the function that's going to be applied and i can do a little wrapping with this lambda, and I can construct my add, my mall, my sub, my div. They all dovetail in this function here. And so here, if we want to do that coercion, or we want to add some behavior, we can do this, and we can be guaranteed that this update anomaly has been eliminated. Now, this is probably just about the local optimum for what you can do with the dynamic nature of Python. You can go a little bit deeper than this, but it ends up getting really ugly and really hard to understand. One of the things that we should consider when we're using better programming approaches is, for the most part, most of the code tools that we use in order to understand a code base are static tools. For example, when somebody gives you a problem to solve in a code base, please go implement this feature. Extend this structure to add in my exponentiate, my exp, or my pow, or something like that. The first thing you're going to do is you're going to grep in that code base or use a grep-like tool. And that grep-like tool is going to operate on that code base as though these are static lines of text on disk. It's going to operate on the lexical information that you have. It won't operate on the runtime information. So if you try to apply more dynamic techniques here, it's going to make it much harder for people to use the actual tools that we use today in order to understand code bases. Not just the static analysis tools, which in the land of Python, some of these tools can be static analysis, but can do some dynamic analysis behind the scenes. But actually, just the common tools like grep will fail to work as you make this a little bit deeper like this. Here you can see it's much harder to read. You're doing something a little bit weird. You're assigning something into locals directly. And the benefit here is quite minimal. I would say even in the previous structure, you can see the asymmetry. You can see the discontinuity in these structures where here it's a little bit harder to read without maybe a little bit more careful formatting. Now, one of the things that you might know is that another approach that I could do is I could have this create a function, and let's fix the bug here, create a function dynamically. We can create functions within functions. And so another way I could do this in order to reduce this update anomaly is I could just have this function created with another function. This may be a very common structure that you've seen before. In fact, this is generally the approach that we take if we were to use decorator. For example, a use case for decorator might be we have some function or it takes some amount of time to run. We want to track over time how long this function takes. This is not a very interesting function. It just sleeps for a random amount of time, but it represents some function that has some unknown or unpredictable amount of computation that performs. And so if we were to update this directly by adding a before and after to capture the time before and after and then print that out, this is going to be a real pain because if we need to remove this in a production mode as opposed to a debugging mode, we have to go in. We have to comment out the lines of code. And if you wanted to apply this to another set of functions, the same idea of timing this, it's going to be a real pain. And so one of the ways that we avoid that update anomaly and one of the ways that we make this a little bit easier for us is we use decorators, which are another one of the tools within our metaprogramming approaches in Python. We write some decorator called timed. It returns a brand new function. That brand new function wraps the behavior of the underlying function. And then we can apply it to our original function. The code's a lot easier. We've eliminated the possibility of these update anomalies, we've eliminated a lot of the boilerplate of copying this before and after lines onto all of our code base. And if, for example, we had multiple functions that we wanted to perform the same operation on, we can wrap these in this fashion. Now, one of the downsides of the decorator as a metaprogramming approach is that ultimately it can only amend either what's going into the function or what's coming out of the function. Functions tend to be generally opaque in Python, and so you're very limited in what you can do in terms of modifying what's happening inside a function, unlike, for example, in a very dynamic language like Lisp, where you could actually see the AST and do mutations on the AST. Now, one of the things that you might find out that you want to do is you might want to amend that decorator to print out some customized message. In the previous case, you're applying this decorator uniformly. Here, you want to apply this decorator with a little bit of customization, like a customized message in both cases. And this is the case where you'd have a second-order decorator, a function that returns a decorator where the decorator is a function that takes a function and returns a wrapped version of it. And this tends to be just about the deepest that you ever go in terms of these higher order decorators. I don't think I've ever seen a decorator that returned a decorator that returned a decorator that returned a function. It just doesn't tend to be very useful in practice. Now, one of the things you might wonder is, as we begin to try and customize the message that's being printed, here if we run this code, and we actually run this code by calling f, you'll see this will print out some information about... Oh, actually, we need to... Don't worry about it. If we actually try and run this, it'll print out running F, and it'll print out the amount of time that it took. Now, if we want to customize this slightly, then we start to need to create our own structures around this. For example, here, you can see I've amended this higher-order decorator so that instead of taking an individual message that just prints to the screen, it takes a function that constructs the message that'll be printed to the screen, and that function takes all the information that should reasonably be available in that scope, namely the args that were passed into the function and the return value that the function generated. And so here, I can have one of these functions just say, I'm running f, and the other one can say, I'm running g. Here's the arguments I was passed, and here's my return value. Now, as you see, as we get deeper in needing to customize the behavior of this decorator, we need to create our own idioms, our own structures, our own libraries or frameworks. But as a technique, it more or less works. One of the downsides of this technique is it tends to be fairly brittle, in that as we take these decorators and we customize what the inputs are in order to generate the underlying decorator, we're creating a fairly narrow API in terms of what we're able to do in customizing this resulting function. Now, I told you one of the other approaches is that we can hook into object construction. This tends to be what people talk about when they talk about metaprogramming in Python. They talk about all the different ways that you can change what it means to create an object in Python. Here we have a basic class. It's called A. It has no methods. If we want to create a base class for this or either a derived class for this, we can create B, which derives from A. If we have on A and B some common functionality, and we want to make sure that these two are updated in sync, for example, they're both timed methods, then we can apply the same decorator that we wrote before to both of these. But you can see that there's always a possibility of a problem. For example, we may intend for every method in these classes to be timed and here we forgot to apply the decorator or we may intend for every subsequent derived class to also inherit this behavior of all of its methods being decorated in this fashion in this case being timed and so here you can see we have this class d and the person who actually created this class d who inherited from this base class may not know that you want to enforce this behavior and that enforcement will be lost this leads us to probably one of the largest ways, or one of the largest hooks into object construction that falls under this guise of metaprogramming Python, namely the metaclass. Metaclasses are often criticized. People say that they're not very useful, or they're just an absurd extent of trying to prove that you're smarter than your co-workers. But I think that in that previous case, one of the things that you can see from metaclasses is they tend to actually be a very narrow but very useful tool. One of the things that a metaclass can do, maybe not uniquely, but relatively effectively among all the different metaprogramming approaches that you have, is allow you to enforce constraints from a base class to a derived class. So in this previous example, where all of these methods need to be timed, even if you were deriving from a base class and you didn't know that that constraint was held, because the metaclass itself is inherited as part of the inheritance tree, this behavior will be inherited along with the metaclass. And so here, as a very simple metaclass, all it does is it looks for the method F, and if it finds that method, it adds this timing here. And so you can see here in the case of class A, B, and C, all of their methods Fs are time, even though we may not explicitly require that. The only thing that we explicitly did was identify that this class is a metaclass. But here in the case of C, we did not explicitly identify that this had some metaclass, but it still was able to inherit that behavior. In terms of how metaclasses are used for metaprogramming in Python, I think one of the key examples you might see is the enum class in Python. The enum class in Python, you inherit from it. It has some metaclass behavior. It goes a little bit further than this, which we'll see in a moment, and it then enforces some behavior on the class that you're creating. Now, the reason why that tends to work fairly well in terms of the enum class is that the enum class tends to be, or if you're creating an enum in Python, you tend to be at the leaf end of, say, the tree of code, where at the very root you might have all the library code and the leaf end you might have the user code. You tend to be at the very leaf end of the inheritance tree. And so it's unlikely that you're going to create an inheritance hierarchy among your own enums. One of the downsides of using the metaclass approach is you tend to infect that inheritance hierarchy. And so if somebody has some particular view of what their inheritance hierarchy should be in order to solve some particular business problem, and then you require the use of a metaclass, you're going to run into problems where you might not be able to, your metaclass may not be the most derived metaclass that applies to the construction of this object. If you want to go a little bit broader at mechanisms that you really shouldn't use. Python has mechanisms like build class. Build class is a function that you can override that hooks into all object construction throughout the program itself. And so here's an example of build class that's actually wrapping every single method of every single class that's constructed. This is very, very broad. This is applying to every single class that's constructing your program, including classes that are constructed as part of objects that are constructed as part of third-party libraries that you're importing. So don't ever to do this. Now, it tends to this is not actually altogether that useful as a consequence of how class construction works in Python. Here, the build class only has the function. It turns out when you create a class in Python, the body of that class is wrapped up to a function that's then evaluated. The local state, the local scope of that function is then used in order to build the class. Here, you only have that outside function. So build class can't really do anything that much more specific than, say, a class decorator. But it's one of the approaches that you have available to you. It is also an example of where maybe you might choose to use additional keyword arguments that you could pass into, you know, pass after the parentheses when you're defining a class. I don't think anybody has ever seen something like this. It tends to be very, very, very rare. The only keyword argument that I've ever seen anybody pass through a class tends to be metaclass. Beyond that, I haven't seen anybody try to create frameworks or structures that require this. Now, maybe a little bit safer. Oh, before we get into maybe a safer or better technique. One thing that I was not able to get working, so if anyone here is interested in this topic and wants to play around with something, what happens in build classes, you get that function that creates the class body. In order to actually perform mutations there without either using a profiler or when you want to perform mutations or enforce constraints while the class body is being constructed, if for some strange reason your class body has executable code in it, what you'd need to do is you'd need to get a handle on the local scope. I don't think there's any standardized way within Python to get a handle on the local variables within a running function without using a profiler. One of the theories I had that I couldn't get to work without a core dump was you could turn the class body into a generator by swapping out the code object and then dropping a yield on the end. And then you can have a generator that has a single yield at the end, and you can actually get a handle on the locals. I haven't quite gotten this to work yet. It'll take a little bit, a couple more minutes to get to work. Now, in terms of safer mechanisms, as I said, enum uses some other mechanisms that are available as part of class construction, like prepare for creating the namespace in which things work. You can also patch out call in init. These are all ways that you can patch out what it means to construct an object in Python. However, probably what you should use when it comes down to customizing object construction is init subclass, which is added in Python 3.6. This is a relatively safe and relatively easy way to patch out a class. Here, any subclass that gets created from base runs through this init subclass, and you can see the code here is just saying, if that subclass had this method, go and wrap it. Now, I told you that another approach that you might take in Python is you might use some of the built-in escape valves or escape patches. There are numerous escape valves, safety valves in Python. Many of these are stored in the sys module. One example is the profiler itself. And so here you can set a profile that gets called whenever you enter a function or leave a function. and then you can perform mutations on the frame that you have there. So this profiler could look at the local variables and perform some mutation such as constructing new objects or destroying objects or wrapping objects that you saw. Similarly there's a trace function, a line tracer so this can be called on every individual line of Python that gets executed. Similarly, added in Python 3.9 I believe there is the audit hooks as part of PEP 551. These really wouldn't be used as a metaprogramming approach but I was involved in the creation of this PEP so I wanted to show this off. This gives you the ability to add audit points within a code base. You can't really use those audit points for anything metaprogram related, but it is kind of cool. There's many, many more of these, and we don't have time to go into them. Another example that comes up a lot is the metapath and the path hooks. If you actually run this code, you'll see that even in a very baseline, a stock version of Python, you have path hooks involved. You might think, when I try to import some code, and if that code is in a zip file, it actually works. Why is that? Well, the import mechanism of Python can be hooked into in order to tell Python to look for code in different places, like to look for code within a zip file. This is actually used in production in some places to load code over the network from object databases, and it's yet another way that you can customize how Python actually works. What may be more interesting than that is trying to figure out how you can construct objects in Python either directly or indirectly. You may have seen the type function in Python, the type function with just one argument tells you the type of an object. Here, x has just the type of a. you can use this type function to create a new type and so this previous code here could be rewritten just like this the type function takes three arguments the name of the class the basis of that class and the body of that class and so this is a way that i could just construct the type similarly if i wanted to extend this a little bit further i could do something like this where i can have a function that creates classes that add some common methods to these classes and so here you can see a has two methods g and h it'll automatically get a method f and a repper. B has just one method, G. And with C, I could even have C take every method named from A to Z. And so you can think that, for example, I had a project a while back in which I was trying to build a network simulator. In this network simulator, I had the definition of the messages that were being passed on that network, and that definition was in the form of a JSON file. What I was able to do is I was able to load that JSON file, iterate through that JSON file, and use create class to create a unique Python class for every single message that represented a unique message in that simulator, and then as part of that simulator, I was able to send messages back and forth and do some fairly sophisticated analysis of, in this distributed circumstance, if this node has this particular error case, how many messages have to get sent back and forth in order to either rectify that node falling off the network or to identify that node falling off the network. That would have involved a lot of boilerplate because every message was just a payload, but I was able to create those payload messages quite easily using something like this. Now, you could do something similar with the function. Please don't. This is very, very unpleasant. But nothing stops you from actually creating a function from its bytecode itself. And so if you wanted to create the superstructure for it, instead of creating just a couple of classes by passing in the methods in a dictionary, you create a couple of functions by passing in the bytecode for these. And you could do a little bit of guesswork on that in order to fill in a couple of these extra arguments. Here, I'm hard-coding the number of variable names, the number of local variables, the number of arguments that are passed. But there's nothing that stops you from directly generating a function at runtime in Python. So if somebody wanted to give you a JSON file containing the bytecode of a function to create, that's when you know you need to look for a new job. Because don't do this. This is really unpleasant. But Python gives you the ability to construct these things in this very direct fashion. Now, the last approach that I want to show you tends to be my favorite approach and probably the approach that I use most of the time when I have to automatically generate a code. And this is what I call constructing something indirectly. It's also known as the take a string, template it, and exec it. And this approach has been criticized from the very beginning. If you remember, the original name tuple, the original collections that name tuple did this. It would create a string containing all the code for the name tuple, and it would fill in the fields dynamically. And for some reason, people hate this. I don't understand why. Because if you actually look at the backtrace for how something in Python runs, it will essentially enter the parser, enter the code evaluator, irrespective of using exec, or if you're importing something from a module. There really is no meaningful difference between execing code in this fashion, where the code is in some string inside your program as a string variable, versus execing code if the string is inside some module in a py file on disk. There's no real meaningful difference here. And as a consequence, I think a lot of the arguments people have against exec kind of fall apart there. One of the reasons why I like this approach is twofold. One, you tend to be able to create very narrow structures that avoid these update anomalies. What you're going to do with this is you're going to see here in this create function, I only pass it the name and the body, and I guess things, like what the arguments are. If, for example, there was a deviation from this routine structure where I wasn't able to guess those, I would just patch this create function just a little bit to handle that, and I'd end up creating probably the most minimal metaprogramming structure that I could in order to create exactly the kind of function to reduce the boilerplate that I want. I like that because I find that the most robust structures that you will create from a metaprogramming perspective tend to be the most minimal. The ones that you have where they try to capture the least number of exceptions as possible. The other reasons that I like this are, one, you can spit out the code. Collections.name tuple used to have a verbose flag that would spit out the code that was generated. One, so you didn't have to go through that generation at runtime. You could do it and just spit it out to a .py file on disk during build time. And two, it makes things really easy to grep for. One of the other requirements is that in some cases you may not be able to automatically generate code. You may be in a circumstance like you're running in a restricted execution environment on Windows where you have something like strict HVCI implemented or you might be running code that runs on a mobile device or on a game console where you may not be able to automatically generate a code as a consequence of some kind of policy that's being enforced on you. And so this approach is one approach where you can, at the minimum, always step back and just generate that code as a PY file on disk while still in some debugging purpose or for the purposes of iterating very quickly run this code or generate this dynamically. So those are the approaches that I wanted to show you. This was Play Stupid Games, Win Stupid Prizes. I hope that gave you a little bit more of an impression of what we're talking about when we talk about metaprogramming Python. I'm James Powell. Thank you very much.

Speaker 2 [25:32]

Thank you very much. I have to admit, I became a bit nervous when I saw this because I thought, if you shoot over, I'm not able to stop you because I never managed to quit Vim. So you could have gone on and on.

Speaker 1 [25:49]

It was just as I predicted, blah, blah, blah, clap, clap, clap.

Speaker 2 [25:53]

So we have time for some questions, but you have to talk as you pose your question in the same pace as he's talking So who wants to have the first question? Okay, you also can post a question in a in your speed Nice shirt

Speaker 1 [26:18]

You as well. One question. What do you think of the metaprogramming capabilities of something like Julia? I don't know that I can. So are you pointing at, for example, the use of macros or hygienic macros, which Python does not have, and Python has been very resistant to adding? As you can see, all of these structures in Python, especially the function, like the AST, you never really have access to an AST in Python. you never have the ability to mutate that. The core developers of Python for the long time have said, is that really something that's valuable? One of the downsides of some of the approaches I've showed you is that individually, they're not particularly useful. You have to start building more and more on top of them in order to make them very useful. And so in the very last example with that code templating, if you looked at the code templating that happened in collections.name tuple, it was quite extensive in terms of what it would take, the kind of validation it would do, and how it would build the underlying class. But it was sufficiently narrow because it was really only serving one purpose and you were given the opportunity if you needed to do a name tuple that had some specialized behavior like default arguments you're you're expected not to build that into the metaprogramming part but just use basic inheritance to inherit from that name tuple you created and replace out new i think that's the argument against more sophisticated metaprogramming approaches like hygienic macros that they end up creating their own universe where you're given a brand new code base you want to figure out how to use that code base and you don't not only need to know the underlying language it's written in, but you also need to know all of the structures and mechanisms that were used in order to generate the actual code that's being executed. Whereas in Python, if you take a much lighter hand, don't provide that functionality. You can hopefully prevent people from doing really crazy things where they're using this, where you wouldn't be able to understand the code base unless you could understand the auto-generation mechanism that generates the code base. Other than that, I would say, if you really like that, use Julia. Okay. Or Rust, or Lisp.

Speaker 2 [28:13]

One more question.

Speaker 1 [28:16]

If you're scared of this, please don't use Lisp.

Speaker 2 [28:20]

Okay, then let's thank James again.