Reaching the next level of abstraction: meta classes and what they enable

Speaker 1 [02:17]

We wait a bit longer or we're good to start? Are we good to start? Are we good to go or should? Okay, good. Thanks a lot for your patience and I'm really glad to see so many of you here. It's super cool, especially for a seemingly niche topic. So people are coming in, feel free to join in. Welcome. So today we're going to talk about meta classes and reaching the next level of abstraction. to give you an introduction there i will talk first a bit about some basics that i think all of us need what is programming what is meta programming what does it mean if everything's an object and i want to segue into meta classes by giving you a small introduction to high order functions and then let's talk about the meta classes but also why not to use them right i I think with all these advanced concepts, it's important to be mindful where not to apply. So, who am I? I'm Valentin. I'm a software consultant at TNG Technology Consulting. We're from Munich, and I did a doctorate at TUM in computer science, software engineering, actually. And I had the blessing or the chance to create a practicum on advanced Python programming. and so that was kind of my job and also just fun to be able to do that and work with curious students on these advanced topics and so that kind of sparked my joy in Python. And my company is TNG. We are a values-based consulting partnership. We are 25 years old now, this year, operating in nine countries and help customers analyze and solve strategic IT problems, as the claim, yeah. So, let's get into it. What's programming? Maybe the first most basic question to ask. And by the way, I'm not allowed to walk, so normally I like to walk around, but I was told to be constrained exactly to this box, so if you're wondering why I'm so stiff, it's for the remote audience, so guys, that's for you. So what's programming? Well, programming fundamentally, I would say, is writing code that controls data, right? So we have some data like primitive values, some container types, or some more abstract object types, and all of these sort of have data in them and are containers for data. And when we write code, we manipulate those data. Now let's contrast it to metaprogramming. Well metaprogramming is also code, but this code doesn't control data, it controls other code. So what does it look like? Let's look at a simple example. If we have this class, and we just add this str, dunder str method to it, now when we print it, what's going to be the outcome? And I'm going to say it for the remote audience. You can shout. What's going to happen? Exactly. And now everyone. I'm a foo fighter. Yeah, exactly. So we change the behavior of this program. Now from here on, I'm going to jump to a notebook. I have everything in slides as well that you can follow along, but I think it's more fun if we're interactive, so let's do that. We've learned about programming metaprogramming. By the way, I hope the font size is large enough. Can someone give me a thumbs up, especially in the back? Yeah, okay, good. And now let's talk about everything being an object. This is maybe a claim you've heard before. What does it mean? Let's look at this example here. We have a class, we have a function, we have an instance of a class. When we print all of those, you will see the same style of output. All of these behave like objects in Python. What that means is we can also treat them like objects. You can see I can assign a new field to each of these, and you can guess in your head what happens next. This time I'm not going to let you show it, it's too easy. We can operate on them just normally. All of these are no different in that regard. That gives a lot of power to handle and manipulate everything at runtime. One way to use this power are higher-order functions. You may know them from functional programming, but let's look at a really simple example. A higher-order function is a function that creates or manipulates other functions. For example, we have here a function producer, make function, and it produces the function that prints success. So what happens when I produce this function, and I execute this? Success! Exactly. Yeah, a very simple example. But what you can see here is, even though this looks like some kind of a special syntax, in the end we just create an object that we can pass around, right? And assign. Good. Now, let's get into the meat of it. Metaclass basics. So, when we try to understand metaclasses, what we actually try to understand is what happens between going the class body, or writing the class body, and getting the actual class object. So, how do we get from the texts that we write to the actual object at runtime? Now, what happens here is, first, that Python runs this class body and produces a temporary namespace. This is not yet the final product. So what we get is we get the name, obvious, basis, none in this example, and some attributes. So here we have one function, and you can see it's already the finished object. So that's how I signal with this box. And now here's the crucial part. You can see that I have a meta class already here. So this is all passed to the meta class, and here is where we jump in, right? Now, we can handle creation of the class object in this meta class. That means we can manipulate both the inputs to type.new, which creates the class, but also the return class itself, and I'll show you a code example on the next slide. So for example, we can add members to this object. And then in the end, we use type, I'll get back to that, to create the class. Is so far so good? Are there any questions at this moment? Is Metaclass something specific to Python, or is it just your chosen name? The question was, is Metaclass something specific to Python or my chosen name? So this is how Python refers to this kind of construct. And I think they were discussing if they want to name it something different. But yeah, Metaclass stuck around. So that's a Python name, so to speak. I don't know if other languages... So the languages that I know don't have this concept. But maybe there's languages that I don't know. I'm not referring to the concept, but I'm referring to the dimension of the MyClass. Yeah, no, no, that's... Good. If there's no further questions, let's jump back. So how does this look in code? Like this. It's very simple. So if we define a meta class, we need two important things to make it a meta class. The first thing is we need to inherit from type. And the second thing is we need a new, dunder new. Now here you can see what we just saw in the figure, right? We get past the name, the basis, and the attributes. And what we return is the object. And everything that happens in between is at our control. So in this example, I'm just printing, right? I'm just printing something. And now here I have a class that has this meta class. So what do you think happens when I execute this block? Does anything happen at all? Any guesses? Nothing. So I hear nothing. Some other guesses? So this already prints. That's the important part, right? What you can see here is we don't instantiate, we don't access or anything. Just the mere fact that we define the class triggers our meta class as soon as we start up this file and interpret this code. So this is the point where we hook in, right? We don't hook in at instantiation, we hook in at definition time. And you can see there's some extra information in here that you can ignore for now, some of that has historic reasons, for example like call name, first line number, but importantly if we added members here we would see those members as well, right? For example, yeah, okay. Now let's look at some examples and use cases. What can we do with this, and why is it powerful? So the first thing that I think is cool to be able to do is creating new or adding new methods on class creation. So for example, let's imagine we want to save ourselves the work to always add a store function that gives a nice output. So we could have our meta class, always add, in this case, a lambda that will be executed. So now, of course, if I use this meta class, and I have an instance, and I print it, I'm going to save you the guessing, we can see that our string function is executed. In this class, there's no string function, and if I remove the meta class, of course, there will be no output. I think that's clear to everyone. But this is a very basic example. I think it starts to become powerful when we talk about reflection and introspection. And to help you with introspection, let's talk a little bit about some of the batteries that are included here. In this case, the Inspect module. So the Inspect module is really powerful. I didn't know about it for a long time, and actually when I gave this talk to some colleagues, one of them was jumping up and said, when was this introduced in Python? I needed this in my last project. I shouldn't say this because we're a consulting company, right? But I told him, actually, Python like 3.6 or 7 or something like this. It's been old, but you don't know everything that's available, right? So I think even if you don't work with metaclasses, this can be useful to you. So inspect helps us introspect our code. For example, we can get the members of an object in nice pairs. And we can limit the types of members we get by these predicates. So, in this case, for example, is routine, which gives us all the functions or methods of an object. And what do we do with those? One other powerful thing we can do is we can get their signature. This is something important if you're introspecting. You want to see what kind of parameter values do we have and what return type do we have. And this, finally, can be used with signature.bind to bind a set of arguments to a given signature. So, if you've ever done this manually, I know I have. Fear not and use this. So let's look at a concrete example and then I'll give you time for a few questions if that was too quick. With a large group, it's always hard to judge the speed. So, let's look at this example. We get our members that are functions. So what do we get? Double it for the next person. So any guesses? happens if I now look at the signature of this function? What do I see? Nobody dares to speak up. Very good. And important, this part, right? Also the annotations. But exactly right. So he said self and value, but I figured for the remote audience you will see the correct answer anyways. By the way, a quick tip. You get back here always tuples of name and object. Just use the dict constructor that will automatically make it key addressable. I think that's very convenient. Are there questions on this module before I continue? Because I will use it now. Good. You guys are smart or disinterested. Either or, let's continue. So, second really powerful thing, I think, and this is maybe something that most people are eager to do when they start learning about metaprogramming is enforcing some constraints, right? And let's now combine some concepts. So, this is going to the speed is picking up. So, feel free to ask questions if they come up. Let's create a decorator first. You guys know this probably, and this decorator type checks my function at runtime. I hope you're already cringing because you're Python developers, but let's do it anyways. People like to do this, I think. So, we're going to wrap a function. We're going to bind the signature, right, to the argument that we are called with. This is really powerful because it allows us to just check, does the annotation here match the type of the object? And if not, we can raise an error. So pretty much we get called as our function, we have certain arguments, we compare them to the signature, and if we have a type annotation, it must match. If we don't have one, we ignore it. So far, so good. Thank you for asking. So what is bind doing is the question, exactly the right question to ask. So if you have a set of arguments that you're called with, for example in this context, you want to match them to a signature. And bind is kind of figuring out, is it a keyword argument, is it a name parameter, is it just a positional, it just matches these arguments to the signature. Where do they belong? Because you know that you can call functions in all kinds of funky ways. You could say, for example, I don't know, 1, 2, or you could say x, a is 2, 1, right? And how do you figure out which argument belongs? Okay, now let's type check a function and look how this one will then work, our decorator, right? So, for example, if we pass a string, nothing happens. What if we pass an integer? You guys are not interactive at all. Yes, exactly. Good. Right? This is the value error. Perfect. And now let's use this in our meta class. This becomes a lot simpler because all the work is done by our decorator. And here you can see that, right? So the only thing we do is we go through all the functions. and then we replace them with decorated functions, right? So why do I use set attribute? Because it's an object, it's not a dictionary, right? And here is just a decorator syntax desugared, yeah? So for those who don't know, if I want to do this, I could instead write it also like this. Okay, so this is syntactic sugar for this, yeah? So this is what I'm doing here more explicitly Cool and then let's write Magic Now final example, and I think this is One that is maybe most attractive and I'll also show you where this is done in the standard lib as well so custom syntactic sugar, I think Before I do this, really no questions? Okay, good. Why did you need the bind method in that example? Why did I need the bind method? In the last meta. Why I didn't do it down here, you mean? Yeah. So I defined a decorator because I think it is cleaner from a code style perspective to define such functionality as a decorator. But I didn't need to. So it's a great question to ask. I could define all of this. in the meta class? Great question, thanks. You see this guy is doing it right. Ah, okay. So the question was, if I use this type checked meta class, I can convert any class into a type checked class. Yes, that is correct. So, for example, I could say any other class, right? right? Give it my meta class, and now whatever I have, and I do all these bad shortcuts, like writing S instead of self, sorry about that. Saving time. And now we instantiate it and then we call X with, I don't know, right? And we get the same error. So that's That's the powerful part, right? We can just apply this to any class that we use. Good. Final example. Custom syntactic sugar. I think what's cool about metaprogramming is that we save ourselves the typing because typing is annoying and my fingers are hurting from too much typing. Very bad. Especially this little one. Really bad. Yeah. I should use agents. That was the comment. Let's talk later about that. You know how much I love software engineering. Okay. So, custom syntactic sugar. I think one thing we could, for example, imagine doing is auto properties. You know, I think a design flaw in Python, now I'm going to get all the flack, is that by default, all the fields are writable, right? So, that's really hard to undo. If you write a library and you have a bunch of fields and they're all writable and you want to undo that, now people may depend on that functionality. And I think it's Ruby, if I'm not mistaken, that does the opposite, right? everything's read-only by default, and then you have to make it writable. And properties allow us to do that. So, what if we want our classes to not have writable fields by default, but properties instead? So, does someone not know properties? Should I quickly show them or are they good? One person? That's enough for me. So, we have a class, right? And property it looks like this. Right? And you need self and blah blah blah. But what it does is the interesting part. When you instantiate the class, you can address it just like a field. But instead it will call your function, right? And so that allows you, for example, to do simple computations like add two fields together or it allows you to have a hidden value that that people cannot overwrite. So for example, what you could do is you could have some hidden field that you return, and then when people try to write to x, it will be blocked. Good, so this is what I want to do, right? Instead of having fields that everyone can write, I want to have properties that are not writable. So I won't go too deeply into this code, but if you want to know more, I'm happy to explain as well. But looking at the time, I'll focus on the actual outcome. And thank you. and continue here. So what we do is, we do two important things. Actually, three. We skip everything that is prefixed with double underscore. For everything else, we create a hidden field that is prefixed with an underscore. That's where we store the value. And then we create a property with the actual name that returns the value. So you can see here, and here where we use the property. Again, it's a decorator, so we just apply that. Now, what happens here if I print e.x? What do you think? Guesses? Nice. And what happens next? Exactly. Exactly what I liked, right? With a very simple code. Now every class that I apply this meta class to will be able to have properties automatically, just like that. So let's jump back into the slides and conclude this. So this is all we talked about. Guys, with great power comes great responsibility. And actually, I had this talk internally, and I said Spider-Man below it. And then I got nerded out immediately by one of my colleagues who said, actually, actually, it's not Spider-Man. And it's actually his uncle who said that. And I was like, I have to correct that. He's right. And I actually looked at the clip and it was his uncle. Oh my God, that's on me. So I'm your uncle now telling you with great power comes great responsibility, guys. We all get shiny new toys. And yeah, let's use them responsibly. Let's look at some simple alternatives. I'll go a bit into the question time because we already had some questions throughout, if that's okay. And try to be quick. So, how can you achieve similar things in simpler ways? Real quick. Data classes, if you don't know them, read up. Really good. They allow you to reduce boilerplate, automatically have init, wrapper, echo, and so on. If you need these basics, right, it's like a record class in other languages. It's really nice. And they are available already in many, many Python versions. Runtime type validation. I hate to break it to you, but don't use Python if you really need this in the standard library. It's a specific use case I think for api's and there's libraries for that don't implement runtime type validation Even though I used it as an example. I think that's not the right approach There is class decorator, so we had one example where we decorated each member of a class That's literally the point of a class decorator right you can make the same Function style as you do for function decorators and just pass it a class and you use the same syntax with an ad So if you want to manipulate a single class, so for example adding attributes or decorating all methods This is the perfect way to go. But importantly it does not inherit, right? If you have a class and you have an inheriting class, it will not receive that decorator as well But you can use init subclass and that does inherit so actually Why do we even need meta class that's gonna be the final question I'm gonna ask you right that init subclass is like class decorator, but all the subclasses also will inherit it. And I will have for the question time some examples if you're interested. So why do we need meta classes? Really quick, I don't think we need them for most use cases. So why am I talking about them? Well, I think there's, or I know there are certain advanced use cases where you still require them, right? But they are very specific, and I want to quickly show you some examples. So if you want to modify the base classes, right? This is something that I didn't get into, but you can figure it out, right? We get the basis passed to us. We can have dynamic inheritance. If you need to add or modify slots, the slots attribute, if you want to prevent set name triggers, so descriptors, if you haven't heard about them, let's talk. I really love those. But they execute dynamically some code on instantiation of the class, right, in the set name function. And if you want to prevent that, you need to be one step earlier. And, of course, you can have custom namespaces for parse class attributes. What does that mean in 10 seconds? If you want to, for example, remember in which order class members were added, you can give a special dictionary that remembers the order, and then you will be able to work based on that. For example, allowing duplicate names in a class. And here's the promised examples. Where are they actually used? Well, for example, have you ever thought about how enums are implemented? Well, there's a meta class. this enum type, and it, for example, adds an iter, right? So you can just iterate over the class, which is not a normal Python feature. And in ABC, of course, we have ABC meta, which, as one example, prevents class insensiation if you have any members that are still annotated with add abstract method. So these are some of the examples where it really makes sense to use a meta class, right? So you can think I'm a library author, I may be contributing to the standard lib or something, there these use cases really emerge. So let's summarize real quick, and then we hopefully have time for two or three questions. What did we learn today? Well, metaprogramming. This is writing code that controls other code. And Python gives us the powers to really do that. Now, why do we have this power? Because everything is an object in Python. This includes classes. We can handle them. We can manipulate them. And with our meta class, we control the full process of class creation. Now, what can we use it for? Well, we can add or remove class members, we can enforce constraints, etc. But we should, I think, use them mainly for fun. Or if you have very advanced use cases. Be mindful of your colleagues and for your future self in one year that needs to read and understand the code. If you try to understand the examples that I showed you, it can be hard. Consider some alternatives, as I have outlined here. And that's it. Thank you for your attention. Hi. If you have any questions, feel free to ask. Always going to need to run a lot, it's a long room.

Speaker 2 [29:17]

I think metaprogramming is not only metaclasses. You can use it just without OO. And the other thing is where I use it before was in C++, and it has a big advantage because it puts stuff from the runtime to the compile time phrase. So you can optimize your code, and you can do type checking even on compile time, not on runtime. I would say this is the biggest advantage, at least for me. This causes me to use metaclasses at all.