Abridged metaprogramming classics - this episode: pytest

Speaker 1 [00:04]

Quick warning up front. When I started thinking about this and turning this into a thing, I wanted to make it a training, which would have been about three hours. Then I proposed the training, and I thought, okay, I can propose also a talk with a shortened version, so I can do that in three quarters of an hour. That got accepted, and I was really happy. And then a few weeks before the conference started, I looked at the time slots, and it was in a half-hour slot. and so I tried to shorten it and I used two groups of guinea pig and tried it on and if I really rush through, talk as fast as I can and rush through the slides, I can make it in 32 minutes but I don't think that's going to be fun for anybody and those seats are really comfortable and you can just fall asleep while I'm talking to you or if you want to go earlier, it's no problem but I'm afraid I think I will need at least 35 to 40 minutes but there's the coffee break after it so I won't hurt anyone. Yesterday somebody fell asleep next to me in the chair here, so feel free. Yeah, so who was in the talk of Raphael just now? Here, cool. Then we might be faster, because I don't have to explain actually how PyTist works, because he did a great job already. Because what I'm trying to do is really to start from scratch. somebody who doesn't know how PyTest works, somebody who knows how to program Python, but never really worked with the internal mechanics at all. You theoretically don't even have to understand decorators, so this is a bit unrealistic anyway, but I'll try. So it's a bit like this image there. So Middle-Earth Epic Jewelry Return Policy is, I don't know how many, 1,200 pages a book compressed. That's what we're going to do with PyTest today. so let's imagine the internet is down we're bored, we have a laptop with Python 3.8 and we want to learn a bit about metaprogramming that's my cat Flopsy I'm her human let's say it like that so the logical conclusion what you do is you re-implement PyTest from scratch about 0.6% of it because PyTest has about 16,000 lines of code and this will have around 100 lines of code If you don't count the tests of PyTest, which is 22,000 lines of code, which obviously is testing itself with PyTest. So there's some really interesting test code in there, if you want to have a look. And we only want to use the inbuilt functionality directly and a maximum of three standard library imports. If you saw the image that came up when I opened my notebook, I'm a fan of Brian Eno, and he has some inspirational things, and one of them is like making arbitrary restrictions to yourself to foster your own creativity. So that's the restriction. And also yesterday, if any of you saw the talk of, I hope I pronounced that right, Luciano, he was talking about it's easier to understand new languages if you focus on their core features rather than saying, oh, this is an object-oriented programming language or this is a functional programming language. You just look at the actual features a language has and then you can have a better understanding what it does and you can better transfer knowledge from one language to the other. So that might be another justification for doing it like that. That's the minimal theoretical knowledge you will need to follow this talk. Modules and functions are first-class citizens. That's the main thing. If you take only one thing away, that's already it. So you can do with functions and modules what you can do with any other object in Python. They are assignable, passable, returnable, which means I can pass modules and functions into another function. I can return a function back or a module back and obviously call it. Or I can inspect them. I can pass them into the dir function, for example, to get the names. Or I can ask for attributes. And these dunder things, as we call them, they are like special attributes and methods, and they give access to the internal language mechanics. which we will need and they're mutable so I can fiddle around with this path which we have to do later on and I can add arbitrary new attributes on the function object for example if that seems useful at that point in time Lukasz you mentioned yesterday that we all don't need really these things maybe in other implementations but in PyTest it's really helpful to be able to do these things yeah and if this is this is all too short and too fast anyway so If you want to start reading up on these things, I would really recommend going to the original documentation of Python and looking into the execution model and the data model and start from there. What is PyTest? I don't have to explain it anymore. Raphael did it already. It's a nice testing framework with a great community, and it's using a bit of metaprogramming, a lot of metaprogramming, and you can just hop in your time machine 52 minutes back and look at Raphael's talk. so meta programming is a programming technique in which computer programs have the ability to treat other programs as their data it means that a program can be designed to read generate analyze or transform other programs and even modify itself while running that sounds scary if we have an abridged version of that of the wikipedia that would say meta programming is using code to do stuff with code. So that's what we're going to do now. And the first feature we're building is the automatic test discovery and execution part of PyTest. So find and run functions with the right name in the right place, which means the functions start with test underscore, they are in modules that start with test underscore, and they can be in arbitrarily nested folders. So if we look at some structure here, that's a simple test suite. This is the basics that we look first, then we implement fixtures and then marking. So there we look first. So this is a test module and this is a test module and that's not a test module. So if we want to write something now that does something similar than PyTest, we will need to make that possible. And if we look into a concrete test module, these are one passing test and two failing tests and one non-test that shouldn't be discovered and those three should be discovered. And if we run them later on, this one should pass and these two should fail. So passing, the idea of a passing test is really simply, I run it and nothing bad happens. If no exception is raised, then I regard this as a passing test. And if anything bad happens, I let the user know. So if we collect them only now, now we have five tests, three you saw just now in that module and two other ones. So first thing is we have to collect the paths. So first standard library import already used because I always always use the path library if I do anything with paths. So we glob in the folder that we got in the path and grab all the modules that start with test underscore and print them out. No metaprogramming involved yet but we have to start somewhere so the next step is we load a test module programmatically if we do that hard coded we use the dot notation and here maybe naively would write a dotted path to that test demo that won't work because it's just a bunch of Python files in the folder it's not packages or anything so I can't do this so I need to do something else which means working with this path which is the module search path in Python. It's pre-populated, usually with the folder you're working in at the moment, and then all the other things that you have installed. It's usually quite a lot of things. These are just the first two here, and I could naively say, okay, then I just add the folder where my test module in is, and try to import it, and that actually works. I don't say it's the right way to do it. That's also important to know when you listen to any of what I'm telling you it just works so let's use this or first let's have a look at quick that would be the right tell corrector way to do this use import lip which is a pure Python implementation of import which already hints to the fact that there is a C implementation of import yeah but I mean we don't need this so let's just try this directly if we rustle around a bit in the built-ins we find a function that is called dunder import that looks really promising I would say so and this function is meant for use by the Python interpreter and not for general user doesn't matter so let's try this and now we have the module again and I you remember I just imported it the normal way just now before adding it to the path and if we look at it those are actually exactly the same module. There's just one module that is important. These are just now two different names for the module. So modules are singleton objects. They are usually only instantiated once during the lifetime of a process, unless you do something, which you can always do, but that is how it normally works. So import foo is syntax sugar for this foo equals dunder import foo let's make use of this we extend our function our picopytest function and add these few lines that then just import the modules append the path make a complete mess but the modules will be there okay now we have two modules that seems to work yeah while working on this talk I had to invent a new mechanism to measure the meter programming s of what I'm doing so because we are all scientists here and you know we're not here for fun so we have to measure this somehow so I invented this machine it's called the I'm a German so it's called dust meter programometer and I will try to say it correctly and I might succeed in the end of the talk so sometimes we will get reads of it. It works quite simply actually really. You fill the code in here, it gets heated up gently when it walks down here, then it gets purified in the popcorn here, walks over to the Zahackstückler that cuts the code in little pieces, then sends it to this globe which contains the collected programming knowledge of all of us in this room, and then creates a readout here that gives us a general description of what we just did and a star rating how metaprogrammy this is. So the first readout would be altering the program itself by loading arbitrary executable code and adding them to a data structure for later use. That already got us a star. I have to breathe in between, I just realized, so let me do that. So now if we pull those test modules in a function, we just tidy up the main function a little bit so that it all still fits on the slide. So now we just have a one-liner here that grabs the modules, nothing else changed. So the next step will be collecting the test from the module. For that you have to know that a module has a namespace. That namespace is in Done2Dict. When you import a module, Basically, what happens is all the code in the module gets executed and is added to the namespace. And code is executed means, for example, the def statements, the class statements. This is all executable code that generates objects and signs them to names. And then I have this dictionary where I can access it. So this is called Dunderdict again. So that's the old school way. I can access it like this, or I can access it like that by accessing the dictionary. or if the dot key is broken on my keyboard, I can also use this. I could use the inbuilt getUtter function, so where I pass in the object, then the attribute I want to access, so this would give back the dictionary, and then I can access the dictionary directly. All the same results, all the same way of, different ways of saying the same thing. So, access to attributes via dot notation. This dot attribute, I think it's called dot notation, is syntax sugar. Let's put this into a function. So, this function accepts a module, and we already have the third standard library input, so we're slowly getting into trouble, and I'm not even halfway through anyway let's do this first we iterate over this dictionary with name and object check if the object is a function type and if the name starts with test underscore and if yes we add it to our list of test functions so then we end up with a list of test functions and we want to keep this import for later so let's do this another way there must be more of these Dunder things that we can use, so let's just use them. We could just iterate only over the values of this dictionary and then look into every object contains the class that it was created from in Dunder class. Every class has a name so we could just look for the name of the class instead of doing this is instance check. And here also if the object is a function then it has a Dunder name attribute and we can check if it starts with the right name. So exactly the same result only one import less and if we put all tests in if we now expand our pico pi test function this is the new code it iterates over the modules I found in the past and then extend my test list and then I end up with list of tests I'm still in the collection phase so that's getting a bit boring now let's execute them those tests what do we need to do for that we need a function that accepts a function tries to call it see sees if an exception is raised if yes give some kind of feedback that it wasn't good and otherwise be happy so here you get a function we assume the happy path so the result is the dot as you see before if a test passed it's a dot we try to call the function and if If everything is fine, we return passed as a string, just as a signal that everything was fine. And otherwise, we will assign f to the result and return the exception. And finally, we will always print out immediate feedback. So if we run this now with some test that we have in our test demo module, we get an f and we get the exception back that is then just printed out here. So, call all the tests and report on the results. That's now the new part of our PicoPyTest function. Here we create a dictionary where we execute the tests, assign them to the test name, and print that out in at least not completely horrific way. This is what it would look like with PyTest. Much nicer, I admit. Not done yet. So, but this is how it looks like with PicoPytas now. Almost the same. Oh, we got new readings. So, run programmatically collected functions from programmatically imported modules. Still one star. Let's build something else. Raphael already explained to you the fixtures. I will still explain them quickly. So we have a function in our test module that we decorate with a fixture decorator. We call it the answer. It just returns some number. It's any kind of test setup, any kind of test data you usually need. So the fixture returns this. And in our test, we ask via the name. So this is the connection via the name for the result of fixture and then we can do anything with it, assert on it. So we have one passing test here where we assert the right number and one failing test where we assert the wrong number. So now we have to figure out how can we get this in here and tell pytest that it should execute that for us. First thing we will have to do is collect the functions from the test modules. We just create a a mapping on some module level which then lives as long as the whole process where we can collect fixtures in and where we have a mapping then from the name to the fixture object which is a function and then we have a fixture function that accepts a function adds this to this module scope map and prints out some kind of success message that it registered that fixture one line of code Yeah, I think the real implementation is a little bit more involved. So if we try this out directly, we pass any kind of function into our fixture function and assign the result to a name. Then we see, okay, the register mechanism was triggered, so that seems to work. What is that here? What's in here? Who's for a function? Who's for nothing? What? what what then if not nothing okay good nothing so because yeah it just doesn't give is it accepts the function and it doesn't give anything back we don't have to give anything back we can just be greedy and put it into our map so now the fixture is in the name to fixture map and now we need this decorator syntax there's actually not that much to it I mean there is a lot to it if you want to do it right and if you want to do interesting things but the actual decorator mechanism does nothing else than pass a function to a function and assign the result to the name of the original function so it is syntax sugar for what we just did so that's exactly the same what we just did and here again I don't give anything back in my fixture so I don't get anything back here but but the fixture was registered, and now we have two functions in the map. Oh, dear. Okay, so how do we test that? Let's just assign our fixture function to the PyTest fixture and try that out so that we have the same API. So now the PyTest fixture function is not the original anymore, but ours. Oh, and that gave a reading. So replacing library code in a running program, also called monkey patching, could also run as metaprogramming and earned us two stars so if we import any kind of module now now we import the test fixtures module and look at what happens the fixture got registered it gives still the the module back and the fixture ended up here in our map so the next step is figure out if a test requested fixture for that let's look at this function it defines parameter 1 and parameter 2 and some any other local name and if we look into the deer of that test function there's one interesting attribute here which is the dunder code attribute if we look at this there's a lot in there but if we look at this one our count and bar names that looks promising so let's try that out so So here we have all the names, and these are not just the parameters. This is also the local names that are defined. So we need the R count to be able to slice the parameters out. So we can then enhance our function to run fixtures. So what I just showed you here, that is what it does. It slices those two parameters out of the var names using the R count. then we have a dictionary comprehension that iterates over those names or parameter names and checks if it's in there it iterates over the dictionary and then checks in the names and then if it fits if there's a request then we can add it to a dictionary and execute the fixture function create a quarks argument and pass that into the function with this star star syntax which means really just Take a mapping and call the function as if the function would be called with that mapping passed as keyword arguments. So and that seems to work. So now we have fixtures implemented. And what we did was inspecting attributes of callable objects from a mapping, selectively calling them to inject the results into another programmatically called function. Still two stars. We have to do something more complicated. I want three stars at least. So let's implement marking also. So marking tests with arbitrary names and selecting subsets of tests via Boolean expression. Raphael showed you that also already. So marking tests are slow and then selecting only the tests that are not slow, for example. So that looks like this. If I run this, I have a test module where I have tests marked with Lucy and or Charlie or both. and this only finds one mark, and if I look into this module where I used it, so this is a Boolean expression which uses the marker names, and if I look into the module, I use this mark decorator, and this here can be any valid identifier. So this is only Charlie, Lucy, Lucy and Charlie, so I can play around a little bit with Boolean expressions there, but first we have to implement the marking before we can do the selecting, so let's do this first, to maintain a list of marks on a function object. And here, I actually have to slow down a bit. So the problem is, how do we create a function mark usable as a decorator that gets an arbitrary name passed via this dot notation? So that is a little bit of a problem. So let's move slowly towards this. The first naive approach would be to just define a function that accepts a function and a parameter m and then it tries to append this parameter to a new attribute which the first time it's a new attribute which will be created as a list or at any later time time it will just be added to that list so that would be the first naive approach and if we do that with a new function we call this a touch mark then with the spam function and with a patty marker and assign it back to the function, then we have a marked function. So that is actually not too hard to do. But how do we get this name in without passing it in ourselves somehow into a function? So I can't do this as a second argument. So there needs to be some other way. Let's try something. yeah for that you need to know how name lookup works at least vaguely in python there's this short acronym no it's not an acronym it's legb it's local enclosing global built-in so i look in the local namespace first or in the enclosing namespace what that is we get to in a second or in the global namespace or built-in this is why i can also overwrite the print function for example if I want if I assign anything to print then this is the new print function so here I have a simplified function then that only accepts the F and somehow hopes that the M comes from somewhere outside and I set it here on the globe on the module level yes so the global level and if I call this with two different M set twice then I actually attach those markers successfully so Oh, that's also good, but still not good enough. We have to make this portable somehow. We have to somehow make this context in a way that we can carry it around, so to say. And to do this, we will build us a closure, which means we take the same function that we have before, we wrap it into another function that accepts this marker as a name, and as soon as I created this function and returned it, it carries its context around. So, this M will be kept defined after I created this. So, if I create this, let's call it MarkAttacherFactory. So, I created this factory. I have this thing. So, I have the AttachMark function that accepts the function, and it is attached to the local namespace of this factory. So, this is basically how to carry around the context. And if we try this, we create a factory that would attach Schroeder and call the resulting function right away with the spam function. Then we already trigger again this attach mark, and so that seems to work. So there's only then the last step necessary. Same with decorator syntax, so we can use this already as a decorator, and exactly the same thing happens. Only here we created a new function object. This is why this is gone. I defined a new function, so created a new function object. So there's just one. And here, the last step, that is what we're trying to achieve now. We try to make sure that whatever we write here ends up somehow in this mark function. If we use the original marker from PyTest, it looks like something similar is happening, only that this looks a little bit more sophisticated, but it's still basically the same. So also PyTest creates this PyTest mark attribute if it's not there yet, and attaches it marks there. So we have to hook into this dot notation, and for that we actually have to write the first class, or at least I don't know any even simpler way. so we create a mark attacher factory factory which will return a mark attacher factory with the right name once it's triggered and what we make use of here is the dunder get utter function which will always be called when normal attribute lookup I know they're running all the way I can understand it when the normal attribute lookup fails then you have a chance to implement the get utter function to implement your own way of getting to attributes. And what we here do is just whatever attribute you give to me, I don't care what it is. I think it's a mark that somebody wants to set on this. So I pass this M into this mark attacher factory and return it right away. So all we have to do now then to make this work is to create one object that lives on the module space, one of these factories. and yeah another syntax sugar announcement dot notation for all attribute access is also apparently syntax sugar so if we try this out directly with an arbitrary name here we will get a marked function back with the right name and if we try this as a decorator it looks already exactly like PyTest is doing it so yeah problem solved I would say let's try this again that's monkey patch this time the mark see if it works yeah so now it's our eco pi test factory and if we now import test marking module the one I showed you before it triggers all these mark activities and marks the functions with the correct names. And if we collect the tests and look at the tests in a different way, then it seems to work. Oh, another reading. So closure creation on the fly and overriding attribute access using a language protocol. Still two stars only. I'm disappointed. So let's do this last step, implementing this the way of selecting the test with some kind of boolean expression using those mark names so if you run this with pi test you see that 10 tests are somehow deselected because I put this there so if I would replace this with this I would have less or if I would only want this one so So how do we do this? That's not the way how PyTest does it, definitely not. It does it much more elegantly, but that's the simplest way I could come up with that is using actually code generation and evaluates that code that we just generated. Did any of you saw that talk by James Turnbull? Was it yesterday? About metaprogramming? he explained everything I explained today in about like a tenth of the time much more eloquently so this is the part where we use strings and evaluate them he showed the exec function which can also evaluate can evaluate everything so statements and expressions but there's a little bit more tame version of it which is the eval version which can only evaluate expressions And as we only have an expression, we use that. So we split the expression up and just stupidly look if that token that we just got is in the names list. So if we have a names list here, for example, x, and we have an expression spam or x, it looks if it's there. If yes, it replaces this token with true. If it's an operator, then it keeps it. And if it's not in there, it replaces it with false. So this would be replaced, stupidly, by calling this function with false or true as a string. We pass the string into the eval function and get a result back. So now we only have to, yeah, we can have, there's some examples. You can look at them in piece if you want later. So if we plug this into a function now that accepts a list of tests and an expression, all we have to do is a list comprehension then that iterates over the tests. and evaluates every test with the expression and fetching this PicoPy test marks that we just generated from the expression. Dynamic code generation and execution in order to decide if programmatically loaded test functions from programmatically imported modules should be executed. Okay, three stars, yes! Yeah, so now we filter those tests. and we do a little bit of output here, and I thought I make this definitely 3.8 plus, so I went with the Walrus operator. There's nothing bad about the Walrus operator. Don't be scared by the Walrus operator. Only don't use it like, I mean, if you're following Anthony Sotili on Twitter, he's a black mage, and he showed some examples about usages of the Walrus operator that made me start crying, But for something like this, it's helpful because we save even one line of code here. So we evaluate this expression, and the expression is evaluated, and at the same time, the result is assigned to this name. That's all. So we can use that here then directly and print out what we deselected, and that's the other little thing. You have seen that already before. That wasn't even a PEP. That's BPO36817 and Larry Hastings. and I wanted to remember both names, but I didn't. They implemented that. Those are self-documenting F-string expressions. So an F-string evaluates the expression and shows the string representation of it. And if I add this equal sign at the end, then it will show me also the expression itself on the left side of an equal sign, which is really handy for debugging or for presentations like this. so if we try this out if we want Lucy or Charlie we get all those three mark tests if we only want Lucy one test is deselected and that's what it looks like then that is the expression expression in this case is just a name and the result and we have deselected one and here we have deselected two tests so that works I'm really really over already so yeah Now, code complete, let's say the refactoring, I already had one standard library import left, so I used that to make the test collection a one-liner with a chain function from IterTools, which is this one here. But that's not really important. The acceptance tests are important, so let's run them at least. So collect only of all the tests, collect a bunch of tests, and only selecting test marked with Lucy that should leave us with two tests I think I broke something I broke something ok Okay, so nine are deselected, and here not Lucy and not Charlie should deselect three. And they're executed, and we get some results. Not as nice as PyTest, but we're getting there. So that's basically it. Thank you.