Python Decorators: Gift or Poison?

Speaker 1 [00:06]

Here is the name of the talk. I have chosen it because of German gift, which is a poison, and also because of the confusion. Because while using the curators, if you are not sure how they work, then it's kind of a tricky situation. So you have to be sure that you know how it works. I will try to explain that during our presentation. So here are the links to the slides and also to code snippets if you want to check them later. But also on the presentation, there are clickable links. You can actually follow and maybe check it on your laptop or maybe on your phone because it's pretty hard to follow. And the last example, if we will get to it, it's pretty long. It consists from three files, which is pretty hard to show on the big screen. so maybe it would be better for you to check the code later on. Okay. A few more seconds. Okay. So, my name is Anastasia. I'm working for Siren. It's in Berlin-based office, basically. And I'm doing backend there and also full stack, so basically everything. And we have 25 billion transactions per day and 1.3 billion of users. That means that if you are really using decorators in your code, you have to be sure that it works as expected. What's in the talk? At first, I will show you the basics of decorators and I will tell you about functions nature in Python because it's really the basic step if you are turning to use decorators. Then I will show you some magic of decorator, then we will dive into basics, and then I will show you some useful examples how I use or how can you use in the future decorators. And then we will see some examples. So functions nature in Python. Actually all functions in Python are first-class objects. That means that you can assign function into the variable, you can store it in data structure, you can return a function as a result of the other function. And I will show you one example which is written over here. It's It's just a simple function, which is printing some text. And also, this is the hello function, which is printing the text on the screen. Then we are assigning this function to the other function and just calling it. Pretty simple, yeah? So let's run it. And as expected, it just printed the text. But what will happen if we will do the next... how do you think will it work yes it's true it will work you know why yeah actually we are not deleting the function itself we are deleting just name so previous name like a previous variable and my func contains the function itself so it's saving everything around so all the variables inside of the function and also the body and like everything which we can use This is a basics of decorators. The next one about magic of decorators. So there are two types of decorators. Function decorators and class decorators. They were introduced long long time ago but we are still using them. Function decorators. Basic function decorator will probably look like this. We have decorated function, just a normal function which is printing something and the decorator is on the top which is called my decorator. It's accepting as input argument decorated function and then it's wrapping, in this case it's printing something on the screen and then returning the value of the result of this function. So what we did in this example, the first one is showing the syntactic sugar with at sign and then the name of the decorator, but you can also do the following. So this part is doing really really the same, same as using the syntactic sugar of the decorator, it's just assigning to your function decorated function. So it's changing the behavior of the function. Let's run it and what happens? So the first function is just doing the same as the second. It's printing on the screen the text which we wanted to use, and then some text, which is happening right before the calling of the function. So something happened in decorator, then my function, and then the next part, which is this one, is the part where we assigned decorator to the function. It's doing basically the same. The next one, stacked function decorator. If you want to use few decorators in the row, so the first one and then the second one, you need to know how they are running. They are usually running one by one, always, I would say, and it looks like this. So you have the first and then the second. And to prove that, we can run the code and then to see in which order is it running. So basically, it's calling the first decorator and then the second. and then the function itself. And then the second way is more visible. In this case, you can really see what happened. So you are assigning to the function first decorator, which is taking the second decorator, which is taking the function as an argument. So it's really stacked one by one. But it's doing basically the same. So the next example. Class decorators, they are pretty complicated and there are a few groups. As I discovered, basic class decorator will probably look like this. We have class, which is a decorator, and we have the other, which we have in this case, we have class and the other class, they are decorating each other. But there is the other thing, decorator as a class, which is decorating the function. So we can decorate function with a class, not only class with a class, but also function with a class decorator. In this case, we have entry exit, which is writing something on the screen that we enter the function, and maybe name of the function, and then we exited the function. So it's pretty simple to track, and it's easy to add, just a few rows, and then you are done. So we can try it out with different functions. What we have here, the class decorator, and then two decorated functions. Two functions, one decorator, and the same result. So we are entering function one, and then we are inside the function one, printing some text on the screen, and then we are exiting the function one. And then the same with the second decorator, but with different text inside. So let's dive into basics right now. How decorator works. I will show you using this example. We have decorator, which is printing something before running the function, then running the function, storing the result into variable, and then printing that function ended, and then returning the result. Actually, we can even change the behavior. We can return whatever we want, and if we will use this decorator, and we will not check what is inside, or it's written by somebody else, and this developer is claiming that, yeah, it's doing exactly this thing which you need, but you could never know unless you will check. In this case, it's correct. It's returning the result, and everything is fine. So we will try to decorate my function with this myDecorator and then we'll print the name of the my function and then docstring. Pretty simple. Should work. So let's check. Decorator worked. Before decorated function we can see then the function worked, then after decorated function, and then we return function back. So when we're printing the name, name is correct. But not in our case, because it's not the name of our function which we wanted to get. It's the name of the previous function which we used. In this case, it's pretty tricky, because if you're trying to track, if it's your function, but you are just using the the curator which was written by somebody else and you are getting the other function then you probably not find the problem where you have some crash or some issues and also the doc string is empty why isn't empty because it's a doc string of the wrapped function so if we want to fix this issue there is a solution of course we can use reps decorator it's a standard decorator which is pretty simple you just simply use it inside of the decorator and it will do the whole trick for you so in the end we still have our function which is decorated and it has the doc string and we're trying to print the doc string and the name of the function, and it simply works. So we have the name, name is correct of our function, and the dogString is the one which we have for the function. So if you are writing your own decorators, always try to use reps function. But when to use decorators? Have you found a situation when your boss is coming to your office and then asking you, can you please add some logins to one class? And this class contains like 200 functions. And you are thinking, okay, I can, but it will take maybe a few weeks or a few days at least. And then somebody else is coming, I will do it. I'm the superhero. I will do it in five minutes. For that, you can use decorators. You don't need to copy-paste the code anymore. Or, for example, you can just debug something with decorators. It will print the timing, or it will track something, maybe some function calls. Or maybe it even will modify the text which the function is returning. For example, if you need to beautify something, you are just returning some tuple, but you need to have it somehow changed or maybe merged with something else. And then you can use a decorator. So I will show you the example with timing with function decorators. So what do we have here? We have the function which we are going to use further and we are going to add the decorator which is doing the timing for the class, for the function and for some different functions with arguments, and then we will check if it works or not. So this is the curator named TimeIt. It's receiving the method which we want to time, actually, and it's printing on the screen the time before and after, the difference between those timings, and then the name of the function and also all the arguments. So we have class with one function which is sleeping quite small time and then we are printing something and then that's it. So we have the other function without arguments and it's just printing the name of the function so it's also sleeping. We have the same function but with one argument which is printing the name of the function and the argument and the last example we have multiple arguments keyword and just non named arguments so it's printing all of them on the screen and we are calling those functions and trying to get some results it's taking time, it's sleeping. So the first function, it took one second. We can check it here. Yeah, this one, it was sleeping one second. Then the second, believe me, it was sleeping two seconds, and then the second one, zero three, and then the last one, the the last function was sleeping 0.2 seconds, this one. So we just used the same decorator and we used it for all the functions. We didn't write any code, extra code, we didn't copy paste. And it simply worked. But we can also make timing with class decorators. Here is the example. we can also create a class and then add a time into all of those functions. But it's not the best idea ever because then if you want to decorate all the functions in the class, then you need to copy-paste again. So you need to copy-paste the name of the decorator everywhere. But there is a solution. You can use it like that. You can decorate the whole class with just simple decorator. We can run it and then try how it how is it working. It's pretty boring, it's really long function. So we have a few methods a, b and c in the class. They are just printing where we entered and then that we exited the function. And let's Sleeping and sleeping, yeah. So, we are decorating, then we are starting timer, entering the A, exiting the A, and then the time. And also the same with function B and function C, which is pretty simple as we use just one line of code here for the class which we wanted to decorate. So we resolve this issue. Examples. How to use decorators in unit testing. So basically, I had a problem. I didn't want to have copy-paste. And we had a bunch of tests. We wanted to build a bunch of tests to test simple functions like input parameters, expected result, and the name of the functions. And I decided to create a decorator which will build all of those unit tests to avoid copy-pasting. and I was searching the internet for the solution and then I found somebody just wrote a simple decorator for two arguments I guess but it was pretty complicated to understand if you don't know some basics of Python so I wrote the same decorator basically this example consists of three files as I I told you before, it's a pretty long one, and the basis of this exercise, I would say, is in one file only, where the most hard to understand the creator is, which is this one. So it's the creating the class, which is then creating those tests. And the thing is, there is a quite big chance that your developers will not get it and they actually didn't want to use it because they didn't understand how is it working so they didn't want to use the simple way simple way for me and i decided to create a workshop for them to explain that it's pretty simple and they can really really use it instead of something which is not written yet so i will um um a bit dive into some and then we'll return back and we will run this example. So, do you know what will the code print? What do you expect? Couple of what? Please. Three. Just use the last initial. Okay. Any other ideas? Yeah? I'm sorry, zero, two, one, zero, two, four, six. Yeah, that's expected result. But actually, yeah, it was correct. We will have sixes. But why? Does anyone know why? Like how many people? Just write the hand. Okay, so this thing is used in my decorator. It's pretty confusing. The thing, why is it so? Why does it work like that? Because we have function. And function has its own surrounding. It has some variables. And as x is not assigned before, then when this function was already defined, then the variable is already looked up until the last value. So the last value was three multiply two is six. two is six. So for that reason we will get all the sixes. So Python has this late binding that means that it's saving the environment inside and if you really want to bind something which you will use later then you need to bind it explicitly like right there in the function. So the solution will be like this to yield the value or to assign it inside of the function. Magic, a little bit of magic, yeah. So now we will run the previous thing. So here I am binding those ones as well. Because then we will have all the tests with the same input and output as the last one, which is in the queue, which is in the input. So those are three files. This is the input. As input we have function, arguments, and expected result. Functions to test. We have three different functions, which is just multiplying, adding something, and then comparing something. And decorator's magic is here, basically. So let's run it. Doesn't have any output, that's why it's failing in this environment. So we got the function which is testing all of them so it's added as a test suit one two and three so the input is there and the output Somehow worked. When I was giving this talk at EuroPython, somebody just asked me about recursive functions decorated with decorator. I'm not sure if it makes sense, but I added it here, so you can check it if we would have internet. So, here we have recursion, we are calling this function multiple times, and this function is decorated with decorator. And it simply works as expected because we are calling this function multiple times, so we are calling the decorator each time when we are calling the function. But it will print something at the beginning, then it will try to call the function, and then we are in the function, which is calling the other function, which is decorated as well. So decorator, and then original function, and then until the last one. So pretty no magic in this case. But there is a weird behavior, there is a weird version of recursion. If we will not assign the created function to the function back, then it will do some weird stuff. So this is the normal one, which is just decorating and then printing the function, so calling the function. And there is a weird one when we try to define, to reassign, to decorate the function and reassign to the other one, to the other function. That's why it's called... it's not reassigning the function back to the original one, it's not changing the behavior of the function, it's reassigning to the other variable, so it's just one time call of decorator, which is here. So there is pretty no magic in this case, it's just wrong usage of decorators. So you really need to assign back to the same function if you are decorating this function. And after everything that you have listened, do you think if it's a gift or a poison? You can use it safely if you know it, then it's a gift. But if you are not sure, then never use it. Because you will never know how will it work. So thank you for coming and if you have any questions, I'm open to answer all of them. Thank you very much, so any questions, any questions, about decorators, oh, yep.

Speaker 2 [27:18]

Thanks very much for your talk. Very interesting, though. I was wondering if that test decorator thingy, whatever it actually was, I'm not sure if I got it anyway, but I think maybe if you have to do a workshop to explain a bunch of programs with four lines of code, then maybe it's not the best code in the world. So could you consider an approach how to implement this in a more, say, readable fashion?

Speaker 1 [27:51]

You mean the way of presentation or the code itself?

Speaker 2 [27:54]

The code itself. No, the code itself. Could you consider a version of this code which is easier to understand for people who don't do a three-hour workshop?

Speaker 1 [28:06]

Yeah, of course I can add more like explanations into the code but if I would add those explanations into the code and then I will try to show on the screen then it would be like explanations of the half of the screen and then a few lines and Maybe if I have like the whole long function then it's not pretty readable So I need to scroll and then it's it's harder manage, but I can do it You can follow my github. I will add explanations. Thank you Just for you.

Speaker 3 [28:42]

So, thanks for your talk. And my question was, when I try to understand decorators, you are a simile between using the decorator syntax with add and assigning it to the call of the decorator's equivalent. That's pretty useful to understand it. Is it really equivalent or is it oversimplifying?

Speaker 1 [29:10]

Yeah, there is no difference.

Speaker 3 [29:10]

Yeah.

Speaker 1 [29:12]

It's the same.

Speaker 3 [29:13]

Okay, thanks.

Speaker 1 [29:16]

Any more questions?

Speaker 2 [29:19]

so thanks great talk it was like really interesting applications um but i like i um i tried to write some some decorators myself but i i really didn't really find an application where i was like so i think yours and with the testing was probably one elegant one i don't know it depends on and if you find it understandable or not but i i have a question to the to the audience mainly like if if somebody has like really good applications where they they uh yeah just use decorators and that helped a lot

Speaker 3 [29:55]

Um, yeah, I like the with statement a lot, but writing your own class, having an enter and exit method for just like three lines of code is very overkill. So in Python 3.67, I don't know, they added a decorator called context manager, which allows you to just take a function, have you build up and tear down logic all in a decorator without needing to write any class with any metric methods.

Speaker 1 [30:33]

I saw some people raise their hand before. Do you mind? Sorry.

Speaker 4 [30:40]

I think generally framework usage for decorators is nice because you keep the framework stuff, be it Flask or Qlik or PyTest or whatever, you keep that in one place and you keep the logic of your actual function or test very readable. but for my own application I've done it once for quickly adding logging in lots of places so I just wanted to have kind of a performance measurement exactly like Anastasia was showing when was the function entered, how long did it run and it's very easy to kick out again because you just have those few decorator lines Thanks for watching.

Speaker 1 [31:27]

Click would be one example where it is used extensively, the command line library. Check this out, this is quite nice for command line tools. And caching decorators are quite useful as well. You have to get the parameters of the function as cache key.

Speaker 2 [31:52]

you derive a cache key and then

Speaker 1 [31:55]

Any comment from Anastasia? Do you want to add some comments? Yeah, Qlik is very useful. I'm using it for my tools, actually, and it has lots of possibilities of input parameters, and then you can even validate the input of the user. So it's nice. Yeah, that's great. Any more questions or opinions about that? Yeah, I saw some hands here before. I'm trying to find you. Any? Any? yeah okay uh if not i think we can kind of uh give her a round of applause very good talking