Aspect-oriented Programming - Diving deep into Decorators

Speaker 1 [00:04]

Thank you very much for the kind introduction. Good afternoon, everybody. So right after lunch, I hope you won't be, I'll help you over the after lunch tip here. My name is Mike Muller, and I've been a Python user since 1999. It's in all of the claims, the last millennium, starting with 1.5. And I stepped with Python all those versions from 1.2.5 up to 3.10 and 11 soon. In my day job, I teach Python. Actually, what I'm doing right now is since I I did my first Python training in 2004 and since 2010 I do it full-time with Python Academy. So if you're interested in training, you're welcome to ask me. But this is not about me, it's about you. So I would like to know a little bit about your background. So who of you knows and uses Python less than one year? Please raise your hand. Very few people less than one year. Between one and five years. See it's more majority between 5 and 10, even more, more than 10, still a few. So just numbers, the times of usage is not the main thing, but I want to get a little bit of background about your price knowledge. Who of you has used decorators, applying a decorator to a function? Most of the people. Who of you ever attempted or wrote an own decorator? Also a few people. So why are you here? Okay. We want to do a spectrum, so I gave it a bit different name. A spectrum on the programming is actually having cross-cutting concerns in your program. What are cross-cutting concerns? Everything has nothing to do with the business logic, so a prime example would be logging. Logging permissions, what these functions are allowed to do, and a few other things. I have some examples, and these are cross-cutting concerns, and Python offers a tool that's called decorator to solve this problem in an elegant way. This is this decorator thing. First I would like to show you some of the basics, then we go through some examples. So we need to cover a few things. So as all of us say, everything boils down to price and everything is an object. So if you understand what it means that everything is an object, also understand decorators. I also dive a little bit into functional programming because when you write a decorator you typically use some principles of functional programming. The Python supports multiple programming paradigms. Please download the material. This will be a PDF. This is the material and if you can always read along. It goes with the PDF. So we only have one half hour which is a bit short for the whole thing. Since I also have some exercises and even the exercises when you do them you might use up one about already for the exercises so you have to cut this bit short we probably do one exercise in other ones a little bit of homework you do will do get a solution document I haven't put it up yet but at the end I get it you get a PDF with example solutions which would be helpful so when you unzip the file you will get the PDF that looks like this plus so the PDF will be this thing so this is a 27 pages there's a lot of covers or maybe like 20 pages of content and also some files and I of course I put the files already here so I'm I'm going to use a Jupyter notebook if you know what it is please go ahead and do it if you want to use an editor or anything else use whatever you like I'm not this is not about the tool you use use whatever you like the notebooks is great tool for presentations that's why I use it and these are the examples here and you can user. So tiny URL slash PyAspect. If you come later, I will put this back on. So basics of decorators. First, I want to start with a little bit of history, because decorators became very famous with this add syntax, but actually the principle exists before, and And there's a few decorators in the standard library, and I want to start with two of them and explain a little bit the basics about this decorator thing. So if I have a class, see, yeah, I can write a normal method, and the normal method would take self as a first argument and return something. So this would be a normal method. And then Python has a notion of a static method. So a static method is nothing special. Actually in Python you wouldn't need it. You can always write a separate function. A static method, call it S method if you want, doesn't take the self, and here I return something like this, and here I return something also. So this wouldn't work the way it is. And in ancient times, when we didn't have this decorator syntax, you could say, now I use the same name, and I call the function, the function static method to turn the thing into static method. So I use the same name, fmethod, and now I have the static method. So if I make an instance on this thing, I make an instance on this class, now if I want to use a normal, call the normal method, I have, I can call the normal method like this. If I want to call it on the class, I do have to supply this instance as the first argument, otherwise I do get an exception because a method on the class is actually bound. It's bound, yeah? So if I say c.method, then you get a function, it's just a normal function, so everything you learn about function applies to these things, it's just normal functions. This is actually also a function, but a special kind of function, and this is called the bound method because you don't put in the self as an argument. Python used it up already and created the bound method. Metabounding goes with descriptors, which is a totally different topic. If I had another hour to spend, I could explain to you, but it doesn't really matter. It's bound. Now, if I use this S method, it's different because I don't put the C in and still it works. And if I say C, S method, then here it gives you a function. So it didn't do the binding. So this S method is just a normal function that just happens to be the class and has no notion of the self. If you use PyLint and you write a normal method and you don't use the self as in here, PyLint here would complain and say, you don't use the self. Please reconsider using static method. So the thing is, it's very important that this is a static method. But I'm hiding the fact here. Of course, this is how Python works. I can only use s method after I define it. so I cannot put it in front, because otherwise the complaint is not there. So if this method is bigger, then you hide the very important fact that this is a static method after you defined it. So if I have 10 lines here under the return 43, I might not see that it's a static method because it's just out of sight. And then they introduce something that's called the decorator syntax. So this is just, as you might see, a static method is a function that takes a function and returns a function. I will use the word function a lot of times in this training. So it takes a function and returns a new function. So hence the function programming thing, because in function programming, everything is a function. And now I use static method here, and this has exactly the same effect. This has exactly the same effect, and this add is just syntactic sugar to do this. And this is the main thing. If you understand that this line goes away, and you just put static method in front, and now you have exactly the same effect, but it's very obvious because you put this decorator in front, you see it, and syntax highlighting makes it, gives it this magenta kind of color, whatever it is, that you can see that it's different. The thing is different. And that's an important message. So this is what people call syntactic sugar, but it is good sugar. It's healthy sugar. Usually sugar is not so healthy, but this is healthy sugar because it makes your program better readable yeah and that's pretty much everything I need to explain to you the rest is details but we cover a few of the details because this just takes a function to returns a new function and we just change it so this one this one work before the syntax I think it must have worked in Python 1 I think because everything is an object and functions are objects and you can put in functions into function calls and get functions back. If you want a fancy word, you call it higher-order functions. So functions that take functions arguments and or return functions are called higher-order functions. Good. This is this motivation and there's another one it's called the class method which is similar I can skip it here doing the same thing in the class method doesn't take the instance but the class is the first argument and class methods are usually used for a second constructor. So Python can only have one init, but there's a from something. You might know this dict from keys. You might know this one. This is some kind of a second constructor that gives you an instance of a dictionary with a different signature of the calling of this thing. Instead of init, you call the from keys method, and there's quite a few things around it. Typically it starts with from. It's just a convention, but it makes sense and have example here. This would be static method, exactly the same principle as, sorry, class method, exactly the same principle as static method. You make it very obvious and put it in the front and don't hide it in the back. That's pretty much it about this one. So I have another example here, but since I want to go a little bit faster, I skip it. You can look it up in the PDF. I'm also being around here till tomorrow, so if you want to ask me anything, you can also ask me later. Good. Real world examples of static methods. I have examples in other libraries and I find a few. You probably know more than these ones. So if you go to PDF, so click is a pretty well used arc pass replacement for writing command line arguments. You use decorator to turn a function into command line arguments. Django is using decorators a lot, like permissions, who's allowed to do something. Flask, use it for routing, you put this regular expression on top of the decorator. Sison is using them, you can use Sison, you can turn normal Python functions into Sison functions without touching the internals, just put the types in front. Numbers doing the same thing. So different examples, web programming, function a numerical problem programming and there are other examples you might see them somewhere and these are all cross-cutting cross-cutting concerns things that don't have any do with your specific business logic but something you can need in many places in your program in many places in your program good so one more concept I would like to explain and this is a concept of a Clojure. Who of you knows what the Clojure is? A few people. There's even a programming language called Clojure. Clojure, yeah. It's a functional language that sits on top of the JVM, so it compiles the Java bytecode in the background, and obviously it's a functional feature, and this is important understand to see how we write decorators. There's a second way of the writing decorators using class which I will introduce later which is also possible and you might argue might be even more Pythonic but most of the examples you see with functions that's why I would like to explain how a closure works and this is has to do with functions so I have an outer function inside this function I can define an inner function. This is totally possible. And now I can do something here. So the example is not useful in terms of what it's doing but this should show the thing. And now I say return and I do return my inner function. So this is a higher-order function because it returns in a new function. You might get fancy you can call the function factory. Now I have this one, and now I can say i10, and I can use my autor to generate a new function. And if you look at this function, it looks kind of funny. It has this funny name with locals and in and out of stuff here. The main is just the name of the notebook, but this is a function name. Now I can call this function, and if I call this 7, what would be the result? 17. There's only two numbers here, and I use a plus, so it shouldn't be that difficult. but this is a function and now the question is how does it function know that there's a 10 somewhere. Functions don't have state usually but this is regulated by the closure. So if I have this inner thing, inner function, then I do have this close, two underscores, have this closure. Why doesn't it come? Closure. That doesn't work. What are you doing? I10. I10, yes, sir. Thank you very much. So typing, talking, looking. So it's a closure, and you see there's a tuple, as you can see. One tuple has a trailing comma, which is a funny syntax in Python. So you can access the first element, and then you can go in here and you say cell content, which obviously is tab completed, and you get your 10. that's how the closure knows this value yeah that's that's how it works internally and then you have this nice another route you can put a people it's called the closure and this is a way to store state in functions so functions programming for function programming is have no state so Haskell has like everything is immutable and there's no like a class where you modify attributes and an instance something like this but you can use a closure to use some kind of state in some way or the other at least maybe it's not the right formulation but at least it has it has the same purpose and this regard okay now we know what the closure is now we can write our own simple decorator and the first i just start with a normal function that it's called hello and i just take a function and i just print hello print hello So you're welcome to type along or just watch, depends how you like it. And then if I use this as a decorator, I can use my hello, add a function, and I will define this function, add, many times because easy enough to understand. So now you see if I do this, something strange happens. Why does it happen? So now you have to go back and do it in steps. So what happens when you put a decorator? this happens yeah now you say add equals hello add down if you write something like this you see now it makes sense I'm calling this hello and put in the function that's a small problem involved here if I want to call my ad it doesn't work because I forgot something important I don't return anything and if you don't return anything, Python returns none. When you get the most famous error message, if you Google Stack Overflow, you probably find 10,000 of questions. This is a typical thing. I forgot to return something. The first part of my decorator is correct. I take a function but the second part to return a new function that replaces my function, which I do here is missing so I have to write a better one good so write a better one so I have a hello that's the outer function hello the outer function and I do my inner function and you can call it whatever you like you can call it wrapped or wrapped, and then we use this thing. Anybody has an idea what these stars are? They're usually called KW args. So this is, maybe I go a step back later and explain actually what it is. This is a function that can process any number of positional and keyword arguments and now I do my hello so my printing hello here is my cross-cutting concern I want to print this hello in many places and then I return my function I do return my function and I use the same syntax here but in a different context you see here I define the function with a def and here I call a function and then the same syntax has exactly the opposite meaning here that means packing and here means unpacking so we come to this in a second yeah so and now I do something I return important I return my wrapped function so the name of the function doesn't matter we see this is not good style yet we have to improve this one but this is the first integrator decorator sorry decorator that works and now I can do the same thing I can just I'm lazy I just copy this one and now you see when I do this nothing happens and our approach before we had some code executed at definition time yeah that's something now this hello is executed at definition time but it doesn't show you any print it just replaces the ad and now if I call my ad function I do get my result 4 plus 5 is 9 in addition I do get my cross-cutting concern which is printing hello so I added something to a function this is one it could be something more than just printing hello so here plug in whatever code you would like to happen for all the functions and that's how this works you can put a decorator in front and it executes yeah good everything clear any questions so far so let's have a look at this star on double star. So if I take this one out here and we look at this and I just print args and I print this kw args. Yeah so this would be my helper function whatever the name doesn't matter here. If I call it without any arguments see it prints something and it prints an empty tuple and prints an empty dictionary. If I call it with arguments so I can give it positional arguments and I can give it keyword arguments as many as you like just do three here but it doesn't really matter something like this and you see now it packs all the positional arguments into a tuple and or keyword into a dictionary and now because if I have this I have two containers and I can move these containers around and I don't care how many items I have zero to whatever many I like yeah and you just need to know how to process a tuple in a dictionary which you hopefully know when you work with Python so this is the first one and now I can do the same thing I can or the opposite thing better I can now call this function this one hands over the argument. So if I call my function add in the common way, I do get a 5. If I have this args in a tuple, yeah, and I try to call my add, this is not gonna work, yeah, but if I use asterisks here, I unpack this tuple into two elements, two arguments, and this will be equivalent. So doing this call is equivalent to doing this call when args contains these two numbers. And the same holds true for this double asterisk thing. It unpacks a dictionary as keyword arguments. Therefore, this function can replace any other function in terms of its signature. It's very important. It's very, very generic. You won't get any more generic than this. You can call any function with it. If it makes sense later, it's not guaranteed, but you will survive this line here because the function will be called. If you get an exception, it will be inside somewhere if you do something wrong. So this is the usual way. In this case, I just move the arguments from here to there. You can also do something with the arguments. You can inspect them. We have an example to doing this. You could modify them. You can modify a tuple, but you could make a new tuple which is longer or shorter or take something out. You could modify the dictionary. Everything's up to you. So in Python, you can do a lot of things. As we've seen, you shouldn't replace 18 with 0 but you can if you like to yeah so nobody stops you if you feel like that's a good idea go ahead the recommendations are not so good this would be a simple decorator here this decorator has a small problem make it a bit nicer and add a doc string to our function add two objects yeah so you can add any objects that work with a plus that's fine so I can still works yeah but if I look at the doc string this which I can do here with a question mark in my notebook then it saves me there's no doc string hmm and if I look at the name of this function, see now, not hello, whatever, add, sorry, that's what I meant, add, still no doc string, and if you look at the name, add this add, now it looks like this, I would like to have the name add and not this strange local wrapped kind of business here, so this is not a good way of doing it, you need to make it better, and for this you need rep from functools. So the standard library provides the functools module and I import a helper which is reps. So it's not the leftover from the lunch here. It's not nothing to eat. But we can use this. No, we can use it here. And I just go up here and take my hello which is missing all the doc strings which in the PDF I do have doc strings but I think we don't need to write in here and now I just put it here and now I use this one and this is finally also used as decorator so to use a decorator to make a decorator better and we wrap this function and I have to say which function I wrap I say funk and because there's it doesn't assume that the first argument is function anything like this you have to put it in and if I do this one now I can decorate my ad again, which I do here, and now if I look at my ad and my doc string, the doc string is there, and if I look at the representation, looks nice. Yeah, the reps, you can look at the source code, the reps is doing nothing else, it's going through some attributes of the original function and adding it to the new function. It's still a different object, if you look at the ID, the ID, it's It's different than the original add I defined. I won't be able to access it because we already placed a name here. But it's actually a new object, but it looks exactly like the old object. And it's a basic principle of Python. Python usually doesn't care for the object or the type. It just cares for the behavior most of the time. As long as the object is doing what you want, it's fine. You don't care what it is. And this is just this principle here. And you just use a different function, but I don't care. doing exactly what I want and if it's function one or function two who cares as long as it works and the result is correct that's fine but so always use reps to make it make it a nice function to keep the doc string and and do something like this one more thing if you write a recursive function then this wrapping will also influence any other recursive function code so recursion is It's not a good tool in Python most of the time, because now the standard recursion limit is 3,000. So you should use recursion only if it's a key kind of problem that's really recursive from nature. And don't overdo it, because if you reach like 15,000 recursions, the program just dies without anything. There's not even an exception, I think, so it depends on your operating system version, but between 10,000 and 20,000 somewhere, it's just going to die anyway. So recursion is not the technique that you should use excessively in Python. It's slow, resource-hungry, and so that's the thing anyway, so I rarely use recursion. If you have a problem where something is referencing itself, then maybe recursion is okay to solve it, but if you can turn it into iteration, it's usually much easier anyway. Yeah, so iteration is usually easier. And very often you can, with Python, iterate some things to turn it. So just a small thing that recursion will also be influenced. This is the basics of decorators. Now I have a few use cases to show you what's going on as an example. And you will see it will be the same thing again and again and again, just with different use cases. The first one would be caching. So I do have all my examples here, and I can load them directly here. And I call it cached because I think there's a name clash with some other thing called caching, so I call the thing cached. And this is one example, cross-cutting concern caching. So if you have a function that is called often and very often produce exactly the same result, you need a deterministic function. So if you call this function with the same arguments, it always gives you the same result. If you have a function that includes time, caching doesn't work, because it would be still yesterday if I called it. So it doesn't work. You have to have a deterministic function. And this is very, very simple. Don't use it in production. There are much better ones, but it's good enough to explain the principle. And very often for cache, you use a dictionary. That's what I'm doing. I put a dictionary inside the function, which makes it not very useful because it's really hard to get to it. You can still get to it, but you need an introspection stuff. So this is a cache that cannot be invalidated and cleaned and anything easily. But it still works. You see I have this outer function and the inner function. This is a typically naming thing. You call this cache and the nested function use underscore cached. This name doesn't matter anyway, but this will go away. This name will disappear. Yeah, there's this wrapping thing which I do here. This function wraps and will go away. Therefore, typically you don't have to invent a name. You just use an underscore and we'll see that's even version is two leading underscores. Yeah, so and then I need to create some kind of unique key for my dictionary. That's that saves the Arguments and that many different ways I use here pickle which kind of restricts my function So if this function would take a open file object pickle wouldn't work Yeah, pick list Python's your C's sterilization module and you can pick the most Python objects but you cannot pickle file handles and sockets and these kind of things that wouldn't work for this but assume it's not the case so if you want to hide the real one you have to bit more effort in this this generation of this key but there are a lot of solutions out there so that's not about this this specific usefulness it's just more example how the decorator works and I create this key, and then if the key is not in cache, I call the function and store the result, the return value of the function in the dictionary with the key. If it's there already, I just return it from here, so I don't recalculate the thing, yeah, and that's all. This one is not very, so if you have a function, the same argument comes as positional, and as keyword it will be a different function call, but assume you don't have so many combinations and make it easy you might have some additional function call and now if i do this i use my cached decorator here cached decorator at my grade function add which we're going to define many times here and let's just add a print to see that something that happens and return my return my a plus b. So now if I call my function add with the two and the three you see the first time add prints called the next time it doesn't print called because it's it just reuses the value in the dictionary. So there's no easy way to access this dictionary. There's many different other ways you could add it to an attribute to the original function and then make it accessible from outside. So you might know at least recently used cache, LRU cache in Python. It's doing exactly this. It's a decorator and they add attributes to the original function and with these attributes you can clear the cache, you can access cache statistics and all kind of stuff. This is not nearly as fancy but it's the same principle, the same principle and now you know how to write a cache. So if I do this and I call this with the B here which is exactly the same but my algorithm is not smart enough to figure this out, this would be a new signature and it recalculates everything. So if you want to do this probably it's possible you need to inspect it to do some fancy stuff with it. But this cache just doesn't make sense anyway because my A plus B is much faster than this cache thing. So this makes everything slower in this case. You have to have a function that's somewhat expensive to make the cache useful. Adding two numbers is not expensive enough because generating the pickle and dictionary and all the stuff there's more time than this one. Good. But as an example caching is a very typical cross-cutting concern and we solved this problem here. The next one I have would be logging and again I called it logged to not conflict with the Python logging sound thing but here I give it a little bit a different thing so I use a constant which I'm going to change it's not a pretty good concept but just an example so this would be in a real the notebook is different from a real program you see I can rerun the same program again and again with the same class definition which you wouldn't do in a real program but for learning experience is kind of okay yeah the notebooks we are different from it and from a real program but I'm doing here so I have this this global module global constant logging which is set to false and this is exactly the same thing as we seen before I have my outer function that takes a function I have my inner function underscore locked which has a reps in front I have my arc stuff and and now this is different. I have this global thing, logging, and if logging is true, I'll do some printing. So here I should do some proper logging, here I just print logged. So there's no logging whatsoever, it's just an example that logging could be there. Now if I do this and I add my decorator, add my decorator if I manage to type the right thing here, add my decorator logged Logged to my function. Okay, and if I call my function now, nothing happens because this log is false. If I set it to true, and I do the same thing again, then you see now the logging will go on. So now I have a global switch which in this case I can change the switch at runtime. If you want to do this a different thing I could but you don't have to. You can set it once and then all the logins will be on and all the logins will go off. So then you have to sprinkle 150 ifs in your code. This would be one. One if and you just put the decorator can turn everything off. Here I can change it at runtime you could even go a step further and just say if logging is off I return the original function without any changes and if logging is on I return the modified function that would be would be no extra overhead. Here I always have an extra function called overhead. If this is a problem or not depends if this function is running for half a second then this extra overhead of another function call doesn't make sense. If the function doesn't do anything like here, A plus B, the extra function overhead was A plus B is very fast compared to the function overhead usually, more or less. This would make it nearly twice as slow in this case without doing anything. Instead, you could just return the function unchanged if logging is off, otherwise there. Then you can change logging at runtime to change the behavior. You have to change it before. That would be a variation of it. Everybody with me what I'm saying? Or too abstract? So if you have anything raise your hand. You can always ask questions. Yeah, so it's good to have feedback. Yeah Yeah, the question if I can form the material about to be printed. Yes, that's our next topic. It is called parameterized decorators you can add yet another layer so three deaths and then you can inform the decorator and give it instead of just print lock you can for instance hand in the text that's to be printed for instance yeah it's that's about it's our next topic i moved a bit back it's called parameterized decorators which you can write is yet make another level indentation or alternatively i also show you how to use classes which saves you in level of indentation and make maybe easy to understand because every level indentation makes it somehow more difficult to follow what's actually going on there. Good. Yeah? How would you return the original function? Let's try to do this. So I call it log2. So you say okay now instead you say if logging you take this out if logging you put this here and then in dent the whole thing so if logging you do this and I have to guess have to indent this one here and then return locked yeah and this one has to be indented. Otherwise, I don't even need an else here, I guess, because I have the return is there, so I can just say return the function unchanged. So hopefully it's okay, yeah, so now let's just type like something like this, and now if it's false, then now if I do the same thing up here with log2, just do this with log2, then the function is just unchanged. The original function. And if I say logging to 2, now it doesn't change anymore. I hope I'm correct. And it was the same thing. No logging. Yeah, because now this this logging was used only here and it's never touched anymore. So now I return the unchanged function, no silver overhead. The only overhead is when I define the function, but this is nothing. It's just once when I write the code. You won't be able to measure, I guess. And if I set logging to true, so copy the whole thing again to stay with the flow, set it to true, then it's on and you cannot turn it off anymore. So if I set it to true here, of course now Now I will return the change function, call it log3, just to, yeah, let's get a little bit hairy now, just changing all those names, but hopefully this makes a point, say log3, and I now call my function at logging zone. And I cannot turn it off. If I set it to false, it doesn't go off because I just have the function which doesn't do anything. setting at defined time so you cannot change at runtime the other one adds overhead but you can turn it on or off at at runtime if you want to do this so you have to restart the whole thing unless you kind of go through and does do some name I don't know if you could we probably could write the solution that's replacing the function but it's not good style would be very indirect very difficult to follow so in person you can do pretty much anything anyway you like but it's not a good style and it's not the recommended way when you should always go the recommended way but here you can return the original function without changing it should be zero overhead if you don't count the compilation but the compilation to bytecode is nothing yeah good any more questions I do have a short exercise and as maybe we spent a few minutes so you can write your own decorator and the example is timing so if you go to the PDF you will find an exercise as a writer function use default as there's different ways of timing so if you know notebooks if you don't have to do this you could use time it which is a ipython magic time it ipython magic that's doing all the heavy lifting and can it's much more comfortable but I show here but you would need to have ipython installed and use a magic you can use a magic a normal script to some get IPython magic stuff yeah and this one he gives you this one so this sir that's twice as slow because it's a m1 machine but I haven't I still the old Python I don't have the arm Python yet because I haven't upgraded yet this happy I figured out how to ask the folks and kind of watch I'll do it so it's it's eight nanoseconds used to be four which m2 minus twice as fast so this would be easy but we say from time it which is the price and standard library module with the same name time it actually using this in the background the present time it import default timer and with the default timer which is actually just the alias for time it timestamp some other function will see you get a timestamp so if I call this default timer I do get a timestamp and this can be a different number depending on your operating system, but we're usually not interested in the absolute value. We just want to have the difference. So I have the start value and then I do this and then I have this one and subtract the start or give it an end if you like, and it tells you how long it took. It took me seven and a half seconds to do this. Yeah, so see how fast I can type. Yeah, that's about as fast as I can go. If you put everything in one, this should be more or less zero but you will see it's not zero this is my measuring oh it's not correct and you see now this is it's a thousandth of a second it's quite a bit so this is your overhead so you cannot measure things at a faster approach would be to repeat it many times and divided actually time it is doing exactly this that's why you get this one is doing now oh it's new one they have there now a thousand separate a much easier hundred million otherwise I was struggling to read it. That's very new. So it's good of us to install the newest version. Now we have a cell phone separator. You do it 100 million times, so one run takes about one second, more or less, and then they repeat the whole process seven times, so it could give you a mean and the standard deviation, and if the standard deviation is small versus a mean, then it's kind of accurate. If the standard deviation is big, it seems like we have big difference with measurements, which indicate there are some caching effects or something, something is wrong. what I don't know but something is not really good and this is much better and you will see I won't be able to reproduce this number with my time I think even if I do something and to my experience always but twice as slow or something for something my measurement so when you measure something very rarely everything is totally correct but typically you just want to compare things and if you make makes the same measurement error with two things and the error is always the same, then you can still compare things, even though the precise number is not correct, because this is twice as fast as the other one, this is kind of okay. So you always have to take it with a grain of salt, and don't look at one or two percent, which can be easily a measurement error, yeah? Good. So use this one, and write a decorator, just take advantage of my examples write a decorator that's using this start and end to get a timing and apply it as we get exercise so maybe you spent a few minutes you can try if you have a questions feel free to ask me I come around for the people in the room the people put here I difficult but you can still ask questions and I will give hints and then in front of everybody. So please try to do something. So you just need to have a Python 3.7 to 3.10. Doesn't really matter. It would be 3.6. There won't be any big difference here. And you can use any kind of tool. You are welcome to discuss, of course, to work together so it's no problem I'm not sure how the online people can collaborate but you can also if you have a means I'm not sure whenever anything then ask me yeah yeah this is exercise so this across a question which one to do so that's a good question so if you go yeah and you have this active first exercises session this would be this one here this 2-8 kind of a little bit shifted somehow yeah this 2-8 on page 11 so the printed page 11 on PDF page 17 they use a timer default timer to do this and also next one reps you can do everything you want so wrap it on one wrap it up in one go yeah I can hear anything Yeah, you can. Yeah, of course. And then, yeah, you can, you can do this. So you can apply your decorator to your decorator function to check if the decorator works correctly. Yeah. Yeah. That's the rest. That's the rest. Can you go up? Yeah, that looks good. Yeah, that's okay. And what does it say? Yeah, it won't get any faster. Put a sleep in there just to make the effect. I guess to see something, just put a sleep a second, then you should see something. That would be a difference. Yeah, if you like, put import time. sleep a second, then you should see the first one should take about a second, the other one should be much faster. You have to make it easier to see the difference, because the difference is tiny, it's very hard to see the difference. Good, so I see first people are finished, so who finished number one? A few people. Number two is very easy, so that should be very difficult. Good. So, since we have a few things to cover, I show the solution. So, of course, I prepared the solution. That's why I turned it off. So, I have solution, and I have this exercise one solution. It could look something like this. So, I have improved the road a bit differently, but it's fine. So, I have my func tools, my time, and my time it. And you see I have this measure time function, which takes a function. I have my reps, and here I call the proxy function, whatever you want to call. It doesn't really matter. And then I have my start and my end, and here I do my function call. I save my result, and then here what I'm doing is just put it together and store it in a dictionary. So instead of printing it, I store all the runtime. The dictionary is a function name, and you call the function the second time overwrite it it's not the perfect but it's totally fine yeah so and this that's what you can do and then I have I just use my decorator and put it on to functions and called actually I could actually load this one here which makes things easier load solutions exercise one yeah and this loaded in here and I can execute it the same thing and can execute it this measure time and that's what I did I use my time sleep to make the point now if I if I run this one you get it takes a bit and I get a dictionary with the entries of the functions and this is a function without the sleep it's a minus six yeah and the function is the sleep is one you see this is much bigger than this or sleep is not exactly one second it's approximately one second and be careful especially windows very course if you call sleep a million times with the millions of a second you end up it's like ten seconds and not this one second It's not very correct. It's approximately a second. Be careful. But it's good enough to make my point. When I put in sleep, how long it takes, I have my time. And you have different timings. You have these measurements. So this is always the same principle. Whatever you do, logging, caching, time measurement, anything, same principle. You can use it as a blueprint. You just need to change your business logic or whatever you want to do. Good. And, of course, I use the reps immediately. so this includes number two you always should do this it's just as a reminder so let's continue we want to get a bit more advanced there was a question already how to make this decorator a bit more general and for this one I can now instead of say hello I use a say and now this function takes a text I would like to display then I have another level and now I'm on my function level so this outer level I add and then I have another one this let's say three underscores you the different ways of the names are totally free to use but this pattern people do and here use my standards arcs kW kW arcs yeah this would be this now I print my text instead of hello and then I do this return business if you have three deaths you have three returns very easy and sometimes you if you have a function doesn't return anything you could skip it but in general the number of deaths and the number of returns should match and then I return and now I return this thunder underscore say it's not that I'm doesn't not not leading trainings just leading and I return at the underscore say so now have three returns and three deaths which would be fine you see them the returns has to be one little more indented and now I can say my function I I use my say decorator and now I put in my hello and I define my function and now if I call my function it should also say hello yeah and now of course now I'm free to change the text and this is called the parameterized decorator because the text is a parameter and I don't have to hardwire any text and for any text, I can write a new function, I can use one, and the outer one is putting a text. Now, you can, if you want to unroll this in one line, it takes a little bit, so, but first you can write the function, then you can say, add equals, yeah, you see, now I have this function, which returns a new function, that in turn calls my add function, I hope this is correct and something is wrong and if I don't have to put this syntax there so and if I call my ad hopefully this works and does it works you know so that what happens so this one the first one returns a function consumes the text and this returns another function so this is a little bit so putting this ad there's easier to understand I guess yeah but that's what happens in the background and now if I have this you can chain these things no I can use one in the exercise he did it already so you can use another one and I can say now say again and I can say goodbye yeah and now goodbye goodbye say goodbye and if I add my numbers and now it says hello goodbye so you can put as many decorators at a function as you like so you can never assume you're the only decorator you never know what people do with your decorator that's why always use assume you're not the only one that can be other decorators yeah obviously your decorators not the only one that can be others you might have seen it some Django is using very often multiple decorators on top of each other. Then the saying goes, if you use more than five, it's a general because it's so highly decorated. I learned it from somebody else. It's not my invention. That's what people say. You can do it. You shouldn't overdo it because decorators, it's not really clear what's happening maybe. So far we used functions to create create decorators, but Python actually doesn't care for function. Python cares for callable. What's a callable? Yeah, int is a callable, but int is not a function. So if you look at the type of int, then you see int is actually a class, a type type is a class. So you can call functions are callables. So instantiating, making, creating the instance and calling functions the same syntax. If you use language like Java, you need different syntax. You have to write a new there to make an instance here, you don't do anything like this. So you actually don't care. So when I do my introductory courses, I say I call the function int. That's a lie, it's not correct, but it's good enough because people have no idea what the cause is, but you don't need to know because this instantiation is exactly the same, looks exactly the same as calling a function. And that makes a lot of things possible, and you have a concept of a callable, so you say callable, and if you say callable int, that's all I need to care. And I don't need to care if it's a function or an instance or something else as long as it's callable. If I define a class, A, and I don't do anything, and I make an instance of this class, and I say it is callable, then it says no. But I can make my class callable. I can make a class, and I have an example here yeah let's say a counter and I can do something like this I we don't have to do this within it but to make my point to make a bit more flexible so I have an init and I have a function and I can store no don't have a function to function other place so different left hand example you could put this function from outside and I say it's self count zero and then I make this thing callable by adding a special method which is called call, done the call and I don't take anything here as an argument I just increment the count so that's a it's not very useful but makes a point plus equal one and now I can make an instance of my counter yeah and if I say now is this is a callable callable it has means true because when you call a function if I call my function, a, b, four and five, three and four, yeah, then actually what Python does, Python just say give me this call, yeah, and it just calls this one, exactly the same, so you don't have to do this, the function is doing this, but then you use the parentheses, Python is using this special method call, that's operator overloading and this is a nice thing because now our instances is callable if I call you now I can look at my instance is zero and if I call my instance and I look at the counter again then you see the counter went up one and so on so every time now we have a function with a state it's not the functions instance for class and now we have a state so that's what I would I would I recommend you could add attributes to an a real function but maybe using this approaches cleaner because when you have instance you expect that you change self something, the function should be preferably what's called the pure function. The function doesn't change anything from outside. Yeah, so Python doesn't have a really pure function because you can always impurify it if you like, but you could strive to write something that's as close to the pure function as possible. Therefore, when you change something I would always recommend use class. When you say class are designed that instances change attributes function of the sign to don't do don't change the outside world yeah so now we have a callable and now we can rewrite we can rewrite our I will say here as a class say so usually you capitalize the class but it would be the same thing in this case you might even consider using lowercase for a class but let's go with the big see the difference so we have an in it as always and I do have my text which I store here and this now takes away a level of indentation and this is yeah so if you run this to pilot with McCabe then taking a very level and indication reduces your complexity you get a better score if you're running for score that would be one way improving your score but in still it might also make it better readable because I do something like this which you always do and I define my call and I do all my business with the printing right here self and I put the function yeah and now I should be able to actually do all this safe thing from from in here so if I go here this would be exactly the same now I should be able to use this one and then you might not one you can call it say so whatever so so this should be okay and I recall it should be fine if I'm not mistaking and of course is a self text here text that's only difference yeah now I have this one and I have a problem indentation problem. It can consist of a notation, a copy-paste problem. And now I can use my capitalized, say, to show the difference, say, hello, to my function, So, did so many times, I should get it, and hopefully now this does the trick, yeah? Now it's the same thing, and now you see we have, and this is pretty easy because you only have one function you return, you don't have this def, def, def, you only have this stuff in here and you have normal methods from classes you know anyway so that's nothing not much new here same thing and maybe better most of the time you see examples with this nested functions but it's the same thing because Python just wants a callable and it doesn't matter how you create a callable so if you feel like you can also fiddle around with lambda which is really not easy of course lambda also creates functions if you want to do but usually using a def with a doc string it's much nicer than doing something lambda especially when you reuse things yeah so this is a concept a different concept Pythons operator overloading the parentheses if you want to call them operators of operator of length you can also overload bit and functions like Lang also the same principle and now you can use a class and instance to do this here. Good. Any questions about this class thing? Anybody totally new to this standard concept? This is two leading and trailing underscores? Yeah, so if you see init, init is a special one because this one, when I make an instance at this place, the init is called. The init is just a special method which is used a lot and And this syntax, this is to leading and trailing underscore, is reserved for Python interns. Yeah, there was a question? I guess this could also help us with our previous problem where it would be evaluated, the function would be evaluated at definition time, and here we could modify this, say, object later on. Yeah. There was a question about this definition time and call time. This would be exactly the same. You can apply exactly the same here. So it depends when you want to do it. so so I would need to try out myself and have different ones but it these method everything will be defined but they won't be called yet at the time you will see this you can import it but then you call the function you get the name out this is not there because Python doesn't check it only when you execute but the definitions will be done already this it's the same thing because the class doesn't doesn't have a namespace class has no in or namespace and if you have this local global bit in has no clause in there so it doesn't it's not different I think but we can try this out and see how about X impact it has but it should be the same so let's look at some examples I have some more examples which now use parameterized decorators and this is a called arc check, so now you have type hints and everything, but you can still use a decorator to check types of your function. You could extend it, not only checking types, but checking values that are between 0 and 100, whatever you would like to do, it's totally up to you. Here I restrict myself to position arguments, so now I have this back to my three level approach. The outer level takes an unspecified number of argument types, I will explain in a second, then they have this check function which takes a function and then we have the wrapped function that's actually doing the work and see these restrictors other to just arcs no keyword arcs here would make everything more complex it's possible but I just left it out to make easy so I have two checks the first one is at the number of arcs and arc types has to be the same if not I give out an exception yeah so use f strings here but it's fine so the old way doing it maybe just make a message and raise a tape power and then it goes through a log step between args and archetypes with zip and arg always needs to be an instance of the archetype so this is like int and this would be a number one and you check number one is type of int everything's fine if this is a number one and this is float then you might get an exception so it's a string there's some other type and you get an exception it's not the instance yeah and I just give out an error message to just say okay I expected this type but I get this type so you could get more fancy there that's fine and then I do my function call and my return three deaths three returns here I restrict myself to to position arguments. So this add before, you can call this anything that works with plus. You can also work with strings. If I now use my arc check decorator, which I need to, I think, execute first before I can use here. No? What's the name? Check. It's called checked. Yeah. Check. So now if I define my function, and here I have to actually, I forgot something. I have to give it a, say I want only allow two ints, and then I can only use ints. Yeah? And now if I use my function with ints, everything is fine if I use something that's not an int then it complains the type error because expected int int but got int float yeah so there's something you can do and of course you not restrict onto types you can do any kind of check because it cannot be positive numbers or can only be dates that are not in the past or whatever you want to yes we have to just improve your checks there and you can restrict something. Yeah. Any questions? So this new function where you can indicate the type of argument, is that something the same, or is it the same with, for example, like depth, s, or like if you define argument, name of argument, Again, it was a question about the argument times, but I didn't understand the question. Yeah, this, this would, this would be type hints there, but they wouldn't be enforced at runtime on this user's special library. This would be a new way of doing it. Here I have a decorator and it has nothing to do with type hints. So type hints are just type hints, which usually are only used at development time to give you an error message. Python, the vanilla Python interpreter still runs even if it's totally wrong. It doesn't do anything about it. And here, you enforce it at runtime. There's a different concept. You could also try to fiddle in type hints and inspect the type hints, and then use the type hints with the decorator and do the type hints as possible. Here, I put it explicitly in front. This would work with Python, too. So, for instance, you don't need type hints for this. It's just an example. And you could use a decorator that's going in and inspect the type hints and does something with the type hints. I think there's tools out there that do exactly this. It's just an example of what you can do. And it's not production code if you want to have something that would be more sophisticated in doing something. But it's up to you. You can always use this type hinting stuff. Good. There's one example, argument checking. Now, let's go and do a little bit different example. This is called registering. So I can register something, and it can be interesting. So we didn't have this before. I have a registry. Yeah, so if you work on Windows, you might know the registry. And this is just a dictionary in this case, just some place where you put something in here in the global namespace. There could be other namespaces. And I have two versions. I have a register at call, and do have a register at death so this at death is a bit different and let's try this first and then look how this works so first if I look at my registry it's an empty dictionary nothing there so and if I say add register at call and I can also give it a say this is simple give it some kind of strings and category or something and I have my great add function so so if I do this and I look at my registry again then you will see it's still empty but if I call my function then my registry will change and will now have something that has a list, just the function will be added to a list. I don't know if it's useful, it's just something happens, see, and if I call this again, this list will get bigger and bigger, so the arguments don't matter here, and if I do this and do this again, then you see my list gets longer and longer with the same function in this case, because they're all rational symbols. So what I did here, I just used this global dictionary, which could be also improved if you want, you can put in the class, modify all the things I never have my dev dev dev approach you know already this is a name my simple and I just say I use the set default here so I don't have to check if it's empty up make a new one all about their pent so it's a just typically use of set default as a dictionary method and then I just put this in here I put the funk append the function and I just return this one that's everything the same only this is different and this happens at call time every time I call something changes okay so and you can also now my registry it's a dictionary and I just can use clear to empty this thing and now my registry is clear so now I use what a name it's forgotten register register at definition time register at death and at the 20th time i define my function here and now if i look at my registry you see my registry didn't change i i need to i need i need to call my number I didn't execute it. Oh, I executed it. What happened? I need to parameterize it. Thank you very much. Yeah, so I have to use it the right way. And if you do it correctly, it works. You see, now this is different. We didn't have most of the time you change the runtime behavior of a function, but you can also do something if somebody defines a function. That's possible. now we have my registration and you have this and if I call my function nothing changes I can call my function and this doesn't change the registry at all nothing happens anymore I'm too late yeah so now that's different you see I lost the level of indentation because I just do this here and I return the function unchanged we did this before with another example I just return the function unchanged do nothing with the function and just do some side effect just this case I register somewhere there's not as common but it's possible and you could write your own plug-in system if you want so you the user writes a function and that they don't have to put it in some directory important it's just decorated and then you decorate I have it I can do something is it yeah it would be one way to do some kind of plug-in style kind of thing if you want to call whatever you want to call it but this would be a way register a function to somewhere yeah good okay let's continue you only have 50 minutes so this is my registry example the next thing is class decorators so far we we decorated the functions and it's been around for a long time class decorators are newer and they are nice because class decorators can solve quite a few problems that you before the time you had to go and write meta class which is something take another two hours the meta classes fortunately nowadays meta classes are not really necessary anymore Python 3.8 so got some some new things so you don't actually meta the users meta class is much much less than used to be and class decorators are nice they can be used and instead of some things you did with meta classes and they're actually easier they're easier than function decorators because there's no nesting I just get the class a class I write a function to cut a class and returns a class and in between you can do something and here you could just access attributes you could also add something that's possible something like this now I use this thing and mark my class and the class doesn't do anything so the class would be empty and if I make instance of my actually I can just look at the class it's a class attribute and I see the new is there and I didn't you don't you don't see when you define a class you don't see that's a new and because I just get the class yeah which is a in this case and I just add this attribute here and then I get it back so this is something would be you can modify it but you can also just access it so and the class has a dictionary and you can access the dict yeah you might know this one dict so this is something you shouldn't reprogram this but inspect me it's fine and then you could see all the attributes in here which are the methods and other things and you could inspect and do something and you can if you don't like it you say okay I don't like this class but don't do it then just raise exception stop the whole thing yeah do class decorators are easier and they're very powerful because you can do a few things there's one thing is they wouldn't work for this inheritance so they don't get inherited you have to edit every class whereas if you fill around with meta classes then inheritance actually is one way to use it anyways because usually you want to hide the meta class on the user so you make class people inherit from and then it would work with inheritance so then that's some small problem there's another solution this but very often you don't need to consider inheritance so there's another example use case. The use case would be for instance assert something, assert fluid. It's my use case here. So I get my class and I just say okay if I say assert that the class dot temperature is between zero and zero and zero less equals zero right so this is anybody's physicist we don't consider air pressure on anything it's it's 0 0 and 100 so just no salt content of the water I think so did that it could change this one and don't forget the return otherwise it's gonna go to work and now I can say okay I have my assort fluid and I say okay this is a class water and I have a temperature so when I just copy this here so I don't miss type temperature 10 minutes left yeah so I times 10 minutes and no temperature and say 25 everybody's in calcelsus here no Fahrenheit great okay then it works yeah if I do the same thing and you say ice and I say minus 25 then it's not going to work hopefully when I do get an exception at definition time so I cannot even define my class I I do get an exception, and usually you get a problem in the instantia cloud, you get a problem here. I cannot do this because I do AssortFluid. So be aware, Assort can be turned off. When you start Python, this minus this dash C, dash O for optimization, then Assort will be turned off. But for this purpose, it's okay. And then I get this exception here. The same thing, if you go over 100 degrees Celsius, So we assume Celsius here, then you get an exception. This would be a verification here. So I have this. Locked, crashed, registered. Good. I have another example in my PDF, and I don't have the source code here. Okay, let's just quickly look at the PDF then. I will upload the source code, I'm not sure why the source code is not here, okay. So there's another example here, we can have a quick look at time. So this one, and somehow I'm missing the source code, but just have a look at it. So this is a little bit more involved, but you can do this. So now I say, okay, I would like to check, allow method names that only the maximum length of 30 or whatever I want to give. So I write this, the thing, def def, you see I specify the maximum length of 30. I take the class name in. I go through the dictionary, items, I get the name of the object, if it's callable and if the length is more than whatever I specify as the next length, I don't like it, I generate exception and raise a name error with this exception method here, and then I return the class and I do this, I don't know why the source code is not there, some inclusion problem, but it's in here, and then if you use it here and you define the method that's a little too long, yeah, here, then it's an exception because the only thing I inspect the dict and I can do this dict. This would be a typical example for a metaclass, yeah, but you don't in a metaclass use it, decorate on the only thing, only logic goes in here, inspecting this dictionary and do, if you don't like something, in this case the name is too long, we just show an exception, you stop the whole thing. So you can write a lot of these tools that do all kind of thing. Instead of the links you could also check if it's the naming conveys the snake case or not snake case whatever so I'm like regular expressions and they are do whatever I like that's totally possible and can be done with a class decorator good this is a thing then we have exercises but we are pretty much out of time so please do the exercises yourself just this is the solution we have now it is time measurement make it parameterized and repeat the measurement many times and divide the time by the Repetition to get a better measurement. You see the time will go down Yeah, and then you can also use a true and false to turn this measurement on and off globally. We did this business thing and The last one would be using a class decorator to work with classes Good Yes, we have questions we have five minutes for questions You can also add them on Slido. I should be able to see them. You already added one on Slido and I didn't see it. Let me know and just raise your hand. I'm still getting familiar with it. But we have one. What would be some anti-patterns for decorators? I think you hinted potentially at that with the pure functions, maybe. Yes. Anti-patterns, so the pure function and the class could be one. So whenever you want to change something, then I would prefer, that's my preference, you prefer a class instead of a function. But let's say, I think least use cache, uses functions and still adds attributes to functions which could be useful to invalidate the cache. Anti-pattern, so I don't have a really good anti-pattern in mind, but the thing is you shouldn't surprise the user. It should be something that the people expect, whatever this means. Yeah, so don't surprise them and do some side effects that take very long or something like this they don't expect. That's usually something you don't wanna do. Thank you. Any other questions? Yep, I'm running over to you. I think I chose a wrong place in Slido. Yeah, anyway, so I wanted to ask when the registration failed because you forgot to add the text, the ,, there was no exception raised. So is there some mechanism of hiding exceptions happening? Or did something else happen that I didn't understand? Nothing happened because it still worked the way I did it. So I have to check because it still returns something, and it just calls something, and there's no exception. OK. could you could put it in and say okay I want to have it has to be a text and so on so you can so I would need to try this out myself but you see it it's of course it's just a function takes a function thing and it's just works and just returned none somewhere whatever I did I'm sure and it's still work some function and work so that can happen but yesterday we had a testing write your test and I just wanted to ask if there is some additional logic to exception you have a link happening, but it's not so because it's, it's not, it's, it's just did what you, what you want to do. It's maybe not what you intended, but it's just did what you programmed. Yeah. Hey, um, my question is, when would you think about switching to the class and callable pattern instead of just using the wrap function? When do you think, oh wait, now I need to switch from wrap to a class in practice? Yeah, so it's up to you. You can always use classes for everything. You don't have to use the functions. If you don't like this nested thing and you get totally confused, use classes. And the classes, you could have more methods and you can do a lot of more things with classes then. You can do the same with functions, but of course you can always add attributes to function, but the class instance has attributes already. So it's probably more familiar with doing this and adding attributes to functions. But all the examples traditionally are always functions that return functions, but you can everything turn into the class. Because in the end, you just want it to callable. And how you produce it, it's up to you. So whatever you prefer, when you feel more comfortable, use it. So if you come from function programming, you have no problem using functions that return functions. If you're more procedural, then the class is maybe easier to understand. Any other questions? Time for one more. yeah i unfortunately i don't see them in the slido app that's my error so do you want to read it out do you mind thank you can you mention a few practical uses of decorators on the typical data analysis science pipeline from discord especially for data analysis you can anything that's that's cross-cutting concerns if you want to measure how long things take then we just did it this would be maybe something you can analyze your data or anything else that you did you want to do all the time that has nothing to do with this specific problem that's usually a use case for decorators so introducing something so if you if you use Scython only compile the stuff to Scython then you can use decorators from Scython this would be an example I can think of otherwise the data analyzers is not that special it's just programming so you will have the same programming problem other people have also I think thanks for that question in discourse hopefully that answered the question and apologies for my slip up with slider that was me Mike but I would like to just say huge thanks to you maybe folks can come if they have additional questions to the PSP stand yes you come to set up I can set around here you can come to me I will be at the stand of the passions or favor bundle so I'm sitting there during the breaks at least yeah I also have leaflets here so if you want to get some you pass an Academy you can pick one up so you can you can also contact me my email is on the PDF so and I will also I forgotten I do have solutions uploaded solutions so you have them so this is a PDF with all the solutions together plus the files so what I showed here and so please do the exercises first before they do the solutions that's that's why I'd like to give them after the fact but we haven't managed to do the exercises yet thanks folks please join me