Stupid Things I've Done With Python

Speaker 1 [00:03]

Hi, everybody. Thank you for coming. I can't believe all of you came to this nonsense talk. And hello to everybody at home. Yeah, the fact that I did used to write medical software is going to be worrying to some of you in half an hour's time. You're never going to go to hospital again. So hi, my name is Mark Smith, although I'm generally known as GD2K online. I'm a developer advocate for a database company you may have heard of called MongoDB. And I've been asked to make something very clear. MongoDB, my employer, does not use or endorse any of the techniques I am about to demonstrate in this presentation. But furthermore, and more importantly than that, I do not use or endorse any of the techniques that I'm about to demonstrate in this presentation. Any harm you cause to your physical person or to your career by using any of these techniques are entirely at your own discretion. On every computer for the past 20 years, I have created a folder called Stupid Python Tricks. And I use it to create small scripts that try out various different features of Python that wouldn't get past a code review because what I've done is too silly or weird. And over the years, I've lost some of that code, which makes me sad. These days, I keep it in a GitHub repo so I don't lose it. Every so often, I take some of that code out, I dust it off, and I show it to people to see how they respond. And if you feel the need to respond, please feel free to tweet at me at gd2k. This talk is in two halves. There are two stupid tricks that I'm going to show you, each using a handful of different techniques. In each half, I will try to describe the stupid thing that I'm trying to do, the strange piece of code that I would like to work. And then I'm going to try and illustrate the thought process that I went through to get this weird code to work. So in a moment, I'm going to introduce you to a library I wrote a few years ago called Ish. I wrote it for a lightning talk as a joke a few years ago for Europython. It was purely a joke, but it's been quite popular and has now had multiple contributors to the GitHub repo. Which is worrying. So let's start by talking about types and equality in Python. I quite like types. Python is very forgiving with them generally. For example, here is how you compare two Boolean values. I'm comparing the same thing on either side because they're literally the same thing. They're considered to be equal to each other, and so this expression returns true. As you'd expect, if you try to compare false to true, those are not considered semantically equal, and so this expression resolves to false. And for historical reasons, one is equal to true, and zero is equal to false. But here's where it starts to get quite interesting. If I compare a string containing the word true to the Boolean true, I get a true. Those are considered to be equal, which is excellent. That's really handy. The only problem is that the word false also resolves to true. And this really isn't what you want in an expressive programming language like Python. So let's try and fix this. So what I want on the right-hand side here is not not true, I want something that's more like true-ish. So this is what I'm going to show you how to build. I want to be able to suffix the word ish to either the word true or false, and I want to be able to compare that to different types and get something that actually makes sense rather than this bizarre thing where false equals true. So I know there's lots of non-native English speakers here. Ish is a kind of slang suffix that you use in English to say that something's not exactly true, it's just kind of true. It's like when you've got a blue-green colour and you don't want to have a fight with your girlfriend, you go, yeah, I guess it is blue-ish. So why? Why would I want to do this thing? You might ask yourself, and I would say if you're asking yourself that question, you are probably in the wrong talk. There is a PyLadies panel next door that I expect is very good. In all honesty, that's exactly where I would be if I wasn't giving this talk to you all. So just assuming you're still with me, let's take this expression that I just showed you, this is what I want to do, and I'm just going to take one step backwards and I'm going to remove that hyphen on the right-hand side, and I know that's very comfortable to German speakers here that are quite happy taking multiple words and smooshing them together. That's a new word I've invented. So we'll create this value truish that we can compare to strings like true or false and get something back that we feel matches those things. So we need to change the way equality works. So let me show you what happens when you compare two values in Python. So like this. So here I'm comparing the string true to my truish value that I haven't shown you how to create yet. Now, what happens is behind the scenes, Python calls this magic method called EQ. Well, I call it EQ, but sometimes people call these Dunder methods because they've got double underscore at the start and the end. Sometimes people call them double underscore methods or magic methods and things like that. These are all a bit of a mouthful. I'm just going to call it EQ. Now, this is what Python calls. It calls the EQ method on the left-hand argument and passes the right-hand argument as the one parameter to the function call. This returns not implemented because the left-hand argument is a string, and a string doesn't know what on earth a true ish is. It's never encountered did before. Strings are part of core Python, obviously, but trueish isn't. If it returns not implemented, which basically means I don't know what to do with this thing, Python has another go. It calls the same method again, but this time it calls it on the right-hand operand and passes the left-hand value in as its one argument. So you can see it's just doing it the opposite way around. If you're relatively new to Python, you may be thinking, How does he know this? And the answer to that question is it's described in the core Python documentation. The developers of Python want you to know this. There's this amazing document called the Python data model that describes so much cool stuff you can do with classes and different special methods and values. So many of the stupid Python tricks that I've developed use techniques that are described in this document. So this is entirely valid to do in production code. So here's a class I came up with called TruishClass, and I've implemented this magic EQ method, and the right-hand value is pulled, in this case the left-hand value is passed in as a parameter, and I just compare that, I just look for it in a set of known true values. So in here I've got what I mentioned earlier, like the true and the one, but then I've got some strings that just contain values that I consider to be kind of trueish. If none of those are matched, sorry, it returns true if it finds one of those. If none of those are matched, then it just tries to find a falsish value, like something a bit falsy. And then if it finds one of those, then it knows that what was passed in was a false type value, and so it returns false. If it can't find anything in either of those sets, then I do defensive programming and I raise a maybe, because I I thought that would be funny. Technically, I think I should return not implemented. But this isn't meant to be correct. So, let's try it out. Here's my oh, sorry, there's the custom error there. So, because most special methods only work on instances, I'm going to actually instantiate that class to create a truish. And then I can compare that to true, which returns true. I can compare it to the string true, which returns true. I mean, this This is also valid for the true boolean type. But we can also compare it to false. We get false. And the string false. We get false. So this is a much better version of true than the version that you're used to. Finally, we can compare it to some nonsense, like comparing it to the string lemons. And it raises a maybe, which is awesome. So I'm not quite done. The way I wanted this to work was hyphenated. So we need to get that to work. We need to know how and you notice that's not a hyphen, that's a subtraction operator. So we need to change the way now that subtraction works in Python. And this works in a kind of similar way to equality. So when you subtract a variable from another, Python behind the scenes calls a magic method called sub. And if this returns not implemented, which like before, Python is going to do, because true has never encountered an ish before. I haven't implemented ish, but again, Python has another go, but the other way around. Only this time, it can't call sub with the arguments swapped, because it doesn't make any sense. If you swap the arguments in a subtraction, and then try and do it, you're going to end up with a negative value compared to what you want, so instead it calls this special method called rsub, which is just sub with the arguments swapped. So now it's calling it on my ish object. And so I can implement this method myself. It's relatively straightforward. It basically says if the left-hand argument is true, then I want to return this truish object that you've seen before. And if the argument is false, then I'm going to return a falsish. I'm not going to show you how to implement a falsish because it's the same as truish but returns the opposite values. That's an exercise for for yourselves in your own time. Here, I've instantiated the ish class for the same reasons as before, and now I can subtract that from a Boolean like true, and you can see that I get my true ish class object, and so we can now start comparing these to various strings. The string true is true ish, the string false is actually false ish, and I can confirm that down below. So all the code you've seen here has been extracted and simplified from this stupid library I wrote, which is on GitHub called Ish. If you enjoyed this, I recommend you check it out. It does a few things that this code doesn't. For example, it supports slang and various words from different languages. It can also do fuzzy numeric comparisons, it understands the 1.51. Thank you. And I've recently removed the code that does this from the code base just for simplification. But if you look through the history, you will find it. My friend Jeffrey French added a neural network to the library that allowed you to compare image data containing faces of people to emotions like happy and sad and angry and it would tell you whether the expressions of the people in the photo matched the emotion that you passed it. So that was kind of fun. The main thing I've been getting to in this part of the talk is that these special or Dunder methods and things are ways of changing the way Python behaves. Sometimes in quite subtle ways. They're a way of opening up a hatch into the Python language and just changing things. So what have we covered in this half? I've shown you how to change the way that equality works. I've shown you how to change the way that subtraction works, so that you can implement hyphens as part of the Python language. And I've also shown you, oh, I should point out that this is a technique that ultimately PathLib does to concatenate paths with the forward slash, which is division, right? But that's just been overridden by the path type. And I've told you about the Python data model document, which if you haven't seen before, you really must go and read it. It's absolutely awesome. So now we come to the second half of my talk, which I have called fun with math. If you want a reasonably accurate version of pi, you can access it through the math module. Here's what it looks like. Now, for all the mathematicians in the audience, you'll know that that is not an exact value for pi. everybody knows the correct value of pi is 3. Fortunately, you can fix the math module's behaviour. If you import math, you can just assign a new value to pi, and that works. Elsewhere in your code, somebody else, one of the developers in your team can import math as they normally would, and when they print out math.pi, they will get the value 3, because that is what you put in the global module of pi, until you shut down the virtual machine or you do something like reload the math module. Now, I have found in codebases that I've done this in that for some reason my fellow developers get kind of upset. So I found it's better to sneak in the correct value of pi over time. So this is what I really want to achieve. I want to import math, and then every time I print out math.pi, every time I use math.pi, I want to get a different value from the previous time. The idea is that the value is different every time you access math.py, and this way I can program it so that math.py gets closer and closer to the true value of 3 over time, and nobody knows really where it came from. And you may ask yourself why, why, Mark, do you want to do this thing? And I would say, well, we're about halfway through the talk, so you've already invested the time. Let's just find out. Well, the code could look a bit like this. So I can import math. I have given an alias as a name, and I stored the initial value of pi in there so I can refer to it later on and, in fact, modify it using this function. So from now on, if I use this pi function to access pi, in this case, it's just incrementing the value of pi a little bit because it's slightly simpler than the code for just subtracting to get towards 3. And then if we use this code, it looks a bit like this. And so it's kind of what I wanted. I can import this module, which I saved in my math, and every time I execute the pi function, I get a different value. But these are a problem. These are a problem. Pi is not a function. Pi is an attribute. So, in some way, I need it to execute this function, but only when you access it. So, it needs to look like an attribute, but behave like a function. Unfortunately, Python has a construct for this. It's the property decorator. So, you would think that we were sorted, but, unfortunately, property does not work on functions. Property only works on methods. So let's go with this. I've now implemented a class, and I've used my property decorator. Let me show you what this looks like when you actually use it. It's not too bad. You instantiate your math object at the start, and then every time you access pi, it's calling a function behind the scenes to slightly change the value and then return it. So, I mean, it's pretty close to what we want, but now this is a problem, right? So we need it to be a module, we need it to be something that's imported. So import math gives us something that has this pi attribute. Let me introduce you to the sysmodules attribute. You may have used this before. When you import any module in Python, it stuffs it into this dictionary. It gives it the name of the module, so in this case, math is the key, and then the actual module that you would be accessing in your code is the value that's put in there. And it's just kind of a cache for all the loaded code in your Python VM. So every time you load a module, it's added to this dict. And to demonstrate what I mean, if I look up math in this module, it's not there. Because this VM is clean. I haven't imported math yet. If I now import math, and then I look in the sysmodules dict, you can find the module. And I can compare what's in sysmodules to the thing that I imported using import math. And those are the same thing. And this is where I get to one of the reasons I like writing stupid code in Python. It's made easier by the fact that almost everything in Python is mutable. And what that means is that you can change it. So let's manually modify sysmodules and pretend that the code has already imported math. So I can take my instance and I can just push it into my dictionary of loaded code. And then later on, when anybody runs import math, it looks in the dict first and it says, oh, that's already there. I don't need to import math. I don't need to load any Python code. I will just return the thing that's in my cache of modules. And so that's what you get is my new math object. So it's kind of, you know, this is a helpful technique. I can put anything I want into this dictionary. It's duck typing. It's just awesome. So now Now I import math. It's in sys module, so that's what I get. And you don't need to do this in the client code. You can do this inside the module itself. So I can just stick this at the end of the module. It's created the class, and then it just stuffs it into the modules at the end of the load, and then I don't need to hide this in the sort of client code base anymore. So now I import math helper, and then I import math, and now every time I access pi, I get a slightly different value. It's getting gradually more correct, more towards the correct value of pi, which is 3. we're done. You would think, but wait, no! No, we're not quite done. Now, there's a problem here which is that math doesn't just contain pi. It also contains a function called seal. And if one of my fellow developers tries to access seal and finds it's not in the math module, they may find out what I've done behind the scenes, that I've kind of changed the value of pi. So I need to go back into my class. And it's relatively easy. I just need to implement seal. I'm not even doing the calculation. I'm just passing it on to the real math module behind the scenes and returning the result of calling seal on that function. So that's nice and easy. So you'd think that now I'm done. But I'm not done. Because now here's the problem. I loaded up the docs for the math module. And it's really big. Did you know that mathematicians have discovered so much stuff? It just goes on and on and on. It was a surprise to me. So I started thinking about implementing a stub for every single thing inside the math module, and it started to get a little bit boring. And then I started to, you know, I opened up my bag of dirty tricks, and I started thinking about loading the math module and then iterating through everything inside it and just making a copy of it into my math class. And then I thought, well, it's a bit tedious, and maybe I need to take a step back and think about what's going on here. So let's look at a call of math.seal. There's actually two things going on in this line. I think some more beginning meaning programmers don't necessarily understand this, but the first thing that's happening is an attribute lookup. The SEAL attribute is looked up on the math module, and then you call it, because it's a function. You happen to know it's callable. So what I need to do here is change what's happening here. So when you're asking for the SEAL, in this case a method, I need to just pass that through to the real math module behind the scenes. And there's a way to do that in Python using another magic method called getAtter. The this works is when you have a value and you're looking up an attribute on it, it will look on that class definition and then it will go up the class hierarchy and try and find a definition. And if it can't find it, then it will call get atter on the lowest level in the hierarchy. So this is like a fallback for just sort of finding a value if there isn't really one defined. There's get attribute, which is the same as this but with the whole get attribute in there that's called first, so you can override method attribute lookup before it really gets into the Python system, but that's it's even more crazy than this. So if I try to access lemons on an object that doesn't have a lemons attribute, behind the scenes, it calls the get at a Dunder method with the name of the attribute that is looking up. So I need to implement this method on my new math class, and this is what it looks Like this is slightly complicated to describe verbally because I've implemented the dunder get at a method, and it uses the get at a helper function, which is a different thing and doesn't have double underscores. That's just a way of looking up an attribute by name on any value in Python. So here I pass it the math module, which is something that I want to look the, let's say, seal up in, and then it passes it the name of the thing that I want to look up, and then that just gets returned. And if it's callable, somebody can call it, but we've handled the attribute lookup using this get at a technique. There's some other stuff I should really do in this math module if I really want to make it a proper swap in replacement for the original math module. I should really copy over the docs. I should change repper so it looks like a real module rather than looking like a class if you print it out. I will leave that as an exercise for you if you can start with this code and see how far you get. So what have I covered in this half of the talk? Well, Firstly, modules and objects are basically interchangeable. You have to manually modify the modules dict if you want to swap in an object for a module. Facebook uses this technique in one of their core libraries, by the way. I found out a little while ago. Modules can't use the property decorator, but objects can, and this is a good reason for actually doing the thing that I've described. If you put an object where people expect a module to be, you can get behavior that's unexpected. Almost everything in Python is mutable, except some things that are written in C, and there are sometimes still ways to change those, or you can wrap them. It's just a new challenge in your writing stupid code. So, finally, you discovered how to magically create any attribute you like on request. It could be quite fun to use this to implement kind of spelling correction where you could call methods with a slightly misspelled name, and it would still call the correct function. There's a bunch of stuff that you could do that would be quite fun. So, this code, or something like it, is also on GitHub in a project I call stupid Python tricks, which contains a bunch of silly stuff, including some more advanced tricks like using meta classes and directly implementing the descriptor protocol. But most of it uses the techniques I've covered today, which is modifying things you're not supposed to modify and using Dunder values in ways they weren't designed for. You don't have to be a Python expert to write weird and wonderful code. You just have to have some time and the urge to be creative. If you haven't written stupid Python code before on purpose, then this is just the start of your journey. There are lots of special Dunder methods and variables you can play around with. Read the Python data model to find out most of them along with what they do. If you don't know what the descriptor protocol is, you should go and find out. It's apart from anything else, from a purely sensible perspective, it describes the way attributes are looked up on objects. It explains how methods work, including properties, class method, and static method decorators. So it's like there's some real behind the scenes kind of under the hood stuff there. Eval is a great opportunity for having some weird stuff happen. You can build up a string of Python code at runtime and compile it into executable code to be run again and again. I really wanted to cover metaclasses in this talk, but it just isn't possible in a 30-minute talk, and I don't usually promote a talk that I have given in the past, but I happen to have given a 70-minute talk on metaclasses at PyCon Australia a few years ago that I think provides a pretty good introduction to how they work. If you have ever wanted to implement a hierarchy of classes but have method lookup happen alphabetically instead of using the hierarchy of classes, that is the video to go and watch. The dis package allows you to disassemble Python code, possibly play with it, and then reassemble it back into executable code. It's a pretty advanced technique, but it did allow my friend Sebastian to add a go-to command in Python to allow you to jump between various bits of bytecode. And finally, the way that Python finds and loads code is all encapsulated in the import lib library. If you have ever wanted to import code directly that's written in a language other than Python, the import lib library is probably a good place to start. There are many opportunities to write terrible Python code on purpose. So let me try to actually answer why I do this sort of thing. So, for me, it's an opportunity, as I said before, to try out a new feature of the language that I might not yet have a use case for. It's kind of a reason to properly dig into a feature without just kind of reading the documentation and thinking I understand it. But it's also a puzzle. So there's two ways I found to approach these kinds of things. I can either decide up front what I want to achieve and then use the techniques I know to try and make that happen, or you can dive into a new feature of the language or a feature you don't know very well and then try and see what inspiration strikes you as to ways you can push that feature towards the limits. I have friends online that I swap these techniques with, and I have met them at conferences when I have sometimes presented unusual code like this. So, it's a great way to meet new, smart, and interesting slightly crazy people. But mainly, I do it for the reaction I get when I show someone the code or I post it on Twitter. I do it because I just love it when I get the response. But why? Thank you very much for listening. And I'm sorry for wasting your time. For more more stupid Python tricks, feel free to follow me on 2D2K. Thank you. We have got some time for questions, if anyone has questions. But not why. We have already answered that three times. Let me check. I'm not seeing any questions here, so if you can raise your hands and I'll just come to you, I guess. Okay. Hello. There you go. Hello. Hi. Yeah, small question. Why you don't override P function? Why you want to override Small question, why you don't overwrite p function, why you want to overwrite with format function? P is overwritable. Yes, but it's not, when you say you mean overwritable, yes, but it would then not be changing each time it was accessed. So at some point you need You can decorate it, but first you need to replace it with an object, which is what I did. Yes, and you can in this model decorate this function. Only one function. You can't decorate a function. The property decorator doesn't work with functions, it only works with methods. You can get into this property. Okay, we can talk about it later. Any more questions? I think our last one here. I think you raised your hand first. Let's see if we have time for more. Hey, Mark. Thank you for your talk. Hi. No problem. Have you ever done clever things in Python? things with Python. No, I'm not actually smart enough to do clever things with Python. Thank you. This is intriguing because I've never had a chance to do Q&A with this talk before. I was wondering is there a reason it wouldn't work to use module level for the PyHack, maybe even patched into the existing math module? Again, that is part of the class data model, so I don't think it would work with a module. In 3.7, there is a module level. I knew they made some changes. It doesn't predate 3.7. I should probably look at that. There is now 3.10, so you can have even more fun. 3.10 is fantastic. Last question. Yes. Does it work with multiprocessing? With multiprocessing? Yes, you can write stupid code that works with multiprocessing.