There Are Python 2 Relics in Your Code

Speaker 1 [00:03]

Hello to Berlin. My name is Miroslav Sredivy. I wish you a wonderful, nice day here from the sunny Vienna. I am standing here in our beautiful office of Treport, together with a few colleagues that are behind the camera, and they told me we are going to attend your talk as long as you find some nice new colleagues for us. So I hope that this will work out and that I will be able to welcome some of you here in Vienna. There is one main question of my talk. Should we return to Python 2? And the answer is no, and that I'm done. Or no, I'm going to tell a bit more about it. Because last year, or even it was two years ago, I posted this tweet at midnight in the first minute of the new year, 2020. asking are you still using python 2 there were some likes some retweets but no real answer like yes or no and hey don't tell anyone there are still companies using python 2 but python 2 is in the development of python 2 is already over and we should now concentrate on the brave new world on python 3 and everything that comes with it very probably i hope so You have already all migrated your projects to Python 3, didn't you? Because, as I told you, Python 2 is bad. Here you see the current year. We are in April 2022. And Python 2.7 has died several years ago. 3.4, 3.5, it's over. 3.6 is already gone as well. And now the current development is concentrated on Python 3.10 and 3.9. And there are some security fixes, maintenance of 3.7 and 3.8. Which means that if you do some project in Python, the question is, which Python version should you support? In software development, in this version development, there is a food chain. At the top are all the applications, the final applications that are using all other libraries. And below, there are the libraries, some specific libraries, and at the bottom, there are general libraries. Which means that if you do an application that is used by someone, you can safely go for Python 3.10. But if you write a library that is used by other applications, then you have to support a wider range of versions, preferably two or more versions, because you want to allow those above you in the food chain to migrate slowly from one of your versions to the next version. In this case, it means that if your library supports 7 to 310, you have everyone has plenty of time to migrate their code that is above you in the food chain to a further version and then get rid of all the versions and with your library 3.7 to 3.10 you just introduce the support for 3.11 drop support for 3.7 introduce support for 3.12 and so on and so it moves but always in a wider range and at the top of the food chain your application just jumps one version after another according to what it is supported from the button. Okay, so let's say that you are really now on Python 3 and that you have already fixed all this old Python 2 stuff, like the print statement, like the iteration of our dictionaries, iter keys, iter values, and so on. And also, of course, instead of xrange, we are always using range, zip, map, filter, and so on. Everything should be in Python 3. also you have very probably already got rid of the six package that allows you to run at the same time code in python 2 and python 3 and all these small switches in the code that work both in python 2 and python 3 now everything runs in python 3 so now you have to make sure that your code runs so because if you want to make any further changes and we are going to make a few of them your code has to be like running and if it is a project on GitHub then just created a nice infrastructure with all the CI-CD workflows, all the tests and the nice configuration, for example, using TOX. The TOX that allows you to run this in several Python versions at the same time. So in this case, we have 3.7 to 3.10. So the configuration is quite easy. And then you need some CI-CD configuration, for example, in GitHub that installs or installs defines all the versions that have to run together with it. Then you need to know what features of the language you want to use. So here, Jürgen, that gave a nice talk right before me, he maintains a very overview, a simple website, a page listing, all the Python versions and all the features that came into it. And which means that if your version or your code should support 3.7 to 3.10, then you can have a look here and see which features it should support and for example if you drop support for older versions you can already remove stuff that was working only in the older versions and that you need the beginning you can replace by more modern code you shouldn't be afraid of touching the code if you are afraid of touching the code then something is wrong if If your code is not well-tested and it is not maintainable, it is not changeable, then something is wrong. Take care of that. Then there will be still plenty of stuff that will remain in the code that will work in Python 3, but that you don't need. So, for example, if your source code is in UTF-8, and it should be in UTF-8, in Python 2 you need this line, but in Python 3 you don't need it because the default is the UTF-8 encoding of the source code, so you can remove it. Also, if your objects or classes inherit from standard object, you just can remove this definition of object. You just define your class dummy without inheritance from object. Also, the super function has some default values, so you don't need to explicitly list them here. Also, if you have some literals that are written in Unicode, in UTF-8, then you can remove safely this U at the beginning of the literal strings. And there's plenty of stuff that you should do, but you don't have to do them manually because there is a very nice tool by Othonis. I upgrade that you can simply install and then just run it over your code base, defining, for example, that you want all the features that are at least Python 3.7. this will modify plenty of stuff in your code automatically but you can also like of course if your code is in the git then you can check the diff afterwards and see whether it is exactly what you wanted but usually these changes are really atomic so they change some line but they don't change something here and here at the same time so you have to check the context it is really all in the context of one line, some tiny formatting, some tiny features. Then you also should make sure that your code looks good. There are tools like Black, Blue, Yap. These are tools that have two functions or two roles. One role is that they make the code of others nicer. And the second is that they make mine code uglier. Of course, you need to have a compromise. So actually, if you don't want to discuss in your team how the code should look like, then just throw it at your code base and it will be fine in general, in average. The best time to introduce Black or something similar was at the beginning of your project. The second best time is now. So do it when you are already migrating to Python 3. And when you are getting rid of Python to Relics, then just introduce it and then the diffs will be much nicer. And also be sure that Flake 8 is happy. or pylint to run some checkers on your code, even introduce them into a pre-commit and make them run every time there is a new commit so that your code looks nice. So, and then there will be plenty of things that won't be updated, fixed by upgrade, that you will have to look at it yourself. And this depends actually on the list of versions that you're supporting of the Python versions, because there are some features that came later. So for example, the F strings came in 3.6, I think. But nowadays, of course, you want to support it. So you should change all your strings, make them nicer and easier to read using the F strings notation. There is one thing that should remain in the old percent formatting are the locks because, well, I don't quite understand why there is nothing like the f-strings or something like that that would be built into the logging module. That would be nice. There are some exceptions to this rule of having the code in Python or in Unicode or UTF-8. Sorry, UTF-8. I have found this somewhere when I fixed some third-party module. There was explicit coding let in one and there were some latin one is all eight eight five nine one uh strings characters and actually to make it work also in utf-8 i had to like rewrite the whole functionality because it was based on the xk values of these strings and their formatting or their conversion from utf-8 so sometimes the change looks a bit more complicated but actually then you also know what you are doing because you are not like blindly changing some simple stuff in your code um if you have utf-8 in your source code you can also write characters that are like beyond ascii so for example my last name you can write it directly using the compose key if you have installed it or set it up on your keyboard and you should um or using the unicode code points which is not so easily readable and the last one is like completely explicit with this backslash uppercase n notation that allows you to write any unicode character explicitly although this is not also not so fluent to read numbers this is something that you will find in your code but now in python 3 this is not useful because math floor that takes a number float number and then goes down to the next lower integer, sail to the next upper integer, and round to the nearest one, they, in Python 2, they returned float. In Python 3, they return integer. So imagine now someone who is a bit younger than us and who started learning Python 3, and they see something like that, they don't understand why it is like that, because it doesn't make sense in Python 3 to take an integer and then convert it again to integer the reason for this was that in python 2 there were integers that were limited by 4 billion or so and then there were floats that went to tend to the power of something much more which means that you were able to you were not able to convert any float to no you were not able to convert all floats or any of those floats into an integer so that's why they kept it at an integer if the number was too high but now integer is unlimited in python so you can have any float to 10 to the power of 300 or whatever what is the limit of float and you can still convert it to an integer so actually you don't need or that makes more sense actually in python 3 to have it converted to integer so just remove this further casting to integer. This is the reason, because in Python, to the maximum integer, okay, it was not 4 billion, it was 9 to the 10 to the power of 15, 16, no, 18. But the float has the maximum of 10 to the power of 300. So now there is no problem. Any float can be converted, or there is an integer next to any float. Another reason to consider integers much more in your code is this in python 2 if you did this so you wanted to calculate the mean arithmetic mean of the number of of integers you calculate the sum divided by the length of the array and this is mathematically wrong because of course the average value should be 3.5 but here it is 3 because python 2 integer divided by integer gave you an integer so this is not what you wanted so what we did usually those who remember python 2 we converted one part another part or both parts with a float or we even multiplied it by 1.0 so this is something that worked in python 2 also works in python 3 but it doesn't make sense and who would see code that shows you like one something multiplied by 1.0 it doesn't make sense so just remove it because dividing two integers in Python 3 gives you a directly float. If you still randomly, if you are still on Python 2 and you have this code that works with the division, then from future import division and that will work exactly like in Python 3, so just edit here, make sure that the whole code works, that it returns floats where it should return floats, and And then when it is converted to Python 3, then just remove that future import line. And then it will work as it should work. Because integers are eternal. God made the integers. All the rest is the work of man. Use integers everywhere where they make sense. Because there are not enough loads. You see, the first number doesn't exist as a float. because at these values the precision of floats is already lower than one or the difference is much more than one which means that there is the number that ends with a two and the next float is the number that ends with a four so there is no such number that ends with a three that is a float in python and this is not a huge number is actually a number of nanoseconds in three or four months so this is maybe a number that you could even use but then it would like that it wouldn't work as you expect because there is not every uh float does exist every integer exists but not every float exists also with floats you have such a problem like 0.1 0.1 0.1 when you add them there is there is no number like 0.3 exactly in float and if you add floats many times then of course will get some number that is a little bit off um so imagine that integer is something that you can count you take if you have uh one book on the table then you add another one you remove one book you can remove and add books on the table any number of times but if i take a glass of water and i pour one deciliter into it then i remove one deciliter at one deciliter and so on then then at the end, I have any amount of water because there is no exact value of one deciliter. So if you can just avoid floats for everything that has to be exact. If you want to add the floats, you see, for example, here, I showed that if I add 0.31 million times, then I get some number that is a little bit off. what is exact error would be using decimal this is something that you should use when you work with finances if you have some financial amounts and you want to calculate taxes or something then do this because finance amt or the financial office or whatever they are called this tax bureau they know no fun you have to calculate exact numbers so give decimals to But if you do use this for some physical values, then you see that you will have already much slower calculation because doing the same calculation with decimals is seven times slower than using standard floats. You can also use fractions, which is also nice, but it is even much, much slower. However, so use integers wherever you can. So, for example, there are exactly 60 minutes in an hour. There are no 60.0 minutes in an hour. If you want to calculate the average, don't divide by 2.0, divide by 2. Also, you have several possibilities how to calculate something when you need an integer result. You can use matfloor that gives you an integer now. You can convert it to int as well if the numbers are positive. or you can use the integer divider with two slashes. There is another function that has changed, that is round. Round returns integer, if you just call it like that, but you can also round to a certain number of decimal places. Think whether you need it, because usually you don't need it. If you want to print something, then take the float as it is and then represent it as a normal, like a string representation of the float with so many decimal places as you need. But taking a float and rounding it to two decimal places, it gives you some random float that is maybe similar to this number with two decimal places. Usually you don't need it. So if I have any float, I can just format it if I want to print it because now the string formatting in Python that's easy and very explicit. So now it's two decimal places. and even if you want to have the name of the variable in it you can just add the equal operator in it and it will show you the name of the variable equals and the value percentages is also something that comes quite often in the code and you see some random multiplications and divisions by 100 only to get some numbers in percentages you don't need them just work with the flows as they are number between zero and one it's perfect because just forget about the 100 you will get some number between 0 and 1 and then if you want to print it then just don't print it as a float and add the percent at the end but you can print floats with percent in the definition of the format and it will divide or multiply this number automatically by 100 format it accordingly and then add the percent sign at the end this is much easier because it is 1 7th It is not like 1 7th multiplied by 100. It's only 1 7th, and randomly it is about 14.3%. Numeric literals, when you are writing some numbers, big numbers, and you have, for example, you want to have the number of nanoseconds, and you have those in seconds, you can multiply them. You want to multiply them by 1 billion. This is billion, so 1 milliard. But actually, you don't do this here. here you multiply you don't multiply it by 1 million by 1 billion here you multiply it by 1 billion point zero so actually you multiply by the float because 10 e 9 is a float but there is no float number of nanoseconds in a second there is integer number of nanoseconds in a second so either use this okay doesn't look good this looks better because you can write the literals with underscore so whenever you wherever you have some bigger integer you can rewrite it like this and it will be easier to read or if you really want to represent a big integer then use this it's the same as te9 but this is integer and not float and logically it is what you want because of the integer number of things in something else possibly past this is anyone using it already no my colleagues they say no um and you should because this is like wonderful thing of object orientation when you are working with uh paths it's like with uh directories and file names on your system look at this this is old python 2 code that goes into a directory it goes over all files takes those that are csv files and then reads them and then process them somehow and this is the same code with path object oriented i would recommend like really have a look at the code and if you work somewhere with uh with paths or directory or file names even if it is only a few lines then define it as a path object because this is built in in python and it also gives you for the debug information here i am not playing with some strings i am playing really with path even if you don't do anything with that, because that will show the reader, because there will be plenty of readers. There is one writer of the code, but plenty of readers, it will tell them here is some thing with paths that you can probably access that you can check whether it is correct. So this is something that you should use. This also works nicely for makes also plenty of operations much easier to read. So for example, if you want to delete the file you can if it exists you can just unlink it and since python 3.8 you can even just unlink it missing okay which means that it won't give you an error that the file doesn't exist so it is really nicely object oriented and in the cooperation with the pi test this is also beautiful anyone using pi test here oh that's better um because in python 2 if you wanted to test something and create a file temporary file you had to temp file named temporary file so some random file somewhere on in your temp directory then okay you process it and then you have to unlink it or like delete it but in a pytest and by possibly path you just use the fixture temp path and that gives you creates a directory temp pytest of user PyTest1, which is the run of PyTest on your code and then it will create the directory, subdirectory with the name of the test function that you have and it will defaultly, it will automatically delete the third oldest or the oldest PyTest directory, so you have always only the three last latest test runs on your file and The names are logical because you gave the names. You gave the names of the functions and you gave the names of the files. So this works like that. Wonderful. Quickly, daytime in Python 2, if you wanted to calculate the Unix timestamp of a daytime object, then you did it like that. That was not so nice because import this, import that, and so on. In new Python, you just use timestamps. So the code will be much easier, but this is you who have to read through it and fix it, no automatic code will do that for you. And now we have also zone info, even in the newest Python you don't have to import it from back parts, so you can just import zone info and then define your local zone and it works beautifully. Time time, which was used, which can be used if you wanted to calculate how much time did some operation take, it's fine. It doesn't work if your system clock has been changed during that. So you can use PerfCounter for short amounts of time or process time if you really want to see the CPU time, how much your process took. This and much more you will find in the newest second edition of this wonderful book Fluent Python that I have read so many times, the second edition, because I am one of the technical reviewers. I hope in the upcoming days, weeks, the book will come to the market and I absolutely recommend you to read because you will learn much, much more than could be written anywhere else. So, there are Python 2 relics in your code. And for the rest of you, here's the plan. When someone uses a feature you don't understand, simply shoot them. This is easier than learning something new. And before too long, the only living coders will be writing in an easily understood tiny subset of Python 096 written 20 years ago. So, I wish you happy coding. Thank you very much. Thank you so much, Miroslav. Okay, question time. Don't forget to drop your questions on Slido. We already have one question over here from Raimar. Is there some inspection tests to show what needs to be updated? there is pi upgrade that will help you with a lot of uh call of details details that can be automatically upgraded uh but this list is something that i got as an experience because last year i just spent some time going through plenty of python projects on github and just trying to improve some stuff so i just read through and i tried to support them and like help them in the migration to the latest Python version and using that I saw plenty of such details so this is like a collection of something that I learned in those few weeks of playing with code of other people that I try to upgrade so there is no official list that I know about Still waiting for questions in the meantime anyone in the audience in Berlin want to ask any questions Okay, here's a question from Slido From anonymous. How often would you propose upgrading the Python version? Once a year because there is one new version every year this is also connected with the version of uh black for example because black also changes its opinion from time to time and i would say that every year or so if you have already like uh this uh uh inner inertial energy and you just you are just rolling um every year you uh upgrade your python or like move uh one version um forward and you also um in the latest version of uh black because black will also like change his opinion the new version comes from time to time and you don't want from time to time have black automatic telling you that i actually want to change this comma everywhere in your 1000 files so keep pin to some version of black and then once a year where you do such a review and it will be easier every year uh you just also pin to the new version of python you see whether there is some older uh python language feature that you would like to drop a new one that you would like to introduce, and you also pin the newest black pylint, flake gate, whatever you are using to check your code and to keep it healthy. Okay, next question. Can you think of something which was better in Python 2.0 than in Python 3.0 from anonymous? Yes, we were younger. Any more questions? Should we close out with that great answer? Okay. Thank you again, Miroslav. Thank you, everyone.