Static Typing in Python

Speaker 1 [00:03]

Hey, everybody. Hi. Hey. So, hi, I'm Dustin. Quick things about me. So, first, like she said, I'm a developer advocate at Google. That means that I represent the Python community internally at Google when we build products for Python users. I'm also organizer of a conference. I organize PyTexas. I'm from Austin, Texas. If you want to come to Austin, it will be May 16th and 17th of next year. I also work on the Python package index. I'm one of the maintainers of PyPI, and I'll have a real quick update for everyone, real quick. We just did a bunch of work to PyPI. You can now do two-factor authentication, including TOTP and WebAuthn, which means you can access PyPI with a security key or your fingerprint. We did accessibility audit and fixes. We also just merged a bunch of changes that adds localization. PyPI is now translatable, so I need everyone in this room's help, because right now it might be a little hard to see, but German is the fifth language on this list. Japanese and Portuguese are 100% translated. So if you want to contribute to PyPI, it's super easy to go and just translate a couple strings. You don't have to do it all at once. Okay, all that aside. Pop quiz. Is Python dynamically or statically typed? Who thinks that it's dynamically typed? Okay, who thinks, ah, this is a talk about static typing in Python. It's probably statically typed. And who thinks this is a trick question, maybe? Yeah. So the answer to this question is that Python is dynamically typed, but it can be as statically typed as you want it to be. And that might not make a lot of sense to you. And if it doesn't, that's okay. You're in the right talk. We're going to figure out what that means. So the steps to understand that are, we're going to talk about types in Python, type systems in general. We'll talk about dynamic typing in Python and static typing in Python. And once we get there, we'll talk about how to use static typing, when you should use it, when you shouldn't use it, maybe, and a little bit more. All right. So let's talk about types. And let's specifically talk about the type keyword that's available to us in Python as a keyword. So we might have done this before. You can call type on any object, and it'll tell you what it is. So if I call type on 42, I get class int. I can do that on a float. I can do it on a string. Do it on a list. I get list, right? This makes sense. And you might be saying, oh, I recognize these. These are the words that I use to turn one type into another. And that's actually right, right? You could have an int, and you could cast it to a float, cast that to a string, cast that to a really ugly list. Who's seen a list like this before in Python? Yeah, that's a type error. Whatever you had written was expecting a list, but it got a string instead, and it just gave you that, right? So we're going to, if you've had that problem, we're going to talk about how to fix that. So it might seem like these are the only types that are available to us, right? But actually, these are just classes that map to built-ins in our language. So int is actually just a class. If I can compare it to, just like I would compare classes, I can say, is instance 42 of an int? And it tells me it is. It is an instance of that class. Int is just a class. So basically, these are just classes. And there are more than just int, string, float, right? So who's seen none type before? You've probably gotten that in an exception. That's the type of none. But we don't actually type none type in Python. Similarly, the type of a function is function, but we don't use the word function to define a function. We say def, function, name, variables, etc. It's also like an ellipsis type. And actually, there's a whole bunch of types in Python. You can type import types, and you get this huge list of types in Python that includes things like in the middle there, function type. And that's just a class. You could instantiate that with a whole bunch of arguments, and that would actually give you a working function. But we don't do that because it's super messy. All right. So, dynamic typing in Python, it means that variables can be any type at all, right? That's what we mean when we say that Python is a dynamically typed language. So in this example, I import random and I assign a variable to the result of random.choice. So what is the type of A here, right? It could be anything. It could be a string, an int, or float. And Python is totally okay with that. It's totally valid Python. Dynamic typing also means that the arguments and the return types of our functions can be any type as well. Just like the variables, the same thing is true for arguments and function return types. So I could write this function for Obnicate. You don't know what it does. Takes three arguments and does A plus B plus C. So in Python, when we write a function like this, how do we know what types it's expecting? So if I showed you this function and you don't know what it does, what would you expect these argument types to be or the return type? Yeah. Okay. You might say, all right, I've declared this function. An int would work, I could give three ints to this function and it would sum them all up. But I could also pass three strings to this function and concatenate them all. So this function is ambiguous about what types it's expecting. And actually, there's a problem. If I wanted to mix types, I couldn't do that. I couldn't give two integers and a string because that's a type error. I can't combine strings and integers like that. So that's confusing. That's a confusing function. How are we going to fix it? one thing we could do is write really long and detailed doc strings about the arguments that our function accepts and what it's going to return who does this anybody yeah some people do your employers pay you enough to do this this is a lot of work right that's a long comment and it has to be right and the thing is that doesn't actually stop any type errors from happening right that's great documentation for your developers but at runtime they can still call that with strings all right how about this we could do this we could assert on the type of every single argument our function gets do our business logic assert on the return type and then actually return it uh i had a joke when i gave this talk once before i was like nobody does this and someone came up afterwards and like yeah we do do this uh it's okay if you do this all right it works it's like a valid like runtime type checking that you're doing uh just know that Like, all those assertions is a little bit of overhead unless you're actually optimizing them out in production, which is possible. Okay, so we don't usually do this, or sometimes we do the previous thing, but it's hard. What do we kind of do in Python instead? We do what we call duck typing. This means that if it walks like a duck and it quacks like a duck, it is probably a duck, right? So we sort of can infer the types of things by how they're used. So in these three examples, you can probably maybe figure out what foo and bar are going to be here, right? So in the first one, I'm iterating over bar. Maybe bar is like a list of integers or some other object. The second example, I'm comparing bar to zero. Maybe bar is like a number or something that's comparable to zero. And the last one, I'm calling bar with some arguments. And so like in the last one, bar could be a function. It could be a class. Usually we name it differently with a capital B if it was a class, but it's ambiguous, right? I don't actually know what the type of that object is. It could be anything. So static typing in Python means that the variables, return types, function types are going to be statically defined. So there are actually lots of statically typed languages. I want to show you some examples of that same function in those languages and see if you can figure out what they are. Here's the first example. Anyone know what this is? I heard C. Yeah, that's C. How about this one? Java. It's a dead giveaway with a public static Int. How about this one? Rust, yes. The person who yells Rust is always the most excited one. Yes. Yeah, Rust has a very, very fine-grained integer type. So that's an unsigned 8-bit integer there. How about this one? TypeScript, yes. Very popular these days. And it's a dead giveaway because JavaScript, everything is a number, right? There's no integer, float, etc. So you can put languages into sort of two categories, dynamic or statically typed. But as I'm going to show you, there's actually kind of like a little asterisk next to Python. And technically, Ruby is about to get that same asterisk as well, because they'll get the optional static typing sometime next year. So like I said before, Python is dynamically typed, but can optionally be as statically typed as you want it to be. So this wasn't always true. Python used to be a completely dynamically typed language. And actually, the story of Python becoming an optionally statically typed language, which I'm about to tell you, is also kind of the story of static typing at Dropbox. So Dropbox is a huge Python shop. They have millions of lines of Python code. And basically, at one point, they figured out that having that much dynamically typed code, no matter what language it is in, is a huge liability, right? Because the inability to reason about the types of variables and functions make it really challenging for new developers to get onboarded to ensure that they're using the code correctly. Basically, there's this whole class of errors that comes just from not having types defined. So not too long ago, they wrote a really excellent blog post about their journey from going from zero typed code to millions of lines of typed Python. But it's not just their journey alone. There were some other things that led up to it as well. So the first thing was PEP 3107 function annotations. And we got this in 2006 in Python 3. And what this let us do was take a function like this and optionally add basically whatever we wanted to the annotations of the arguments and the return type. So here, anything that is valid Python is there. I can put anything I want in there, and it has zero effect on the function execution. what it does give me is the ability to call dunder annotations on it and get the results of all those annotations so cool what can we do with this uh so in the pep they kind of described what it could be used for and some of the ideas basically there are a lot of ideas that basically boil down to we could do something like static typing and some other ideas basically boil down to and maybe also documentation so yeah static typing is kind of a valid use of these annotations you could do something like this. You could specify the actual class or type that each of these arguments has and the return type. And then you get these annotations that correspond to what you as a developer had written and expected those arguments to be. However, that's all you get, right? This doesn't give you a type checker. It doesn't give you any way to validate it or to determine at runtime if it's actually being used correctly. They're just annotations. So around the same time, Yukalik Tosla was working on his PhD thesis at University of Cambridge, and his thesis was about the unification of statically typed and dynamically typed languages. Interesting. He wanted to use the exact same language for everything from a very tiny script, where you don't really need static typing, to a sprawling multi-line code base like what Dropbox had, where you kind of start to need static typing. And he was focused on having a gradual growth, basically being able to take code that was untyped and slowly add type annotations until it became typed, which means that you don't have to do it all at once. You don't have to type your entire code base at one time. So he published his thesis in 2011, and his conclusion was basically that trying to add a static type system to an existing language that was dynamically typed would be really invasive. It would require changing basically everything in the language and the runtime and all the tools involved. However, you could build an optional type system that wouldn't actually change the runtime of the program, but would allow you to do sort of static analysis of your code. And that sounds pretty interesting. That sounds great. So at PyCon 2013 and PyCon US 2013, he introduced something called MyPy. So if you've heard of MyPy before, this is not quite what you know of it today. At the time, this is how he described MyPy. MyPy is an experimental variant of Python that supports writing programs that seamlessly mix dynamic and static typing. So at this time, if you know MyPy today, MyPy is a type checker. But at this point in time, MyPy was this weird variant of Python that he created just to do his research. Because he couldn't just use Python. Python didn't have the right semantics and built-ins in the language to do the research that he wanted to do. So he just made his own language instead. And that's not that absurd for research purposes. So this is what MyPy looked like. And you can see that it kind of looks and is inspired by Python. So the issue was that even with the function annotations that were already existent in Python, there were some other things missing. We couldn't annotate variables, for example. So he gave this talk at PyCon, and he presented it, and basically afterwards, Guido came up to him and said, hey, let's just do this in real Python and drop that variant thing. And they decided that sounded like a pretty good idea. So the next thing they did, Guido authored PEP483, which is his sort of unifying theory of type hints. And this is basically just some broad ideas about how typing should work in Python. So this is kind of directly inspired from Yuka's research. So one idea was it should be optional. Adding annotations shouldn't affect the actual runtime of your program. An annotated function should behave exactly the same way as one that is not annotated. And I think this kind of represents some lessons that we learned from the migration from Python 2 to Python 3, right? We don't want it to be too abrasive or hard to adopt. And it should be gradual. We shouldn't have to type all of our code bases all at once. We should also have variable annotations. So we don't just want to annotate functions. We want to be able to annotate variables and things as we're using them in Python. And so building on PEP 3107, this implied that we could do something like this, where using a comment syntax, we could specify the type for a variable when it's used inline like that. This comment syntax also meant that we could add type hinting backwards in Python, too, because even though people stuck on old Python versions deserve to be able to use static typing. And because function annotations didn't exist before Python 3, there was also a syntax for doing function annotations in a comment as well. So same function, Python 2, Python 3, both are statically typed. This pep also introduced some special type constructs, which gave us building blocks for doing static typing in general. So these are things like any. Any is a type that matches literally anything. Union would be the combination of two types. Optional is the combination of any type and none, and some other things as well. So in addition to the types that we already had, we got some new extra types as well, which allows for functions like this. So from the key can now take either an int or float at the end, and that means that it could return either an int or float as well. It's typed correctly. This pep also gave us container types. So these are classes for things like lists and dictionaries that might contain a specific type themselves as well. right? So for example, you might have a list of only ints, and you want to be able to define that statically. Or you might have a map, a dictionary that has a value of integer and a key of string. And this allows us to type that dictionary. And also generic types. So when we don't really care about something, but we want to just have a property, like something is iterable, we have a generic type. So we can just type something as being iterable, containing a number of tasks or something like that. And that just satisfies the type checker there. And the last thing is type aliases, which allow us to take multiple types and combine them into one. This allows us to be like JavaScript and just combine all of our number types into a single number and use that. All right. So that was PEP483. And then came PEP44, which basically standardizes everything in PEP483 and basically MyPy's behavior at the time. It introduces a typing module to Python. And we'll talk about that in a second. And it also contains a lot of details about edge cases and specific use cases. Basically, it's a guide to how to write a type checker. And it, like I said, leans really heavily on what MyPy is already doing. So we got that in Python 3.5, which was released in 2015. And then we got PEP 5.26, which had syntax for variable annotations. So in Python 2, these comment annotations are necessary, but in Python 3, they're kind of janky. So in the first example here, maybe I don't want to use comments. With this pet 5.6, I could just put that inline like that. One problem with comment annotations is that it doesn't make a lot of sense to initialize something if it doesn't have a value. But with inline type annotations like this, we could just declare the type and not set a value for it. And similarly, for class variables, we could do something like this, inline. So we got that in Python 3.6. And at that point, we basically had everything we needed to do static typing in Python except for one thing, which was coming along from that MyPy variant, which is a static type checker. There's actually two types of type checkers. It's a little bit confusing because you'd call them static and dynamic. Static means a type checker is going to analyze your code at rest. Dynamic is going to be like those assert statements and happen at runtime. By this point, MyPy had become just a tool that you could pip install, and it still is a tool that you can pip install. It's basically just a type checker now, and it's available on PyPI. and it works kind of like this. You could pip install MyPy, write a file with some static types and use it incorrectly and it will tell you that it's wrong. So it's pretty straightforward. And there are actually a bunch of static type checkers. It's not just MyPy. Basically, every large organization that had a huge Python code base came to the same realization at roughly the same time and they all started working on type checkers as well. So Google has PyType, Facebook has PyEar, Microsoft has PyWrite. PyCharm has a type checker built into the IDE, which is cool, But actually, you can integrate any of these type checkers with your current ID. The integration is pretty easy, no matter what you use. And there's also some dynamic type checkers, but since they incur overhead, they're maybe a little bit less popular than the static ones. So disclaimer, I work at Google, and like I said, PyType is our equivalent of MyPy at Google. And the question I always get when I talk about static type checkers is, so what's the difference between all these? So if they all implement PEP484, aren't they all exactly the same as one? Like maybe a little bit faster than the other one? And so, yeah, PyType is also available from PyPI, and you can also install it. And the key difference between these two, which I'm most familiar with, is basically a difference of philosophy, right? And I'll talk about two things, cross-function inference and runtime lenience. So basically, the difference in philosophy is that PyType will never raise a type error if it's not actually going to produce a runtime error. And here's an example. So if I define these two functions, one returns a string, the other calls the first function and tries to add an integer to it. If I try to run this, I get a type error, right? You can't do this. However, if you run this through MyPy, it does nothing. It can't actually infer the types between multiple functions, so it doesn't say that anything's wrong here. But PyType will say, yeah, I see that that's going to raise a runtime error, and you probably actually think that you're using that wrong, right? And so the inverse is true as well, right? In situations where there won't be a runtime error, PyType will not raise an exception, where sometimes MyPy will. So in this example, I start a list with a string, and I add an integer to it. And if I run this through, if I run this, it works just fine. Like, there's no runtime error. But if I run it through PyType, it also doesn't complain. There's no runtime error. But if I run it through MyPy, it complains. It says you're trying to mix types of this container type, and I consider that to be invalid. So those are some subtle differences, but it basically depends on if you want your runtime errors to be raised when you're actually typing your code. Okay, so you might see all of this and say, okay, why? It's great that we have this in Python. why would I want to use this? So first I'm going to say when you shouldn't use static typing. And this is basically never. There's not really a point in time when you don't want to have static typing. One argument that could be made is that static typing is not a replacement for unit tests. So this tweet sort of outlines that. But basically, there's kind of an argument that sometimes unit tests feel a bit like type analysis, right? You pass certain arguments and you make sure that errors don't come out. But really, you probably need both. You should keep your unit tests or write your unit tests and do some static type checking as well. So when should you use static typing? Well, basically, use it liberally. Use it as much as possible. If you are millions of lines of code of untyped Python, you probably want to use it. And if you work at a place that has millions of lines, you probably have already started thinking about this. Dropbox found that at their scale, dynamically typed python code was needlessly hard to understand and actually was impacting developer productivity and this is true across all these other organizations as well the lack of static typing in python was like a serious liability for the adoption of python at that scale so if you haven't yet you should look at this graph if you haven't had static typing yet that is so basically the idea is that as your code base increases your desire at typing annotations is going to increase, but the ease of adding it is going to go down. And so you're probably here right now. This is the place where you're going to want to do it. And that's probably where you actually will do it. But keep this in mind when you're getting 10 millions of lines of code. Okay. Another place you should use static typing is when your code is just kind of confusing. Like, let's be honest, I've written confusing code. I'm sure you've all written confusing code. It can be said that static typing is basically machine verified documentation for your code, right? If you run a static type checker over your types, it's going to tell you if they're right or not, if you're being used correctly or not. And your developers can look at them and understand how to use your code. If you feel like you need to document the input and output types of a function, then that's probably a good indicator that you should just statically type it. Another time when you should use static typing is if your code is for public consumption. So like maybe you're writing a module for PyPI. If you add type annotations to it, it would help developers know how to use it. It will help IDEs know how to do auto-completion for it. And also, if anyone's using static typing in their code base and they're also using your project, they will really appreciate it because it will come with types as well. You should use static typing before you do a big migration or a refactor, right? If you go through and add a bunch of types, move everything around, and the type checker doesn't complain anymore, that's probably a good sign. It means your code's probably going to work. Whereas the type checker going to help you find lots of bugs that might have been introduced when you're doing a migration or doing some refactoring. And finally, you could just use static typing to experiment with static typing, right? It doesn't hurt. You can start with like a little easy little file, a little corner of your code base, or like the hardest, most hairiest part of your code base. It doesn't matter. All right. So to wrap up, how to use static typing in just five easy steps. So first, Maybe consider migrating to Python 3.6. That's going to give you all of the latest steps. You can do type comments in line and all that stuff. And let's be honest, you should migrate to a modern Python version anyway. But yeah, that's technically optional. You can still do type comments in Python 2.7. Next step, install a type checker locally. I don't actually care which one you install. You can install multiple if you wanted and have them sort of compete against each other. That's totally fine. Install it locally and then integrate it into your editor. Once you start adding some static typing, your editor will start to tell you if you're using that typed code incorrectly. Then start optionally typing your code base. Start with maybe the easiest thing or the hardest thing, like I said. Don't try to do it all at once. Remember, it's gradual. You can start with a single file or a single function. Pick the most critical area, maybe, and then just start there. Next, you can start running a type checker with your linting. Integrate it into your CI environment. Have it run the same time you run black or flay gate or whatever you're using. You should use black, though. And finally, you should convince all your coworkers to join you in static typing. And if you need any help convincing them, you can show them this video on YouTube. Thank you, everyone. Sure. Yeah, so now there's like a couple of minutes for questions time. Anybody want, okay, oh, yeah. Hi, thanks for the enjoyable talk. Is there any work or is there any plan to have work on doing type inferencing so that you can essentially have the types inferred by the way they're used? Yeah, so the static type checkers, the question was, is there going to be any work done to do type inferencing, right? to be able to automatically detect these types. So if you read the blog post from Dropbox, they said that they originally thought that this was a great idea, and then they realized that that is actually a terrible idea. And the reason is because, kind of like trying to write a script that will automatically migrate your code from Python 2 to 3, something that's going to automatically infer types is going to generally be right, but occasionally be wrong in ways that you're not going to pay attention to. And so basically they found that when they tried to do that, it required a human developer holding the inferrer's hand all the way through and basically checking everything. It ended up being just as much work, if not a little bit more. So yeah, there are some things that will kind of give you hints or suggestions at what types should be. But yeah, generally you should maybe do it by hand. It guarantees that you're not doing it wrong. Or maybe not guarantees. Any other questions? I played around a little bit with stack typing in Python, and I think anyone who's spent some time with it has probably run into the problem that when you get to more complex data structures, it seems to be very difficult to set up types. Do you know, going forward, is there going to be anything that are going to change to make that process a little easier? Because once you get out of the box in terms of data structures, it becomes kind of a nightmare. Yeah, I can't quite hear you, but I think what you're saying is once you get into some more complex data structures, it's getting harder and harder to type, right? Yeah, so that's definitely true. And in my experience, the situations when you are happily going along typing your code and it suddenly becomes difficult are kind of an indicator that your code is maybe not ideal, right? Like maybe it could be simpler. So in those instances, I would generally recommend doing a little refactoring to make it easier to add type annotations. But yeah, if you have super complex data structures or just really weird functions, they are going to be challenging the type. You should consider, though, that those things are probably also hard for your developers to understand what they're doing. Maybe one more question in the front here? Yeah. Right, on your right. Thank you for the talk. I have a question about if it's possible in Python to select like put the type of arguments for example you are taking a list of something and you will return the list of the same thing like it could be the list of string or int or whatever like it's something that you can do in Scala for example but I don't know if it's possible in Python I'm not sure I understand the question but let's chat afterwards about it cool so thank you so much for Dustin I'm sure he would love to answer more questions if you find him later but yeah thank you