Break your API gently - or not at all

Speaker 1 [00:02]

Okay, thank you. So it's great to see that there's so much interest in API. I'm Tim Hoffman. And if you want to look up the slides afterwards, they are on GitHub. I work in the semiconductor industry. And essentially, we are the people making your computers faster and more energy efficient. Also, I'm a Matplotlib core developer for two years now. And that's somehow what brought me to this topic. So Matplotlib, as you know, probably has a lot of API issues as well. So it's not the best API in all parts. So I could go there and say, yeah, I know how to do it better. Refactoring is great. I just change the API and make it better. But if I do that, I will have a lot of users growling at me and saying, you broke my code. So it's not that simple. In particular, if anybody else uses your code, and that's not only true for open source projects, that's also true within your company. You may have libraries. if you don't have complete control over the usage of your code, it's much harder to do refactoring because you will have to not access to all places where you would have to change the calls to your API. Or maybe you could, but it's too cumbersome inside your company to change everything. So you have to be really careful what you do. You don't want people moaning about you breaking your code. So, what can you do about that? First, and this is maybe the most important message in this talk, try to make the API good from the beginning. So, what can we do so that we don't have to deal with API changes later? One point is reduce the footprint of the API. Really make it a habit of making your functions, your methods or attributes private. Mark them with the underscore. This tells all other people it's none of their business and I'm free to change that anytime I want. If they use that and I change it and it breaks their code, it's their fault. It's not mine fault anymore. Another important point in this is use keyword-only arguments. They also narrow down your API Because with a keyword-only argument, your user has not the freedom to call the parameter positionally anymore. That narrows your API. And we see later on why this gives you much more freedom in changing things afterwards. And, yeah, common wisdom, if you don't need something, don't write it. Then you cannot break it afterwards. the second point what you should do is really try to do it right from beginning if you start something and say I will change that later on that later will be much later and then that code is used a lot so it will essentially be like a public API and you have all the troubles we will see you have to go through if you want to change that so you know essentially how to write good code. I mean, there are all these principles. If not, and you are planning to write an API like that, really learn these design principles. One point I want to really emphasize with this, which is essentially simple to do, but really has a large impact, think about your naming a lot. I mean, it's always said, think about the naming, but this makes a difference in two ways. One is if you think about your naming, you think about your API, you think about what do I want to really do with this method and this prevents bad designs from the beginning and the other thing is it makes the life for your users easier because they have proper names and you don't get into trouble that you want to rename it later because you think, well, this function should be called something completely else. And if I do that, my users would have to change their code as well. So, and this is really an example from matplotlib. We had a function called use, which had a parameter arg. So there's no way of knowing what this actually does. And what this does is it sets the backend. So what we right now did, we changed from here to there. So that's a step better. But this is also an incompatible API change because your users might have called this as a keyword argument. So they might have said use arg gleich qt or something. Not very likely here, but matplotlib is a very large and widely used library. And if there's something possible, you will always find someone who is doing that. So we are right here. I would like to go there, but we are not going there because this really means our users would have to change and adapt their code. And you could have either both at the same time, which is the duplicate API, which is not nice as well. Or you would have to deprecate that and go to set backend, which means all your users would have to change their code. And this is, in that case, we decided it's not worth it because it's a burden on the user. We don't want them to take, we don't want to change anything. So we are stuck with this medium bad API because we don't want to go further in this place. And this is always something you have to consider. Do I want to change the API? Is it worth the effort also from the user side? And then there's one thing with naming or with parameters, which is, I think, not well described. Your parameters should always describe one logical concept. So we have here a very simple case, which is from my company work. I had to plot an image and I had a special function doing that. And it took two parameters for the color scale for a minimum value and a maximum value, which seems like a reasonable API. And I could have done it differently like that, which would actually have been much better, which I didn't realize in the beginning. But it's probably because you're used to that from matplotlib. And that's probably because you're used to having simple types as parameters. but that's not always the best way to do it because then here I realized I would like to say I want to have the color scale out auto scaling but symmetric. I want to have the same negative and positive value and I cannot do this reasonably if I have two parameters then I would have somehow to say the one parameter has to depend on the other parameter and that's really hard. If I had done it directly this way I could just have added a parameter symmetric, which would have allowed me to do that. And that's really just the idea that there's just one scale I want to set and it's not two parameters, but it's one concept and Pratton allows us to have complex types or like two complex types, like tuples, which can express this one concept in one entity. Okay. Now, if I have to go further and really have to make changes, there are different ways I can do that. I can just do an extension. That means I have an old API and I add a new API and they just stay both. This is good from the user point of view because they don't have to change their code, but it's also bad because I have now two ways of doing things instead of just the one way. If I don't want that, I can switch off the old API. And ideally, I do it in a gentle way that I have a transition phase in which both APIs are available. And there are just some cases where that's really not possible. And I have to do a heartbreak. so I switch on the one and I switch off the other and that's really awful for your user because in that case of the heartbreak your user when he switches the version of your library he immediately has to adapt his code and likely it's not the time when he wants to adapt his code so if you have this transition range he can update to a new version of the API to a new version of the library, use his old API at some point, say, okay, now I will rewrite that. And then he can later on migrate to newer versions of the library again, which doesn't have the old API anymore. And here it's also important that the user knows about that the changes will happen. And we'll come to that in a second. So coming back to the first thing, just extensions. What can we do with that? We can add stuff, classes, methods, functions, attributes. We can append parameters. We can just append parameters, but only with defaults, because I might have this function A, which is called just with a single parameter. And if I would add another positional parameter without a default, I would have to provide two parameters, which would be a breaking change to the existing code. And here we get back to keyword-only arguments. If I have keyword-only arguments, I can insert keyword-only new parameters in the list of keyword-only arguments, and I also can reorder the keyword-only arguments if I say, okay, these parameters logically are better grouped together without changing the API. But that's about all we can do with extensions with non-breaking changes. So let's have a look again at this keyword-only parameter. Really make it a habit of making almost all your parameters keyword-only. This has two benefits. One is your users are forced to write more readable code. There are no five parameters one after the other. without giving information what they are actually are. So usually there shouldn't be more than maybe one or two positional parameters that can be accessed without keyword. And also it gives you a lot of more freedom of changing your API. And this is an example from the standard library. In former times, there was a compare function which was provided to sort. And then one realized, well, compare functions are actually quite bad performance-wise because they have to evaluate twice. And so one introduced the concept of a key function. And one correctly did that by introducing this as a keyword-only parameter. If they hadn't put in the keyword-only, they couldn't have gone from the step two to the step three without breaking code because then it would have been valid to have the original compare function as a single parameter and that would have been interpreted as a key function in the third case. So there would be, if I don't look at changes I would get a completely different behavior probably an error, because I switched from top to bottom without the star, with just the single parameter. Okay, so when it's not enough to not break things, we should do it in a controlled way. So you should always announce the breaks before. So when you change something, say you rename a function, you should document or you delete a function, whatever, you should document it in your release notes. But that's not enough because no one reads release notes. So if at all possible, and it's most of the time it's possible, you should really warn when this deprecated API is used. So that's the warnings method. And you can just issue a warning. It is deprecated. Ideally, you also say what should be used instead. And there are different types of warnings. Most used is probably the deprecation warning, which targets developers using your library. Then there's the future warning, which is emitted in all cases. So this is targeted at the end user. likely something like your configuration file format has changed. So really the target audience is who can do something about that. If it's a fixed program and the user cannot change the code, it doesn't make sense to warn him on change this function X to Y because he cannot do anything about this. And there's a third type of warning, which is a pending deprecation warning. I think it's not used that much, but it can be used if you want to really be careful and take it slow. You can announce a deprecation. You don't deprecate. You don't warn about the deprecation already. You just say, I will deprecate that in the future. And if you want to look into this further, there's the standard documentation and PEP 565 changed a bit when this deprecation warning is issued. Okay, so let's get to some practical stuff, some examples. We rename a function to new function. And what we do for that, we change the name. And to make it backward compatible, we reintroduce the function with our deprecation warning, and we call the new function. That still works. And if the user calls func, he will get his warning. it's similar for classes you rename the class you introduce a new class which inherits from the base class and also issues a warning on creation you can also rename attributes then it's getting already a bit more complicated the most straightforward way to do this is to replace the attribute by a property and use the functional character of the property to issue the warning and then return the new value. So we're just changing here size to radius. And if we call size, we warn. And after a deprecation period, we can just remove the properties and we have migrated. So these are the simpler things. Now you can also rename a global variable. And if you're on Python 3.7, or larger. You can do this with a get attribute. We have now this module get attribute, and you can do it this way. Then upon usage of that old name, you still get the warning. I thought if you were on Python smaller than 3.7, it wasn't possible until last week, where I realized actually it is possible. You don't have module-level properties, You just have properties on the class level. You don't have a get attribute. So I thought there was no way of warning when accessing a module level variable, which is not exactly true. You can do that. There's a project called mprop for module properties, but it's really doing a lot of magic in the background. So essentially, it patches your import mechanism, replaces the file you want to import. buy some rewritten file, which still does some magic stuff, insert some initialization mechanism, and then can warn when you access that variable. But it's really, really, you should think about if you really need that. Okay. Going from variables to parameters. Renaming a parameter may seem simple. So I have here a, b, old, and c, and I want to rename b old to b. And if I just think about this in the standard signature, I would say, okay, maybe the function is called that way or that way, but that's not the only ways you can call the function. You can call positional parameters using a keyword argument. So I could have really have here this b or b old, and you can also conversely call keyword parameters if they are not keyword only as positional. And you have to accommodate these cases because they may also be valid calls in your user code. So that makes it really more difficult. If you want to do it in the function, you would have to do something like this. I don't think you can get much easier. You would have to have some sentinel values to really see if I called B or if I called B old, I rename that positional variable, give it a default value. That means also I can just do it with the last positional variable. I reintroduce a keyword only old variable and I check which of these are code. And if the wrong is called, I issue a warning and I issue an error if someone gets to the idea of providing parameters both way. So this is already quite complicated. You can simplify that a bit or abstract that logic away with decorators and I will come to that a bit later. Okay, so changing return values. You have not much you can do. One way is you introduce a control parameter like this full parameter in NumPy Polyfit, which is not only for API changes, but if you want to have more flexibility with your control parameter, but that's about all you can do. It's not really a nice way of changing an API. If you have complex return types like a dict or a data class, you can just add stuff. You can add more entries to the dict. Yeah, name tuple, not really, because they may be converted to regular tuples and then you cannot not just add stuff. Only if you have these already in your code, you can do that. Change of behavior is even more difficult, and usually it's the best to just not change the behavior but introduce a new function, and this was done in the standard library with system calls already twice. They changed from the OS system to the subprocess call stuff, and then later again they simplify to the sub-process run. And they have quite a good documentation how one should migrate one to the other. So in the last five minutes, I will take you to some more fancy stuff. What we do in Matplotlib, because we do a lot of refactoring of the API, we have decorators for common operations. We have a decorator for just deprecating a method. and we have also decorators for renaming a parameter. So this was this original use arc and we have now a decorator which says I can also have this arc argument but it should actually be a backend and if it's arc, then I will issue a warning. You can also delete a parameter and really nice, you can have a decorator that warns if you use a parameter as a positional parameter that makes it possible to change a positional parameter to a keyword-only parameter later on. And that way we can recover if we didn't introduce keyword-only parameters in the beginning. So then there's another example from Pandas. Pandas has this awful plot method because it's awful because it just does way too much. It has a kind parameter which decides which kind of plot you want to plot. And then there are a lot of arguments and they are specific to the kind of plot and you have to document this parameter is only if your kind is this type and it's a really bad API. So what you could do is just ditch that method and add separate plot methods. But that's not really nice for the data frame because then you get additional three methods or 10 methods or whatever, how many plot kinds they have. And the data frame already has a lot of methods. So that's not so nice. So what they did is they just replaced this underscore with a dot and put all the plot methods in a namespace. Is that backward compatible? Yes, it is. So I want to have plot as a function and also as a namespace. And you can do this If you use a class, which is a namespace, and you can call a class if you define a call method. And that way you can really have a plot work both ways and make the API much nicer without breaking it. And then we come to the last example. This is even a bit more tricky. This is an example from Matplotlib. we have the access, which is just the access of the bar, and there's a list of tick labels. And there's a reset ticks. This should be reset ticks. There's a reset ticks method, which clears all the ticks and populates the list with one default tick, which gives you the properties of all the ticks you should have. You just then copy from that. And this is really bad. Because this reset is called many times when you create a figure with multiple plots side by side. So this was the problem which I was facing in my company. I have to plot a lot of data and say like 30 plots in a grid. And then this whole process of creating the ticks and resetting the ticks because they have to align to each other takes an awful lot of time. It's super slow. So I would like to change that. But it's hard because this is all public. The ticks are public. It's a list. Users can use that. They have to expect that there's at least one element in that list. And I have to create that beforehand. So how can I do that without changing the public API and without rewriting major other parts of my library? And you actually can do that. you use a cheap placeholder object instead of that list. And that placeholder object is replaced up on first access. So when I just reset it and reset it again, this placeholder object stays the same. But when I first use access ticks, this is the descriptor protocol. When this object gets accessed, this descriptor is called and this descriptor replaces itself so the tics object replaces itself with this instantiated list and that way I think this is an API change which is 99% backward compatible I think you can break it if you really try hard but it's in since matplotlib 2.2 and no one can play it and this really made the library effector far faster without breaking any API. So coming to the end, what you should do is really try to prevent later changes, limit the API, and write good API from the beginning. And if you have to break, do it really gently. Warn your users, give them time to adapt, and a smooth transition path. Okay, thank you. Thank you, Tim. So, any questions, people? We have room for one or two questions. No hands? Ah, there was that one first. I'll need somebody to help and pass the microphone. Thank you for that amazing talk. I don't see really the difference between replacing that underscore for plot underscore line and bar and so on with the dot. Maybe there are code issues, maybe, but from an API side of view, I don't really understand. Could you please explain what's better there? So, what you have here is you have all the methods in one large namespace. And the data frame namespace is already very large because you have a lot of functionally in there. And so from my point of view, it's better to not add just more and more methods to that plain namespace. But here, maybe it's actually that flat is not always better. then here it's better to just group all the plotting stuff together in one namespace and not have it playing alongside with the other stuff. So, thank you. Thank you, Tim. Thank you for the question. We don't have that much time left, so you will have to find Tim outside for further questions. Thank you for attending his talk. And if you've enjoyed it, please share to the Twitter. Hashtag PyConD. hashtag PyDataBerlin. Thank you, Tim. One more round of applause, please.