Beyond Paradigms: a new key to grok Python & other languages

Speaker 1 [00:04]

Thank you very much. I'm really honored and proud to be here. It's my first talk at an European conference. I've spoken in the U.S., in the Dominican Republic, Argentina, and China last month. But in Europe, this is my first one. So I'm very happy to be here. And also my first time in Berlin. Visiting lots of friends next week when I'm going to be on vacation. If you want to invite me for coffee or beer, I'll be around until next weekend. Okay, so beyond paradigms, I'm going to start by quoting my favorite author, who is Jorge Luis Borges, an Argentinian author, who is a representative of this style called Fantastic Realism. and it's really amazing he has a book called Fictions in English Ficciones in Spanish that is a collection of short stories I love short stories and this thin book has two the two best short stories I ever read so I highly recommend it but one of the things that Jorge spent his whole life working at the Argentinian National Library so he knew a thing or two about classifications And in one of his stories, he plays with that. So there is an article on Wikipedia about the Celestial Emporium of Benevolent Knowledge, which supposedly was a Chinese encyclopedia that included a classification of all animals in 14 categories. And the categories were those that belong to the emperor, embalmed ones, those that are trained, sucking pigs, mermaids or sirens, fabulous ones, stray dogs, those that are included in this classification, those that tremble as if they were mad, innumerable ones, those drawn with a very fine camel hairbrush, etc. super important category those that have just broken the flower vase and those that at a distance resemble flies ok so this is how this is if you read the article on Wikipedia there's a lot a lot of intellectuals that have quoted and talked about this classification anyway so paradigms programming paradigms that's a way of classifying languages, right? So, the first language that I learned was the notation for programming this calculator, and the first interesting program for me, that I wrote, was a part of the Lunar Lander program that came in the manual of this calculator, I ported it to that one. And the notation looked like this. It was very similar to assembly, except that the displays of both of these had no letters, only numbers. Kind of sucked. So you really had to program in a paper, in form, so that you could actually edit your program by looking at many instructions at a time. Anyway, so, I like this book, it's a pretty traditional textbook, a survey-style programming languages book that discusses many different programming languages and their semantics and their implementation and the book starts with an example that is the greatest common divisor implemented in assembly and then the author shows it implemented in C, OCaml and Prolog and I implemented it here in Python two styles imperative style and functional style you can see the functional style because there is no loop construct looping in functional languages is always done with recursion and there is also no assignments see here there are assignments and here there are no assignments which one would you code as a practical matter left right which one do you find prettier left right yeah interesting I agree with the majority of the opinions there so yeah the thing is this is a bad fit for Python because Python doesn't have a feature and that's the key word for this talk features it doesn't have a feature that is pretty much required of any real functional programming language which is tail call optimization which is an optimization where when the return value of a function is a call to the same function then the compiler or the interpreter just jumps to the beginning of a new execution without adding a new frame on the stack so this becomes as efficient as a jump instruction in in assembly. Python doesn't have that, so this style is not very good in Python, but sometimes this is so expressive that it's a good solution for things like parsing and so on. So, talking about programming language classifications in that same book, the author wrote a box, it's literally a box, called Example 1.3, and I think it's interesting because I wrote a book and I understand something that, you write a box maybe when you want to do it as an aside, you know, you don't think it's so important, it's optional, whatever. In this case, the author is clearly not wanting to commit to this classification. So he called it an example. It's an example of a possible way of classifying languages. And then he he offers this classification, but why is Python here and not there? I attribute this just to academic prejudice against what they call scripting languages. They don't respect us. Anyway, so there's no reason really to put Python here and not in the object-oriented line, according to common criteria, okay? Alright, another book, this is a Brazilian book by two professors at the University of São Paulo, that also offers a classification, there is a mistake there, but it is not important the classification itself, it's just that if you look in detail, and then there is also this other site called the language list that's kind of broken, but it offers another classification, the thing is, if you look at several classifications you will find that they don't agree completely. It's all kind of subjective, you know, where to put particular languages. This whole site, like I said, is kind of broken. It's very promising. It says that it has information about more than 2,500 programming languages, but the information about the languages itself is missing. But there's a whole section called language categories that is a lot of pages long. All right? So, lots of categories. But categories are kind of something out of the 1900s, and even classifications that we consider rock-solid because they are based on science, sometimes have flaws. Like, for instance, this is a masterpiece of science. I love this. I actually, when I was younger, I had a poster of this in my room because I just loved this whole thing. but Clay Shirky who I'm going to quote soon observed that it's kind of weird that this column is called the noble gases because at the time when the classification was invented it was thought that those elements would always appear in gas form but later it was discovered no, it's not true if you make them cold enough they will liquefy so the name gases is no longer correct and the word noble come on you know why are they called noble because they don't mix up with the rebel you know it's nobody would use that anyway in these days so there's a misunderstanding about the nature of things and also cultural bias in this name. So, this example is actually from a paper called Ontology is Overrated, which I highly recommend that you write this down this minute if you work with information. Who works with information? Everyone. Okay. So, write this down because it's super important. I was certainly somebody who overrated classifications or ontologies and categorizations until I read this article, okay? Please do. And it doesn't just criticize classifications, but also offers an alternative, which is actually the basis of this talk. As another example of a classification that is overrated, is gender assignment at birth. You know, most people who haven't studied the matter, think that there's XX and XY, no, this is a simplification, okay, there is more variations, and there's a lot of other things that determine somebody's sexuality and gender, so this is also another thing that is complicated, more complicated than most people assume. All right, so what is a better approach for thinking about programming languages, because the idea is why would you want to classify a programming language actually this talk started from a conversation at a Golang group in Brazil where somebody asked is Go object oriented and it occurred to me that Go, just like Python is a very pragmatic language that doesn't really follow strictly the dictates of this or that paradigm and I had I had this reaction because I had studied a book so this this guy Sri Ramakrishna Murthy is a professor of computer science at Brown University and he wrote a paper years ago that said programming language paradigms are a moribund and tedious legacy of a bygone age it's great modern language designers pay them no respect this is certainly the case with Guido van Rossum and the core developers of Python and it's also the case of the core developers of Go you know, they're not really interested in checking boxes in some you know, paradigm list Anyway, this paper argues that we should abandon this method of teaching languages. And it describes a textbook that explores these matters. And then the textbook, which is freely available on the web, is called Programming Languages Application and Interpretation. And this is really awesome. The idea of this book is, instead of being just a book that talks about programming language semantics and paradigms and so on, what this book does, and also, instead of being a compiler book, where you're supposed to spend one semester implementing a compiler for a very simplified version of old Pascal, and then you just learned that one thing. This book has a great approach. The approach is, he provides a DSL written in Racket, which is a Scheme dialect, and then you use this DSL to implement interpreters, several interpreters, and each interpreter adds or removes features, like an interpreter can have the concept of a class, without inheritance, like Go. Go has, they don't call it classes, but they do have classes, except that they don't have inheritance. And then there's another, maybe an exercise or another section that introduces inheritance. You know, so you can have functions at first, but they they don't work as closures. And then you implement lexical scoping, and then you have closures. So anyway, it's a very interesting approach, very hands-on. It does require to learn a little bit of Scheme, or Racket actually, their dialect, but it's great. And after studying about that, I actually discovered that there is another older book from the 90s that follows the same approach but writing interpreters in Pascal. And one of the many projects that I have started and never finished is a porting of the interpreter examples from this book, from Pascal to Python. I've started it, I have about two chapters done, and the book has about 10 chapters. So, anyway, if you want to help me with that, I think if I had more people working with me, I would be more encouraged to actually do it. Anyway, so the key here is talking about features separately, right? And what kind of features? It's important to understand that when we talk about features, we're not talking about syntactic sugar, and And we're not talking about libraries. Stuff that is just, you know, synthetic sugar is only sort of decoration. The least people always say that. And on the other hand, libraries is something that anybody can, anybody who's a user of the language can create a new library. The thing is the features that are core features of the language that you can only, that can only be evolved if the core developers of the language decide to do that right so features like for instance first-class functions and first-class types which is a fancy way of saying that the language treats functions as objects and classes as objects okay so Common Lisp does. Iterators Common Lisp doesn't have iterators but it does have synthetic macros so with synthetic macros you can implement a new statement, like for instance, in Python we didn't have generic iterators before, and then the U keyword was introduced. Because Lisp allows you to introduce new keywords with synthetic macros, then that's why I put the star there. What is the variable model of the language? In the case of Lisp's references, what is the type checkings, dynamic, what is the type expression, that means how does the runtime determine if something is adherent to a type or not, by looking at the features of the type. Anyway, I won't have time to go over this in detail and talk about different stars that appear here and there, but the idea is to show that if you take core language features like that, and maybe you know two, three, four different programming languages, try and build a table like that. Probably you're going to come up with some interesting insights. Now one practical reason why it's important to understand the language features is because sometimes you are tempted or told to apply design patterns okay so everybody knows the book by the way do you know why this book is called the gang of four who knows someone knows yeah so there was a corruption scandal in china and at the time when this book was released and four high officials of the party were arrested because of corruption and they were labeled the gang of four and as soon as they were arrested the corruption problem went away you know because this is a very good way of fighting corruption you label a group and then you arrest that group and the problem is solved it happens over and over in different places anyway uh so all right so one of the things that uh this book is very hard to read and to conf i i have to admit i haven't read it entirely because i have i read the introductory material which is excellent i highly recommend and i read some probably most of the patterns by now but not all of them but the introductory material is really excellent and one of the things that they say there that maybe people sometimes forget because a lot of people is introduced to the content of this book in the context of Java and the book doesn't have Java examples it has C++ examples and sometimes here and there a small talk example but there is no Java examples in the book so because of the fact that people are introduced to the concept of patterns in the context of Java this kind of spreads the idea that, oh, these patterns are universal, and this is just wrong. Because the authors say, our patterns assume small talk and C++ level language features. If we assume procedural languages, what some other author would call imperative language, we might have included design patterns called inheritance, encapsulation, and polymorphism. So what does that mean? It means, for example, I'm going to give you a concrete example. Who here is old enough to remember the Palm Pilot? Remember those devices? Yeah. Anyway, the Palm Pilot was programmed in object-oriented C. Not C++ and not Objective-C. Object-oriented C. It was just plain C with a strict discipline of how to organize your structs and the functions that touch those structs. The functions always have to have a first argument. Ooh, remember that? The first argument is the target object. And it has, I don't know what the conventional name was. Maybe it was this or self or something. But anyway, so it was object-oriented C. you can do object-oriented, see if you're very disciplined and you implement, you find a way to implement the relevant patterns that make a language deserve the title of object-oriented. Okay, so this is very important, but also the reverse also happens, right? Not only they say that maybe they could have talked about other patterns if they were targeting languages with less features, but also some patterns are not relevant if a language has certain features. And this is really important in the case of Python. After studying design patterns for several years, I stumbled upon this great presentation by Peter Norvig, who is a professor at Stanford and also a director at Google. And in this presentation that was published two years after the Gang of Four book, Peter Norvig says that design patterns have fewer language limitations and less need for bookkeeping objects and classes, and less need to get around class-restricted design. So study of the design patterns book means that 16 out of 23 patterns have qualitatively simple implementations in Lisp or Dylan than in C++ for at least some uses of each pattern. And then he gives a list that is worth its weight in gold, no much more because it's It's very light. Anyway, so this list is amazing. This list says, you know, specifically, if your language has first-class types, which is something that Python does have, but Java does not have, you know, for instance, in Python, classes are full objects that you can actually create at runtime, right? Because we have the concept of metaclasses. In Java, there is a class class, but the class class in Java is only used for reflection. You cannot instantiate a class at runtime in Java. You can also, you can always make your Java code, generate bytecode, and actually some advanced frameworks of Java do that, but in the language you can't. But anyway, if you have first class types, that means you have to rethink those patterns, because it's likely that for your use case, there's a much simpler way of doing it than repeating the pattern as described in the book. And the same thing happens with first-class functions And then Okay, so here's the thing Iterator Why does it say macros? It's because of that thing that I said before, right? Even There's this concept of a true iterator Which is not just an iterator I'm going to talk about this distinction soon But anyway, Python has true iterators Which is provided by the U keyword And if your language doesn't have a true iterator You can still create that if you have macros like this has right anyway so let's talk a little bit about the iterator pattern and let's talk about the classic recipe of the iterator pattern this is a very fine diagram that is super hard to understand because they decided to make it symmetric, but this is much better. Okay, this explains that the idea of the iterator pattern is that the lady programmer here, the PyLady here, actually is a Java lady, when she wants to traverse an array list or an array, which in Java are very different beasts, she doesn't need to know the specifics of how the data structure is organized because she can get an iterator for that particular data structure. And the iterator implements an interface that is very simple, a next method. So she doesn't need to know how the thing is organized inside, she just knows that if she gets an iterator for that thing, then I can call next and get items one by one. Right? And this is leveraged in Python by the for loop machinery right so the idea is that this iterable object is something that will provide an iterator and it's important to understand the difference between iterable and iterator right the iterable is the thing that provides a method which in the case of Python is called under iter, that gives you an iterator. And we may take this, this is a little bit confusing about Python, I don't really like this, the fact that they decided that the iterator is a class of iterable. I don't like it because as a teacher I know that people often confuse the iterator with the iterable, and I think this relationship here needlessly complicates things to understand the separation. Because the key thing here is to be an iterable you have to implement dunder iter, and to be an iterator you have to implement dunder next. And how is dunder iter implemented in an iterator? It's just a simple method that returns itself, because if you want to iterate and you already have an iterator, then you get the iterator itself so that allows you to put both iterables and iterators in a in a for loop so if I wanted to implement a train that was iterable this is a way of doing it following the design pattern right so train is the iterable it implements the iterator and when you call the iterator it it instantiates and returns a trainIterator, and the trainIterator has the dunderNext method and some machinery to take care of the state of the iteration, right? Here's the thing, it's super important that, it's kind of tempting, actually, to implement dunderNext on the iterable, right? But this is a mistake, because then you will not allow two different threads or two different coroutines to iterate over the same object. You'll create confusion, you know, because what you need is, you have this data structure, and each iterator has its own state, and each client has its own iterator, so that they can advance at different rhythms, you know. If you implement next on the iterable itself, then you have only one state, and this is probably going to create nasty bugs, so don't do that. Anyway, now this whole very advanced way of doing things that we have in Python that involves yield and so on, was actually created by Barbara Liskov, who probably everybody knows as the the L in the SOLID principle, the Liskov Substitution Principle, she and six men wrote the CLUE programming language. Now, it's important to note that the six men were her students. She was a professor at MIT, and her students, they together created this language, the CLUE language. the Wikipedia has a pretty good article about it, and it's really amazing the things that... Clue was not a success commercially, but it was highly influential. So it says key contributions include abstract data types, call by sharing, which is what Python uses also, iterators, a form of parallel assignment like Python has, you know when you have several variables on the left hand side, type safe parameterized types which Python now is growing to have type safe variable types and and it also had classes with constructors and methods but did not have have inheritance actually it's a language very similar to Go but only written in 75 all right okay so iterators is a feature before the The kind of construct that people call foreach these days, which us Pythonists take for granted, but in other languages, like in Java, they had to wait like 16 years to have that because the for loop initially was just this stupid for loop of the C language. And the iterable objects are a key. And it's another thing that we take for granted, that, oh, you can create iterable objects. That's nice. you could also do that in Clue in 75 but some languages don't offer this flexibility C has just no concept of iterables and in Go only some built-in types are iterable five built-in types are iterable and the language itself is not expressive enough to allow you as a Go user to create an iterable object that will work with the for range special form of the for loop in Go. This is something, I like Go, but this is something that makes me sad. Anyway, but the Pythonic way of doing an iterable train is not to use the design pattern, but to do something like this, right? Where you have the dunder iter function is actually a generator, right? So what you have is a lot less code and a lot less opportunity to make mistakes, right. This is, you know, managing the state of the cursor is kind of tricky, you know, very easy to make one by one errors and so on, but this is simple because the state of the iteration is kept as the state of this generator. Another example, and this is familiar to people who have maybe read my book, yeah, so in Fluent Python there's a chapter called, there's a part of the book that is called First Class Functions, or Functions as Objects, and one of the chapters in that part talks about design patterns a little bit and so the example that I have in the book is about this strategy so this strategy who knows who has read my book or who has started it all right awesome yeah just a suggestion the book is split in six parts part one is just one chapter. I recommend that everybody reads it. And the other parts, you can consider as five different books of about 120 pages each. So you can actually, if you're more interested in, I don't know, object-oriented programming, there's a chapter, it's not in the beginning of the book, that is a part that talks about that, a few chapters that talk about that. And then you read those 130 pages about object-oriented programming in Python, and then you celebrate, you finished a book, right? When you finish the whole book, you've finished five books you can put five stars in your reading list for the year okay do that and you will be happier okay so anyway in the design patterns book they talk about the strategy pattern like this so there's a strategy that apply that is used in a certain context and the example that I give in the book is the context is an order class that represents an order from an online store all right and it has several methods and attributes but the most important ones for our purposes are two the total method gives you the total amount of the total cost of all the items in the shopping baskets okay and the do method gives you the amount that is due which is the amount that the person will have to pay which may be less because there may be a discount applied okay so that's the idea and then the idea is our store wants to when they create an order they want to look at the at the customer and say okay so this person is a frequent buyer or whatever so we're gonna give give them the fidelity promo or this this order has a thousand copies of the same item so we're gonna give them the book item promo or this this order has more than so many distinct items so it's a large order so we're gonna give them the large order promo discount right so there's a strategy which is an abstract class in the book in the Gang of Four book and also in my book the the strategies of an abstract class in our case it defines the discounts they specifies the discount method and then the concrete strategies implement that interface okay and the way to use this is when you create an order you pass it a user, a shopping cart and the promotion that's going to be used and notice that there is an instantiation here right so if you have if you are a very big store and you have a million orders at a time you're gonna maybe have I don't know six hundred thousand instances of the fidelity promo, because for each order you instantiate this object. And so this may be costly, and in fact the design patterns book talks about that and recommends that then you apply another pattern, called flyweight, to solve the possible problem of the cost of having so many strategy instances in memory at one time. So, yeah, the best solution for complexity is always to add more complexity. Anyway, so you instantiate different orders or different promotions to do that. So, how do you do that in Python? I'm going to show four different implementations. This is what the Gang of Four book describes, okay? Like I said, the promotion is passed to the constructor of the order, and in the do method it's used like this. If there is a promotion, then we are going to call the discount method of the promotion passing the order as an argument so that the promotion can look at the order and maybe the person who ordered to see how the discount will be. all right and this is the implementation so we have promotion as an abstract class and we declare this as an abstract method and we have all of these different classes this is one concrete strategy these are two other concrete strategies so this is one way of doing it if you read the design patterns book and you just came over from another language maybe you're gonna do that but then a better way in my opinion for this particular example is to use the first class function so the idea is the API we are going to provide this instead of providing an instance of a class that implements a certain interface we just pass a reference to a function right so if you look at the diff the call here is simplified because you don't have to go through the promotion and discount you just call the promotion itself and you don't actually have the whole ABC anymore right so this is gone you don't have the ABC and each one with each one of the promotion, the concrete strategies become just a function, right, and yeah, usually when you have to, when you think about providing an interface that implements only one method, and the instances of, the concrete instances that will implement that interface don't have have any internal states, you shouldn't really be using classes for that case. You should just pass, instead of providing an interface that defines a method, just pass the function and be done with it. Now, sometimes things are a little bit more complicated, because sometimes you may need to do a parameterized strategy. So, for instance, in this case I want to be able to say this is the book item promo but the discount percentage is going to be this and this is a larger order promo with this percentage discounts okay so you can implement that using a closure right because then when you call this function here it will return a function that has this arguments in its closure so it will be able when it's invoked later it will be able to recall the percent argument right and so this is the way to do it I'm going to skip ahead because I am running out of time but the parameterized strategy with callable actually I've already published these slides on speaker deck, speaker deck slash Ramalho, my last name, you will find it there. But anyway, you can also implement that with a callable. So the callable is a class, so you instantiate it, and then the class implements the dunder call, right, so the instances of the class behave as functions. Now, which one is more idiomatic? I think the classic strategy feels too verbose for Python. And I put a star here to say, yes, this is subjective. I'm talking about style. And I think people don't realize, I think we who work with computers try to think of ourselves, we are big fans of Mr. Spock, right? So we like to think of ourselves as very rational people. But actually, I think for a lot of people that won't admit, In my case, certainly, and I do admit it, aesthetics plays a big role in liking or disliking a language. And so this is a matter of, it's a very subjective thing, it's a matter of style. So I think the classic strategy feels too verbose. First class functions are very common in the standard library. For instance, the key argument for the sorted built-in is a very good example of using a first class function. the use of closure now for the parameterized version the use of closures is common in Python and the non-local keyword that was introduced in Python 3 makes it better provides better support for closures in Python but callable objects is something very uniquely Pythonic, this idea of implementing a class that has the dunder call method. And Graham Dumpleton, who is probably, I don't know, the pope of decorators, he has written awesome stuff on many blog posts about how to do industrial strength decorators that survive metaprogramming and frameworks and so on. He highly recommends using callable classes for decorators because then it's easier to manage the whole context. So to wrap up, the idea is that instead of trying to place languages in these broad categories called paradigms and some languages overlap and others don't fit, what you should do is focus on the features. Learn to look at those essential features that are not Synthetic sugar and are not libraries or functions in the standard library They are things that are in the core of the language And if you identify those things Then you start to get a better understanding So from this to this And then you can organize your thoughts by doing a little table like that you know, oh, these are the languages and these are the features oh, this is interesting, oh, yeah so that pattern that I used in JavaScript makes sense in Python too because of this common feature ok so, that's it my message, inspired by the Michael Scott book Programming Language Pragmatics is why should you learn the fundamentals because it will allow you to learn new languages faster that's why I promised in the subtitle of the talk in the program that this was going to help you not only understand Python but also other languages so you'll also be able to better leverage language features you're going to be able to choose among alternative implementations what makes more sense in the context of this language to make sensible use of design patterns and not use a design pattern just because it was anointed as a design pattern in 1994. And debug hard problems. And this is very interesting, emulate missing features when they are helpful. So this is one reason why when you learn a new programming language, you probably become a better programmer at the old programming languages that you already knew because you're going to learn new patterns and new ways of doing things that sometimes are also applicable or you can imitate in another language. Okay, vielen dank.