Dr. Schmood's Notebook of Python Calisthenics and Orthodontia

Speaker 1 [00:04]

Who here has used Jupyter Notebooks? Okay, that's why we're here, I guess. Okay, cool. So there's some interest in Jupyter here. I won't be talking so much about Jupyter specifically. We'll be talking a lot about Python specifically. So I guess we should start at the beginning, which is this unusual title for a talk, Dr. Schmood's Notebook of Python Calisthenics and orthodontia, don't get bit by misaligned state and output. So, one of the primary complaints about Jupyter notebooks, of course, is that there's this hidden state problem. It's very hard to reason about your code in Jupyter notebooks. And I'm going to kind of contextualize how we might actually think about this from the context of the 1970s. The title of this talk is a play on a 1970s computing journal called Dr. Dobbs Journal for Computing Orthodontics in Calisthenics. And it was a reference journal for users of home computers distributed by the People's Computer Company in the 1970s and 80s. And in fact, all the way up through 2014 is when they finally stopped publication. This is what it looked like in the 70s. This is what you get in the mail. You see the newsprint here is a very cheap way to distribute this information to a broad number of people. And basically, it would be oriented in different ways depending on how the code would be formatted. So, you'd get this in the mail, and then you'd sit down at your home computer and you'd type in the program, right? And that's how you distribute software in the early days. When the editor came on board, the editor James Warren came on board, he kind of contextualized the reason that we do something so foolish and so silly as like sit down and type in all these commands. And he said it as thus, the mass information processing power will truly become available to the general public. And I see that as having a tremendous democratizing potential for most assertively information and the ability to organize and process it is power so this is the big idea and in fact actually we're still struggling with this idea we're still struggling with the ideas of power and computation and also how to distribute that power in the guise of code and data and how it's organized and this is what jupiter notebooks are really about right it's a way to organize our code and data in a way that is purportedly easier to read by those who are going to download the Jupyter Notebook and go through it and da-da-da-da-da. So that's kind of why we're here. Now, one of the complaints about Jupyter Notebooks is the fact that there is a hidden state problem, meaning it's easy to get lost in what the value of values are in your particular notebook. And my argument today is that this is more of a problem with Python than it is with Jupyter. And we're going to go ahead and look at the problem in detail and then talk about how this might be avoided. Before I go on, I want to contextualize this also in the sort of environment I'm using. This is a tool called Next Journal. And so this is another notebook authoring software with specific features around immutable stacks that allow you to take your computation environment with you and your data environment with you. So you can run a notebook in one click. So that's what I'll be presenting in today. And I will be running code from within this notebook. For example, running code in my presentation is as simple as clicking that button. So I'm going to get back to that though in a second. Okay. So we have to start at the beginning. What is the equality sign because that's how we determine a lot of state in Python. And when we grew up in grade school, we learned some things about equality. Equality has these characteristics. It's reflexive. It's transitive. It's symmetric. Transitive, if A equals B and B equals C, then A equals C as well. Symmetric, A equals B, B equals A. But the problem is it doesn't quite work like this in Python, right? Syntactically, there is some sort of disconnect in Python. That's partially because the equal sign in Python is assignment and equality is the double equals. We'll get to that in a second. But so when we look at sort of a very basic Python, a trivial Python example, A equals the string of 10. I set B equal to A. I update A. And of course, a is now equal to 100 and b of course is not updated and that's because string a string is what's called an immutable sequence an immutable sequence of characters and we can see that a and b are not in the same location uh in memory okay this is a great premise except for the fact that of course not every data structure in python is immutable there are also a mute also mutable data structures as well, mutable sequences. And so here we see the same code roughly. We have the B equals A sort of business. And here when I update A, also B has been updated. So the equality operator works kind of differently depending on what type you actually give it. And so here we have a balanced operation. And when I look at where A is a memory and B is a memory, they are indeed in the same place. So the code is very similar, but the equality operator is behaving a little differently. So this has been a controversy in computer science for some time. This is not new to Python. Niklaus Wirth, let's do it correctly, Niklaus Wirth is one of the primary proponents of an alternative way of assignment. So much so that, and he was so, he lamented so much how people pronounce his name, especially in the United States. He made a joke about it because he was so associated with values and equality and assignment. He used to say, you can call me by my name Wirtz or call me by my value Wirth, which is what most Americans call him, Nicholas Wirth, not Niklaus Wirtz. And so he had this assignment operator, which was a colon equals. This is what look like. This is him in front of the Lilith computing environment that he built, but he's probably most famous for having created Pascal, most famous for creating Oberon and Modula, those languages. And this is what his assignment operator looked like in context right here. We can see it in this atta code. I had the fortunate experience of doing some atta code when I was in college, so for a couple years. And this is what this assignment operator looks like. So what's the big deal, he says? Well, he would say that, he says that a notorious example of a bad idea was a choice of the equal sign to denote assignment. It goes back to Fortran in 1957 and has blindly been copied by armies of language designers since. Why is it a bad idea? Because it overthrows a century-old tradition to let equals denote comparison for equality, a predicate that's either true or false. But Fortran made it mean assignment, the enforcing of equality. In this case, the operands are on equal footing. On equal footing, the left operand A, which is a variable, is made to be equal to the right operand and expression. So therefore, x equals y but that does not mean the same thing as y equals x and then we read this for example and we think about the state or what state is held in a we can read this particular statement as set a equal to 42 or a equals to 42 this is sort of how people read these things whereas he felt like this is a more clear syntax because a is assigned the value of 42 and because 42 is immutable, which it is in Python, because it's the fact when we have actual assignment, A becomes 42. But of course, there's no guess that 42 becomes A because 42 is immutable. It will always be 42. And when we stop using the number 42, it will be garbage collected. So this is part of the change with garbage collection as well. Okay. Now, this isn't the only way we can define state in a language. For example, closure. here i have a def a define function a define function that just defines the c we mostly see it more like this this is the same as we see in python right def a to 10 and the difference here is that is the wording the words do matter assignment is to specify a correspondence or relationship between a and this other thing on the other side to define is to define its essential qualities. And in fact, the essential quality, for example, of C is nothing. C is just not even nothing. C is just a place in memory that's held. And that's all it is, right? But in these cases, A is defined as 10, the string that we would expect, B equals A. And because strings in every data structure enclosure is immutable, we get the result that we see in Python when we deal with also immutable features or immutable data structures. So we have this difficulty of managing state in Python because some things are mutable and some things are immutable. And we're going to see this directly bite us when we talk about execution order in notebooks, which is one of the big complaints about it, about notebooks. But before we get there, we have one other thing to talk about with hidden state in Python, and that's about scope. Most of this should all be familiar to this room. I have state in a global place, and then within a function, I update that state. And then when I run that function twice, of course, I get two different outputs because it's updating the state in the global variable. If I still introduce the state but pass in the state and keep it local, then the two functions behave differently, where of course they behave as we might expect as good programmers, where they are equal to each other. So here I run the function twice, they are not equal to each other. Here I run the function twice, they're equal to each other. It's a little difficult to read. Probably the best way is not to introduce any global state at all, but of course just run it by passing in the immutable value itself. This is the easiest to read, and of course, as expected, running the function twice, it will equal itself. Now, this quality of actually referencing state outside of the scope of a function is pretty common and can be pretty innocuous seeming. Here's a very simple statement with an anonymous function, where of course, I'm going to pass in this one to this anonymous function, and I get two. I have the exact same anonymous function, but oh, do we remember what X was set to? Okay, we don't remember. And this is what people complain about with notebooks, because I've gone through this whole notebook, and I've gotten to this point today, or at this point in this talk, and we forgot what X is set to. And this happens all the time in notebooks, where people set X to something in one cell, like way in a different section, and in this cell, it gets updated, and we get this output, six. What does six mean? Why was x five? We have no idea, right? This is what we've got to stop doing in notebooks and Python, but Python makes it so delicious and easy to do something like this. So, this is why we can't stop and help ourselves, right? Okay. Well, the way, of course, we solve this is through local bindings, and local bindings work differently in different languages. In a functional language like Clojure, for example, this is how we might work with local bindings where we use these let statements. This is yet another assignment operator, isn't it? So we've talked about the assignment operator. We've talked about equality. We've talked about defs. We've talked about lets. There's so many ways to assign state. And this is why it's so hard to think about state. What's the best way to think about state, to assign state? How do we set up our scope? This is what makes it difficult. Not notebooks, but just fundamental computer science. In this case, in Clojure, we have a lexically scoped let A2, which means make the symbol A the integer 2. Don't assign it or whatever. Actually make it that thing. A will never change. A is going to remain 2 for as long as that let is in memory. So it's actually not even a variable. And so we're going to always try to keep our scope local, like we would in Clojure. For example, let statements don't actually even intern anything in memory, like def statements in Clojure. It doesn't even put anything in global namespace. This is the evidence of that. um and so and so um when we when we go ahead and think about how we're going to apply this to our everyday to our everyday uh practices and notebooks i think one of the best places to start one of the most practical places to start is with something like pandas because pandas has an opinion about immutability and they have an opinion about immutability because this is the thing that always catches people, right? Scope, immutability, variable assignment. And this is their opinion on it. This is from their documentation. In general, we like to favor immutability where it's sensible, okay? So that's kind of a middle road to take. Okay, so this is what it looks like. I read in a CSV file. I set it to a global variable here. Then what I'm going to do is take a look at the headers of this CSV. I'm going to drop a column named ascension number and then i'm going to look at the headers again and here we have ascension number and here we have ascension number twice because of course it's immutable i dropped the column but it didn't change the state of this global of this global variable that's what it means to be immutable uh but pandas does offer explicit immutability you can explicitly change something to mutable and uh through this uh key uh through this boolean here and i run the same thing. And the first time I have a sentient number actually listed and the second time I don't because it's been dropped because I made it mutable. The problem is, that's actually not really the problem with pandas. The problem really becomes that data mutation actually is possible within pandas. And so, for example, here I'm looking at the art of Robert Blake. and um in this table i have an acquisition year and that's what i'm going to take a look at 1922 uh i'm running through my ink function that i defined earlier a pure function and increases it to 1923 but it doesn't change anything because of course i am not actually changing the state of that table but i can actually change the state of that table uh if i like through of course um this equal sign again. And so I run these two cells, for example, and I run them in the order that we see here, where I have true run first, and then I have false run second, where I actually set up the date. And the true is just to see if it's less than or equal to 1922, the acquisition year. And the second time, I updated the state by increasing the year. I see if it's less than equal to the acquisition year. It's false. This is what we'd expect. But the problem is, if I now run this again, it's now false. The cell is false, right? This is the problem with the mutation. This is the problem with reasoning about code in Python and in notebooks, because the state is mutating somewhere in some cell that's kind of hard to track. It's actually kind of hard to track everywhere in Python, but this is just something we usually accept. So what we need is something with guaranteed immutability. So something like StaticFrame, for example, is a library that offers guaranteed immutability for table mutation. There are all sorts of tools in Python and libraries coming up that are guaranteeing immutability to solve this particular problem, and it's going to make working with notebooks much easier. So if I use StaticFrame, for example, I'm going to do the same thing. I'm going to load in this CSV into my global namespace. And here we have, I'm going to pick up the acquisition year again, and I'm going to increase it as we'd expect, because this is immutable as we expect. The acquisition year, which was 1922, even though I increased it, is still 1922. It is immutable. The thing about immutability is what it says is if you change the data, it is new data. We'll see that in a second but if you change the data it is new data it's not the same data but different it is new data so you can change something in um you can change something in static frame but you have to give it a new name and once we give it a new name and we set it to a new data frame now we have of course as we expect that new data to reveal after the increase of the one value that of course it doesn't pass this Boolean. It fails. So, this is what we're focused on here when we are thinking about how to improve not only our notebook code, but also our Python code in general. We're thinking about where we're holding state and how we're going to tell other people that read our code how we're managing the state, right? This is the problem eternally because someone else, it's it's almost entirely code is almost entirely written for other people to read and only incidentally for computers to execute right so we have to communicate that to someone somehow and immutability is one of those ways that makes reasoning about the code easier and the other way that we saw actually is through declarative syntax the thing that numpy and pandas are starting to bring to python and changing with how people think about python is one of the first rules in data science with these libraries is don't use for loops. Don't write your own for loops at least, right? Use our packages. We have optimized them and declare what you want and we will take care of it for you. So pandas is showing this tendency towards immutability. Pandas is showing this tendency towards declarative syntax, right? And if we take a look back at the state of the art in 1976, we actually might discover how little has actually changed when we actually write code for other people and actually how we write code. So, from Dr. Dobbs, you can't read that. This is a page from Dr. Dobbs, though. And this page is a typical printout from what you might receive this quarter from the journal and what it is is a simpler uh for a for loop in basic in the programming language of basic so you type this in and you now can write for loops in basic right so this is the code that actually does this and the thing that i love about this is it's actually very readable if the print is big enough it's actually very readable um it's well commented It's beautiful code today. I mean, I can read it today. Okay. And this is a symbol for 6502 microprocessor. Okay. It doesn't take a genius to read something like this, but the thing is writing for loops, because that's what a manipulating state manually and doing all this, this is 1970s style programming. And the reason you'd actually write something like this is for other people to cut and paste, for other people to go to Stack Overflow and cut and paste, right? How much has changed since 1976? We have for loops, we have unmanaged state, but we don't have to do this anymore. We just do it the same way. And then we distribute it on the internet, on electrons, and they can't even read, they can't even run the code, okay, in a web browser on a computer. This is lunacy and it's 1976. It's still the same as it always was. And so if we actually want to be programmers of the future and not another iteration of Dr. Dobbs, we need to consider how we handle mutations. We need our global variables to be immutable. Otherwise, they're going to be impossible to reason about. Our syntax, we should favor declarative expressions over imperative statements. So our maps, our reduces, and allow our Python library or Pandas libraries allow the underlying machinery to actually figure out the implementation details. We want to stay local as much as possible. And finally, we want to, I think, publish the computational environments and the data with our code so it's runnable. And this is one of the things that we can start doing with Jupyter. We don't have to rely on GitHub. And so, for example, something like Next Journal here, this notebook, I ran it in this presentation, and now I'm going to actually publish it at this URL. And now it's available at the URL on the screen, and anybody can come in and rerun it, rerun the closure code, rerun the Python code, use the data that I have. It's all right there in one package. Okay, so go forth and program. Thank you.

Speaker 2 [23:04]

really just in time so we have enough time for questions are there any questions i mean i i think he also accepts mutable questions so you can change the content yeah once they are flying in yes

Speaker 1 [23:16]

Yes.

Speaker 3 [23:24]

Thank you. Are you aware of any maybe like static linters or something that might detect mutable changes or global changes? If, say, like a team wanted to have a policy to say no value changes in their script.

Speaker 1 [23:41]

script i can't think of anything does anybody that seems like a very difficult problem because yeah please my pi has a finally statement and then the static type checking checks whether the variable ever changes and okay my pi yeah there's probably a way to limit this with very with various uh narrow to narrow tools but a linter like a like a broadly applicable linter to take in like all your code and and try to deduce um uh global mutation that's a unique challenge.

Speaker 2 [24:28]

more questions not okay then let's thank Daniel again