Refactoring Large Programs

Speaker 1 [00:06]

Two years ago, in the same place, I gave a tutorial on the basics of refactoring. This tutorial session is where I want to level up my own game a bit. I tried to come up with something more challenging. And I heard that for some of you it's challenging already because there's a dependency that's hard to install. I'll comment on that. And this is exactly one of the reasons why I want to talk with you about these larger refactorings today because this is what you find when things get a bit tougher. I will make a few remarks at the beginning, tell you what we want to do. And in the meantime, if you're still installing dependencies, take your time, try out the program, see if the web app launches. The web app should be the easier one to launch. See if you can launch the GUI interface. I'll ask in a couple of minutes for whom of you, one or the other, is actually working. And then we see what we can do with it. So for the whole thing to work, I think the web app is the more important one because that is what we want to have running in the end. but the GUI is worth thinking about so if you want to have something a minimum viable path through the tutorial then focus on the web app it's the more important one to have running in the end so I've written a game where there's a space traveling panda I put it in my git repo github.com krotor slash space this is a game that i started writing i think six or seven years ago and i used this to try out different programming concepts every now and then and it grew and grew and grew over time so i had several rounds of refactoring in it already just for the for the fun of it and trying new things the first attempt at the graphical interface was with the arcade library and that tends to get in the way in many things. For instance, that it's difficult to install and it's also not compatible with more modern paradigms. So the main goal that I want to pursue today is actually to make the game work with a modern fast API-based web interface and HTML template. normally I would allocate for this kind of refactoring with a arm with a program that has roughly 1,000 lines or above a couple of days at least and maybe maybe anything up to a week yeah I tried to condense this into a 90-minute tutorial anyway. Because what I want to talk with you about is a systematic approach to this type of refactoring and disambiguate that a bit from the easier stuff that people also refer to under the name refactoring. Let's take a look. There are easy refactorings yeah where the workflow is very simple assuming that you have any type of automated tests you run the tests then you modify the code and then you run the tests again and if the tests are green after step 3 then you can move on to the next refactoring or start adding a feature does any of you know an example of such a easy or small refactoring? And I'm going to repeat your answers to the online audience. What could be an easy refactoring? Yes, please. Moving some functions from one file to another, reorganizing things. Moving files from one function to another, reorganizing things. What else? Yes. Abstracting some common classes Creating an abstract class from common objects, Ina? Correcting errors that may occur in the code or modifying the code so it becomes more robust against these errors? Making the code more robust. Anything else? I can take one or two more suggestions. What examples of small refactoring? yes fixing typos well if the test should pass before otherwise it's a bug fix they are not a refactoring so that the definition of a refactoring is we we want to improve the structure of the code without changing its functionality Not in my head performance optimization yes why not why not yes absolutely performance optimization could be could be a reason to refactor your code absolutely so there's an entire book of these about these small refactorings by Martin Fowler that is a classic in the field and what Martin Fowler writes that these small refactorings they can be done in minutes now what I want to talk today is about larger refactorings yeah what about what if your refactoring breaks the tests or you don't have any tests to start with yeah or your code is total mess I assume that some of you has any one of you found encountered this type of situation in the real world yes yes yes I spent most of last year doing stuff like that yeah and then this easy refactoring workflow doesn't really work that well on the other hand you may have to do these small refactorings to make the bigger refactorings possible. In preparation of this tutorial, I took my code and I did all the small refactorings that I could find to make the bigger issues easier to see. You should do some cleanup work. If your code is a total mess, make it a little bit less of a mess first and then we can talk about complex refactorings. Most of the bigger refactorings that I have seen. They are about putting some type of interface somewhere. And then the workflow becomes run the tests. If you don't have any tests to start with, your step zero is write a test, at least one. Because usually you can get to something like 50% test coverage with relatively little effort. Then you have to define the new interface. This is the hard work. This is really the hard part. I'm taking a shortcut here, because looking at your code and thinking about what the new interface should be, this is the part that could take you and your colleagues days, if not longer. Maybe it turns out that your first attempt was wrong and you have to throw it away and start over again. This happened to me a couple times and I believe it's part of the game. Then you write tests against the new interface and implement the new interface with the old code. This is what I want to do here. Then you run the tests. The actual refactoring, if you have done steps one to five, should then become a bit easier and you run the tests in the end to get us a little bit into this space game code i tried to to break it down into into some overall structure i draw a dependency graph of the python modules i was thinking whether i should also make a class diagram depending on what you're working with this the step drawing things. This can be a very very good idea. I got away with one diagram because the class diagram would be a bit similar. You have to decide which diagram is the good one but you start seeing things here. In my case I drew the dependency graph of a program that I have worked with over a period of six years and I saw this a a couple of days ago and thought, oh, is it really that bad? What do you see here? Sorry, which? So we see which Python module imports which other one. Yeah. yeah why why do you think is this is this a bad example of a of a dependency it's we have tight coupling of of everything to everything and this is the reason why I want to refactor so the we have two modules here the GUI and the views py that that do the graphical interface and then we have at least two modules, the game and the location, that handle the actual things that happen in the game. And they are somehow all interconnected. And I don't know when looking at this, where should my web app that is there on the right currently in some kind of prototype state, I don't have any spot where I can connect this. the problem that I want to fix in the next 80 minutes yeah so that's the whole point of the exercise yeah so I want to approach the left part yeah and to make connecting the right part possible I do have automated tests the the automated tests they they are a bit crappy because they hijack the keyboard yeah and then then push keys they are not very readable and they tested the graphical interface and so we have to think about what we do about these when when you think about your your code on a high level you may start to see some some patterns emerge if you are lucky you will see something where you think hey there's a design pattern that I read about. Sometimes it works, sometimes it doesn't. I picked here a case where you find one of the simplest design patterns possible, that is the facade, because this is a well-documented way to engineer an interface. I show you the next graph and we see what we are talking about. The The facade provides an entry point level interface that other parts of your software can talk to and it hides away complexity. This is what we want here. So I want all the game logic somehow to become encapsulated so that the GUI can connect to the facade and we can use the same interface to connect a web app. that's the idea. If you look up the facade, has anyone read about the facade pattern before? Anyone familiar with it? A few, because that is one of the, from this famous catalog of design patterns. If you look it up, it writes, yeah, the facade is a class. I don't care here, it's going to be a module because in Python, the difference is not so big. Yeah, the main point about the facade is that we define an interface in the first place. Good. So this is what we are going to build. In a few moments I want to dive into the code with you, see what we find, maybe get some hints from you what we can fix. Let me get a few, a quick show of hands. Did anyone manage to get the web app to run? Please raise your arm. It's quite a few. Did anyone manage to run the graphical interface? That's a few less people. Okay, Arcade remains a bit elusive. So we may not need to dive too deep into that part of the code. So I'll keep that short. Is anyone stuck with an installation question I think depending on the type on a annotation that I used it should be Python 3.10 or above 3.10 work I use 3.11. I did not dare to try 3.12. But 3.9 definitely should not work because the type annotations work in a different way. This is part of the fun that you can have when you do bigger refactorings. Thank you for the question. Good. So let's take a look at some of the code. Actually, I want to start with defining the facade that we want to build. Let's make this a bit bigger, or a lot bigger actually. Is this big enough? In an ideal world, if the arcade library is working, if it's properly installed, you should have six tests that pass. In a less ideal world, the program is already a mess at that point, and that is also something that might happen in a complicated refactoring situation. These These are tests against the GUI, they are no good when we think about the façade pattern. So we need to replace them by something else. What I want to have is, I want to create a façade. I'm going to create in the folder with all the Python modules a file façade.py and open that in my editor. Now, before anyone objects that, hey, facade py is really a crap name for a module that actually does something, you are not wrong. But renaming the module to something that is more descriptive, this is a small refactoring. That is something that you can safely push to a point where things are working. Then you can come up with better names, if you don't have a better one right away. What I want to have in this facade is two functions. We need two functions for the game. This is all documented in the documentation for the tutorial, but I have that stored in my head. We need a function that starts a game, that requires no arguments and comes back with a game data object that we need to define and then we need a second function that allows the player to execute a command and there we need two arguments. One is the ID of the game. For simplicity I'll make that a string. And the actual command, which is going to be a string as well. And again, we get back a game data object. This is the interface. This is basically what the web app requires to work. Now, if we dig a bit deeper, what does game data consist of? There is something prepared in the app.py file. I'll drag that over into the facade. So I go here, I have two Pydentic classes in the app data and I move them to the facade so that we have a data model here. Anything that I should add? The imports, we need imports. Can you tell me what should I import? from pidentic import base model from typing import optional okay so we have we have two objects here one of them is the locations these are like planets and and other places that you can visit in the game. They have a name. They have an image. And they have a description. All of these get onto the screen one way or the other. And then we have game data that catches the entire game situation there. We have an ID. I make this a little bit smaller here to save one line. Then we have a reference to a location. Then we have cargo. That is items that you can carry around, basically, because we are making an adventure game. Then we have a crew. You can collect characters. If you manage to play the game later, then you can check how many characters you can collect and actually solve the game by collecting all of them. But I'm not giving away any solutions. Then there is a list of things that you can do. I call these commands. They get printed as well in the menu. Yeah, like if I go to my web app, is that still running? Ah, no, I stopped this. OK, sorry. And then we have a message that is optional stuff that appears on the screen. So this is all we need. This is all pretty static data. What we have here is a so-called data exchange object. If you look into the game PY, you will find that there are very, very similar objects that store the internal state of the game. But they have their differences. These ones we use to pass data around, between the core game objects and our graphical interface, or between the core game objects and the web interface. And defining these can be quite a bit of work. I think getting to this took me like two or three attempts from a program that I was very familiar with to start with. And I think Pydentic is a great tool to model this type of thing because it helps you to avoid code clutter. Any questions or anything that you'd like to add at this time, at this moment? So you are right that everything should be imported from the package space game. Maybe there's something wrong with the install path. What I usually do to temporarily fix this kind of thing is that I modify my Python path and put the folder there that contains the space game folder. So there's a question in the back. Yeah. So I approached this from two ends. So, okay, I repeat the question. Thank you. So what was the process? is how I got to this data model iteratively. First of all, I reviewed what I had. So in the game.py, if I look into this, we see that there's, oh, let's look at location. I think this is more descriptive. We have here more attributes. It's a slightly similar class, but it contains more attributes because the locations, they are interconnected. There is a fully connected graph of planets and places that you can travel from A to B. So what I did is I thought about, hey, what is actually needed by the web interface in the end? And my biggest headache was that I cannot put anything into a JSON that contains like a circular dependency. That was the hard part. And then you basically go through the list and think, decide for each of the attributes, is that something that you want to go into your data exchange object? Or do I need to come up with some different structure altogether? Because this is where the internal logic happens. So I basically tried to find a good subset that provides everything that my user interface needs to do. What I did on the user interface side is that I defined my endpoints and had it create some fake data that I can then render and see, ah, does that really contain everything that I needed? So for instance, trying out some cargo item, if I want to see that on the screen, it has to be somehow part of the data exchange object. So it was a little bit back and forth movement from both sides. In a real world project, this also having discussions with colleagues around exactly this topic, this usually helped. There's another question in the back. Yeah, so the question is, could I briefly show the game how it looks like? Let's see if it's still working or whether I broke it already. Let's take a look at the game. We are on Pandalore, the home of the pandas covered in bamboo forest. What can we do there? We can pick up bamboo, then we can warp to the Turing planet. We can warp to Kubernetes, a planet inhabited by powerful whales. We go to the city on the planet, a harbor city with impressive docks. And we can talk to the whales. The whales trade bamboo for a strange piece of DNA. and the game is then around about you take the DNA to some other place and you probably will get something for it. So that's the entire game is like this. At some point, additional crew members will turn up. Does that help? Okay. So assuming that at some point And we get to this facade that this is what we want to have running in the end. And you have some basic agreement on this. This is a great piece of work already done. You definitely, definitely should write a test for it before even continuing. So I'm going to write a test. I take the one that I prepared from the from the actual tutorial material. Let's see if I have that here. I don't need this. Let's create a fire test façade. Let's see if we can write a test for the façade. The name façade is still not good, but I stick to it because I had it in the actual module. I want to have a test start game, and for simplicity, I'll grab the code from my own tutorial. Let's see, where did I put this? I think it's in the refactoring.md file step number 8. You can copy this from github page, it's a little bit more readable. I copy that out of step 8, leave a comment here, code from step 8 in refactoring.md. Oh yeah, and And of course we need an import of our two functions and as well the data exchange object. And these tests are very, very simple. All we want to know is does the start game function really return a game data object. You can argue, yeah, if you have some good type checker in your editor, you maybe don't need this test function at all. It's an absolutely valid point, but I'm sticking to unit tests right now. And here in the execute command, we want to modify the data, so we actually run the first command from the list executed then we get some new game object and we want to make sure that the two are different and but they have the set they should have the same idea so we want to make sure that something is happening we don't we don't even want to go into details and now we are already at a point where we could run the tests for the facade so I go back for the command line we at this point we do not need to have arcade installed I can say pi tests test test facade and oh yeah of course the tests fail what did I think hmm why do the tests fail. Oh, we did not implement anything yet. At least it's clear what we want to implement here. I'm now back in the facade module. So what do we want to implement here? We need to create one of these game data objects that has the actual game. Usually at this point, when you do a larger refactoring, it becomes a little bit of a tail chaser. You have to jump through different modules that you have worked on previously and bring together pieces of code. I'm not saying that it's going to be pretty. So it's a little bit of puzzles. sometimes you may have redundant pieces of code flying around, and it's OK. In this case, I go into my GUI module, and I borrow a similar function that has been hanging around here. There is a start, a function for starting a game. But starting a game is actually nothing that a GUI should be responsible for. This is one of the reasons why we have tight coupling in this code. So this is a bit smelly. So I might take this function out right away, move it to the facade, and say, OK, to start a game, we need to create a galaxy. We need some configuration variables. And we need to create a space game object. The space game object is from the game module. And I delete the return statement because it's wrong. Now, we see that we also will have to borrow some of the imports. What do we need to borrow? We need to import Create Galaxy and Space Game. I can delete these from the GUI right away. So we are decoupling code as we move on. I can take this entire section here and put this here on the top of it. And everything should be clean by now. And now we get to a point where I need your help a bit, because we need to create one of these game data objects. So how can we create a gamedata object? Anyone has a hint? We need to pick at handpick attributes from space game. So if some of you could look into the actual space game class, see what attributes we have there. And they should be named rather similarly, so that we can create a game data object here, because that is the return type that we need. Let's go through everything one by one that we have here in game data. Because pydentic requires all those attributes to be filled in, and it requires all of those to have explicit names. Does space game have a game ID? No. So then let's fill in just some arbitrary string for now. Because we want only one game to be active at a time. For the web interface, we might want to have at some point the feature that you can run multiple games in parallel. But this is an important point in an advanced refactoring. Making multiple games in parallel possible, this is a new feature. It's not a refactoring. It's very tempting to say at this point, hey, Let's build that in right away. But then if you do that, you are making your life really hard. So let's keep this as simple as possible. Does the space game have a location? Yes. OK. But it's a location. It's not location data. So we need to translate this into a location data object. And that one, if I recall correctly, had three attributes. Name, does the location in the space game have a name attribute? Yeah. OK, what can we then write here? Name dot location dot name. Oh, that makes sense. Thank you. What else has an image? Does the space game location have an image? Easy so far. Okay, game.location.image. Description, thank you. Description. say exactly the same way, right? Game.location.description. So that is a characteristic that I've seen more than once in these data exchange objects. That you move these, that you basically are copying attributes around. This is an indication that there is further clean-up possible. Because I start asking myself, huh, why don't we use these data objects in the first place? Yeah, it's a valid point to think about. But I don't want to think about these now. I'd rather create an extra data structure to have a clean interface. What else? We need cargo. Okay, the cargo is optional. We could say it's empty, but let's have it there. Does the space game have cargo? If the cargo, is it a list of strings or is it just a string? Oh yeah, but the cargo that we need is also, it's an optional string. If it's just a string, that should satisfy all the type requirements. Yeah, the other way around would be more problematic if you run a type checker over your code that would complain. So, I can say game.cargo, crew, does the game have a crew? Oh, it's a list of crew members, it's a list of crew members. Interesting whether, so we have a a little bit of an inconsistency here, because while doing these small cleanups, I included a literal. These are all the crew members that exist in the game. This is a very cool thing to make your code cleaner. A type checker would start complaining at this point. I've actually done this myself in the past, that I've been thinking maybe enums would be a better solution? Do you have an opinion on that? I switched from enums to literals because the type checker works with the literals so nicely. I don't want to get into an opinionated...

Speaker 2 [39:14]

I remember to repeat the questions.

Speaker 1 [39:16]

oh yeah sorry so the question was would it be possible to use an enum instead

Speaker 2 [39:16]

Oh, yes.

Speaker 1 [39:23]

and what do I have an opinion which is the better solution I've made good experiences with literals yeah because if you heavily use type checking which is really not the topic of this tutorial they make a lot of unit testing unnecessary. Yeah, so they keep your tests short and they catch a lot of issues early on. So for this first attempt, I think I'm going to ignore the difference between a string and a crew member. And I say, yeah, it's some sort of string. Let's buy it. Yeah, so let's take the game.crew. But it's important. Thank you for pointing this out. It's, by the way, something that I did not notice in the five test runs that I did with myself on this tutorial. Oh yeah, it's a list, sorry. So we could 4s in game.crew. You mean something like this? Okay, that should at least satisfy the type checker, thank you. Yeah, why not? Works without, but let's see. What about commands? Is there a commands attribute in space game? There is not. Where do we get our commands from then? Oh, there's a method. You know my program really well. Thank you. Get commands. What's the return type of get commands? What's a command? I don't remember. Let's check. Command, command, command. Oh, here. It has a name and a callback. I thought I made callback functions. This is actually another design pattern. This is the command pattern that you get past callback functions. Really nice for a GUI. I think the original design patterns were invented for making graphical interfaces interfaces some 40 years ago, no, 30 years ago. That was when the, 30, sorry. It's less useful for making a web app. We are only, for the web app, we are only interested in the names. So we need the name for every command that we get. How can we check? How can we get a list of the command names? List comprehension, or whatever the name is. list comprehension. I love list comprehension. cmd.name for cmd in. And we are throwing away all the callbacks. Yeah, let's, we need to get them back sometime later. Yeah, but for now, should be fine. Anything else? Message. Message should be easy. Is there a message in the game, that was one of the smaller refactoring to add the message to the game in the first place. I think this was the hard part. This was the hard part. What could we do now? We We have implemented the first façade function. Is it still the matter of createGalaxy? Oh, with the createGalaxy, we did import the function here. So it's there. The imports come from the original GUI code. Run your tests. Run the tests. And they will fail probably because you need to import OS. to import OS. Thank you for pointing out import OS. This one was under. This OS was underlined. This was probably hard to see from the back row. So let's run the tests. Tests for the facade. One pass, one fail. That's not so bad. I push back dealing with the GUI a little bit because it's not working on most of your machines anyway. This is the risky part. This is a very risky decision because if I say let's ignore the GUI for now, I'm basically be throwing away the power of the tests that I already have. So it would be nice to make these tests run at some point later, but for now, I'm happy with what I have. Let's implement the second part and see if we get the facade to work. And at that point, I would be happy enough push the code to my solution branch. So to execute a command we need to call one of those callbacks. Call the callback function given by command. So all we have is a game ID. But we need a game so that we can do game.something. How might this possibly work? We need an instance of the space game again we need some we need thank you we need to keep and mapping of the games that that we remember this is why we introduced the ID yeah that is kind of anticipating that somewhere in the future we may our web server may want to run many games in parallel, so we need some sort of persistence. Again, this is a very dangerous point in a refactoring. Why? Because when programmers hear persistence, they usually get very excited. You get all kinds of hints like, oh, we could use a SQLite database. No, let's use SQL alchemy. Maybe you should deploy this on Snowflake. And then you get lost in the details of implementing a database adapter. This has nothing to do with implementing the facade. It's a separate refactoring. It's a hell of a good idea to build in persistence. There's a reason why many of us are passionate users of databases. But it really does not belong here. We want to get the thing working in the first place. So we can do something really, really primitive. Dictionary, thank you. Games equal dictionary. And then we can persist. If you don't like this, you can leave a to-do. Move this to a database. A database. A S A P. Because there is a nice addition done to the Pydentic library very recently. That's the SQL model. Yeah, I learned about this a couple of weeks ago and I was really getting super excited. Yeah, I tried it in a different, very similar project. It makes your objects go from the API to your database and back in with a lot of, no, with very little effort. Maybe, maybe, yes, okay, thank you for the comment. Maybe SQL model is immature. I like the design. You can use something else. the trick is that to implement your program in a way that or your persistence layer in a way that the rest of the program does not have to think about it this is the sign of the result of a good refactoring so another thing that I had in mind that could be useful is to make like a database module in here and stuff everything that has to do with the database into that but I figured out that 90 minutes are not enough to do that but it's a good idea so let's take our game and store it in the games dictionary like this and my game ID is still one because I want the one only to appear here once I make a variable out of that right away. Yeah, and now we can extract data from our dictionary. Yes, it's still necessary because that game data is the data exchange object that the GUI will need and that the web app will need. The game data object, this is what they draw.

Speaker 2 [49:46]

So the question was if it's necessary that start game returns something.

Speaker 1 [49:47]

Thank you. Is it nicer to store the game data in the games database? The thing is we need to store the original object because the game, if we go to the game class, the space game class, has a bit of a different structure. And the location class that I think I closed it. Did I close the location class? maybe, has more attributes that our data structure here does not contain. It has an internal state that is hidden, that we do not want to expose to the web interface. Sorry, where I'm coming from, like now it's dark games, like, I don't know, you put a function as a method, like it changes the state of the database, and also it returns something, and I wanted to solve this. Oh, I see, I see, so the objection was, now we have a function that has a side effect, yeah, so you mean that it changes some internal state, and it also returns something. Yeah, good one, but I think in the facade pattern we need something like that. So we could, I mean, we could delegate the modification to some other module, yeah, that we hide away the state, but that is, again, I would treat as a further clean-up step. Yeah, but point taken, one might make a to-do out of that. So we have this persisted dictionary of games, and we can pull out our game out of it like this. And I don't care about key errors at this point. If I would care about key errors, the first thing that I would want to do is write a test against it. I skip that level of detail for now. And now we can go looking for the command and use the callback. How do we find the right callback? Sorry again? From the get command? Yes, we need to call get commands again. That gives us a list of command objects. For command in game.commands. Do we need to execute each callback? We need to check the name, so we only want to execute one. We only want to push one of the buttons, yeah? And we need to find out the right one. I implement this in a very straightforward way. if command.name equals command then we can say command.callback in the old design pattern catalog you would need something like three classes to do something like this. This is why I like Python and I don't like the old design pattern book by the Gang of Four that much So the question was, do we need to call the callbacks in the opposite order? No, but we are only executing a single command. We assume that all the commands are unique. we could put in a break statement thank you so that to make sure that exactly one command is called good one yeah good so what we need and then all that's missing that this command callback this changes the internal state of the game it moves us to a different planet or gains as a crew member or picks up the bamboo or talks to the whales or whatever. Now, all we need is to return the game data. Yes, please question. So we are the get commands is here from the game py module. We have game as an instance of space game. And there, we have a get commands method that returns a list of commands. So isn't game a game data object? Let's see. So what we store in games is the space game. So it's our core business object. It's not the data exchange object. The data exchange object is this one. This is what the facade is using to communicate with the outside world. And this is a core business object. I borrow the name from Uncle Bob, who has a couple of hours of video lectures on the topic. So, how do we create a return from the execute command? Any ideas? We have to repeat the cycle we did before with the game data. We have to repeat the cycle we did with the game data. So we need to do this entire thing again and again. And we have to do exactly that thing once again. So that's a perfect excuse to make a function out of it. So I make a function. And because we are in a facade, I mark this as it's intended to be private. Get game data with a game ID and a game instance. I could add the type annotation here, but no, not now. I move that little helper function to the top. Is it a bad idea to make this as a class method of a game logic? So is it a bad idea to make this a class method? No, it's not a bad idea at all. You can make this a class method if you prefer. The good thing is anyone using the facade should not notice. It's because it's really internal. Our facade does not change. It does not break any of the tests. So that means now we can call here the get, no, sorry, it's underscore getGameData that takes a game ID and the game. That's simple enough. So the facade is really mostly passing through data. And we do the same down here. I don't even have to edit the names of the attributes. Now, if things went well, there are two things that should happen now. A is that our tests for the facade, they should be working. Let's see. This is where I wanted to be about one hour into the tutorial. Good. So we have a working facade pattern, just in case. In case you missed out any of the code that did not fit on the screen, this is the moment where I'm happy to push stuff. I'm online. Good. Git, add, tests, test, facade, git, add, space game, facade, git, commit, implement, facade. Oh, I cannot commit because I was extra smart and built in a pre-commit hook. Ha, ha, ha. Yeah. So what does the pre-commit hook actually prevent that I actually push half working code? because the changes that I did, they break the graphical interface. So now we have to operate, before I can push, I will have to operate on the GUI so that it actually uses the facade. This is why these pre-commit hooks, they actually do good things. I was just trying to cut a corner and say, okay, yeah, we have six unit tests. Yeah, let them fail. It's fine. Yeah, but my own continuous integration doesn't let me get away with it. Yeah, let's try and make the web app work first for the sake of time so that we have a bit of occasion for Q&A. Yeah, because I think if I go to the GUI, The process on the GUI side should be more or less clear from this point. So we need to make the GUI and the facade play together. We may find some interesting things there, but there it's mostly cleanup work that we say, here, remove this. We don't need the create galaxy. We also don't need the space, the base path. Instead, we need from space game.facade, we need to import start game and execute command, and we also need the game data. But I'm not going into the details of the implementation right now. I think if you are aware of the order of things, then that is the part that you would figure out in your own project on your own. Let's see whether the web interface is working and what we actually need to change there because it's a bit easier. I'm going to submit afterwards, I'm going to fix the GUI, upload it. There's a branch solution in the repository where you can grab it afterwards. Let's look at our web app. This is a fast API script. Two weeks ago, I was in a sprint on fast API with a couple of colleagues. One of them, Tim Weber, found out a very smart thing. He figured out how to build in a switch that uses the JSON output of FastAPI and plugs in a Jinja template so that you get HTML code out of it. And I thought, okay, I can make a graphical interface for my game using this and still use API functions. so the code here stays really short and here we have two endpoints that I filled in with fake data for testing the HTML part we can get rid of that entirely now so we don't need this and we don't need Faker even though Faker is a fantastic library to write interesting tests quickly so we have two endpoints and they look very very similar to our facade so all we need is the API to pass through the the calls what will we have to import here. From space game dot facade import start game. We have here now two functions, start game and new game. Not a beautiful naming, but it's a small refactoring to change it, not the topic of this tutorial. Execute command, name analogous to action. Maybe you can think of something better. And we do need game data for the type annotation. So all we need to do here is to say, what do we need to put in here for the new game function? return start game that's it start game returns a game data object that is what fast API expects of us how about action return execute command And game ID and command. That's it folks. Yes we also don't need random. So now we have an API for the game that almost, if you have a bigger screen or use a smaller font size than I do here, then the entire thing fits on one screen page. Let's see if the whole thing is starting. The facade is tested after all. I'm not sure because I'm doing this in front of, I don't know how many you are, 100 people. I give it a 50-50 chance that it's going to work. So let's start uvCorn minus minus reload because I don't trust myself to work it in for it to work in the first attempt, then space game.app colon app. Let's see. It's starting. Okay. Complete startup. No errors at this point. The way the frontend works is I have to reload this. We get the front page, so nothing has been called yet. This is pure HTML here. Now we are on Pandalore. This is something that we did not have before, or maybe we had that before. Let's go to the bee soup planet. At this point, the previous endpoint came up with random data, and now we actually have the information from the game. Let me see see if I can pick up an item. I go to Adalof. From there, I go to Colabo. In Colabo, you can collect notebooks. Great, but I'm not telling you what to do with the notebook. Who might need this? It's working. I don't have a test that checks the entire game. I it with the GUI, but at this point, we've done quite a bit of the circle, and I prefer to have the time that we have left for a little bit of Q&A, for if anyone wants to share an experience about a larger refactoring they have, I'm happy to give space to it. I was just wondering if you do this new test and you disable the test of the UI, what do you gain if you know where you were refactoring before? I mean, over the advantages, of course, that you have, like, a more decoupled testing, but for the specific task of refactoring... Ina, could you hand over the microphone? Could you please repeat the question? Yeah, I was just wondering, we redesigned the test so that it's not a test of the GUI anymore. For the specific task of refactoring the code, the test will still be fine. Of course, it is better to separate the test and do it on a lower level, but for the specific task of refactoring, why would you just break the old test and create new tests? okay so so my uh so the the next thing that i would do here um would be definitely to to fix the old tests again they should still be working if we fix the the graphical interface yeah because even though the the tests they have their disadvantages um there is still value in them Like, my basic assumption is if there is a test, it's potentially useful. In this case, if I switch back to my architectural, to my dependency graph, the tests, they run against the graphical interface. If the tests would be testing something that is now hidden behind the facade, they would be probably a lot less useful because the structure that is now there in the gray box, these are the parts that I can now rearrange very, very freely. A, because we are decoupling them from the rest of the program, and B, because we have all the tests going in from the top through the facade. yeah so in that case you might if you see that you have 100 tests against location but you're not happy with the structure of that again you may might decide at this point hey here are tests but they won't help us anymore throw them away there's another question over here in the middle It's quite easy because there are no side effects there. So you go to the database presence layer. If you add a presence layer there at this point, so for example, the games are played somewhere, and the test will become a little more complicated before the market database. Yes, yes, yes, yes, yes. And not a database, but something like that. What would you do in such a case? So let me repeat or rephrase the question. So if we face the more complicated situation that our tests have side effects, like, for instance, we are testing this persistence layer, what would I do? Yeah, that we are engineering the persistence layer. The first thing is that I would seriously think about is to actually create that test database, create myself some mock database or write something that starts a Docker container with an actual database. One of the good things, if you isolate this persistence layer into a separate module, then that is something that becomes a lot easier to mock with test data. Because then it's independent of what is there actually running in the background. And so you can limit the amount of actual database operations run during your tests. Because if you imagine the situation that you have a test database, yeah, ideally all your tests, they would fill the test database with fresh data that might take a lot of time. So for example, if we're speaking about using some ORM, yeah, like a scale model or something, you would not deploy your local test database. You rather would just mock this wrapper of this database, yes? I idea is, so the question was, would I mock the wrapper of the database? Yes, there is a rule in testing that I find very, very helpful. only mock what you own. If you write a database wrapper, you own that code. And then you can freely mock it without having too many side effects that are created by that. It's pretty much a long question. Long question. Please, microphone. Thanks. So I kind of have a large project that I've been struggling a bit to refactor in the last couple of years, basically, mostly because I don't have time and other stuff. So the thing is that sometimes I sort of cheat when I write code in the sense that I discovered recently large language models in the GitHub Copilot and it's pretty useful for very nice boilerplate code. And a couple of times I've been tempted to just ask Copilot to just inspect my workspace and maybe try to prepare a plan for these kind of big refactorings with a more step-by-step solution. but I don't think that it's smart enough to do this kind of stuff. But I don't know if there may be tooling that would be able to allow this automation to just make a plan of the refactoring. To make a plan for the refactoring. So I haven't used... I did not dare to use any co-pilot or other large language model to actually do a refactoring. In fact, most of the situations that I've been in, the situation was too complex for me to trust an AI to do a good job. The situation that I can think it's useful in the context of a refactoring like this is coming up with a suggestion or identifying code smells or something like that. yeah but I have the somewhere somewhere deeper in the deeper layers of my my head I would believe that to you cannot figure out what is the best refactoring to do from the code alone because you have to know what is the code actually doing on the on the business and you have to have understood the requirements If you find a way to consistently feed your requirements to a large language model, then our work is basically done. But I don't see us there. I hope that some of you will maybe agree with me that figuring out what the requirements exactly are, this is what makes up a majority of our job and not not typing in the commands think we still have 15 minutes do we have any questions on slide or

Speaker 2 [75:41]

Should we have written a test first that covers the entire refactoring?

Speaker 1 [75:49]

So it's an interesting question. So we could have written a test against the FastAPI interface. But look how thin this interface is. If we now... We could use the FastAPI client interface to write test functions against these two endpoints. I don't think that would add much value something that I thought about is that could be actually quite useful is to add a front-end test I was playing with the thought of of using what's the name of the library playwright playwright is great to make front-end tests yeah but on the other hand playwright is even better when the front end is already working, because let me take a look at one of the existing tests for the space game. Oh, now I'm giving away part of the solution. The test for picking up the Python character with the original GUI says, you have to push 111432342 on the keyboard, and then the Python should end up in your crew. It's kind of a crap test because it's impossible to understand what is happening. With our facade interface, then it would become a bit better because you could rewrite this test and fly to the Adalof system, pick up the fish, and then go to the place where the penguin lives, give them the fish, and you have the penguin. This is not the solution, but I'm not spoilering here. With PlayWrite, this test would become a lot more readable. There is value in that, but it's easier to write because PlayWrite comes with a program that allows you to record the test from actually playing the game. One thing that I can do is I go home, ask my kids to play the game while the recording is running and they will create a test for me. So that was part of my thought process why I did not come up with a front and test from the very start.

Speaker 2 [78:41]

All right, so there is another question in Slido. How do we refactor code with deeply nested inheritance hierarchies?

Speaker 1 [78:50]

Oh, yeah, yeah, let me see. I've seen how do I refactor deeply nested inheritance hierarchies. The longer I work with object-oriented programming, the more I come to the conclusion that hierarchies are quite a pain to live with. I've seen thank you I did not expect that I've seen hierarchies that were sort of useful but none of them was deeper than something like two layers of hierarchy if if it's any if it gets any deeper you may want to think about ways how to replace it by some compositional patterns if you don't have tests at that moment then you have a problem then you should write you should figure out what is the entry point, what goes into the program, what comes out to catch that without actually using the entire object hierarchy I hope that gives a direction

Speaker 2 [80:13]

i think so yes and it also basically answers another question that we had which was your thoughts on favor composition over inheritance so yes answered now as well then there is the question you mentioned coupling but not cohesion would you say high coupling always goes together with low cohesion or are they independent

Speaker 1 [80:39]

So low coupling goes together with cohesion.

Speaker 2 [80:45]

Would you say that high coupling always goes together with low cohesion?

Speaker 1 [80:52]

Not necessarily. It really depends on what you're modeling. I'm very wary of using the term cohesion because it's so hard to measure. So I haven't used it as a concept in refactoring a lot.

Speaker 2 [81:13]

Alright, then there's the question, are there any other refactoring patterns apart from the facade method that are useful in common refactoring scenarios? Which are the factors to choose?

Speaker 1 [81:26]

Yes

Speaker 2 [81:27]

Yes.

Speaker 1 [81:28]

There there are there are quite a few the ones that I have used in the in the more recent past is the the BCE pattern I've used a strategy pattern quite a lot. I've used the factory pattern quite a bit and The Builder pattern also appears repeatedly on my desk. Yeah You can write quite decent singletons with Python classes as well. I think these are something like the top five or six from the catalog of famous patterns that have ended up in my fingers during the past couple of years. many others can be usually solved by some other features of the Python language. Then there are patterns that don't really have a name but they keep reoccurring, like let's say you write a decorator to add logging to a function or something like that. So they are useful but they don't, this is a useful strategy but this this is not a pattern so I try I find this this thinking about patterns or or reading a bit about them I find this useful but I try not to come attached to it because many times it's outdated knowledge even especially since since the language is evolving and Python has a lot of useful things which are not classes.

Speaker 2 [83:07]

All right, thank you next question, what are the first steps to start a complex refactoring project when you don't have unit tests.

Speaker 1 [83:17]

Read the code.

Speaker 2 [83:18]

such as first create dependency graph visualization.

Speaker 1 [83:22]

So reading the code and trying to summarize the stuff in a way that you can sketch it on a whiteboard or a piece of paper is an extremely useful approach because before deciding for a way that you want to edit the code, that is possibly costly because you may spend a couple of days working with the code. You should have some clear idea what the approach you want to take should be. And drawing helps me a lot personally. Usually when I do this, if I work with people in person, then a piece of paper usually does the job. It does not have to be a nice graph like this one. I heard some people use Miro boards or similar web technologies for this it's the same this doesn't have to be formally clean as long as you and the people you want to do this together understand what it's about

Speaker 2 [84:33]

All right, and another question. Now we're getting questions on Slido. Do you recommend domain driven design?

Speaker 1 [84:46]

I don't know, so I'm not entirely sure where to put the term. So I don't have an opinion on this one.

Speaker 2 [85:02]

All right. That is it with the questions on Slido now. We can take a second to see if there are some other questions on Slido, if someone else in the room has any questions. We have about five minutes left now.

Speaker 1 [85:21]

Then I think we are running we are running out of questions. I have a bit of cleanup work to do Keep an eye on the repository because I'm going to push the solution to the to the GUI there as well. Yeah, so It's not a lot that is going to change there But I might might leave some some comments there so that you see what has happened That's it from from my side. I I wish you safe space travels.

Speaker 2 [85:52]

One more question just came on Slido. How long does a refactoring project normally take?

Speaker 1 [86:02]

longer longer than you expect it usually like if personally if I if I'm the main person responsible for the for the refactoring and I don't see that there is an then there's an end in sight after after something like one or two days I I might realize that I've maneuvered myself into a dead end and may have to go back to the last working stage and start again. This is something that does happen. Good. I wish you happy refactoring and enjoy the rest of the conference. Thank you.