10 ways to debug Python code

Speaker 1 [00:03]

Hey, good morning. Can you hear me? So, yes, I will talk about 10 ways to debug Python. So I'm Christoph, I'm from Heidelberg, and I'm a Gamma Ray astronomer and have been using Python for about 10 years and just wanted to share some tricks and things how to debug. So what is debugging? The definition is pretty straightforward. It means to identify and remove errors from your programs. and um i mean sometimes the story is told that uh the the this weird word like debugging and bug came from this bug that was caught in a computer in 1947 but actually according to wikipedia the uh the word bug and debugging has been used already before that so but that's what it means um you might ask yourself like why should you learn debugging um shouldn't you try to avoid bugs in debugging. And yes, you should try to do that by writing clean and simple code and writing tests. Debugging is something that is often annoying, frustrated, unpredictable how long it will take. So you might sit there and do late-nighters. And then this goes at the cost of your personal life, like not having time with your family or friends. And also there's a huge economic cost to debugging, which is by various studies estimated in many hundreds of billions of euro per year that is spent paying programmers to debug um but yeah you should learn to debug because there will be bugs um and debugging if you're a programmer also if you're a data scientist it's the same thing um you have to debug your code and your algorithms and your data um so debugging is unavoidable um so you should learn and train to be ready and efficient at debugging when you need it. So it's kind of like learning to defend yourself when you have to, then you should be ready. And the basics are simple. So with little time investment, learning how to use a debugger, you can save a lot of time and this will pay off. Why 10 ways? I think different tasks require different tools. So if you're coding in PyCharm, then you should debug in PyCharm. If you're working in the Jupyter notebook, then you should, you have to, or you should debug in the Jupyter notebook and not just because you don't know how to debug where you currently are, switch to some other tool. So if you have bad performance, then you need to learn how to do profiling. If you have bugs in production where you cannot use PyCharm on the customer machine or something, then you have to use logging. So that's why you kind of have to learn multiple ways to debug. So this is an overview for beginners, just showing you what you can learn, like you won't walk away from this and really know and be good at debugging. This is something you have to learn after by yourself, but it'll give you some ideas what tools you could look at and what techniques. So these are the 10 things I'll cover. There's many topics that are not covered, like I will not talk about concurrency or C-extensions or web apps. So really debugging is a big topic, but this should give you a nice start. So before we start, let me get a quick feel for the room. Who uses a debugger regularly? Wow. So maybe you're in the wrong talk, because this is really for beginners. So who uses a debugger sometimes? and who has never used a debugger? Okay. So I'm an astronomer, so usually there it's the opposite. Like everyone codes to do their analysis, but 90% of the people have never used a debugger. And then what fraction of your time, like your development time, do you spend debugging? Raise your hand if it's 0%, 10%, 20%. 30%? 40%? 50%? More than 50%? Okay. Yeah. So I think studies show that it really ranges from 30% to 70% of the time of people is spent debugging and not just writing code. Okay. So number one is reading code. So this is weird. Is this really debugging? But I think to debug, you need to learn to read code. So when you have code like this, point.py file, analysis.py file, and then you type Python analysis.py, you need to have a mental model for how Python executes the code. And it's mostly top to bottom. And then function calls create stack frames, import statements execute other files. And also you have to know that everything is an object. So def and class and import statements create function class and module objects. And variables are references to objects. And just to illustrate this, there's a nice page called pythontutor.com, which visualizes a Python code execution. So you can put any Python script you like here. I've put this example, and then you can step through it. So if you click forward, then you have this, always this pointer to the next line that will be executed. And this is very similar to what you see in a debugger, only that you have this nice visualization here on the left. So what happened is that in the global frame, a variable math got created, which points to a module object, which is the math module. And then the dev statement creates a function object, the class statement creates a class object, and then the code starts running. and when you call a function or a method a new stack frame is created so we were in the global frame now we are in this point dot init stack frame and you have local variables and so on so basically this is how python executes code and what's on the right is the mental model you should have when executing code or also when reading code you kind of do this in your head to think about how the program state changes. Okay, let me go back to the slides. Okay, so just to sum up this section, have this clear mental model how Python executes codes because it's the basis of everything you do, code reading, writing, debugging. You can play with this pythontutor.com. There's also a Jupyter Notebook extension which kind of does the same thing locally. Or you can use a visual debugger, which is also very similar, and we'll see that later. And here's some nice resources if you're new to Python and want to get started. The second thing you have to learn, and especially with debugging, is to read tracebacks. So because debugging often starts with an exception and a traceback, and the traceback is this function call stack that we've seen before. so if you execute this script above here then you will get this traceback and you will get the error so typically the way you read this is that you check the exception type in this case a file not found error and the error message first and then you read the traceback some people read it bottom to top or top to bottom it's up to you but this you can see that this is the function call stack For example, in line 9, we are calling the countLines function, then this is executed. And in line 5, it executes the path.retext method, and that's where execution moves to another file, which is the pathlib library in this case. And often, when you're debugging, you can ignore the frames and the code in the other libraries, because usually the bug will be in your code. um okay so also um so exception and error often and usually means the same thing these are instances of class of a class that derives from base exemption for example name error is an exception you can see that it derives from from base exception there's two errors that you will get very frequently which happen um on import time so right at the start after you write your code. This is a syntax error and the indentation error. And then all of the other errors that you see on the left here, like name error, attribute error, index error, key error, and so on. They happen when you execute the code at runtime. So this is normal. With Python, you get errors all day. It's a feature, not a bug. Explicitly in this end of Python, it says that you should write your code such that errors don't pass silently, and you should refuse the temptation to guess. and this means that errors will occur as you run your Python code. Another little bit more advanced thing is chained exceptions. When you see one of those, there you have two exceptions and two tracebacks. This occurs when a second exception happens in the accept part of a try-accept statement, like in this code example above here. In this example, I have a bug in my error handling code where X is a variable C which doesn't exist. But you'll also see this sometimes from libraries that use try exception for control flow. And I mean, yes, they are more complex. Like typically exceptions are really long so they can span like two pages or something, but you just have to be calm and read them. Sometimes you only care about the second one, but sometimes you have to read both to understand what's going on. Okay, so to wrap up for exceptions and tracebacks, you should learn the common exception types and the common bugs that cause each one. And I've shown the most common ones on the previous slide. The Python documentation for this is pretty good. You can just read the pages on errors and exceptions and built-in exceptions. If you have an uncaught exception, the Python interpreter prints the tracebacks and exits, and you have to learn how to read them and how it connects to your code. And often this will be enough to do your debugging. So the debugging really is reading the tracebacks carefully, reading your code carefully, then you understand what's wrong and you fix it. If this is not the case, then you would go and use a debugger. Before we go to the debugger, let me talk about print quickly. So print debugging is a very common way to debug. You just add print statements in various places to try to figure out the state of your program and what variables contain. The problem is like where to print which information. So usually it's an iterative process where you add a print statement here and then a print statement there. And first you print the X and then you print self.x and then you print the type of self.x and you always rerun and add print statements. So it's just annoying and slow and error prone and not the best way. Yeah, so the reason why I think it's not great is just that it's slow and because you have this slow process and then also it's error prone because you can forget to remove some print statements or if you want to know what's going on with the state in some other library then you kind of have to edit files in from the installed libraries inside packages and so on and this is pretty messy so don't use print but learn how to use a debugger. Okay, so moving on to PDB, the Python debugger. It's a very featureful command line debugger that ships with the Python standard library. The common ways to start it is to say Python minus m PDB and then give you a script. Or you can also put a breakpoint explicitly by putting this function called breakpoint into your code at the point where you want to stop and look around. Since Python 3.7, if you're using an older version of Python, you have to write import pdb pdb.set trace. And then you need to learn five to ten commands how to use it. The most important one is the age or help, which prints you the available ones if you don't remember them. And then the most common ones are listed here. So you can get help, you can quit, you can print or pretty print the content of variables you can list or long list the source code in in the current function or frame where you are you can use w to print the stack trace of of the frames that you have and then you have various ways to step through your code so you can step over or step into where step into goes into functions and follows each line of python that is executed and step over just goes to the next line in your in your file and you can continue which runs to a breakpoint or exception or the program end. Then you can set breakpoints or remove them and you can move up and down in the stack frame. So this is just an example using again the same code I had previously. Let me just quickly do a demo. Yeah. So if you do this, you start the debugger and it's on the first line. So the error always points to the next line that will be executed, not the last one. And then you can, for example, with C, you can continue. In this case, an exception occurs and then the debugger stops at this exception. And then you can look around. So you can look at what is DX or what is self.x or what is the type of self.x. So you can evaluate Python expressions. You can use W to see where you are in this call stack. It's like really not very well visible, but like this little caret shows you in which frame you are. So I could, for example, if I wanted to check the state in some other frames, I could move up. And if I say W again, you can see now I'm in the frame one up and I could, for example, look at the content of point. So if I want to see what's in this point. You can do stuff like this. And then when you're done, when you figured out what the problem is, in this case, I put strings instead of numbers into my data. You can use Q to quit. Okay. So to sum up for the Python debugger, it's always available. It's part of the standard library. It's a command line interface. I think it's a bit hard to learn and remember, so I don't use it much. I'm not good at it. You can try PDB, and then I suggest you also try to use a visual debugger, and we'll see this next, and then you see what you like. There's multiple ways to start it, and there's multiple ways to poke around, and you have to learn these commands and play around with examples. So there are some really good tutorials both in the Python standard library documentation and also from Doug Hellman with this Python module of the week tutorial. And on realpython.com, there's also a very nice tutorial how to use PDB. Okay, then IPython and Jupyter. They provide a nicer interactive REPL and debugger than the normal Python. And then there you have IPDB, which is kind of the same as PDB, but also a bit nicer. So you get things like color, multi-line editing, tab completion, magic commands, and so on. The magic commands that you'll use most often is %run to run a script or %run-d so that you run the script and when an exception occurs, go directly to the debugger. You can just run something and then if an exception occurs, you can say %debug and this will enter the debugger after the fact. And then you can also with %x mode, you can control how verbose these tracebacks are that are printed. You can also put this line, import IPython, IPython embed somewhere in your script. And then what will happen is that and then run your program just through the normal Python. And then you will be dropped into an interactive IPython session at this point in the code. And you can again look around. With Jupyter, it's very similar. Like everything you can do from IPython, you can do from Jupyter. it also has this rich output that's often useful to check data so you have HTML tables and plots and so on but concerning the debugger it's the same so when you enter the debugger you're just dropped into an IPDB session in the output cell there is the questions like why is there no visual debugger for Jupyter or JupyterLab I think there is some work in progress to create one and last year there was this Pixie debugger that was developed I haven't tried it myself but I mean maybe there will be like a general solution for a visual debugger in Jupyter in the future okay so just to wrap up I mean IPython and Jupyter they have IPDB it's very similar to PDB and generally I think it's just nicer to use and I recommend it and there's some good resources here where you can learn more if you haven't used IPython or Jupyter okay then moving on to ide's like pycharm and visual studio code so there are really many python editors and ide's um the key point i want to make here is not just to so i'm going to demo pycharm because it's what i use um but the key point i want to make that there are these visual debuggers and they are really nice and you should you should try one um so with pycharm you have the Free Community Edition, which has a debugger. It's a very advanced IDE and code analysis. And then Visual Studio is great as well. There you need to install the Python extension extra, but then you also have the debugger. And there's many more editors and IDEs with varying levels of debugging support. So for PyCharm, the way it works with a visual debugger is that you can set breakpoints here just by clicking and then you can say debug this code and then you get the debugger window down here where you have the same things again you have the stack frame which shows in this case the main function was called and here you have your variables so at the moment there are no variables but if you for example step over to execute the next line of code then you will create this point object and you directly see the type here and the value and you can look around you can also evaluate expressions so you could do things like p.x like float and you can evaluate the expressions you also have a console where you can look around with the debugger. You have to say show the Python prompt and then you can look around. So this is like being in IPDB. And here you have the debugger view. And here you have the stepping part and here you have other ways to restart or continue or stop the debugger. So it's much more beginner-friendly, I think. And also for advanced users, like if you use the shortcuts, then you have this nice visual debugger. Okay. And then with Visual Studio Code, yeah, I think it's very similar, the experience. So I've tried it quickly yesterday and I've done a screenshot and it looks to be very similar. So, yeah, I mean, I suggest you learn in IDE and you try its Visual Debugger because it's easier and more pleasant to learn than PDB. and at this conference you have JetBrains which produces PyCharm and Microsoft which produces VS Code so you can visit their booth if you have any questions about these ideas and there's also very good tutorials and resources online showing you how to use them in detail okay number seven is testing so if you have too many bugs and are doing too much debugging so you're spending like more than 30% of your time on debugging, then I think you need to have a systematic effort to improve. And a very common way is to add tests, add simple test cases to establish what works and what doesn't work. And then, so first you write a test for something that should work, but currently doesn't, it's failing. And then you debug and fix the issues via the tests. Here I'm showing this again in PyCharm. I can also demo it quickly. It's very similar to the normal debugging, only that you have a test and you have this little triangle where you can run or debug tests. For example, if you want to stop here, then you can say debug this one test and then you will enter the debugger here and you can look around and you can step and so on. Yeah, so for the test runner, I suggest you use PyTest and then you use the visual test runner and debugger. If you prefer to use PDB and not, whoops, and not an IDE, then you can use the PyTest minus minus PDB flag to enter the debugger from your test runner. Okay, the number eight is profiling. So there's this common saying, make it run, make it correct, make it fast. You can use debugging and testing to make it run and make it correct. If it's not fast enough or if you run out of memory, then what you should do is define a real world benchmark that you care about, measure or profile the CPU and RAM usage, and then try to improve the performance using other libraries or restructuring your code to do less computations and so on. So this is not covered here. But let's look at some profiling tools. There are really a lot of profiling tools in Python. This is just three that I picked out. The first one is if you want to do process level profiling, you can use PSUtil or PSRecord, which just measured how long does my program take? What is the CPU utilization over time? And what is the memory usage over time? You can get these graphs where PSUtil does process profiling and PSRecords then just does this as a function of time and makes a quick plot for you. If you want to do function level profiling to see how much time do you spend in each function, the Python standard library has this built in C profile module, which measures the profile. And then you use PSTARTS to analyze the profile. There's an example on the left side and an alternative way to just doing these tables of how much time do I spend in which function. You can also visualize this with a third party tool called SnakeWiz. Generally for timing and profiling, I think IPython and Jupyter are great because you have these magic commands. You can say percent time to time the execution, time it to time it repeatedly and get a better measurement. You can use PRUN to run C-profile and get this function-level profiling. You can use LPRUN to run the line profiler and get a line-by-line profile, how much time you spend on each line. And then you also have tools, %MEMIT and %MPRUN, to do memory profiling to see on which line do you allocate how much memory and how much memory does your process use. And often you have these two variants where you have the single percent sign, which is for a single line. So percent time it for a single statement. And if you have a cell with multiple lines, you use the double percent for the cell magic command. And there's a tutorial here showing you how to use these. Yeah, so for profiling, for process-level profiling, you can try PSUtil and PSRecord. Function-level profiling, Cprofile, PSTAT, SnakeViz, line-level profiling, the line profiler. And then I think using them from IPython and Jupyter is kind of the nicest way. And here's some tutorials how to use them. For logging, I think logging is useful if you have long-running processes and sometimes it's the only debug information you can get. If you have code on production on a server or on the user machine, then you have to do logging. I just quickly mention it as a debugging technique here. I won't go into more details, but there is good documentation about the Python standard library logging module available. The last point I have is rubber duck debugging. Who has heard of this? Half of you. Okay. So the idea is that you explain the bug in code to a rubber duck. And just by stating out loud your assumptions on what your code does and what the inputs are and so on, you will find the bug yourself much quicker yourself than if you just look at the code and do debugging. if you don't have a rubber duck you can also talk to a colleague this works as well and there's other like general debugging tips so i generally like i said you should avoid debugging as much as possible by writing clean and dumb code in tests so don't try to write clever codes keep it simple and avoid late night debugging and too long debugging sessions like i said if you're debugging for hours then you should take a step back create a minimal test case that fails, put it as a test and then just debug this simple test case. Exactly. So just to wrap up, these are the 10 ways to debug Python code that I quickly mentioned. Like I said, many topics are not covered here. And if you want to learn about any of them, you will have to follow up by yourself. And this is just the summary. You should avoid bugs and debugging as much as possible, but there will be bugs in debugging so you should learn to use a debugger. If you prefer to work on the command line there's pdb and ipdb and if you like to use an IDE then you have PyCharm or Visual Code with a visual debugger. That's it. All right, thank you very much, Christoph. Judging by the time I think we have room for exactly one question only. And then we will all head to the coffee break. So is there any questions from the audience? It seems like everything has been said. Oh, now there's a question. It's more an organizational question. In your first slide, you said that the slides would be available on your website. I looked it up and they're not. I'll post them right after okay thank you sorry well now that that's clarified I think we just wrap it up here and all other questions go to Christoph personally thanks again for being here and thank you to Christoph