Commenting code — beyond common wisdom

Speaker 1 [00:03]

Okay, so thanks everyone for coming. Yeah, okay. So actually, it's more than 15 years. But yeah, okay. Some things about me first. So I'm using Python for about 20 years now. The version I started with was 1.5.2 or something. I'm a full-time software developer since 2000. Became a freelancer in 2005. So I wrote a book, Workshop Python, despite the title, it's a German book. And it's also quite old, as you see, it was Python 2.2 at that time. That was brand new. Okay, and this is, I guess, the 16th conference talk. And I'm also the maintainer of this FDPU-Till client library since 2002. Actually started as an exercise in this Workshop Python book. the introduction for this talk or some overview first some introduction what brought me to this talk some in case you didn't read the summary the abstract on the website this talk is about okay we'll come to this this is part of the introduction actually yeah but then yeah okay since I have this in the overview common wisdom relates to things you find on the internet about commenting there's a contradiction and I come to this in the introduction then I have a few words about commenting or when to comment more or less and some tips for good comments which I think are good suggestions and also which are also in some books and I agree when i agree with them and if there's time i will say a bit about comments versus commit messages okay what started this talk um okay i'm since always kind of i'm interested in writing maintainable software this is really my top priority um i'm writing lots of comments also sometimes rather long ones depending on the situation of course but on the other hand if you use a search engine about good commenting practices there are many which say keep them as few as possible you don't need any comments you just write self-documenting code and this was my own practice and what I found online this obviously this contradiction and I was really curious why this contradiction why I have this impression that for me it's different or why don't I agree and still these are this advice is still around okay so I thought a lot about this and this talk is about that this was the starting point for this talk okay why do we comment anyway comments help with understanding the software better so you can change this more safely and also help understanding the software faster so you can make changes faster but still correctly this is important of course for maintenance if you have a larger software project yeah you want to go fast. I mean, of course, there's always a compromise and you need to refactor everything. You can't get infinite fast, of course. What's important is this holds even if the software design is already quite good. Many of these websites or articles say comments are often a workaround or a patch for bad design. But my talk is explicitly about having a good design and commenting in addition to that. So yeah, when you these articles you find online, one statement you find is document the why, not the what or how. And I mostly agree. One tip I found in a book, but which I have used anyway for a long time is this idea of section comments. The idea is that you describe a code section on a higher abstraction level even if you could derive the same information from reading the code. So some people say it's already in the code so you don't need to add comments but I think it's still useful to have these comments so we are more easily able to scan the code and by reading these headings, so to speak, inside a functional method you can orient yourself faster in the code. This is an example for a section comment. This is from the Python standard library from the inspect module and you don't need to understand this in detail. It doesn't matter that much. It's just to illustrate this idea. You have some code and even though you could derive this information in the comment from the code there's still this comment so you can get a quick overview then we have extract the function or method instead of writing comments that's a trade-off on the pro side the individual functions are shorter when you I mean, the original function is shorter and probably easier to read. And extraction, that's also a main idea. Extraction introduces hopefully meaningful names for function and methods and their parameters. So the name is supposed to tell you something that you would otherwise explain with a comment. I mean, that's the idea, but it falls short a bit. Okay, but there are also downsides. The code becomes more fragmented. The overall understanding, if you have to jump around, doesn't necessarily get better. And one thing I find most problematic is, I mean, this is probably related to code becomes more fragmented, that a reader has to find out from where this extracted method is called. This is an example. I mean, not a very original example, but to illustrate the idea. This is the original method. in a class and I previously this what is now a new method was here also an original method and now I extracted a method and call this from here now the question is from where this I mean if you come to this code maybe some someone else did this refactoring and you come to the code and wonder from where it's called I mean you can use your IDE in many cases or ideas help there But if you forget to ask the IDE and unconsciously, subconsciously assume that the new method is only called here, it may not be the case. Now, then another one is use better names instead of writing comments. And I think the which is also known use a better name instead of commenting a bad name. And this is also something which is partly true. of course you should have good names but you can still have the situation where an additional comment makes sense so it's not either or but comments are often useful so you improve the design do everything necessary have good abstractions and good names for example and if something needs clarification, add comments. This is also often found, comments easily become obsolete. So the idea or the argument is if they become obsolete, they are not so useful and are just confusing. uh i think that um a problem comes from uh not taking the comments as seriously and i can't even imagine uh that this is some um a circular conclusion or some some self-fulfilling prophecy in that um people think comments are not so useful they don't pay as much attention as to them as they should be so the comments become outdated and this is taken as a confirmation that they are not so useful, that they become outdated a variant of this is I can't test comments, so if I had everything in code this would be I could test this but I'm quite certain that usually you have some code which is not tested and And I mean, in that case, you take the code extra seriously because it doesn't have tests. So this is also not a comment, not a con, nothing against comments. One general piece of advice is that comments are more likely to become obsolete the further they are away from the code they apply to. So this is, I mean, for me, it's definitely true. you change something in a functional method and you see the comment maybe shortly above the code you are changing and you update the comment no big deal but if a comment for example in the class two or three screens up on the class level refer to something you just changed then it's easy to forget to change a comment or doc string or whatever in the class so if you can everything that is that is specific to the function keep this in the function or method this is similar to the one before avoid comments because they could deviate from the code or contradictory comments are worse than no comments yeah as I said pay attention to updating comments and I think since the comment should be on a higher abstraction level. I mean, usually it is. I would say that for my code. It's likely that the code is wrong and the comment tells you what was intended. So, actually, you might have differences between code and comments, but they help you. So, you can point to a bug. And also, even in cases where it's true, the pros of comments outweigh the cons so overall you're still better off with comments even if occasionally you have such a contradiction between code and comments okay whether to comment more or less and okay this is an argument for less what you shouldn't comment is comments that literally repeat the code. Like in these first examples here, these two, increase index by two and add client to client's list. These are typical examples which are cited when people say comments are useless. On the other hand, if you do this on a high abstraction level, I would say it becomes useful like in this case if you don't have the comments what does a non-value mean for max age development context this is also the idea that you can't give general advice but that the amount or detailedness of comments depends on the project or on the development context. One is developer knowledge, and this is a point that can be divided further, which is actually in the next slide. The size of the software, for example, if you have many modules, it's more difficult to really have this overview over the software. Software complexity, in this case, refers not only to the size, but also the problem you're trying to solve in your code. For example, it's a difference if you have quite straightforward code or if you have, for example, you're developing a new algorithm for machine learning. I mean, not only tuning an algorithm, but developing something new. In that case, maybe you don't even have the terminology for this new algorithm, so you are bound to comment more or you should comment more. um application versus library there's also applications you know you have more freedom to to refactor um in libraries sometimes uh okay the ideally uh people or users of your library don't have to read any comments but in case they should or they do it's good to have them to have the comments yeah and this is not as much true for an application where usually users don't read your code or don't need to read your code team size is important especially if it's a small team of one person um yeah so um yeah so yeah actually and team fluctuation is related um actually that's something that happened to me um and this is because this development context is also to be taken with a grain of salt um one project i worked in there was some someone who developed the software this was about 30 000 lines of code and um he was alone so nobody was there that depended on his comments and this code base was hardly commented and uh yeah but he got an offer from a company left and two weeks after he had left i was supposed to take over this code base And this really took several months to really get into this code because it was also very clever code, so to speak. Time between modifications is also relevant. For example, in this open source project, FTPuter, sometimes I don't work on this for half a year or even longer. So it's important for me that when I come back to the code that I can get an overview quickly about what I'm doing. Okay, developer knowledge. Usually or often you find only this distinction between general programming experience and programming language experience or knowledge. But after thinking a bit about this, I found there are many more. So, yeah, of course, as I said, general programming knowledge where experience with concepts like loops or if statements, algorithms, these things. Programming language knowledge, both referring to the language itself, but also the standard library or other common libraries for certain problems, the ecosystem, as we usually call it. problem domain knowledge is important for many for much software for example if you work on software for machine learning or factory automation or accounting or chemistry you need some obviously this is additional knowledge you need to understand the code and for example if you have a a project where all the people working on the software are physicists also and um they they are yeah they they all know this concept they all have this problem problem domain knowledge but if you have the case that um you have a team and uh one or two modules i use some uh domain knowledge and you have only one person on a team with specialists there and so they should probably comment more in the code about the problem domain or what they are doing maybe at least referring to articles or where you can find additional information maybe for Wikipedia articles or something company knowledge can be important if you have if your software depends on things like business processes or things that are company specific and architecture knowledge would refer to other systems your software talks to for example if you have a REST API you need to know something about the servers you are talking to if you are a client and maybe there is even more but these are some items I came up with or that occurred to me when I was thinking about this. And, yeah, do pair programming and code reviews for knowledge transfer because there are so many different kinds of knowledge. And even especially if you have only one person with specific knowledge on the team, it's good to spread this information and do pair programming. I mean, apart from comments. sometimes comments are just not enough so my summary in this case is more or less if you really want I mean if you want it you could ignore existing comments but you can't conjure up comments which aren't there I mean if you have done maintenance programming especially on larger code bases you appreciate comments so my advice is if in doubt comment rather more than less so okay something about what I think are good commenting practices comments should make the code easier to understand I mean this is a starting point or the basis comment what a reader might want to know for example yeah I'm just a bit okay yeah for example yeah right previously I worked on some NumPy code and I was wondering what happens if some of these values are an N, like not a number. And so I thought about this and saw that the code I'm writing already works with N values, but I still commented this because the next person coming along and is wondering the same, they don't have to go to the same thought process as I did. So in the comment explained that the code works with N values and that it works correctly, or supposedly at least. I'll comment what you found out while working on the code. This can refer to both when you initially write this code and, on the other hand, if you work on code that someone else wrote or that you wrote some time ago. yeah for example if you need to look up read some documentation read articles on something you are working on in this module then it's i think good practice to comment about this like yeah like this what i said a link to an article or wikipedia article can also be some intricacies of the language or some library. Maybe you need to work around some problem in the library you are using. Also, if you are working on the code that is already there, if you are maintaining code, sometimes I notice I read code and I have to really think how it works, especially if it's NumPy code, which is so abstract that sometimes you don't have a clear picture in your mind. I mean, at least not initially. If there are too few comments or not extensive enough comments, this can be difficult to find out. And what I'm doing then is that I'm adding comments that explain these things so that, again, the same idea, the next time someone comes around and works on this code, they don't have to go through this again. a comment what uh isn't obvious and okay um quite often it happens that um you write something and in this moment especially if you worked on similar code or involving similar concepts the code seems quite obvious to you but in the long run usually it doesn't even for yourself It happened to me that I wrote some parser code and I also added, yeah, I tried to explain rather well what I was doing and I think two or three months later, I had to return to this code and change something. and it's still despite the comments it took me a while to really understand everything what is going on in this part of the code and I mean if you are evil you could say this is a case against comments because they were useless I needed to think anyway but of course yeah if I hadn't had these comments it would have been much more difficult check comments for ambiguity this for For example, in one book, insert data into the cache unless it's too big. And this it doesn't refer to the data or to the cache. So you can either, instead of the pronoun, you can use the noun again, or you could say, unless the data is too big, insert it into the cache, or find a different wording which removes this ambiguity. One other thing I noticed from time to time is if you say something like the clients and their accounts, you lose the information whether there's always one account for a client or if one client can have several accounts. And I usually write something then like the client's accounts or account. Clients with apostrophe S. Also, sentence structure, I don't have an example for this right now, but some sentences can be understood in several ways. So the only advice I can give here is that you read your comments and think about how they could be misunderstood and change the wording so that if possible they can't be misunderstood. Yeah, comment to avoid simplifications that could break the code. This is actually a special case of comment what a reader might want to know. This is, yeah, you have some code and it uses, yeah, for example, some workaround, which is not, I mean, this would just be an example that is not really obvious from the code. And the comment is to prevent that someone looks at the code and thinks this is too complicated, this is much simpler to do than what is actually in the code. And you take measures, you add a comment to avoid that someone simplifies the code and introduces a bug by doing that. this is just one example from this one example from ftputil where you could cause an infinite recursion by the way I go one slide back many of the comments I write are from things I stumbled over in during development yeah so I write something I yeah how I expect the code to work and why for example when I'm writing the tests I'm sometimes surprised I notice something I didn't think of before and I add comments about this about these things I discovered this would be referred to the third point start comments with an uppercase letter this ends the period and this makes it easier to extend comments maybe if you read pep 8 they also recommend starting with a capital letter and ending sense sentences in a period i was when i initially read this i was yeah i was wondering why and this is actually the the reason is that it's uh that it encourages to extend comments In the past, before I knew Python at PEP 8, I usually started with a lowercase letter, and then I wanted to add something, I wrote a semicolon and something after that. And a while later, I was wondering, do I append something else with a semicolon? No, I go to the start and make it uppercase, and how do I structure this now? But if you, from the start, write sentences, it's easier to extend those. so you have less overhead I use XXX to do and fix me comments don't pretend that you know everything or that there are no problems often I mean at least that's the case for me when I'm writing code I think of so many things that could go wrong or edge cases and I add a lot of these comments where the code might break um of course ideally in an ideal world your code wouldn't have any uh bugs yeah or you would everything fix uh fix everything right away but uh you have to make compromises if you i mean i can understand this if you uh write software for medical appliances also or for flight control software or something you don't want to have any bugs but usually with most software it's easy but with most software you enter diminishing returns and you don't expect to have zero bugs and in some cases you say okay this edge case is really really unlikely Of course, you could be wrong, but you make a compromise usually. But in that case, it's, I think, much better to have these comments than not having them. The third item is inspired by some time ago. I tested an IDE, and the default color scheme for comments was a light gray on white. yeah so this was really weird and i i think that's a bad idea i think it was well intended i think the idea was that you um that most people read the code and uh don't have to read the comments if they understand the code also yeah this idea that i mentioned initially that the comments are less worth or something than than the code uh they refer to and uh but uh it has exactly this problem i mentioned that yeah if you can't read the comments they are more likely to get outdated so so you should so adjust your color scheme if you have have one that where comments are hard to read then another thing i noticed is sometimes you find functions which are structured into blocks so with empty lines between them and very often I think maybe in 90% of all cases I see this is a block but what is this block about what does the separation tell me why is it a block so I would advise that if you use empty lines to structure your code add a comment what the block is for some things about comments and commit messages of course there's some overlap you explain things in regular high level language and not the programming language comments are far more visible than commit messages so put things in comments that reader must not miss that they really should know for example these comments that I mentioned that should protect against accidental simplification of code. If you put this in the commit messages, you probably have the bug introduced before anyone looks at the commit messages. I think that's likely. So there should be a remark on this one in the code. Explain in commit messages why you did not do something. For example, if you find that you can do something in three different ways or algorithms approaches I usually write in the commit messages the trade-offs between the algorithms or why I implemented something this way and not in another way this can be interesting if someone else returns to the code so they have the trade-offs explained so if if they read the commit messages. Okay. Put information in commit messages for code review. In one project, my colleague and me, when we looked more into writing better commit messages, and we also had a code review previously, we would often have questions. Why did you do it this way? and not another and when we were starting writing longer commit messages I could just say yeah look at this commit message that's explained your question is answered okay I have used some books and websites that I would recommend the first one is the art of readable code this is This doesn't discuss high-level design, so not so much design patterns or something, but it's really about writing clearer code. And it's also a funny book. It's a not-so-long book. It has cute pictures or cartoons. and yeah so it covers things like it covers things like structuring code for example if you have a function which does lots of different things that you should pay attention to that each section within the function is about one task so to speak so that you can reorder the statements or doing things one after the other not in parallel I mean I'm not talking about the execution, but about how the statements are ordered in the code. A philosophy of software design, this sounds quite lofty, but it's actually, I think, a very good book. John Osterhout is the author of the Tickle programming language, and he's a computer science professor. And there's a very good talk, in my opinion, at YouTube, which covers a part of the book. So the book is mostly about software complexity and what you can do to reduce complexity. And one chapter is also about comments. One article I highly recommend is How to Write a Git Commit Message by Chris Beams. this also applies to other version control systems of course and the last point this is actually not a book about computer programming but this is how to edit your own writing this is much more interesting even if you for things like writing reports, writing articles on a wiki also how to write clear text And this also has many examples of sentences you can misunderstand. Okay, we could either stop here, but maybe we go directly to the questions and answers. I could also show some comment examples from FTP Util. so I don't know what you would like what you would prefer yeah okay because to my understanding this is still for the talk and we have anyway five minutes in addition to questions yeah okay this is from one module in FTPO2 and I mean even though I talked about good and not so good comments I most likely have also some which are not ideal but I can maybe show some which I think make sense this for example is some general concept which holds for this code which has to do with the FTP state machine so to speak when you're talking to an FTP server this one for the remote host this RFC about FTP says that it depends on the server but every FTP server I've seen accepts these three like in local file systems on POSIX but in case it happens that it's different, there's this command also this one don't use the list A option because originally in the previous version this was the default was set to true but it turned out to be a problem because if the server doesn't understand this minus A version it interprets this as a file and you get an empty list back for a directory instead of the list of files so this is not so great and so this refers to to the ticket maybe one more yeah here's something like why I or some problem that the code might have depending on circumstances this is usually or what I identified here usually it shouldn't be a problem for the code but in case you run into problems and read the code you have this confirmed or there's a limitation and I think we are time for questions and answers yeah you need the microphone sorry I have quite a loud voice maybe it carries hi it's interesting I was super excited to come to this talk today although maybe from the outside it seems dry but i've always felt conflicted about comments and how much i should write um perhaps the same as you um i was speaking with a colleague um before i i came here and i told her i would go to the talk about comments and she said oh but like isn't that what documentation is for and i see these very extensive comments and i'm wondering like what is your opinion on um how much crossover is there between comments and documentation do you have sometimes the same comments in your documentation very good question etc etc yeah yeah yeah um yeah i i try to separate the abstraction levels so so usually um i don't have documentation i mean maybe apart from auto-generated documentation i don't have documentation about details of any functions or methods. Documentation is more about general concepts of how the module works, for example. On the other hand, you could also put this in a module doc string. But sometimes you have concepts which span many modules. So the big picture, so to speak, of the software. And I think this should definitely go into regular documentation. this is again yeah this is very good that you that you said this avoid duplication yeah you don't want to duplicate information in the code commands and in other documentation and and one way to work around this in some cases is auto generate documentation yeah with sphinx or something um it's my turn um so sometimes i update the code but not the comment, which is probably bad. Yeah, it can happen. I mean, it happens to me as well, probably, sometimes. So, when is the best time to comment? When you write the code. Yeah, right. That's actually, I wrote some, I read in books, I forgot to mention this in the talk because it's, for me, it's completely normal to write the comments when I write the code. And this is actually a strong, I think this is in the book, and also in the talk, no in the talk not but in the book by Osterhout that you should write the comments when you write the code because then what you're thinking about is still fresh and your comments will probably be better. I mean especially these comments things I learned during coding something. They will probably, these memories will probably be gone if you comment the code a week later or something or a month later. Okay. Hi. So both of us started using Python in the last century. So because of that, you remember Zope, right? And the Zope project was where doc tests were born. Ah, right. And I want to ask your opinion about the practice that they had in the doc test because comments are for programmers, right? Yeah, yeah. Actually, people who are going to touch the implementation, not just users of the code. Yeah, I understand. And in the ZOAP world, they had this idea, which they called, besides embedding comments and doc strings that could be extracted for documentation, they also had this idea of what they called narrative documentation. Do you remember that? Do you mean something, okay, I don't remember, sorry if I, since that I interrupted. I know SOAP and I probably, maybe at some point I looked at the code, but I wasn't, I was never working on SOAP. So, but I remember that, I don't know if it was SOAP, but maybe, probably, which used a lot of doc tests. um yeah i think that um these doc tests make sense if they um that's something you usually put in uh in doc string yeah again and doc strings which is um user documentation so to speak i mean i mean i prepared a slide originally originally this was part of the talk um difference between API, which is the name and parameters and the doc string on the one hand, which is for users and implementation comments, which are in the code itself or in the body of the functional method, which are for developers. And the doc tests, as long as they make sense for the user, I think that's fine unless they become so many that they completely obscure the whole doc string or something. doing things only with doc strings, with doc tests I'm rather skeptical it doesn't scale very well yeah so that was all the time we had for questions the rest of you I need to redirect to Stefan directly thanks man