Testing in Python - The Big Picture

Speaker 1 [00:03]

Good afternoon. Welcome to my talk. Who am I? My name is Niklas. You will not be surprised that I'm a Python developer, although I must say I think I'm somewhat of an oddity here at this year's PyCon DE, as I don't do any data science, which seems to be what everybody else does, so I don't know. Maybe I don't know what the big hotness there is. I've been using Python for about five to six years, now and I use mostly web stuff and database access so those are the things that I usually concern myself with and occasionally I contribute to open source projects here and there. I work at a company called MPS which is a medical software company. We have one main product which is called Chemo Compile which is a chemotherapy management system and you may not be surprised that But having said that, we are very interested in software testing because it's very important for us and, of course, the patients that the software works correctly. We are actually also looking for some new people. So if you're interested, speak to me. Where am I from? I brought this nice map of Germany. We are here right now in Karlsruhe, and we are located in Freiburg, which is even farther south from here. It's a nice city. we have an old church, we have water running on the streets for no reason, and we have these people. So, come visit. But why am I here today? Four years ago, I was at EuroPython, which was at the time in Berlin, and there was a keynote given by Emily Bach, and it was titled, Will I still be able to get a job in 2024 if I don't do TDD? Which is, if you don't know, test-driven development. Um, yeah, I was just fresh out of university and I had learned some, some TDD there and I thought, okay, I'm on the right track. This seems to be, uh, something that, that, uh, in the industry is picking up because in this talk, she sort of, uh, outlined how test driven, uh, is going to be getting more and more important. Uh, and, um, yeah, we as developers should, um, should, um, yeah, embrace it and make it one of our core tools. So, this was four years ago, she was talking about 2024, we're not even halfway there. So, what is the state of testing software like today? Who here does testing, automated testing, that is, on a regular basis? I would say about half of you, which is interesting. I personally would say we are on a good track if you speak to people you get the sense that people think about testing they know I should probably write some tests here it's an important topic it's probably not where we would like it to be so I thought today maybe I give an overview over software testing in general and the state of the software testing tools that we have in Python today available. There are many and many exotic ones also, so let's have a look. This is just some collection of concepts and tools that you have maybe heard of or seen in the Python ecosystem. I want to say Before I start, most of the topics that I only scratched the surface of today probably deserve their own talk, so if you're disappointed that I don't go into the depth, that is normal. That's what's to be expected, and maybe you can give your own talks about those topics if you're more interested. So I want to approach the topic of testing from three different angles, if you will. The first is the why test. So this is sort of the motivational part. Why should I even bother with writing tests? How do I do it? And how am I doing? So how do I know when I have a test suite, when I have tests set up, that they're actually doing a good thing? So let's start with the why test. And again, we will look at this from different angles. The most obvious one probably is you should write tests for your users and your stakeholders because they are the ones who are interested in that your software is working correctly. So this is the most, if you speak about testing, you most often speak about functionality. So is the product doing what it's supposed to do? Is it working according to specifications? But you can also test other things. You could test performance, is it fast enough, is it handling large sets of data quick enough. Accessibility testing, is your website readable to colorblind people or something like this. User experience testing, all these kinds of things are also tests. But in the classical sense, we talk about functionality testing. The second group of people that you would want to write tests for are your colleagues. Or if you're a library developer, an open source library, then the users of that library. And there you can see it as tests as documentation. So oftentimes, everybody here probably writes documentation and keeps it up to date all the time, right? Yes, I thought so. So documentation is notoriously out of date and you may take the time and write one day, write a nice documentation page for your library and then two weeks later it will be out of date and you will not bother with updating it, right? But if you have a good test suite, then the tests can be the best documentation that you can offer for your colleagues or for your users of your library. They can just look at the test and see, how do we use this function, and then be done with it. Easier code review. If you do code review in your company or your project, your colleagues will probably first look at the tests to see, because the tests are where you can clearly see the intentions of the developer. So it's much easier to just look at the tests, see what goes in, what comes out, aha, this is what is expected, and then I look at the actual new feature and do my copy, yeah. The same goes for quality assurance. So for the maybe less programmery people, if they test the functionality of the new feature, they might look at the tests and see, and yeah, get a sense of what the programmer was thinking, so it could be easier for them to understand, again, how it works, but also maybe where they should take a closer look because there's maybe something that the programmer hasn't thought about as much. And the last group consists of only one person, and that is yourself. And I think this is the most important one. If you do real TDD, as they say, so you write the tests first, you write the tests first, and then you write the code, you have a great advantage because you know when you are done. So you've probably all written some software, some program at some point without bothering about the test because it was just a script. And at some point you thought, I think I'm done, but I'm not really sure. So this could be very stressful, I guess, when you don't know when are you actually done with what you wanted to do. If you sat down before, write all the tests, and tests in this concept are also a bit of the specification of your software. So you think about all the edge cases, all the things that you want to do, write them down as tests, and then you know when all your tests pass, then you're done. This is nice. If you have a good test suite, you're not afraid of touching anything. So, refactoring. Maybe one day you stumble upon a section of your code, and you think, ah, this could be done in a much better way, but you know it is used in production in that one corner case that you don't understand quite as well as you would like, and the test is very incomplete for it, and you're just afraid to touch it. You leave it be, and you never fix that part of the code, make it nicer. if you had a nice complete test suite and you could be confident that when you do a refactoring and the tests will run through after that that, yeah, you didn't break anything. And overall, you have a better understanding of your software. You have a much more clearer picture of how it behaves. For example, functionality-wise, yes, we've talked about this, but also performance-wise. If you do performance testing, then you understand which parts of your software are the bottlenecks. So maybe these are parts where you don't want to add new features, where you want to do a refactoring next, something like this. And also you have some knowledge about your dependencies. This, of course, is true if you do continuous integration, so automated test runs, and they every day or so install all the dependencies fresh. then you know there's a new release and the dependency just breaks anything in your code you know that right away so you have a better better understanding of the software so this is the end of the motivational part now let's talk about how can we actually test software and well you will I already mentioned that there are different different angles that you can look at this functionality performance so on. Again, functional tests is sort of the classic class of tests, I would say. And you might often see the testing pyramid, which is a theoretical way to classify functional tests. The idea is that you have a base on the bottom of the pyramid, of course, where You have the simplest tests the most simple tests. They are easy to write They are easy to execute and they are very fast when you execute. These are usually unit tests so the idea is that you have many of them you run all the time whenever you make a change to run the unit tests again and That's cool. So it's no no hustle to To run them Run them all the time and the higher up you get in the pyramid and more complicated the testing becomes. The more complicated it is to write the test, to maintain the test, and also to run the test. On the very top, you have manual tests. To execute manual tests, when you write a hot fix Saturday night, you have to call all the testers, call them into the office, sit them down, do their manual tests. We'll never do this. So manual tests are executed very rarely, and they are complicated to maintain because you have to maintain protocols or whatever so yeah complicated tests everything in between yeah so these lines they are not clear cut so anything in between sort of mixes and mangles if you have ever written a Django application you've probably used the tests thing client thing for Django and you would probably considered as a unit test but other people might say this is actually an integration test because you're talking about over a fake web protocol so this is a theoretical thing which is nice to have in mind but yeah I don't know how much it actually benefits us yeah other tests I already talked about this speech testing how fast is my application load testing how good can how well does it behave with larger inputs. Best practice in style testing goes, so linting basically does my code, so this is testing not on the running applications, but on the code, is the code according to best practices, does it, for example, is it PAB8 compliant, so on. Of course, security testing is also something that you could do in a web app or whatever. Right, so let's dive into it. Unit tests, I guess everybody here has heard about unit tests, has written unit tests. Unit tests are really the bread and butter of any software developer. They are, as I said, lightweight, run them very often, and they make sure that the most basic parts of the software, the units, that's where the name comes from, work correctly, and the hope is that if my small parts by themselves work correctly, then if I put them together, then it might also maybe work, so that's the hope. In Python we have three, basically three, we have, I think over 20, I looked it up, there's a listing on python.org, over 20 frameworks, but these three are the ones that you hear about most often. Unit test is the only one that is bundled with Python, so it's part of the standard library. That is also its most unique feature. Then there is PyTest, which is, in my opinion, the best one because it's the most, yeah. I guess it's still under development. It's getting releases every month or so, and it's becoming, every month, it's becoming more easy to use, more features. And finally, there's Nose. I think NOS is now in maintenance mode, it's not really developed anymore. It used to be, I think it's something, you can think of it as something in between, it makes writing tests easier as unit tests, but then sort of PyTest sort of took over, so this is getting into politics. Yeah, so what is a unit test? This is what a unit test looks like, actually two unit tests in PyTest style. Very simple, you have, for example, in this example you have a calculator and you just check if the addition and subtraction methods were correct. It is, well, depending on how you implement those methods, but this should run very quickly, so you could have hundreds of those tests and it will be done in under a second. Then we have a cool tool in Python which is called TOX. And this sort of goes on top of unit testing. And it deals with a problem that you might have when you, sometimes you have the situation where you run a test and everything went fine. You check the code in and then your colleague checks it out the next day and says, well, the tests don't run through. And this can be due to many circumstances. Maybe you have forgotten to add a new dependency to your project or you had some cached files somewhere or you made an assumption on the system that is actually unique to your system and not, for example, present on macOS or whatever, some other system. And TOCS sort of tries to deal with this by taking your test suite from your development environment into every time you run the test, It creates new environments, virtual environments for you, puts the code there, and executes it there. There's a fresh install of everything, so you know I'm removed from everything that I have in my messy development environment, and I have it running in a clean environment. And you can also, this is what, so this is a configuration file. You can also do this for many versions of Python. So many, some of you might remember that there was once a version of Python 2.7, it was called. Nobody uses it anymore today, I'm sure. But some libraries still want to support it. So TOCS makes it easy for you to just specify many versions of Python and runs a test with once in every of these versions. Right, next cool tool, hypothesis. One drawback of unit tests is that you have to write them by hand. You have to think about them by hand, well, by brain, I guess. So, yeah, you have to find all of the corner cases. Usually when you write tests, you think about, yeah, you think about the edge cases in your program and you write tests for them. But what if you don't, if you're bored and you don't want to find them all, you don't want to write them down? That's where Hypothesis is nice to use. you can specify ranges of input. So instead of writing a test and specifying one input for the test, you can actually define ranges of input and say, okay, this function has to work for all the integers. This function has to work for all text. This function has to work for all booleans. Well, there are only two. But, yeah. And it is optimized to try to find edge cases. How can this work? So this is a hypothesis test. you can see again my little addition test from earlier so actually this is a little bit different it takes one integer and it tests that if I add this integer to itself it will in fact be this integer multiplied by two and I want this to be true for all integers so I can there's an import missing I need to import given as well so you say okay this is true for all integers and this is done with the given decorator and the integers sort of sequence generator. And this turns your one test into actually a generator function which generates by default 100 tests. And this number you can specify and yeah. It's 100 tests, it's not actually testing all of the integers because that would take forever but it is optimized to sort of test, it also doesn't only test one through 100 because it's boring, it tests a couple of, so it tests zero, it tests negatives, and yeah, it tests very large ones. You can read it up in the documentation. It's very interesting. And there's generators for integers, for text, for all kinds of data types. Right, we have supportive tools to make it easier for, because in the end we want to have it very easy to write tests. So we have a mock which makes it easy to hide away stuff. So if you're reading a file and you're doing something with the contents of that file, In your tests, you maybe not want to deal with file operations, so you mock it away. Say, hey, program, if the test comes along and wants to open this file, don't actually open it, but just give it this content. So this is cool. This is mock. Mock is in Python 3. It's in the standard library as well. And faker. Faker is one example for a test data generator. It makes it easy for you to generate realistic-looking test data. For example, you can say, hey, Faker, give me a person, and you get an object which has a first name and a last name and an address, an email address. Give me a company, and so on and so forth. You can do that in many different locales. You can say, give me an Italian person, give me a Polish company, and so on. This is nice if you want to quickly have realistic test data. Behavior-driven development is an interesting concept. So, it wants to move, let's say, the stakeholders and the developers closer together. So, the idea is that if you're producing software, you have users who tell you how the software should behave. Then you as a developer have to understand this, write the test for them that the user cannot really understand anymore. So let's cut off the middle man here and have something that both the end users can understand and also the programmers can understand and also the computer can understand and execute. In Python, there's an implementation called behave, I guess. And yeah, let's look at an example. So the top one is what you would then write as test cases. So you define scenarios, you have a couple of keywords in here, these examples only have given and then. So given defines preconditions and then defines assertions on those preconditions. And this is arguably something that a less technical person could write and understand and also it is structured enough for the computer to understand. But you as a developer still have to do some work because you have to define all of the lines that you can use. I access location placeholder URL. You can then use this sentence and give it any URL, and then you have to write something in Python that tells it what it actually has to do in this case. I don't know if anybody... I've never really used this in production. It's interesting to think about how far can you go with it. I don't know. Right. Time for one little excursion. Let's think about correctness. And something that is sometimes maybe misunderstood is passing test suites says nothing about the correctness of your code. Correct code can have failing test suite and incorrect code can have passing test suite because tests are also programs written by humans who make mistakes, so you can have faulty test suites, you can have incomplete test suites, and you can have misconfigured test suites. So don't think if your tests are running through that your program is running correctly under all circumstances. Tests are merely a set of assumptions on your program, how it should behave in the wild. if you think of this circle as all the possible situations in which your program can find itself, then the tests will only cover sort of one little part of this area. This is not to scale, the orange part is much smaller. So we will never completely understand, you will never completely prove that a program is correct. It can be done, there are theoretical approaches to theoretically proving a program's correct, but it's not fun, and I don't know if it can be done for things that are more complex than a counter or something. So keeping this in mind, we have to think about, okay, how can we optimize this orange thing that covers, I guess, a large portion of our domain, and most importantly, the most critical ones. So let's talk about test suite quality. How can we improve the quality of our test suite? Well, this is something I have mentioned before. We have to make it very easy for ourselves and our colleagues to write tests because if it's not fun, if it's hard to write tests, then we will not do it. It's just human nature. We will avoid this, so make it easy. Use tools like Hypothesis, PyTest parameterization. I haven't really mentioned this. This is also something that makes it easy for you to have one test run under multiple sort of scenarios. And analyze the code coverage. What is code coverage? Code coverage tells you which parts of your code got executed in any given run of the software. You can use this in production if you want, but most often it is used during a test run. So you run your test suite and you run this with coverage. There's an implementation for Python which is called coverage.py, which is a fitting name, I think, and then you run your test suite with this and coverage will tell you afterwards which parts of your code actually got executed. There are two different kinds of coverage, line coverage and branch coverage. So line coverage tells you which line actually got executed in your run and branch coverage tells you which branches were taken. So if you have an if, it has, okay, there was one run that went into the if and one that went around the if. And they are not the same. So, yeah, this is out of two different types of coverage. So, again, we have the problem 100% coverage that only tells us that all of the colon and branches were executed, it doesn't really tell you anything about where they properly tested. They could be run in a very trivial case because coverage tells you nothing about the different values that the variables could have taken. But, you can be sure, uncovered test was definitely not tested, so make sure you have 100% coverage in your project, and pro tip, start right at the beginning. It's very easy to start a new project and enforce 100% coverage from the beginning. It is very, very hard to do it later. So yeah, start doing it right away. And include the test code in your coverage because if you know that the test, if you have a 100% coverage and your tests are actually in that 100%, you know all of the tests were run. There are many different reasons for which tests might not have been run. So, yeah, do that. And the last thing I want to talk about is something which is a bit exotic called mutation testing. I've never actually used it, but I find it interesting. And it has, it operates under the assumption that a good test suite should be so specific that if I change something in my program, so if I introduce an error, that my test suite should break it, so it should detect it and fail. So the idea of mutation testing is take the code, change something, run the tests, and assume failure and start over. And if you change something in the code and the test suite is still good, then your test suite has a problem and it will record this. There's something for Python which is called Cosmic Ray. It does exactly this. It basically uses Celery as a task runner and it just spawns, yeah, it spawns these, it takes your code, does one mutation and spawns a test run for this, does another mutation, spawns a complete test run for this. There are different operators who, for example, go over your code, look for all the break keywords and change them into a continue, and look for all the continue keywords and change them into a break. Change constants, flip booleans around, so if you have an if statement, it just add a not in front of that, it's fun. It does that by modifying the AST. Yeah, and as I said, it's running in Celery. This picture has nothing to do with the project. I just thought it was cool. And now this is actually the last thing, manual testing. So you will find people who say manual testing is no good. Testing has to be automated 100% because you never know what the people do. They might have a bad day and not do the test at all. do them in a bad way you just get frustrated with it but i think you can think about it another way automation is just not creative even the mutation testing thing it just does yeah basically random stuff and it's not creative it doesn't think about the tests in a human way so for example if you test the website and you have completely messed up colors but the tests only test for the contents of your fields then the test will never say something but a manual person a human person I guess would say hey the colors are all wrong what's going on here so you can use if you have the capacity do manual testing for example for releases if you do that use test protocol, so it has at least some standardization. And yeah, basically use this only rarely, because on top of the pyramid, you use it for big releases depending on your capacities. I think I'm at the end, if you still want to talk to me. You can use Mastodon if you're really bleeding edge. You can use Twitter, my website. And this is my face. If you see this on the conference, you can talk to it. and we can have a conversation about testing. Thank you. Thank you very much, Niklas.