Introduction to automated testing with pytest

Speaker 1 [00:04]

just a few quick things about myself I'm one of the development leads for the cookie cutter project and I've been contributing to PyTest as a core developer for a number of years by now my contributions to PyTest core it's not a lot of things that I do but I try to kind of advocate for the project in events like this and teach people about PyTests If you are interested in this kind of material that I produce and the content, you can follow me on Twitter or you can read my blog if that's interesting to you. I live here in Berlin and I have to say it's very convenient to live in the city where the conference is. So I highly recommend either going to local conferences or even running conferences in your own hometown. And for a day job, I'm a test engineer, a senior test engineer at Mozilla. and I yeah again I post I try to post regularly on my blog but depending on how much time I have sometimes there are several months between blog posts one ask to you if you read one of my blog posts and you find it interesting or useful or you find any problems errors anything with it please get in touch I struggle sometimes with the amount of feedback that I get so sometimes because I don't have any analytics on my blog. I have no idea if anyone is actually using it. And as an author, I think it would be quite helpful to know if people find it useful. I go by HackerBroad on GitHub and Twitter, and I work at the company behind the Firefox browser. If any of that is interesting to you, please come find me after the talk and we can have a chat. Before we get started with the actual kind of introduction, I wanted to make an announcement from the PyTest core team. So we recently started releasing the 5.0 release series, and that's the first version to drop Python 2 support. So I know everyone is behind on their Python 3 migration maybe, but please make sure that once you upgrade to Python 5, you actually have Python 3 compatibility. There is a 4.6 series that will see security fixes and bug fixes to Python 2 as well. And if you have a new version of PIP and setup tools, those tools will make sure that you get the version compatible with your Python installation. So you're not in danger, but please make sure to upgrade. If you want to find out more about this, you can take a picture of this. That's a blog post with kind of the strategy that can follow. i don't know about you but i personally find it super helpful to learn new concepts by applying them on real problems and what i mean by that is i when i went to conferences and people explained libraries frameworks all of that stuff to me i was mostly okay by reading the foo bar hello world examples but typically what i realized is that a few days after the conference when i was going through my notes i wasn't really sure what exactly i was learning um just because i kind of as humans we're we are very good at remembering stories um rather than theoretical concepts um so my attempt with this talk is kind of give an introduction into pi tests by working on an example project and i really like nature documentaries and before i wrote this talk I watched this documentary, I think it's called Earth, One Amazing Day. It's with David Attenborough. And so I came up with this idea of creating this Earth project. And our task for today will be to kind of switch our role, our mindset into a test engineer mindset. So the kind of fictional story is someone from our company comes to us. could be a maybe a product manager some some VP saying hey we have the super super important product and it's going to be released in two months we have to use this one very particular tool but the person developing that's who she left the company so we have no idea how it actually works but we know we have to use it can you please have a look and give us kind of a like make an assessment if there are any blind spots that we need to fix before we actually use it in production. On a side note, I wrote a blog post about this too, so if you kind of want to come back to the content of this talk, there won't be slides that have the contents, but you can follow along on this blog post. And the source code for this demo live coding session is available on this repository so it's just my github handle under the earth there is a specific branch which has commits for the individual steps we should be able to follow along cool so what I did I cloned the repository and without looking into the implementation of this library we want to kind of get a better understanding of what it does so what I typically do is I would run the tests due to my being a test engineer. There is also a README file, so that's maybe where I would even start with the documentation. And what we can tell from the README here is that the Earth library is using Python 3.7, and the only requirement is on attributes, address. And there is also some code snippet here for how you would use the library, which is super helpful. It has a code of conduct. Okay, that's all we need to know for now. So we set up a virtual environment using Python 3.7, and we install. I think there is also requirements.txt, so we install from that. Yep, some development tools, attributes. Cool. So the first thing that we do now, if you check out a new project and you don't want to read the source code, is maybe like grab for imports on stuff which might actually cause damage to your environment so like things like the OS import so if the tests have some broken setup or tier on code which deletes files from your file system you could check for this or yeah so that's maybe what I would start with can't find anything so that should be should be good then we run the tests let's see what happens and we get a super super interesting error message it's saying that it's a syntax error and it's apparently using Python 2 syntax and it has some more information here which is maybe something that I'll get back to later. So then the next thing that I would do is I would open up this file and have a look. And one of the amazing things about pytest is it can be very specific about what kind of tests you want to run. So as we can see we have this invalid syntax here under Python 3. So what we can do is we can commented out at a to do comment can we remove this later and then if we want to skip this tests we can we can import pi test and use a feature called markers so what we do is we use a decorator here pi test dot mark and then we use one of the built-in markers which is called skip we provide a reason to it outdated Python syntax and maybe because we're still on Python 2 we're very disappointed cool so that should be should be fine for now let's run the tests again and see what happens. Cool. So we are able to run the tests. Maybe that's a good point to explain why PyTest kind of crashed on our tests during the collection phase of the test run before actually running the tests. There is some kind of black magic how PyTest provides error messages. And one of the steps in this process is looking at the abstract syntax tree of your Python files. And it tries to add some extra information to it. And since the print statement is invalid Python syntax under Python 3, it can kind of complete the step and that's why we see the error. So we have an error here, a zero division error, and then we have a test failure. So we can kind of have a look into how the other tests, for instance, look like just to get a better understanding of PyTask. We can actually do it here. So a PyTest test doesn't need to be more than just a function and you can have assertions in it using the assert statement and different from the built-in unit test library you don't have to call into specific methods on a class that you have to inherit from but instead you use the assert statement and write your comparison just as like this. You can add custom failure message as a string after it, but typically PyTest is smart enough to provide you enough information about why the test is failing and what the values of the comparisons are. Cool. So now that we see we have a test suite with a number of tests, I would be curious to see if it actually does anything so one of the ways of you how you can do it is you can use the coverage plugin against your library run the tests again and see how much of our code is actually covered with tests and we see that something takes a lot of of time so we have 53% test coverage which isn't great and that might be a good point to check if any of the tests actually import our library because because we can't possibly generate code coverage for our library if we don't even import it so we can grab for the name of the library in our tests and that's just one test which imports the library so probably all of the other test do something but they're not useful at all. So there's a good chance we can possibly just remove them. As we saw the test took a while to complete so what you can do is you can use the durations built-in comment line tool, sorry feature option to give a list of the slowest running tests. So we provide a number here, that's the number of tests that we want to display in this list, and sort them by the time they take to complete. In this case it's pretty obvious to us, but this feature is super useful if you have a large code base. So we see here that in the calling phase, so the phases of a test run are you run all the setup code similar to what you would do in unit tests, then you have the caller phase where you execute the test, and then you have the teradon phase. So we can see here that the execution of the test itself, test string methods, takes two seconds or more than two seconds to complete. And what we can do now is since we've seen that there is not a lot of code coverage for a library anyways, we can run this particular test and generate the code and see what the code coverage is for the earth library it's zero so what's the point of having a test that takes a lot of time to complete when it's not doing anything useful. So how about we get rid of this? Okay, there is a comment here, so at least the developer documented that something is wrong, but it doesn't help us now. Let's skip this entire test module. We import pytest again, but rather than decorating one specific test, we want to skip this entire module. So we call PyTestSkip provides a message. This does not generate code coverage. And we have to specify this keyword argument here to be able to tell PyTest that it has to be able to skip this entire module. Cool. running this again you can see one test was skipped you can print the reason why test was skipped by dash dash R for reason and then s is for skipped there's also X for a different marker Yeah, this does not generate code coverage. Awesome. So we have a test suite that's fast to run, still doesn't produce any code coverage. So what I typically do is, since we have an example, I would maybe run the example code itself and see if there are any bugs in it. Also to get a better understanding of what it does. Let's run the example. so there's a bunch of prints here hello adventurers and then there are folks in accepting invites people are packing that are traveling and then it's meets welcome to PyCon US in North America let's start with introductions so from that we can kind of tell hey maybe it's a library for organizing events probably our product manager already told us cool so we can work with this example works so one of the best things that we can do now is to generate create a new test file we call it test earth because that's maybe just a side tip if you want to kind of test the entire library you can help with the naming but so the name of a test file and the test functions and all that should indicate what it's trying to do so rather than just having directory with numbers attached to them you can maybe use something like test name of the library to indicate this is something that I test the entire library test earth and we want to have it on the test level and that's the point where I realize I'm not using this all often where was it here anyways cool so let's open up the readme file and copy paste the example from here we don't need this here and then we go back to test earth add this here. That looks fine. We import PyTest and then this should be our test function. So we'd say test of and that's all we need to do. I think it's a common misunderstanding that you always have to have some sort of assertions in your tests if you're really just trying to get a what you maybe call a happy path tests or like for a library this maybe wouldn't apply but an integration test sometimes it's enough to kind of run through the code and see if it produces any unexpected errors so you don't have to check for any values in this case what we want to do is we want to make sure that the example that we provide on our documentation always works so as again as a side note if you maintain a project and you provide example usages on your documentation please test them as well because as a user if I come to a project I find an example I try to run it and it crashes in my face I'm probably just going to move away and find an alternative project so let's run this since I don't want to type out the entire file name all the time I can use markers again and this time I'll apply a custom marker. So I use WIP for work in progress and you can also apply multiples. So I'll also add happy indicating this test is a happy path test so it doesn't have any surgeons but it tests the entire library. Then on the comment line I can select those markers. I can say I want to run my work-in-progress tests and a newer version of pytest will now tell you hey you're trying to select tests by a marker but you haven't registered those markers so maybe you did a typo in your test selection you can get rid of this warning by creating a new file to configure a pytest it's called pytest any and then you have a new section pi test markers is it like us work in progress tests aren't happy happy puff tests cool so it passes that's awesome So now I'm curious, did this change our code coverage? And how did it change it? The PyTest Cough plugin can generate an HTML report for you that allows you to inspect the code that is being tested. Is anyone familiar here with the concept of code coverage? I think most people are. So we say we want to generate an HTML coverage report. By the way this error here isn't necessarily a problem it's an existing test that was already failing and I don't see since it's not causing us any performance issues we can ignore it for now I think. We can open up the coverage report in our browser we are at 84% so we have a 30% increase of code covered just by running the example which is awesome looking at the first file it's covered probably doesn't contain anything we can look into the next file of the project and let's see so it has a class adventurer all of this code is being executed then there is one line here which isn't covered something else isn't covered and then there are a bunch of different helper methods for generating new types of adventurers so we can see that most of them are covered but three of them aren't so the next logical step would be to use the existing test and extend it by the functions that are not covered but if we want to kind of keep to this concept of testing all of our examples one like exactly like they are we wouldn't just extend the existing test but instead what we will do is now we copy the test that we have and generate a new one. But first, before we get to this, there is another concept that we can use here to kind of optimize the structure of our test, and that's called fixtures. So what we do is we separate our test setup from the actual test execution. So, rather than defining the list of adventurers here, we move this outside of the test, call it friends, and then we can return this list. And next, in order to be able to use it in our test, we have to specify it as a positional argument. There's also an event, and that is just test-telab as well. Let's move this to the beginning of the file. I test fixture, event, and we return the event in here. we're adding the event and just to make sure that this still works we should run the test again still works so that's good going back to our tests so now we have this here we copy it call this let's call it test large group and friends this doesn't really help if we have multiple test scenarios we'll call it test small group this would be our small group so the new tests will get a large group. We copy-paste this. Okay, so we need a number of new adventurers. Let's start with a panda. Does anybody have a name suggestion for a panda? George, there we go. Do you know a panda named George? Okay, you never know. Cool, next up is bears. You bear. I have a co-worker from Canada, I think he almost looks like a bear I like him and I have two friends who live in California are called Daniel and oh that's a random fact about from the earth documentary bears and warm clients don't hibernate and if you look into the implementation here bears have a very limited availability they're only available to attend conferences in summer so if we have a bear and it's good to have some like different kind of test scenarios and maybe not just one example of one bear but have multiple with different parameters we have a much better kind of test coverage not only by lines of code but different combinations are very very sensible so the availability availability that's a very difficult word is all of the different months and if this looks weird to you then you're probably not using Python 3 and you're not using enumerations which I highly recommend for tests up so somewhere in our code here in the year we get a enumeration of months which maps from like abbreviations for a month to a longer string of the month and this is kind of how you it's I think it's I don't know what Python 3 version that was but improved unpacking kind of support in Python allows to unpack an enumeration into items that can then add to a list cool anyways so Daniel and and also Audrey and then we need foxes foxes are based out of Europe two names here I want to be in this test as well and since we don't kind of want to risk that there is some implication on the order we just add a fox somewhere higher in the list as well whoops cool I think that's that's all of the different functions let's let's run us there is a name error friends is not defined okay that's interesting oh we forgot to rename this that's a good point for me to point out that there has been a change in PyTest, which is already some years back. I think it was in 3.0 where we introduced it. But you can now add a keyword argument to fixtures called name. The way that PyTest matches the fixtures with the tests is by using the function name like this, and it's matching it against the positional argument. But if you define your fixtures in the same scope as your tests, and everything in Python is an object, Python will actually find an object called friends, because it's a function in the same scope, and it tries to iterate over the function. In this case, we have an arrow, which is awesome. But there are some cases where you use Python's duct typing capabilities for, say, you just assert and then just the the name for instance so it maybe if you're checking for a boolean but you're not checking for is true but sorry maybe just something like this assert friends and friends you expect to be a fixture that returns either true or false this would pass cool so we have to speed up a bit but this is really important one of the ways how you can kind of work around not work around this issue, actually solve it, is by adding an explicit name which doesn't interfere with kind of the Python scoping. So what we do is we call this event and rename the function. Fixture event. And some of the PyTest core developers raised that this is just a lot of typing and it's annoying and I'm totally with them but I'd rather have my tests work than having to type my hope is that maybe plug-in extension authors would maybe add support for this but it's my bad I haven't raised this with them so but this is how kind of how you can work around this now if we run this we get a much much better error message because now we are being told that friends is actually not defined and Python can't find it. Let's go here and change this. Run this again. Okay, so the first test passed immediately and now our terminal stuck. What we can do is we can run the same tests but this time we want to show the standard output. So all of the print statements that we have in our library by default, pytest captures them, stores them in a variable so you can run tests against the standard output as well. But in this case we want to see and maybe if there are any indicators what's taking so long. Yeah, George is eating. Okay, so from this documentary what I've learned is that pandas are very particular about their diet. They only eat bamboo and the nutritional value of bamboo is so low that they have to eat up for 14 hours a day which is unfortunate if you're planning to organize an event and you have panels there so you need to take account for that but in this case we want to kind of we want don't want to give up on the code coverage because partners are just as important as everybody else so we want to have a test that also includes them but at the same time for the sake of developer experience we want to have a very very fast test suite that we can run incrementally whenever we make changes to library. What do we do? We create a new test. We go to the large group and we should rename this as well. We copy paste this and since we are running out of of time I really want to get to this part where I tell you how you can do so much better small group large group okay I feel bad for doing this but let's call it no pandas group. No pandas group with George Seab. Now what's interesting is we have a slow running tests and we already learned about markers. It would be kind of cool to be able to select only the tests that are fast. We can do this by adding another marker and we call it slow we don't want pipe test to yell at us so we have documentation right away slow slow running tests so this is a slow test it's using a large group and we have happy here we have test no pandas group no pandas group changes here to don't so though we can the same error again cool let's run this and this time we can use a slightly more powerful way of selecting the test we want to run all of the work in progress tests but not the slow ones and we support expressions and those markers they can be they can even evaluate stuff I think but this is one of the best ways how you can kind of define a syntax for selecting tests sorry on it they pass immediately everything is awesome except we have so much code duplication our tests so that's probably a source for errors in the future so to conclude this talk what we will do now is we make this much much nicer by yep so we define a new test or let's let's copy this and we define a new test tests earth except a group it accepts an event it's still a happy puff test it's still a work in progress tests and now we define a new fixture fixture name is group fixture group what this well let's maybe start with yeah so what we do now is we accept this built-in request fixture here. And this is really just the quickest. It's not the the smartest way of getting there, but since we want to use the existing fixtures, this is how you would maybe go about it. We define a dictionary. It's just a mapping, so we map from the string of a fixture name to the actual values large group and no pandas group. No pandas group. We have to... So fixtures can also use other fixtures, so that's the mechanism that we use here. Large group and now pandas group and then we want to return from the groups mapping we want to return the name of the group if this looks funky to you it is I'll explain it later I'm more than happy it's a built-in way of getting access to the current context of the tests so requests will have information about the test that we're currently running and param that's something that we will see here we use another decorator called pi test mark parameterize and if you don't like that word there's a plug-in which is accepts a number of different spellings of parameterize depending on if you're British American or if you just don't want to type it there are variations what we want to do is we want to parameterize the group value and then we define a list of parameters and we can do this simpler but we do it like this for now so we want to have the small group and I know I'm really out of time so let me just skip that we have the large group oops and we want to keep track of the existing markers so we can provide them here as well so that was yep that was the slow marker and it's also the there was also a different error but it didn't occur for now so let's just go ahead, no pandas group. And the no pandas test wasn't slow, so let's get rid of this. So not the no pandas group, but group event. So what happens now if pytest runs this, it's generating a test for each of those parameters, the items in the list. And then we also need to specify this keyword argument here, indirect true. What happens now is pytest will pass this string into this fixture on the request param. So it's a way of parametrizing a test and then adding that value to a fixture. So let's run this again. let's run all of them that are work in progress or happy oops so a small group is fast large group takes a lot of time but it's hopefully hopefully passing as well I should have said this very to a different number okay so for the sake of of kind of not overrunning what this this does exactly the same as all of the tests before afterwards you can get rid of all of the tests we will have to keep the fixtures for now but then there are different ways of how you can kind of go on if you kind of read the blog post you will find other usages of different markers as well which are super powerful and something that I wanted to mention as well. This pytest is based on a plugin system, so maybe if you've attended the tutorial with Florian on Wednesday, you will have heard that there are more than 600 pytest plugins. You can do all sorts of interesting stuff. For instance, if you're like an emoji person like myself or Marietta as well, you can run your test with dash dash emoji with emoji plugin and it will print emojis for you that's super important and then if you want to generate a test report that you can maybe copy paste to the github issue your pull request you can generate a markdown report and look like this so it's generating a markdown that you can copy paste to your github issue with all of the kind of results of your test run so I hope you enjoyed this talk thank you so much for having me and if you have any questions please come find me after. You can also message me on Twitter or send me an email. I'd be more than happy to help you. Thank you.

Speaker 2 [39:10]

So, thank you very much, Raphael. I think it doesn't matter whether you're a PyTest for a long time, like five years like me, or just started. You learned something new today, and we are happy. We have maybe room for one question. Is there one question?

Speaker 1 [39:27]

hi in your parametrization example why do you need the indirection and the passing of a string and then going via dictionary couldn't you just have parametrized via a list of groups instead Yes, you could if the groups weren't defined as fixtures beforehand then we could just specify the list of adventurers directly but since I wanted to reuse all of the code that we've written before, we need to use indirect If we wouldn't use the indirect true PyTest would override group and not call into the group fixture That's kind of what indirect does It's putting the string... Yeah, I know, it's complex it's using that string value passing it onto the fixture as extra context and then still running the the code this makes much more sense if you maybe develop something like a strategy pattern where you're testing your your code against different implementations say different databases or something so you have some teardown code which connects to database setup tables and stuff but on the test you maybe specify hey I'm want to run this test against Postgres and again again SQLite or something so you don't have to rewrite all of the setup and teardown but you just change one parameter

Speaker 2 [41:04]

So there's room for one quick question. Anybody else? No? Let's say thank you again with a warm applause for Raphael.