pytest - simple, rapid and fun testing with Python

Speaker 1 [00:03]

It's nice to see that this room got so full so quickly. It's going to be interesting, because the workshop I usually do at conferences is intended for three hours, so I had to cut it down to half the time. But we will see how that goes, especially with so many people. And someone who wanted to help me with questions not being here, so yeah. But we will manage. So, a real quick introduction, I'm Florian, I'm using Python since around 2011, originally to modify an electronic typewriter to write out tweets from Twitter, which was quite an interesting project and I at that point I mainly programmed C and bash scripts and trying to parse HTML in bash or C is not a good idea but that's how I got into Python. In 2013 I started Qt Browser a Vim-like browser written in Python which is still my main project today and in 2015 I got do PyTest as part of the AdoptPyTest month, where PyTest maintainers helped projects to start using PyTest. I fell in love with it, started contributing, and somehow ended up as a maintainer of PyTest at some point and gave PyTest courses at various EuroPython events and also at various companies. So what will we cover in this course? First of all, how to install PyTest, which hopefully is something most of you already did by now. Then how to write tests with PyTest, how it avoids boilerplate, and what nice command line options there are, and some other basic features. a very big PyTest features which are fixtures to do setup and teardown of tests. Then we will take a quick look at how to run existing test suites and how to migrate to PyTest. We will see an overview of what plugins there are and if you still have some time left at the end there will be an open space or just room for discussions and issues. So if you have another run, hopefully nowadays everyone should be using Python 3 because the year is almost over and then Python 2 is finally dead, or kind of dead. So if you have no Python 3 running yet, could you please raise your hand so we can get it sorted out? Good, no hand. So, what I recommend is creating a virtual environment, because then you can play around with PyTest and plugins without affecting your system-wide installation. If you already have used Virtualenv for vEnv before, could you please raise your hand? Cool, that's maybe three quarters of them. So what VirtualRAMP does, it just gives you a subfolder with a Python install in it. And whatever you do in that subfolder, whatever Python packages you install, are just contained in that folder. So you can play around, install different versions than what you have system-wide. And when you're done experimenting, you just delete that folder and everything is back to normal. so what you can do is install pytest in a virtual or otherwise, whatever you prefer and call pytest.h to print its help to see how things work and if this doesn't work out for some reason, please raise your hand so we can see what's going on In the meantime, I have some questions for you, just to know you a little bit better. Can you please raise your hand if you have written unit tests before? Okay, keep your hands up if you've written unit tests in Python. Okay, keep your hands up if you've written those with PyTest. Okay, cool. Now please raise your hands if you know what a decorator is. If you know what the yield statement is. Okay, cool. It's fine if you don't. I'll explain. No worries. So it looks like everyone has PyTest up and running. Awesome. Then we can get to the more interesting part. So what are the fundamental features of PyTest? PyTest tries to avoid boilerplates to make your tests as short and as readable as possible. It tries to give you useful information when a test fails. It allows you to parametrize tests so you can have a single test run over several datasets or several arguments, for example. It allows you to have model or setup and teal down of things you need in tests using fixtures. And it's very customizable, so there are a lot of options to customize it to your needs and also a lot of third-party plugins. So the easiest way to run PyTest is to run PyTest. It used to be called PyDotTest in older versions, mostly for historical reasons, but then we got rid of that dot, which wasn't without controversy, but it's gone, so nowadays you can just call the PyTest tool. What you often see in PyTest projects are two files, ConfTest.py files, which have some global objects for the test suite, things like fixtures or hooks, which we will look at later, and a pytest.ini file for configuration. So it's a more declarative thing where you can say, for example, I like those command line options. I'd like you to, like if I run pytest without any options, just act like those options would have been given. For example, for more verbose outputs, so it could always have verbose output without having to call it with dash v or verbose. If you just run pytest, it will run any tests it also discovers in the current directory, or it can also pass a file or a directory to it, and it does what you would expect. It collects all tests from there. So how does it discover tests exactly? It looks at any files starting with test-underline prefix, so test-underline-anything.py. In there, it searches for functions and clauses which start with test-underline or test, and also any clauses which derive from unitest.testclause, so the standard unitest way of writing tests with the Python standard library. And something we see here and also we'll see later is that there is a bit of magic involved and even some concepts which might seem a little strange at first, but at the same time, nobody came up with something better yet. So once you get used to those concepts, which might be a little weird, PyTest is something which is really easy to use. so how does that test code look with pytest well you have a function you do something in there and then you just use python's python's assert statement to test things so contrary to the unit test library there is no things like self dot assert equal or assert less or whatever you just write whatever condition you would write with an if statement for example but use the assert statements to make your test assertions you can use classes but you don't have to and many pytest tests just define three functions so the main reason to use classes is just to group related tests and there you don't need to inherit from anything you just add a class as a test group basically and that's it so as I said we have those kind of simplified assertions for most things there is a corresponding assertion in the unit test library but still I feel like that's the first big thing I noticed when migrating the Python test oh my assertions get so much nicer to read and to write because you don't really have that discrepancy of how you would write a statement ideally and how you need to write it for your tests you just use the assert statement there's also something I've heard about pytest which is the best API is no API which kind of is visible nicely here but of course there is a little bit of API in PyTest for example if you have an exception and you expect your code to throw that exception for example a zero division error here there is a PyTest.raises context manager so you have this with block and whatever is in that block when that code is finished or anything in that code should raise that exception now maybe if you're not used to this you'd ask well how is that different from try-except the difference is that code really needs to raise that exception if it doesn't your test will fail because you say hey I want to test that it does raise the exception so now I'll show a quick demo demo of how a failure would look with pytest. There is a file called failure-demo which which just tries to show various different kind of failures. So what PyTest does is it's, again, does some magic to be able to introspect what happens in the assert statement. And then shows the output like this. So it tells you your code called assert F equals G, G, those function calls, and that boils down to assert 42 equals 43, and that assertion failed. And if you have some more complex things, for example, if you're trying to compare strings, it tries to give you the difference between those strings and tries to make the output is nice and as obvious as possible because I think that's one of the main goals of a test runner to show you where your errors are as obvious as possible so you don't have to debug and don't have to dig down into them but instead you just see what's going on. There are many more examples there for example if there's just one character difference it tries to highlight that character and so on, or tries to skip if they have a big data structure, like a big string or a big list, it skips whatever is equal and only shows you, hey, there's a difference there at this position. So, PyTest has a lot of command line options. I can't really cover all of them, so I'll just show some which I use almost daily. One thing PyTest does as well is it can captures all the standard output from your test. So if your test prints anything or your code under test prints anything, with the Python unit test, you can't really do that because your terminal would get messed up with all those printout outputs. But with PyTest, if you know in a test something is interesting, you could just print it, and PyTest will hide that output but show it to you when the test fails. and with the dash s option you can tell it hey just disable that capturing show me all the outputs even of pausing tests with dash x you can tell pytest to exit instantly if there is some failure you can use dash k like keyword to filter with a given expression So, for example, only run the tests with whatever in their name, in the function name. And you can use //pdb to start the Python debugger on an error if you want to dig down into a test failure. So, we'll start with the first little exercise. if you haven't downloaded the code yet there is a URL here at the bottom where you will find some example code for future exercises for these exercises just write some test functions play around with options a bit create a failing test and insert a print statement so you can see the output capturing also we have Rafael Piazzina here or Hockedroth who is working on PyTests as well and CookieCutter mostly. He'll also help me or help you if there are any issues. And feel free to ask, of course. Do we have the slide? Unfortunately, no, because the person who originally wrote them asked me not to share them. So, I mean, it's been a while and I modified them a lot but still I need to If you have a basic test running, you could also try to create a pytest.ini file, so a config file, where you persist one of those command line options, for example, dash x, so whenever you run pytest, it always stops at the first failure. The pytest.ini file should always be in the folder you run pytest in, so usually the root of your project. What you also see in some projects is that they use an existing config file like a tox.ini or a setup.cfg, where PyTest also can read options from, but I personally prefer to have a separate file for PyTest, because then it's clear what file is for what purpose. Filters or marks, exactly. so the concept of those fixtures it takes some time to to get used to it so if you feel lost now if you feel like okay he's talking about fixtures but what what what are those really um please let me know and we'll try to clear things up I'll just reproduce it for a second. So we have this test fixture.py, which is the same as on the slide. And if we call PyTest with, there's just fixtures there, it will show us the fixtures defined there. If that fixture has a doc string, like those strings at the top of a function you use in Python to document them, PyTest would also show it here. and what you will see when you use skip inside a fixture for example something like this so we inside the fixture we say hey we can't run this test for some reason for example if you add a fixture maybe for a database backend and that backend isn't available you could call skip inside the fixture and then what you will see if we add another test function here which uses that fixture and now run PyTest on it it will skip both of those tests so the gist of it is that a fixture can say hey I can't do this I will skip this test and whatever test is using that fixture will automatically be skipped because there is some kind of dependencies and kind of object it needs which isn't available for whatever reason PyTest, the reason you pass to the fixtures, PyTest doesn't print them by default, because in bigger tests you likely have a lot of skipped tests, or skipped or x-failed tests, but you can call it with "-rs", like reason skipped. And then at the bottom it will tell you what tests were skipped and why, and from where. There are also other arguments to that R, like you could say, give me a summary also there again of all failed tests or of all X failed tests or whatever, depends on what letters you use of the special so let's recap about what fixtures are fixtures are returned from a fixture function, so a Python function decorated with PyTest.Fixture. Each fixture has a name, which is the name of that function. And a test function can request or can use that fixture by having an argument with the same name. What we'll see next is how fixtures can be cached and how to do clean up or tear down from fixed truth. So sometimes we have something which takes some time to set up. Here we just used a time.sleep call, but it could also be a database or something else which just takes some time. Then maybe there is a good reason to say, okay, I don't want to set it up again for every single test. In an ideal world, tests would be independent from each other, but sometimes we just can't help it. We have some dependency which takes some time, so we just want to set it up maybe once per test file or even once for the whole test session. PyTest fixtures can do that as well. In that fixture decorator we can pass a scope which is the caching scope. By default that is function so the value is regenerated or the fixture function is called again for every test function. Now take a look at the test fixture scope file. Modify that scope to module which means it is cached and just generated once per test module and take a look at how the runtime duration changes. You can also use the duration argument for PyTest to get some additional info how time is being spent. What you should see is that your test takes two seconds without that caching. And if we edit that file to use scope module instead of scope function, PyTest caches that value for us and it only takes one second. So the expensive setup only happened once and PyTest then reused that value for all tests in the same module. Quite often, if we set up some kind of object for tests, we also need to somehow clean things up again. Let's say for a database, maybe we would want to close the connection or roll back all changes after every test, maybe. So what we can do here, instead of returning a value, we use the yield keyword. now what the yield keyword is or does exactly doesn't really matter for you to use that feature all you need to know is that it kind of returns a value which the test can use but then after the test runs the rest of the fixture function runs so the setup here at the top runs before the test then here where the yield statement is the test runs and anything after the yield statement is teared down which runs after the test like it depends on the fixture scope if you have a with scope function for example where the value is regenerated for every test then the teardown also runs after every test if you have for example scope module then before the test module runs, the setup runs, then the whole test module runs, and then after that the teardown runs. And there are also some other scopes like class for every test class or session for the entire test session. We have some print statements here in that A object. When it gets created it prints A setup. When finished it's called the prints A teardown. Let's say we have two tests here and now if we run pytest on that file with dash S to see the standard output, so it doesn't get captured. What we see is a setup, then in the test 1 teardown, setup again, in the test 2 teardown. Now if we would tell PyTest to catch that value by using scope equals session, for example, and run it again. Then we can see the value is set up once. PyTest uses it for both tests and at the end it is turned down. Now, one feature I don't want to get into too much detail about but is useful to know that you can also parameterize a fixture. For example, if you have a database backend and a fixture database, for example, and you know your code can run with, let's say, either MySQL or Postgres or two different libraries or that kind of situation, you can parameterize a fixture and then all tests using that fixture will run twice it will run once with database a and once with database b and now that that's the params syntax i decided to to move that out uh to cut it out because of time reasons but just so you understand what's going on here and what you then can do to kind of compose those fixtures is that you like you like you would use a fixture in a test by having an argument named the same you can also use the fixture inside another fixture so there is a another exercise file test fixed using fixed write a test function there using the onSphere2 fixture, also parametrize onSphere2 to return two different values similar to what we did above, then run pytest with "-v", for verbose, and see what tests or what combinations of tests are being run. What you should see when you write a test using that answer to fixture and when we run that test is that I test runs it with the value of 10 and 20 so we didn't parameterize the test itself. All we did is parametrize that fixture onto one. Now if we now try to parameterize answer 2 as well. Let's say we have parameters 2 and 4 here. Now parameterizing fixtures, the API for that is kind of weird in my opinion, like Like the bad kind of weird, for once. You have a special request fixture, which is built into PyTest and gives you information about the test which requested that fixture. So it could also say, hey, print the name of the test which is running at the moment. But it's also used to access those parameters we use above. so if you parameterize the second fixture as well so we have answer 1 which is 10 and 20 and answer 2 which is 2 and 4 and now run pytest again what happens is that pytest runs all four combinations of those values so basically what I think you'd expect if you use two fixtures which are both parametrized. But I think the basic thing to take away is that you can use fixtures inside fixtures, which is quite nice to build up some kind of more complex structures. If you have some kind of object you need in your test and that object needs another object, you just have a fixture for each thing you need and then you combine them however you see fit. Now, how are those fixtures visible? What tests can use those fixtures? What's kind of special is that you don't need to import those. PyTest also discovers them just like it does with the tests. so whatever function you have decorated with that pytest.fixure decorator you can use immediately now if you have a fixture method inside a class it's only available in that class if you have it in a module likewise it's available to all tests in the module and if you have a contest.py file somewhere in your directory tree you can define a fixture there and it's available to all tests in that directory and also all sub-directories. So that's kind of a nice way you can use to organize your fixtures inside your test suite. If you have plugins defining fixtures and that plugin gets loaded, the fixtures are available anywhere. so i want to quickly look at some built-in pytest fixtures which can be quite useful pytest can give you a fixture to create a temporary directory for each test it gives you a fixture to temporarily modify external states for example patch some function away and it gives you fixtures to influence the capturing the outputs and logging capturing so if you need to know what a function prints or what ends up in the log you could use those so what's the temp path fixture could you maybe raise your hand if you have used path flip before from the standard library Okay, that's more than I expected actually. So what poflib is in the standard library, it gives you a pof object. So instead that you use the os.pof module with strings, you can use those objects. And for example, use the slash operator to chain pofs together and it will end up working on Windows and on other systems, because Windows uses backslashes for the path separators. And for a long time, PyTest had something similar with the fixture called tempdir, but with its own kind of path object, because pathlib was only added in Python 3.4, I believe. Nowadays, there is another fixture, temp-path, which just uses popflip and the popflip API. So for consistency, I'd recommend always using that. But if you see tempdir in a test suite, it's kind of the same thing. So what it does, it gives you a temporary directory where you can for example create files, write stuff to them, and then use them in your test. There are people who believe you shouldn't do file system access inside a test. It's kind of not pure enough. I personally believe you shouldn't do network access because then that will end up painful because then the server is down and your test fails or whatever. But I think often file system access is okay and makes things easier because it's just a temporary file on your file system and what PyTest does if you print the directory there it keeps those around for a while. So if you run your test suite like five times, it keeps like I think the last five temporary directories it did for those tests. So if something fails with file inputs or maybe with some temporary file or whatever, it's just there and they can inspect it and run things manually and see what's happening, which I find quite useful. Then there is the monkey patch fixture, which you can use to patch things away. Like I said before on the dependency injection slides, sometimes there are better measures like pausing in some object instead of creating it. But sometimes it's okay to patch things away. Then there is quite nice PyTest documentation about those fixtures, but the main things to use are setAutor, which allows to set an attribute and cut it away, and setEnv, which allows you to set an environment variable. There are also some example files in the code directory, but I unfortunately have to skip them. The last thing about fixtures, sometimes you need to just do some setup inside a fixture. Maybe create a database or patch some things away and you maybe aren't even interested in the value, so you just use it to have a nice place to put setup code. Now what it can do is pass auto use equals true to pytest and fixture and pytest will automatically use that fixture for like here for the whole session or use it for every test without having to explicitly request it. You can still return a value and then use it maybe in some tests but you don't have to. You don't have to have a DB argument for it to be used. So, wrapping up about fixtures, you can use them to inject resources into test functions, interact with those tests, you can manage the lifetime of those resources by caching, you can set up things implicitly using auto-use, and you can also rerun tests with different configured resources, which I couldn't show in full, but I hope you got a first impression of how things look. So, if I convinced you that PyTest is cool and you want to use it, how do you migrate to it? Usually, if you use Unitest or the Nose library, or Nose Test Runner, PyTest should run your existing tests just fine. If it doesn't, that's often considered a bug in PyTest, so it's good to investigate what's going on. I would recommend if you switch to PyTest, don't aim for keeping compatibility with Unitest or with Nose, because then you can only use a small subset, you can't really use PyTest features. I'd prefer committing to it and only using PyTest because personally, I mean, I'm a bit biased but I don't think there's any downside to it you need to install it, sure, it's not in the standard library but nowadays that shouldn't be a big problem so you don't need to rewrite how your tests look you can start using it incrementally start using fixtures incrementally there is however a project called unitest2pytest actually there are two projects called like that one is from Dropbox and not that good and one is from the pytest organization and better so what that does is kind of a whole automated rewriting of all those assertions you will still need to fix things up by hand but it makes the process a little bit less boring one thing i also couldn't really cover are plugins there are hundreds of them uh for whatever purpose you you need really so if you need some if you use some framework say jongo for example there's a plugin for it uh testing for for cute gui application sure it's plugin. Things like asyncio, twisted, there are plugins. Even if you have JavaScript or C++ code and want to test them with pytest, there's a plugin. So you see some kind of pattern there. There are lots of plugins and if there isn't a plugin for whatever you want to do, it's really easy to write one. All you do is you start by writing some plugin hooks. You can even put them in your ConfTest.py to start customizing PyTest that way. And whenever you are at the point where you have something maybe even usable for other projects, create a plugin for it. At that point, I should really give shout out to Cookie Cutter, which is a project giving you a template for a new project. so you can use cookie cutter and the PyTest plugin template and it will set up things for you like it will give you a setup.py which installs your plugin and all that so it's really easy to get started and to start contributing even outside of PyTest of the core. Some plugins I want to highlight are hypothesis, which is only a whole PyTest plugin, you can actually use it directly with Python even. What hypothesis does is, because it's a property-based testing, so instead of writing tests you tell hypothesis, hey, give me some strings or give me some integers. And then I have some property which should always hold. For example, if I have some encode method and the decode method i should be able to encode something decode it again and get back whatever i had at the beginning so it's just like two or three lines of code you call hypothesis it calls your tests with like some hundreds of values and it's really clever about it and finds issues you would never have found by hand so i'm quite fascinated by that usually in the three hour version of the talk there's an exercise about it but I unfortunately also had to cut it out there is pytest instafail which can be useful if your test suite runs more than a couple of seconds because it prints the output immediately instead of printing it at the end and all kinds of other reporting plugins want a slack message when your test fails there's a plugin for that there are different plugins around repeating tests for example if you want to run a test 10 times for whatever reason or want to write benchmark tests as I said there are integrations in all kinds of frameworks there are integrations for the coverage and the mock library which can also be quite useful Coverage gives you an overview of what part of your codes are run by your tests and what part isn't. So it can give you kind of an idea of what needs more testing. And yeah, lots more. So that's pretty much it from my side. We will have maybe five minutes left for questions at the end. you can contact me here if you have any further questions and can't find me anymore or whatever you can find out more about pytest there and i'm also or i also did those workshops in companies so if you think that would be a good fit for your company please let's talk thank you very much