pytest - simple, rapid and fun testing with Python

Speaker 1 [00:08]

from my side as well if you haven't done that yet please take a look at the repository for this training that is the compiler as my github username and then you should find a repository called pytest basics or i also posted a link in the discord we will have a code there for a couple of exercises the slides are there as well there is a solutions pdf if you already want to take peek at the solutions, though I would not recommend doing that already. And after this, I will also upload all the live demos I'm doing there. So I'm Florian Bluhin, and I got into Python originally when I wanted to modify an electric typewriter with a friend as an art project called Die Schreimaschine, where we wanted it to print tweets from Twitter and similar websites. Back then I knew a bit of like QBasic, a bit of bash scripting, but kind of was at the end of my knowledge. So I looked at Ruby for a weekend, looked at Python for a weekend, and now that I'm standing here, I guess you know where the choice went. 2013, I started working on Qt Browser, which is a Vim-like browser focused on keyboard FOCUSED ON POWER USERS AND IS STILL A PROJECT I'M VERY MUCH WORKING ON ALMOST 12 YEARS LATER NOW. AND VIA THAT I ALSO GOT INTO PYTEST WITH A THING CALLED ADOPT PYTEST MONTH WHERE MAINTENANCE OF PYTEST HELPED OPEN SOURCE PROJECTS TO SWITCH. AND AS THOSE THINGS GO IN THE SAME YEAR I ENDED UP BEING ONE OF THE CORE MAINTENANCE OF PYTEST and I ended up giving my first training at EuroPython 2015, I believe. So a lot happened in that year. Then I studied IT and started my own company, Bruhin Software, where I do this kind of thing, except not one and a half hour, but usually three days. So I teach PyTest to companies, or Python as well, And I also teach Python to first semester IT students. Now we'll start with a little stretching exercise, a.k.a. some show of hands. Who of you have used PyTest before? Nice, that's almost all of you. We'll skip the second question then, actually. Who of you have used PyTest fixtures before? That's less, maybe a third or so, two thirds. Who has used Python decorators, the thing with the at before a function? Nice. What about context manager with, for example, with open? There we go. What about yield? Oh, that's surprising a lot. And then who of you have already set up pytest and are ready to go? All right. so I Have the opposite problem many people have when I prepare for a training I don't start at zero and have lots to prepare but I start at having material for three days and then having to decide which parts to cut out to get to one and all of hours from that and That always ends up with me having a lot more to show than I can show so I try to cut it down as much as possible We will start looking at Why do we write tests? just very briefly. Why should we use pi test? i noticed someone asked in the Previous talk about testing, if maybe some of you were there, Asked via slido, does it matter if i use unit test or pi test? I mean, obviously i don't quite have a neutral answer to that Despite being swiss, but i would clearly prefer pi test And i can maybe tell you why and then we look at the like fundamental pi test features how you can configure pi test and then look at the fundamental features that being marks that being parameterization and then of course as a quite big topic fixtures. I would have a lot more of material. Most of it is grayed out here. That's things we are mostly going to skip. But I want to maybe show a couple of tips and tricks how to deal with failing tests. I want to show you a quick preview of the whole ecosystem around PyTest. What plugins there are. And maybe a very, very quick preview of how you could write your own plugin. So, why do we write tests? The previous speaker also asked that in his talk around testing, and the answer somewhere from the audience was confidence. And I would agree with that. The main reason you write tests is to be confident that your code actually works and continues working even if you change things and we usually roughly look at three sizes of tests either one unit and there it's somewhat debatable what one unit is but it could maybe be one function one class then integration tests where we see that multiple things cooperate nicely with each other and functional end-to-end tests where we really test an entire system so if you have let's say a Django web application written in python you could test a function on python level you could call your is user logged in function or whatever you could write an integration test where you maybe use Django's test client and kind of simulate a web request or you could automate a browser with something something like Selenium or Playwright, and actually use your web application the exact same way a user did. And all of those types, all of those sizes of tests have their benefits and drawbacks. And often you see this testing pyramid, also in the previous talk, that says you should have a lot of unit tests and only a couple few end-to-end tests. And I would say it's not quite that easy. really depends on the circumstances. So for example I have a customer that tests washing machines with pytest and there you're kind of forced to have those more like end-to-end tests. So they have things like a camera looking at the screen of the washing machine, they use OpenCV for image recognition, they have a touchscreen robot. Those are very much end-to-end tests because there is really no other way they could test that at least from python they could write unit tests in i assume c and inside the test you roughly have three steps you're doing arrange you set something up act you actually call the code you want to test and then assert you check the results somehow and with pytest that will be using the assert statement from python Now, there is this kind of purist view that you should only have one assert in your test function, and I would not quite agree. It's fine to have multiple assertions, but they should roughly test one thing, one aspect. But it's totally fine to have, like, two assertions if you have maybe a list and want to test the length and then only one of its elements or something for some reason. And that's where pytest comes in. there is this part of the title of this workshop rapid and fun testing with pytest and sometimes people tend to love this a bit and say what are we having fun with testing how and i hope that if not fun at least that pytest will make testing less painful for you So I have a couple of slides here on setup, but I am assuming that you are mostly all set. So you can maybe look through those if you haven't something set up yet, but I will mostly skip through those. Here's the link again to the repository, clone that, install the requirements TXT or just pip install pytest at least, and you should be all set. So, why do I say you should really use pytest? I would say you should really use pytest because it reduces boilerplate code a lot compared to unitest in Python. Boilerplate code is the typical kind of code that needs to be there but isn't really of any use. So if you have written a hello world before in Java and have written a hello world in Python of course, the Python version will have way less boilerplate code than the Java version will. PyTest also tries to give you useful information when a test fails. It lets you parametrize tests very easily, so run the same test function with different inputs and outputs. It has this very nice very modular concept of fixtures we are going to look at which lets you nicely arrange setup and tear down steps and it's always quite mind-boggling how customizable it is and then and how a wide array of use cases it's used so that's I would say the first thing that surprised me when giving those company trainings how many companies don't actually write Python code when I'm teaching them PyTest, to just use PyTest to, for example, like I said, test washing machines. And what they had was like they had their test steps in an Excel sheet, like one line in the Excel would be one step. They exported that as a CSV file and had a custom Java test runner that runs those tests. And they wanted to switch to PyTest and I helped them develop a plug-in so that during that migration they could just run PyTest on an Excel file and it would interpret it and run those steps. So that's the kind of thing you can do via its plug-in API. And last but not least, there is this survey by chat brains where over 25,000 Python developers Answer what kind of tools they use what kind of things they work on when they use python And I would really love this number here people that use no test frameworks at all to go down Every year I look at it. It goes up by one percent or so Sadly, but also the pytest number goes up by one or two percent and last year. I think we finally cracked Over half of people that use Python also use PyTest, which is quite nice, I would say. So to use PyTest, you run PyTest, for example, by your ID, or especially during this training we will also look at it as a command line tool that you would just run from your terminal, Because there are a lot of useful features in it that you can only access via the terminal. A quick hint before I continue. In the corner of the slides, you will find those QR codes which link to usually appropriate PyTest documentation. So if I'm showing you a feature and you want to know more about that, you scan that QR code and you get to the right page there. So you run PyTest as a command line tool, you can pass a directory to it, you can pass a file name to it, and it will automatically discover your test functions based on a naming convention. You can also add the name of a test function or of a test clause with this double colon syntax. So if you want to be a bit more strict on what you actually want to run, run, then this is one way to just run one single test. And what happens if you just run pytest, or if you point it to a directory, for example, is that it will discover tests based on this naming convention. It will look at all files that start or end with test underscore, and in there then look at all functions and all classes that start with test it also looks at all classes that derive from unit test test case but that's mostly so that you can run pi test over your existing unit test test suit if you are migrating to it so to write a pi test test all you need to do is to follow this naming convention Run PyTest, and it will automatically discover everything. So if you have a function like this, it just takes two operands and an operator, and depending on whether that's plus, minus, times or divided by, it will return the appropriate result, or raise a value error if you pass any other operator to it. Then if you just focus on this addition part here, the most simple test we could write with PyTest would look like this. So we have a file testColc.py, we write a function testAddInIt and in that function we call the colc function we want to test and we use the assert keyword to check the result and when we then run pi test what we will get is something like this so it tells us it found one item and with a green dot here tells us hey this test passed if you compare that to unit test there is a lot more steps necessary there so there you need to actually implement the class that inherits from unit test test case and you need to use those special assert methods which are actually like inspired by Java inspired by J unit and it turns out that that way tests are just much harder to write and to read and pi test takes away a lot of that complexity you can still put your tests in classes but those classes are only really used as a namespace. So you use a class to group those related tests together. The class doesn't inherit from anything and there's really not much that changes. All you need to do is add the self argument and then put things into a class. We will later also see that this can be very useful when it comes to PyTest fixtures. So if you write a fixture with some set of Teradon steps a clause can be very nice to group that fixture together with the tests you need it for. With the assert keyword of Python we can use any condition as an assertion. So anything you can use with an if statement you can use with this assert keyword, with the difference that if the assertion fails, the tests will fail. And PyTest tries to give you nice feedback there, to give you a nice output, based on the types as well involved in those comparisons. So if I run PyTest on a file here where I actually didn't follow the naming convention, so it doesn't get run by default, but I can still point it to it, then I have a couple of failures, and you can see that PyTest really tries to point out what the differences are. So for strings you get a comparison, but if there's only one character difference, PyTest will highlight that, like here. If they are mostly identical, pytest will skip over those. And it also does that for various kinds of data types. So for lists and dictionaries and such, it will always try to give you this little explanation here about what actually happened. And to some degree, you can get this kind of thing when you call pytest from your IDE. I think VS Code recently had a bigger change where it actually now shows you this output, but often I think also in PyCharm you only get a visual diff from PyCharm, which can be nice in other ways, but it's also something you lose in a way if you don't run PyTest as a command line tool. And this kind of thing has like specializations for lots of things, not just built-in types but but also things like data classes, things like, I think, data frames, all kind of things. And in the end, a test can have one of six outcomes, one of six results. It can pass with a green dot, which means everything worked. There was no failing assertion and there was no other exception. it can fail which means some assertion some condition wasn't true or some other exception occurred it can have a result of error which means an exception happens somewhere outside of the test for example in a fixture during a preparation step and then there are a couple of more special ones which we will look at in detail later a test can be skipped and say for example this test only makes sense on Windows I don't want to run this test case on Linux and it can be expected to fail or marked as X fail which means I expect this test failure I know about something being broken but I cannot fix it right now but I still want to keep my test and then if this still passes it's an X pass so an unexpected success a test that I expected to fail actually did pause not only does PI tests give you nice results if an assertion fails but it also tries to give you a bit more context in tracebacks so if we call divide here with the divisor of zero that will raise a zero division error and we will get a With a traceback that's a bit different from what Python usually would show us. Where PyTest also shows us part of the source code, mostly in the function where the exception failure occurred and also shows us the arguments to that function. So again it tries to give us a bit more additional information, a bit more additional context about what's going on there. There is a couple of useful command line options. We can for example tell it we want a bit more verbose outputs where it also prints every test name. We can tell it to be a bit more quiet. And with "-k", we can easily filter tests on the command line based on a keyword, so based on a substring of the name of the test case. So that's something I like to use if I just want to quickly say I only want to run this one test without having to type its full name. start just with a quick exercise, maybe four or five minutes or so, or even a bit less, just to see also that your setup is working and everything, open the test calc.py from the exercises, add a test subtract function there, which as the name says would test the minus operation, run pytest on it and then also play around a bit with those options. So try using dash K to only run your new subtract test play around with verbose and quiet and see how things change I saw there was one question in the meantime on Slido probably tying into the previous talk if I have used inline snapshots or dirty equals and what my opinion on those is I have used something similar a pi test golden as a plugin I really like that for a couple of use cases for example I use it to test the command line tool and test output of it I wasn't aware of inline snapshots which offers a bit more like the automate or like half automated the refreshing of snapshots so I definitely want to take a closer look at that and same for dirty equals I haven't used it yet but it sounds amazing so So often at this point people ask also how would my directory structure look like when writing tests. There are a couple of philosophies around that, but what I usually would do is have a test folder and have that be a one-to-one mirror more or less of your source folder. So if you have a utils.py, you would have a test utils.py. If you have an export sub-package in here, you would have one in your tests and then corresponding test files, at least as a rule of thumb. If maybe you find a clever way to write tests for all those exporting features in just one test file, then it's fine to have a test export pi here. And if maybe your utils pi is small enough to be in one file, but your tests get more more complex for some reason then it's fine to split this up into into sub package here but the rule of thumb I would have this one-to-one mapping then you often see contest of PI files which we look at later it's a way to put things for PI tests like fixtures or plug-in hooks in a kind of central location where PI tests will find them and will make them available for all of your tests. And finally you have a configuration for pytest itself. This could be a pytest.ini file, it could be as part of a pyproject.toml file. And you have dunder.ini.py files, normally as part of your Python package. And there are different recommendations there, but mine would be to also have them as part of your tests and the sub packages in your test folder because what that means is that pytest can import your things from inside this test folder so it can basically do the equivalent of import export dot test trace and export to get to your test file and if you don't do that you will get into trouble as soon as you have two tests files that are named the same even if in different subfolders. So that just makes things easier. Someone asked if I have uploaded my slides yet. Yes, they are also in the Git repository I've showed at the very beginning. Or if they are, then please shout, but they should be in there. And erpion and calc does not work. Not much I can say without more details there. But maybe we can check that as part of the next exercise or so. All I can say is it works for me. I have this source folder here that's based on the so-called source layout. I linked two blog posts here with a bit more details about why that is a good idea. Another question popped up, if I have my package, so here this would be my project and this example here, if I have that installed in the environment where I run the tests or if I run them against the source code. I have to admit the clean way would be to actually install it, maybe as an editable install with pip install dash e but I usually don't do that and pick the lazy way of configuring pi tests with Python path equals source I think that should also be in this dunderini.py where you can then sorry not dunderini it should be the pytest.ini.py where you can then just run pytests and it will import things from your source folder it's admittedly a bit less clean if there's something broken about your install process you wouldn't notice that but it's just a path of least resistance i see a lot of questions that popped up Installed via PyCharm, no utils module. Does that work for others or does everyone have problems and nobody dare to speak up? I see thumbs up at least. Let me just check the repository real quick. So there is an rpn-calc.utils. What might be is that you get into this issue when running rpn-calc with like the play button in PyCharm. You would need to tell it to run rpn-calc as a Python module instead. But we will get to that later. I have not used pytest-bdd-ng. I only use PyTest BDD itself, so I cannot say anything about the question, what I think about that. Then someone says, I have seen a template with a fixture folder in the root folder. And if I have seen that before and why it would be useful. Not in the root folder. If anything, I would expect something like that in the test folder. What I've seen before is that a project ships something like a test-utils.py as part of its sources, or even ships a PyTest plugin as part of its sources. And that can be useful if you want to make testing easier for users of your library. So personally, I think in an ideal world, every library that does something with an external resource, Let's pick PySerial, for example, to talk to a device over USB, over a serial interface. I would find it amazing if PySerial shipped some sort of PySerial test support package, or if Requests shipped some way to mock Requests right as part of its code. And at least in the case of Requests, other projects that fill that gap exist. So it would be amazing in my book if projects would care more about, or libraries would care more about also testing. Then there is one question about parameterization features, if it can be extended to handle dynamic runtime generated test cases while maintaining traceability and reporting clarity. There are quite some ways where you can tie into parametrization. Those are kind of out of scope for this more beginner-focused training, though. You might want to take a look at PyTest Cases, a plug-in. And you might want to look at the PyTest Generate Tests hook that let you generate tests dynamically based on a custom marker, for example. So in this PyTest.ini file, or in a PyProject.toml file, there is more like key equals value like configuration of various PyTest settings. For example, the PythonPath equals source setting, which ensures that PyTest knows where find your sources here and can import from there. Now so far we didn't really use any API from PyTest. We never had to actually import PyTest as a module and that will change now when we look at PyTest raises. So with PyTest raises we can test for an exception that we expect. So we can say in our test with PyTest raises zero division error if I would call call quit three and zero and divide as an operator I expect and I want to make sure that a zero division error gets raised. So kind of similar to a try-except, with the difference that if the exception doesn't get raised, the test would fail. And with that we already have a second quick exercise. Write another test there with PyTest raises, but testing for this value error here when an invalid operator is passed, and then play around with it a bit. So check, for example, what happens if no exception is raised at all, if you call it in a correct way, and what happens if a different exception is raised. For example, if you divide by zero again. And optionally, and especially if you have used raises before, take a look at its match argument. With this match argument you can pass a regular expression which is then matched on the exception text, on the exception message. So that can be useful for certain cases, for example for value error which is a quite generic exception that could happen for various cases and i think i'll maybe show you a quick example of this later as well to make the test a little bit more strict and a little bit more expressive as well so again i'll give you maybe four or five minutes to take a quick look and then i will continue Someone mentioned maybe a solution for PyCharm. It apparently has problems discovering those imports, but running PyTest works. And the person says, if you run PyTest from inside PyCharm once, the red lines stop. So that seems to be... I don't use PyCharm, so maybe Maybe I've never seen that myself, but apparently that could be a good workaround. And someone asks, what's the goal of having a PyProject.toml file in a repo or for testing cases? I'm not quite sure if I understand the question, but PyProject.toml originally started as a file as a replacement for setup.py so to turn your project into a Python package which others then can pip install where you do things like specify your dependencies of your project and then more and more tools started supporting also configuring them via pyproject.toml so if you use rough as a formatter and linter, if you use rough, if you use mypy, if you use pytest, maybe some other tools, instead of needing to have five different config files, you can configure them all from this pyproject.toml. So I want to show you another example of why this match keyword can be useful. Let's say I write a parse positive number that takes some string and returns the number as an int. So it converts the string to an int, but then also checks whether the number is positive. I have a bit of lag here in this screen preview, so typing is a bit of a challenge. Maybe one little hint at this point. If you raise an exception, it always pays off to... If you raise an exception about some value, it always pays off to include the actual value in the exception message. Because then if you're debugging something and you get this exception, it makes your life much easier and now if I want to write tests for this there are actually two value errors that could happen here so I could call my function in two ways here I could call it well or in three ways I can of course call it with a valid number in a string. But I could also call it of course with a negative number and I would get my value error back. But let's not forget that I can also call it with a string that makes no sense and I get another value error back, but this time from Python when trying to convert that to an int. So when writing tests for this. I use ipytest here, which just runs this cell in the notebook via pytest. I could say if I call this with a number as a string, I get a number back. and then I could say I expect a value error if I call this with a negative number, but I also expect a value error when I call this with some other string that's not actually a valid int, and those tests all pause. But now if I had a typo here, this test would actually test something very different. And in fact, I could now remove the code that handles negative numbers, and this test would still pause. So for cases like this where I have a very generic exception, like value error here, this doesn't make sense for a zero-division error. There's only really one reason why a zero-division error could happen. And sometimes it's maybe better to start writing custom exception classes, which are a bit more fine-grained, so you can actually then test for those. But sometimes, from a code point of view, it's perfectly fine to just use value error. And I would say here it doesn't really make sense from an API perspective to have a negative numbers value error. But then we should make the tests a bit more strict and a bit more expressive. And that's where match comes into play. I can here now say match equals no negativity allowed, just it's a substring match with a regular expression pattern on the exception message and here I can say match equals invalid literal for int. I probably didn't get the casing quite right, that should be Uppercase here and then the tests pause but now if I had the same typo again the first test now fails and PyTest tells me you expected a different exception message so with that my tests are a bit more strict and I would say they are also way more self-documenting in a way so I can really see what exactly am i testing for not just a value error and same thing if I had maybe it's some kind of parser and I have a parse error then I don't want the missing comma parse error and unexpected end-of-file parse error and whatever but if I test it I I really, really want to test the message as well, so I can see when looking at my tests what kind of failure exactly is this test trying to provoke. Now next up I want to show you marks. in pytest on a basic level are something like tags or labels for your tests so you can use those marks to categorize tests even across multiple files so we could tag this test here with slow and web test and another one with just web test and then we can filter based on those markers when we run our tests but But marks are actually more than that, they also serve a second use case, namely they attach meta information to a test. So people sometimes wonder why is parameterize, for example, why is that a mark? That doesn't make sense with the mental model of those tags or labels I've just told you. But if you think of marks also as a tool to add information to a test, that makes more Because the parameterized mark itself doesn't do anything. It has no logic in it. It just attaches that data to The test case. And then pi test implements This plug-in hook i've just told you about earlier for a question. This pi test generate test hook. And it actually checks, is there a Marker there, a parameterized marker? and if so, it gets this data and actually calls a parameterize function internally. So that's why parameterize is a mark. And I won't show this here in the basic training, but you can also do that for your own fixtures. So you can have a custom marker to attach some data, maybe the name of config file or whatever else makes sense to a test. And then you can read that marker from inside your fixture and change its behavior Accordingly for this test case you will need to register markers in your pi test In it so you will need to list all of them in your config file and that is to Avoid typos in marker names so if you had an accidental second W here or something you would otherwise maybe never notice because if you filter by marker name this test would just not be run so it would be a silent failure and that's why you need to register them someone commented still with issues with our pen caulk I didn't know there was an I have no idea what that is. It's not my project, so you might not want to blindly install that. It certainly won't work if you do that. You also shouldn't have to clone anything else on GitHub. It should just work out with this repository. And from what I can gather, it did for almost all people. So I'm not quite sure what's wrong on your end. And probably we don't have the time to debug that, I'm afraid. Maybe whoever you are, maybe in the next exercise, check with your neighbors if they can help you get that sorted out. So there's another question. We now checked that the function works for one particular input. How do we know it works for any numbers? And that's actually the perfect point maybe to show you parameterize as a first step towards that go, which very easily with one of those mark decorators lets you parameterize a test case. So we can say I want to parameterize the arguments a, b and expected, so those are arguments here to my test function, and I want to run the same test case with different values. And now if one of them fails those are really independent test cases so that's the very big benefit of this over just using a for loop inside your test function that those are independent and if the second one here fails the rest will still run and you also nicely see here as the test name what the arguments actually are in your parametrize. If you just parametrize one single argument the syntax gets a bit easier you don't need to pass a list of tuples you can just pass a list of arguments there. Now the question was how can I check for any numbers and I would say you cannot check for any numbers because there are like infinite an infinite number of numbers but there's a thing called hypothesis which I would really recommend you take a look at if you are if you know like how pi test works how parameterization works hypothesis takes that to the next level and lets you say I just want some sort of integer or maybe I want the positive integer or I want the list with five integers and this in that range and then Then based on that, it generates random values, tries to make your test fail, and if it can do that, it minimizes those examples to give you an example that's as simple as possible. It's an amazing library. Unfortunately, I won't have the time to tell you more about that. Someone asks, when parameterized tests are independent, can they be run in parallel? So there is pytest-xdist, which lets you, as a plugin, parallelize tests across multiple CPU cores. And I believe that also works for a parametrize test. And do we need to also register marked parametrize in pytest-ini? You don't, because that's a built-in mark. So pytest knows that exists already and internally already registers that with itself, basically. I would actually have another exercise here on parametrizing, but I would say we skip over that. I think you see on this slide how it works. If you have never done it before, I would encourage you to really take the time to try this exercise maybe later, and not just copy-paste the example from the slides, but actually type it because sometimes people tend to struggle a bit with the nesting if you don't have like much Python experience that can be tricky sometimes to have like this decorator and then a list but tuples inside that list and such so if you've never done that before maybe give it a try but I would like to also tell you about two more marks built into pytest and that is skip if which skips the test based on a condition so we could say skip this test maybe if you are on Windows because it only makes sense on Linux and the result will then be skipped, as shown with this yellow S. Oops, sorry for that. There's also XFAIL as a bit of a more special case. XFAIL is used to mark your test as expected to fail and I usually use that if I get a bug report and then if I get a bug report maybe the first thing I will do is write a test for the bug because I'm going to try reproducing it a couple times anyway so I might as well make my life easier by writing that test first and then I see okay I cannot fix this bug right now maybe it's just not a priority or maybe it's a bug in Python or a bug in some sort of third-party library and I open a bug report there but now what do I do about my test if I cannot just have some sort of workaround I could delete it again I could skip it but there's actually value in continuing to running that test case but marking it as X fail because then I can say okay I know of this issue I know this test currently doesn't work Either way in the idle world it should and then I can configure pi tests with a thing called X fail strict where I can say if this test now starts passing that should be a failure. So I said I expected a failure with this test it's broken it started working so that test fails and my test run fails and only a couple of weeks ago I actually had a test do that because of a bug in CPython that I reported like five or ten years ago. Someone else reported it, it was clarified as a security issue and fixed even in all the patch versions and I would never have found out that, I long forgot about the issue if it wasn't for this X failing test. Someone asks if I can repeat the name of the package to parallelize the tests, that is PyTest x dist, so like this. Can you set up a skip-if decorator with a dependency to the failure of another test? not with like pytest built-in things but I bet someone has written a plug-in for it. How do you handle larger test data like data frames size kilobytes to megabytes? In the end you store them somewhere could be in a separate file could even be in some sort of database whatever for like more end-to-end tests and then you find a way to get that into your test case I mean in the end the the parameterize here is just a list of values and how you get this list of value it's up to you I mean if you for things that you can run and import time so if you just read something for a file I would say that's perfectly fine I mean Python reads lots of things from files if you import something, then you just call a function here that does whatever and returns this list. For more complex things this PyTestGenerateTests hook would come into play again, where you can do a custom mark, PyTestMark parameterize data frame from JSON or whatever and it would read that JSON file and internally call a parameterized function then with the result of that then another question the skip if test is passed even if not on Windows no if well yes this one would pass if not on Windows true because it doesn't actually test anything but the point was to show you how you would skip it on Windows. And of course I'm just ignoring that macOS and other operating systems exist, but to keep things simple there. And someone asks is it always recommended to write your tests first or sometimes implementation first, tests second? So personally for Box I try to write a test first for the reason I just just told you about it's just going to save me time for implementing it really depends if I have this scenario where I start on a blank state personally I cannot write tests first because I have no idea where I'm going I have no idea how my API will look like I need to get my hands dirty first I need to write a rough prototype to just get this mental model of where this thing is going and And then at some point, I start to write tests when I feel like things stabilize a bit. But as long as things are very fluid, personally, I would not start out with tests. But I think everyone is different there. I mean, there's also document-driven development, where you write your README first, write your user documentation first. There are different approaches, and I wouldn't say TDD is the one right approach. Whatever works for you. as long as you do write tests. Someone asks, when using PyTest raises, do you always retype the error message or do you gather the constants somewhere? Depends on the error message. I mean, in the end, same thing as with strings in a function outside of PyTest. If you use it once or twice, maybe it's easier to have it there. If you use it 10 or 20 times, Sometimes you probably want to have a constant somewhere. Any advice on writing tests for frontends like Streamlit? I barely know what Streamlit is, I'm afraid, and I really don't do any data science, so I cannot say much about that. Someone asks, they have emitters coming into an image processing function. Can I test for infs, nons, or non coming in? I mean, yes, you can just parameterize with those in a test, or if you use hypothesis, it's really optimized towards testing strange values. So in the newest version, if you tell it to give you strings, it, for example, has a thing called the big list of naughty strings, I think, where it tests all kind of weird Unicode things of weird values, and that can help a lot finding those corner cases. And someone asks, is there a clever way to reference or call your fixtures when parametrizing the tests? Yes, but I don't have the slide on that in that slide deck, unfortunately. So there's a thing called a request fixture that has a request.get fixture value. And that lets you, inside a test, dynamically request a fixture by name. that's something you could do. Okay, back to XFAIL and raises. People are sometimes a bit confused about the difference between the two. I would say the main difference is that PyTest raises tests for a bad case, but intended behavior. So if we call calc with an operator here that's not known, we expect the value error, while XFAIL is for unintended failures. So in an ideal world, this test case should work, but it doesn't. So before we get to fixtures, I will need to quickly tell you a bit more about the example code, because we will need something with a little bit more content than just this one calc function. And what I implemented is a thing called RPN, reverse polish notation, which is a way of using calculators, mostly known with old HP calculators. Like this one here, which is apparently HP's longest and best selling product across all their products, which I find quite impressive. And I'm doing this not because I'm a fan of old calculators, but because it just makes things much easier to implement. So to add two numbers with this RPN notation, the idea is that we first enter the numbers, then enter the operator. Every time we enter a number, it just gets saved on a stack. And if we use an operator, it takes two values from the stack, adds them, puts the result on the stack, which means we can then continue doing calculations with it. And this is much easier to implement because as soon as I get an operator, I already have all the information I need to do a calculation. So I implemented that in Python. You can run it, you can play around with it a bit. There is two special commands, there is p to print the stack and there is q to quit. And the implementation, with a couple of bugs, fits on one slide. With the bug fixes it doesn't, so I'm showing you the buggy version first. It's a class that has a stack. It handles those p and q inputs. And for everything else it calls evaluate. And evaluate checks whether a digit was input, if so it converts it to a float, appends it to the stack, or if plus minus times divided by its input, if so it gets two things from the stack, calls the calc function we have already seen, appends the result to the stack and prints it. And then we have this dunder name equals dunder main block, which checks has this script been imported or is it being run directly. For this special variable dunder name here is set to this special string if we are running this as a script. So if so, we call this run function. If not, if we are importing it, we don't do anything. So this is crucial for testability because then in our tests we can actually import this module without the run function being run, otherwise we would wait for user input already. Now how do we start writing tests for this? The first thing I would do is actually recycling the example I had on my slides. So this is kind of against what I've said earlier, that one test should only really test one thing. But I think there is a lot of value if I have some predefined example to have a test for it. So if I have some sort of specification, some sort of reference implementation, anything that gives me predefined examples, it really pays off to turn them into test cases. Now I can be sure that the example on my slides actually works in the same way I've shown it in my code. But of course if you want to write unit tests you would get a bit smaller and have a test that only really tests can I add numbers to the stack and can I do calculations. And that's where parameterize comes in again. I can now easily parametrize this with plus minus times divided by and then I only test those operations. So I prepare the stack from the outside and only evaluate the operator and then check the result. Now there's version two of the calculator that fixes a couple of bugs. Maybe some of you already saw or smelled some of those bugs. The isDigit check doesn't allow negative numbers. It doesn't allow floating-point inputs. Then I was lazy here. I did some substring matching. So plus minus was actually tried as a valid operator. And then there was no error handling at all. So if there was an invalid operator, nothing happened, which would be quite confusing. And if there was a zero division error, or if you entered an operator with no elements or one element on the stack, it just crashed with an unhandled exception. So in version two, I fixed all those. And I added also a new config object, because of course we need new features. We can't just have bug fixes. We need new bugs as well. So you can now pass the config object to it to customize this prompt, this greater than sign that is shown when waiting for input. And I support multiple inputs on one line, just separated by spaces. So let's take a look at those bug fixes. And again, here the first thing I would do for a bug fix is write a test or maybe extend the test. nice thing about parametrize if you already have a parametrize test sometimes a new test case just means adding one line so here i parametrize the test and say i want to test that i can actually input one in the half and that i can input minus one and that fails meter with rpm version one because its digit only allows for digits but it passes for both of those values with version two And the fix could look like this. Instead of using isDigit and checking beforehand, I just try to convert it to a float. And if I get a value error, I handle that. So as a kind of error handling philosophy, those are often called look before you leap versus it's easier to ask for forgiveness than for permission. And Rodrigo actually has a very nice blog post in his blog from Matt's PP. Here's a short link to it where you can see a bit more about this philosophy and the benefits and drawbacks. But here it seems like the nicer approach because it doesn't accidentally filter out things that would actually work. It doesn't filter out negative numbers and floating points. And also the other way around, because this actually lets any digit through, even in different writing systems, or even a superscript 2, for example, where float will then fail. And finally, for unknown operators, for things like at, or this plus minus thing, I have a test. And for now we don't know how do we test for this error message being printed. So this one passes, but at least the second one fails with version one and with version two they both pass. Because I stopped doing this lazy substring matching here and instead just use a list of strings. So I don't have too much time left but I can tell you about fixtures at least for a bit. So fixtures are a way where in PyTest you can extract setup and teardown code from your test cases. What we did is we added a config class to the RPM calculator which just holds a prompt, So this greater than character. And the config is passed into this R-PAN calculator from the outside when it is instantiated. But that means now we have two steps we need to do for every test. We always need to create a config and pass it to the R-PAN calculator. And here you could say that's simple enough, but when things get more and more complex, You don't want to copy-paste your setup code into every test function. So that's where a fixture comes in. A fixture is simply a Python function that takes care of setup and optionally of teardown and provides those kind of objects to the test. So a fixture function creates some sort of thing, returns it, it's registered with PyTest a special decorator and test functions or also other fixtures can then use them. And how that works is a little strange if you have never seen it before or if you are not used to it. This works by having a fixture called arpian, by having a function here named arpian decorated with PyTest fixture and then using it in the test function by having an argument with the same name as the fixture. So what PyTest does is it looks at your test function and anything that's not parametrized is referring to a fixture. And then if I have this argument rpn, it means PyTest looks at the fixture named rpn, calls this rpn function, and passes the return value to my test. Which means I can now put my setup steps into such a fixture function, and then just have an argument Rpn in every test where I need such an Rpn calculator instance. I would recommend you write a docstring for fixtures because this can get a little chaotic. That's maybe one thing I don't like about pytest. I mean fixtures are a great idea or a great concept really make things way more modular and reusable but them all being global in a way can get chaotic and one way to counter this a bit is to write a doc string which explains what this fixture is what it does what it represents and then you can call pytest with dash dash fixtures and it will show you a list of all the fixtures it knows about with name with where the fixture is defined and with this documentation here. So that can help to keep things nice and tidy. I would also recommend... Where did that go? Sorry, I skipped over a slide accidentally. I would also recommend writing type annotations for your fixtures, because again, things are quite implicit there, and while at least PyCharm and VS Code are clever enough to know about how PyTest fixtures work, so they know if you have an ARPAN argument in your test that it actually needs to look at the return value of this fixture function, but still I think here pays off to be explicit and have a type annotation so that for a human reader of your code it's a bit more clear what this RPN argument is especially if the fixture isn't defined right here in the same place and I say this makes things way more modular and that's also because you can use fixtures from fixtures in the exactly same way you just add it as an argument of your fixture function which makes things really nicely composable so we could even split up things further and have a separate fixture for a config object again debatable if that makes sense here but imagine things needing a bit more to maybe set up a config object and then we can have a test for this config object which isn't concerned with the RPN calculator, it just needs a config, but we can also use this existing config fixture in our RPN fixture. A couple of questions popped up. I think I will pick up most of them later. There was a question about PyTest fixture classes, which someone mentioned in a previous talk. I've talked to a speaker there. I would love to bring that into PyTest Core, the idea of decorating a class as a fixture, but I've never used it so far. I actually didn't know it was a thing until I talked to him, I think yesterday. Someone asked, can I replace a fixture by using a test class and define an attribute in the dunder init? You cannot, because PyTest will just warn and ignore any clause with the dunder init.py. Because this could work in the trivial case, but in a more complex case, PyTest would not know what the arguments are. So that won't work. the other questions I think oh five more questions appeared to pause results from one subtest to another we also use fixtures or do you have another approach I would say ideally tests should be independent from each other but if that's not a possibility that's possible to use a fixture to hold some sort of state between tests as well someone asks how to avoid recreating the config fixture and I think I will have a slide on that later so let me continue what we have seen so far is defining a fixture in the same file as the test and that's nice if we only need it in that one file but sometimes we want to share this fixture across multiple files multiple test files and that's exactly where this contest of PI file comes in you can move fixtures there and they will be automatically visible available for any tests in the same directory or any sub directories so as soon as you need a fixture in more than one test file you You move it there and often what you see is a ConfTestPy right at the root of your tests folder with like global things that you need in your entire application. But sometimes it also makes sense to do that in a subfolder. So if I have some sort of exporter helper fixture, I can define this here in this export folder in a ConfTestPy and it's then scoped to this folder. So I cannot use it outside in my testUtils.py. That would give me an error message. And that can be a nice way to keep things tidy again. There's a couple of more places where I can define fixtures. I can define them as methods in a test class and it will only be available in that class as we have just seen I can define it in a test module it will be available in that module in that Python file I can define fixtures in a contest PI file and they are automatically available in all my test files in the same directory so you don't need to import anything and I would say it's actually a bad idea to import things from contest of PI except maybe for type annotations. But as soon as you start importing fixtures from a ConfTestPy file, it actually results in PyTest seeing the same fixture twice and that's not something you would want. And finally, fixtures can be defined in a plugin or built into PyTest and as soon as that plugin is installed, those fixtures are available globally. Now there's a lot of stuff that PyTest exposes as such built-in fixtures and I won't have the time to go over those but you can for example access output that has been captured by PyTest so we could test for an actual error message being printed when we use the calculator in a wrong way. There is monkey patch to patch things temporarily for a test function. So we can say we want to maybe replace this value, replace this function by some sort of fake function. And there is a very nice fixture temp-pov, which provides me with a temporary Directory for a test case and temp off will actually keep around those temporary files for a bit so it will create a directory for every invocation of pi test and then keep the latest three around so I can still go in there and look at some sort of maybe generated input files maybe some sort of measurement data, log files, output, whatever I want to store if my test fails maybe. For graphical tests it really pays off to store screenshots there if tests fail. And then I can look at them again after the test failure. So, we have seen how PyTest gets those fixture functions or those fixtures value by calling a fixture function. Each of those fixtures has a name, the function name, and test values can use it by having an argument by the same name. Then we have also seen how this is quite modular, how fixtures can use fixtures themselves, and how we can define those in different places, for example in conf-test.py to share them. We will now take a look at a couple more intermediate fixture features, which hopefully answer almost all questions I've seen pop up, maybe not quite all of them. Namely, we can cache fixture values, we can reuse them across multiple tests, and a fixture can also do cleanup, so it can do something after a test by using the yield keyword. There's a lot more advanced concepts around fixtures. You can, for example, parameterize a fixture instead of parameterizing a test, but those I won't go into detail about. So as for caching, the fixture function can declare a caching scope. And often people are confused there a bit, and it's important to say that scope here has nothing to do with where the fixture is visible, where the fixture is available. So it's not scope in the sense of the scope of a variable, but it is the caching scope. So across how many tests the return value should be reused, instead of calling the fixture function again and again. So by default, if I don't have any scope argument here, it's the same as scope equals function, and my RPN fixture function will be called for every test. And normally that's what you want, you want tests to be as isolated as possible from each other and you want a new RPN calculator for every test. But we could change that to module and then all tests in this test module, all tests in this Python file would get the same value back. So the value gets cached and RPN only gets called once. And you normally would do that for performance reasons. So here I just have a sleep for two seconds and you don't want to wait those two seconds for every test that would maybe be too expensive but you really need to take care that your tests are still isolated nicely from each other so here the second test would actually fail because we reuse the RPN calculator but test A appends something to the stack, and test B then checks that the stack is empty. So that won't work out. And there's not really a one-size-fits-all solution. Maybe the return value should be immutable in some way, it should be impossible to change it, then it's no problem to cache it. Maybe you can copy it somehow, maybe using the deep copy module in the Python standard library. Maybe you can reset things somehow. Or maybe you can take some sort of snapshot. So if you have a database, some sort of test database, and generating data is very expensive, maybe you can open a transaction before every test, take a snapshot, you do your testing, and then after every test you do a rollback. So you just discard all changes that have remained since then. And one way to do that is by using a second fixture that you basically have a cached fixture as a kind of backend where you do caching. And then the fixture you actually use in your test can still do some sort of cleanup before or after every test. Let me take a quick look at the question. Someone asks, how does the metric with fixtures work so there's no need to import them? And if I could say how that's implemented. So basically you have an argument to your test function. And from a Python point of view that's just an argument. That name doesn't need to exist anywhere in this scope. And then the fixture decorator registers all fixtures by name. So imagine it like a dictionary, a global dictionary. And the decorator just stores the function with this name in this dictionary. Then PyTest sees your argument, RPN, config, whatever, and looks in that dictionary, finds the fixture function there, calls it, gives you the return value. That's a very simplified version of how that works. Someone asked what does yield do now in the body of a fixture. What it does is this. It's basically the same thing as return, as far as pytest is concerned. But it allows pytest to pause this function and do something after your test case. So instead of return client, I do yield client here. Everything before the yield happens before my test case. Then where the yield is, my test runs, and I get the yielded value as an argument, just as before just as if I would have done the return and after the test my fixture function gets resumed again and everything after the yield gets run as cleanup code so that's how I can kill some process clean something up in whatever way after my test case finally I want to leave you with a couple of tips and tricks around failing tests there are quite some nice command line arguments pi test has to influence its output you can for example tell it you want shorter tracebacks without printing all the source code which can be really nice to get a first like bird's-eye view if you have lots of failing tests there are various arguments to select tests so if we have a scenario like this with a passing failing passing failing passing test you can say I only want to run the tests that fail last time or I want to put those first or I want to put everything you haven't seen last time. If there's a new test 6, I want to run those first. Or there's even a thing called stepwise, where you can say run until the first failure, and then if I run pytest again and again, it just runs this one failing test until I fix it. Then it runs from there to the next failing test until I fix it, and so on. So that can save you quite some time for bigger test suits. If you are confused about how fixtures work under the hood, there's a very nice argument setup show, which will nicely show you step by step how fixtures are set up, which fixtures are actually used, even implicitly with a thing called auto use, which I didn't have the time to explain, and how they are torn down. So that's a very nice way to get some transparency there. Finally, one thing I want to say, there is a very big ecosystem around PyTest. There is over 1,600 plug-ins. Most of them are probably just people trying things out and such. But there is really a lot in there if you want to customize PyTest, change its behavior in some way. And it's very easy to write your own plug-in. You actually did already, if you ever use the ConfTestPy, because that's a PyTest plugin, and you can put plugin hooks in there to, for example, influence PyTest's output. So just put a function there, PyTestReportHeader, and you add some extra information PyTest should print when it's being run. That's like the hello world of PyTest plugins, those two lines. Finally, I want to recommend the book by Brian Ocken, Python Testing with PyTest. I was a technical reviewer and I read it like front to back and back to front again. It's an amazing book. I can really recommend it. I always ask the publisher, hey, could I have a discount code? So they gave me one. With this code here, you get 35% off the e-book until my 10th. And I would say like everyone who does something with PyTest should have one. I have a copy here if you want to take a look. Please leave the book here. Everything else is free to take. I have PyTest stickers. I have some pen if you want. Please help yourself. If you want to hear me talk for more than one and a half hours now, for whatever reason, I also have a PyTest tips and tricks talk I've done at previous conferences, which you you can also find on YouTube, where I go a bit more into detail about all those things I couldn't tell you about now, about the more intermediate to advanced things. I also have a talk about property-based testing with hypothesis, which again you can find recordings for. I will be back at Europython with a three-hour training, so a bit more time with a bit more exercises. And in early March next year, I will also do a three day training with Python Academy. So basically everything I told you here and lots and lots more. And of course, I'm also happy to do custom company trainings. Thank you very much for your attention. That's all I have. And I hope you learned a lot of new things and made testing easier and maybe more fun.