pytest tips and tricks for a better testsuite

Speaker 1 [00:07]

Okay. Hello, everyone, and welcome to this session. I'm Mashid. I'm the moderator today, and it's very nice to meet you guys. I give just one more minute to the people who want to sit. Please, yes, have a seat. Okay. Our speaker today is Freya Bruhin. She is a long-term contributor and maintainer of both the PyTest framework and various plug-ins. So please note that the talk is one hour and a half, so we do understand if someone wants to leave the talk earlier because of the other talks or you want to read something else. But just for not interrupting Freya, please use the left door. Thank you very much, and for sure quiet. There will be some exercises during the talks, and you can ask your questions about the exercise during the exercise time, but if you have any other questions not related to the exercise, please postpone it to the end of the session. My colleague and I will give you the micro, so please just raise your hand in case you have a question. And also you can write your questions in the panel, the talks panel. And so with this short introduction, I give the stage to Feria. Okay. Welcome.

Speaker 2 [02:30]

Hi, I'm Freja Bruhin, which might or might not be what you expected if you have seen any of my previous talks. Consider anything you know from previously deprecated API, and as you know, you should not be using deprecated API. I have a GitHub repository for this workshop, so if you haven't cloned that yet, that would be the first thing for you to do. There is a read me in there with some instructions, but it basically boils down to just installing PyTest, maybe installing a hypothesis, not quite sure yet if you will actually get around to that, and then you should be good to go. I started with Python actually as a funny story behind it. I started Python for an art project called die Schreimaschine, which only really works in German, unfortunately, where a friend and I, we modified an electric typewriter to automatically write out tweets from Twitter and some other, or like the website that used to be Twitter, and some other websites. So maybe at some point I should like clean the dust off this thing and try to get it to work again and then we could have it somewhere at PyCon Germany printing tools from Mastodon about PyCon Germany. So that's how I got into Python. Then I had this crazy idea 12 and a half years ago to start writing a web browser in Python which is a project that still very much keeps me busy today. It's called Qt Browser and it's focused on keyboard usage and on power users, so similar to things like Vimium, for example, but a standalone project based on Chromium, and with that project in 2015, I started discovering the hard way that I should really test my code, because users started finding bugs before I found them, and that's where I got into PyTest. There was a thing called the adopt PyTest moth, where maintainers of PyTest would help open source projects to switch to it, and at the time, PyTest wasn't quite as popular as it is today. Many projects were still using unit tests, and as part of that initiative, I switched to it. I started contributing to it to scratch my own itch, which I think is the best way to contribute to open source projects. something you yourself really want and you really need. And then I was asked, do you want to take care of the next release of PyTest? Yeah, sure, I can do that, but I don't even have the commit bit yet. I don't even have any, well, I'm not a maintainer, basically, and then I was, as those things sometimes go in open source. Also, at EuroPython 2015, I gave my first PyTest training, jumping in for someone, the founder of PyTest, who couldn't make it on quite short notice, so I did the training instead, and that's basically how it all started. Nowadays, since 2020, I teach Python to first semester students at a university of Applied Sciences. It's called Ost in German, or Ostschweizer Fachhochschule, for once the German term is shorter than the English one, in English it's the Eastern Switzerland University of Applied Sciences. And in 2020 I founded my own company, Bruhin Software, where I teach PyTest and Python to companies, so I pretty much made Python and open source into a full-time job. So, we will learn various tips and tricks all around PyTest. If you have been here maybe last year, then there I did a basic thing. So, more basic PyTest topics. Today we will have more advanced PyTest topics for people who use PyTest already. If there is anyone in the audience who will be at the master class tomorrow, then it's probably not worth it to attend this training, I'm afraid, because the master class is basically just a much longer version of this here, but it's a superset, so to speak. So you will see tips and tricks around marks and parameterisation, you will see tips and tricks and advanced features all around fixtures. We will look at various built-in fixtures that come with PyTest, and you will see tips and tricks around debugging failing tests. And then I have a couple of chapters where I have a much longer version, but I want to include at least a short preview around ecosystem, around PyTest, around hypothesis, the framework to generate test data, and around writing custom PyTest plugins. Before we start, I'm sure everyone knows by now what a schnitzel yoghurt is. It's the chase of the schnitzel. Do you know what a schnitzel coma is? It's a German word for the coma, so to speak, you have after eating lunch. So you're busy with digesting and falling half asleep. Maybe some of you are feeling that right now. I hope you got some coffee, and also I'm on the last day in the schnitzel coma slot, so I hope I won't fall asleep, because if I do, then you all will, but let's hope that doesn't happen. Finally, a few things I want to point out about my slides. If I show a code snippet, It has a file name header like this, and you will find the code in the code folder as part of the repository you cloned earlier. Sometimes in the corner here I have a short link. You can either just enter this link directly or you can scan the QR code and it will take you to the PyTest documentation about whatever topic the slide is about. And finally, when we have a couple of exercises, they have a tag like this, and you can search for this tag in square brackets, and it will take you to the right place where to solve this exercise. So could we do a quick raise of hands? Who of you has PyTest up and running? is almost all of you. Well, I won't actually ask who of you has used virtual length, that's more interesting when I do business trainings, but I hope all of you did. Nowadays, I think with PyCharm, with VS Code, with other environments, people often use virtual environments without even realising, which is great. Who of you cloned the repository with the example code? Okay. You are almost all good to go. And finally, who knows what an RPN calculator is? Reverse Polish notation. Okay. I only see 10, 15 hands, maybe. I will explain in a minute. It's just a special way of entering things into a calculator which will serve as an example for this training. So last chance to scan the link or go to the GitHub repository to get the example code, you will also find a PDF of the slides there, and we will use any supported Python version with the newest PyTest 9, but most of the slides will also apply to PyTest 7 and up. If you are not up and running yet, that would be the slide to look at. Open the code folder, for example, in PyCharm, for example, in VS Code. Make sure you install the requirements that come with it, and then you should be able to open a terminal and run, for example, PyTest version. We'll mostly be using PyTest as a command line tool. I think it's very useful in day-to-day development to use it with the nice and graphical integration into your IDE where you just need to click a play button, but various tips and tricks I will teach you today require that you also know how to use PyTests on the command line, for example to pass custom command line arguments to it. And I think it's always good to know how things are actually working under the hood. So personally I use both of those interfaces. So, we will start at the basics, but hopefully quite quickly get into more intermediate to advanced territory. Could you raise your hands if you have used PyTest Aprox before? Okay, that's maybe a quarter, maybe a bit less. Do you know what 0.2 plus 0.1 equals according to computers? That's almost nobody. Then who of you has used PyTest races before? Okay. That is three quarters, almost everyone. What about the match keyword for it? Love it. Nice. Okay. Those who don't, outside of those 10 or 20 people who raised their hands, you will after today, after the first exercise already, actually. And finally, who thinks PyTest is around five years old? Well, I kind of spoiled you when I said I got into PyTest in 2015. Oh, well. What about 10 years? 15? 20? Okay, most of you said 15. About five of you are right and say 20. It's 22ish or so, so it's a really, really old Python project. It actually originated from PyPy, the alternative Python interpreter, where they had quite a big focus on testing, and then started to factor various internal tools out into a helper package called pi, and hence also the name, originally pi.test, just a sub package of this pi package. And everything else in this pi package isn't really used anymore, but pi test, of course, as you all know, is going strong. So, let's start with pi test raises. We have this calc function here which takes two operands and takes an operator as a string, plus minus slash for divided by or times, and then returns the appropriate result. So we could write the test, it's kind of questionable if that's a good test because we're more testing Python behaviour rather than our code at this point, but we could write the test that says If we call this calc function with three, zero, and divide, then we must get a zero division error. Sometimes people ask what's the difference to a try-except. The difference is that the test will fail when this exception isn't raised. Now there is this nice little tool which is the match keyword, and I'll show an example of where this actually can be useful in a minute. But first we will actually already start with a little exercise. So with this match keyword here, you can pass a regular expression to pytest which gets matched on the exception text. Now here it's kind of questionable whether it's useful or not. But again, afterwards I will show you a more elaborate example. But we will start with the exercise. You write another test in test raises.py which makes sure that this value error here is raised if you pass in an invalid operator. Then you run your tests, and then you add this match keyword to additionally match on the exception message, and maybe play around a bit, maybe check what happens if you don't posture the right regular expression and then we will continue in five minutes or so.

Speaker 1 [15:58]

just a short hint for whom joined us recently if you have a question regarding the exercise you are welcome to have the mic but if your question is not related to the exercise please post money to the end of the session either by mic or writing in the panel Thank you. Thank you. Thank you. Thank you. One minute left for the exercise. Thank you. If there isn't any point or question, we will move on.

Speaker 2 [19:34]

All right, so after this, I hope you all know that you are actually properly set up as well, because if that worked, then you can be sure that all the rest works as well. So it's sometimes a bit questionable, of course, does it make sense to match on the exception message or not? For zero division error, of course, this doesn't make sense at all. There's only really one reason for zero division error, there's only really one reason why that would happen, and that's a division by zero. And sometimes if you end up matching a lot on exception messages, it's maybe better to change your code, and actually introduce your own exception classes, maybe even with a little hierarchy of exception types, and then you can just check for that. But sometimes you have code where, like, from the code's point of view, it's perfectly fine to just raise a value error, or maybe you have some sort of parser, and that raises a parse error, and it doesn't make sense to have a missing closing parenthesis parse error, unexpected end of input parse error. It's just a parse error, but you still want to match the exception message there. For one, to make your tests a little bit more self-describing, it's clear to see what the intention of a test case is, but also to make them more strict. Let me show you another example of that. So if, for example, we had a function parse positive integer that takes a string and returns an int, and not only does it convert the string to an integer, but it also raises a value error, and I always like to format the actual value into an exception I raise, makes debugging way easier. And now I can call this with a string like this, and I get the integer back. I can call it with a negative number, and I get the value error. But there is actually a second error condition there. I could call it with any other strings, and I also get a value error. So if I start writing tests for those negative cases, I might be tempted to just say test negative and say if I call parse positive integer with a negative number, I expect a value error to happen. And that test works if I actually run it, and I will use a special thing here called IPyTest which can run test cases right out of Jupyter. But now if I want to write a test for the other error condition of there being an invalid input, then for one that will look almost the same. So I can't really see any more easily what the intention of my test is. But also if I accidentally make a typo here, then my test isn't actually testing anymore what it's intended to test. Now I could remove the test checking for negative ints and my test still pauses. And that's why for cases like this I really like to say here I want something like no negativity allowed, and here I want something like invalid literal for int. As you can see, those are substring matches, and I think it's lowercase, not uppercase. As you can see, those are substring matches, and I use this r prefix to make this a raw string, which is useful because backslashers are literal. And if I don't use any, I think this is good practice for regular expressions. Now, if I want to go further than that, I can use the ask keyword to get an exception info object from PyTest. And with this exception info object, I can programmatically access various information about the exception that happened. So I can, for example, as access.value, and I get the actual exception value, and I can look at it as a string. Or if I have custom exceptions with custom attributes, like the parse error, which may be as a next token or expected token or something, I can do assertions on that. What's important is that you keep this assert outside of this with block, which is counterintuitive to what you might know with using with files, for example, where you would only use the file handle inside the with block. But the problem is here an exception happens, and then there's no way for Python or PyTest to run anything else inside this with block. So if you have this inside the with block, it would never get executed, which is kind of a foot gun, unfortunately. And finally, there is PyTest warns, works very similarly to PyTest raises, but checks for Python warnings, so things like a deprecation warning, for example. Then let's move to floating point numbers. So if I add 0.2 and 0.1 with my calculator code, you would expect that I get 0.3 back, but I get not quite that back, but a tiny bit more. Because floating point numbers have a limited precision, in Python there are eight bytes, and that means not every number is exactly representable, just like we cannot represent and one-third exactly as a decimal number. And this is not just a Python thing, that's a thing in almost every programming language, and someone actually did a funny thing with subdomains, so if you go to that number.com, you will find a website which explains what's going on and shows examples in various languages. So you would use something like math.isclose maybe or numpy.allclose if you are using numpy, and pytest has its own helper for it called pytest.approx where you can quite nicely express this value should be approximately 0.3, and it will nicely integrate with pytest, it will give you a nice output, and you can also override the tolerance. So you can say, for example, you don't you measure some sort of temperature, so I have various customers who test electronics with PyTest, I have a customer who tests washing machines with PyTest, so you can measure temperature and say this value should be 20 plus minus 2 degrees, for example. And the default is something that usually works well when comparing floating point numbers. Then let's look at the configuration of PyTest and look at a couple of tips and tricks there. Starting with how PyTest handles warnings. Personally I like to configure PyTest as strictly as possible. I think pretty much anything that is a warning should actually fail your tests. So if you run your tests on CI, for example, or you run your tests even just with your ID integration, you actually notice what's going on. And by default, that's not the case. If you use any kind of deprecated API from Python, for example, you get this deprecation warning, but it's just shown in the output. So if you're not closely watching the output, you will never notice. And if you just want to override this for one run, you can actually run PyTest with capital W error to turn all warnings into errors, and that will actually fail the test. But what I like to do is to actually persist this in PyTest's config file, so for example in PyTest.ini or PyProject.oml, and there you can pass a list of warning filters with the filter warnings option, and the nice thing is you can say all warnings should be errors but then exclude certain warnings. So what I do is I upgrade to a new Python version, for example, and usually I get some sort of deprecation warning of one of my dependencies, especially if I try this Python version very early with the first alpha or so. And then I open a block in the upstream repository and tell them, hey, you're using some sort of deprecated API, and then I add an ignore line like this to my config. So with that, I can say this particular warning message of this particular warning category in this particular module should be ignored and not shown at all, but all other warnings should still be errors. And I think this is the right approach to deal with warnings in PyTest. I actually have one more slide on strictness, but it looks like it didn't actually make it into the presentation. My bad. There is a strict mode, new in PyTest 9, where you just set strict equals true in your PyTest config, and that's also something I would really recommend. That turns on various strictness options, for example, it errors if you use an unknown config option, it errors when you use an unknown marker, and it errors when an X failing test, more about that in a minute, starts passing. recommend in older PyTest versions you can do the same thing but there are separate options and starting with PyTest 9 you can just set strict equals true. Now let's talk about marks a bit, which is I would say one of PyTest's central features. And let's start with a little show of hands again. So who of you has used PyTest mark skip before? I would expect that to be a lot of you, and indeed it's almost everyone. What about Xfail? Probably some less hands. That's about half maybe. What about parametrize? There we go, stretching for everyone, beautiful, against the schnitzel coma. What about PyTest Podom? That's only five or ten people, excellent, they will learn something new in just a minute. What about indirect parameterisation? That's two or three people, yeah, that's about what I expected. What about PyTest Mark as a global variable? It's also two or three. And who of you has used data classes or etters or Pydentic? Okay, everyone that doesn't have your hand up, look in the data classes, they're awesome. All right. So on a basic level, marks in PyTest are something like tags or labels. So I can use this mark decorator to mark those test cases, and then we can, for example, filter them and say I only want to run tests that are marked with web tests. But marks also serve a second use case. Marks also are used to attach meta information to a test case. And people sometimes ask me why is parameterise a mark? It has nothing to do with this idea of tags or labels. And that's exactly why. Because the parameterise mark itself doesn't really do anything. The parameterise mark just attaches this information to the test case, and then different logic in PyTest itself actually turns that into a parametrization. And same with skip or xfail, the mark just serves as a way to attach this data to the test. Now, before we look at that in more detail, let's look at a couple of tips and tricks all around markings. Starting with this here, you can apply a mark as decorators in Python in general, at least from a syntactic point of view, you can apply that to a class of tests instead of just single test cases. And essentially what happens is that this marker here then just gets applied to every test case in this class. If you want to go one step further, you might want to set a a mark for every test in a file, in a test module, and you can do that by setting a special global variable called PyTestMark. You set that variable with that name to either a single mark object or a list of mark object, and those marks will be automatically applied to every test in that same module. That means this also gives you an easy way to skip all tests in the file, so I just say you want to skip all tests in this file as work in progress for now. Maybe you want to push something, some big refactoring, you wrote some new tests, they are not ready yet, but you want to push and look at the CI output, so skip them, and then of course don't forget to remove that again before actually merging them. Or you can say if you have some sort of optional import, there is a helper called PyTest import import or skip, where you pass the name of a module, and if it could be imported, you get the module object back. If it couldn't be imported, then the test module is skipped. So we skip tests if it cannot run for some reason, if some sort of dependency is missing, And the test result will be skipped. And most of you said you have done that before. But there's another thing called XFAIL, which stands for expected to fail. And you mark a test as XFAIL if in an ideal world this test case would work, but something is broken right now. So maybe you get a bug report, and then one of the first things I like to do when I get a bug report is to write a test case for it, because I find test-driven development in general quite difficult. I need to get my hands dirty and write some code before I can write tests. But if I get a bug report, I'm going to try and reproduce this bug like five or ten times anyways while fixing it and making sure my fix works. And in the end, I want the regression test anyways. So I can make my life easier by just writing that test as the very first step. But then maybe I discover, okay, this is actually a bug in the Python standard library. Or this is actually a bug in an upstream project. Or maybe I see where the bug is but I cannot fix it right now for whatever reason. Maybe it's just not a priority or maybe I know there is going to be a big refactoring of this code anyways. Now, I could just throw this test away again, or I could just skip it, but when I mark it as X fail, as expected to fail, then the test case still gets run by PyTest. But the result is then X failed, lower case X, instead of failed. And by default, when it pauses again, that's an X pass, an unexpected pass with a capital X. And that's where one of those strictness options come in, which I would really, really recommend you set, and that's X fail strict equals true, or actually newer Python versions for consistency, strict X fail equals true. But I think both of them still work. And then a test that is expected to fail but now pauses counts as a failure. Sounds confusing but can be very, very useful if you have this thing especially where you figure out there is actually a bug in the Python standard library. So I had this with the IP address module. In my browser code, I had some tests for validating URLs with IPv6 addresses. And some invalid IPv6 addresses were actually allowed to, were actually validated despite them being invalid. I reported that to CPython probably like ten years ago. And I marked the test as X fail and I forgot about it. And recently it suddenly started failing after a Python patch release even. And I figured out someone else reported it, and it actually got treated as a security issue and fixed in the next patch release. And I would never have found out about this if it wasn't for my ex-failing test, because it's ten years ago, I won't remember. The only reason I can see why this isn't the default, other than historical reasons, is because sometimes people tend to use this for flaky tests, so tests that sometimes fail for reasons outside of your control, and there I think it's much better to use a plug-in like PyTest rerun failures. I will have a slide on that later as well. Now, parameterise. We can use PyTest mark parameterise to parameterise a test function to turn it into multiple independent tests that run after each other. Now a couple of tips and tricks all around parameterisation. Starting with this, you can stack parameterised decorators, and you will get all permutations of those values. What is difficult then of course is to also parameterise the expected value. So here I cheat a little and actually recompute it, which can be a bit dangerous because if things got more complex, then at some point I wouldn't need tests for my tests because maybe I have a bug in that logic if I have a bug in my code as well. But where I really like to use this is if some function has some sort of Boolean flag And I have a collection of tests which should run the same way, no matter what that Boolean flag is set to. So maybe I have some parsing code with a strict flag. But for the invalid cases, they will be invalid no matter if I run strict true or strict false. So I have a parameterise for my test cases, and then I have a second parameterising strict with true and false. And all my test cases will run twice, once with a strict flag, once without. Then we have seen how we can mark a class of tests or how we can mark an entire file. So we kind of zoomed out, and now we will zoom in again and see how can we mark one single test case in a parameterize. And we will start with a little helper clause called PyTest.param, or helper function, but I think internally it's a clause. They can pass those PyTest.param objects to parameterise instead of just tuples of values. Now this is the exact same thing, with the difference that we can now pass keyword arguments to PyTest.param. And there's only two of those, one of them being marks, where, again, we can pass a single mark or a list of marks to attach to this particular test case in a parametrize. We can say this particular one here should be X failed because I haven't implemented explanation yet, maybe. The other thing I can do with PyTest Param is I can override the names of the tests, the test IDs. So by default, PyTest generates those names by just concatenating the arguments with dashes, at least for simpler arguments like strings. But here with the minus, this can be a little confusing because we end up with a test named like this. And if you have tests where you parametrize something with long strings, long inputs to a parse function again maybe, then your test names get really, really unwieldy. And if you have something like a dictionary, then PyTest will not turn that into a string but just count up the argument name. So if a test fails, you need to kind of look, okay, which one was it again? And that's where ID can help you, so you can say ID equals add or ID equals sub to say this test should be named test call add and this should be named test call sub. You might have seen an alternative syntax for this, an older syntax where you can pause a list of IDs to parameterise instead. That still works, but you need to do this kind of mental mapping of the first ID here being the first test, the second one being the second, and that works if you have like five tests, but if you have 20 or 50, then this gets really unmaintainable. So I would recommend using pytestparm there. And finally, a word about data classes. A data class helps you if you have a class that primarily holds data, and it avoids having to write all these boilerplate code for things like having a nice Dunder wrapper, a nice debug representation, or being able to test things for equality. So instead of writing all this code, you just write those four lines. So you declare your clause in a declarative way, and you use this data clause decorator. And there's actually quite a nice tool from Trey Hunter, another Python educator, where you can paste the data class and it shows you the generated code behind it. Now why am I telling you this, if this is about pytest, I'm telling you this because data classes can sometimes come in useful for very complex parameterizations. So if we define a data clause for a test case, we can use named arguments, we can do things like default arguments, and we can say the first thing we pass into it here should be used as the string representation. And then in our parametrize, especially if we have, if we parametrize a function, or if we test a function with many conditionals, many booleans maybe, then we don't need this true, true, true, false, true, true, true, true, whatever in the parametrization, but we can rely on those default arguments instead. And by passing a function as ID's argument here, we can tell PyTest just runs string over every call case to get the test name, and then it results in the first thing here being the name of the test case. So we have default arguments, we have type safety, we have nicer autocompletion, we have more readability. The downside obviously is that we need to actually define this data class for every test case, so I wouldn't do this if you just have two or three arguments to parametrize, but if things get more complex, then I think this can be very useful. And finally, PyTest 9 added an alternative to parametrization called subtests. This used to be a plugin which you can install with older versions called pytest-subtests where you get a subtest fixture and then can use as a context manager subtests.test and during a test case basically say this assertion should be handled independently. So if this fails, don't fail the entire test case, but continue with the other subtests. And then in the output, pytest will show those as failed subtests, but will continue to run the entire test case, and then, of course, if subtest fails, the test case itself also fails because it contains failed subtests. I would prefer parameterisation to that because it happens at collection time. The tests are really independent, you can reference them from the command line, plugins can reference them, they are really just normal tests. While with subtests you have a little bit more flexibility, but that comes with a couple of drawbacks. So, for example, you cannot just refer to an individual subtest, and plugins or even some things like reporting inside PyTest need to do some, I wouldn't call it hackery, but still, it's not as nice, let's say. Now before we get to fixtures and all sorts of tips and tricks and advanced features around those, I want to tell you about why I have this here. So this is actually HP's longest and best-selling product across all of their products, which is crazy because I've never seen it before actually developing this training. It's a calculator from originally 1981, which is still in production today, and apparently the gold standard in the financial sector, which uses this thing called RPN, reverse polish notation, as all HP calculators used to do at some point. And what RPN means is that you first enter the operands when you want to calculate something, And then you enter the operator, which makes it way easier to implement, because you don't need to parse a mathematical expression. You just put everything on a stack that's entered, or in our case, a Python list. And then as soon as an operator is entered, you do your calculation and you store the result on the stack. And then you can continue doing calculations with that result to, for example, calculate a term like this. And as a little example, oh, wait, my output is frozen. Nobody is telling me. Sorry for that. So let's do that again. Two inputs are entered, two integers are entered, and they are stored in a stack. Operator is entered, calculation is done, and output is stored on the stack again. And then we can continue with that and can say we add five to the stack, and two times, for example, and the result is stored on the stack again. Now I implemented that in Python. You can run it. You get a prompt. You can enter operands, you can print the stack, and you can do those kind of calculations and finally quit it again with queue. If you want to play around with it, you can run Python with dash m to run a module as part of a Python package, and then RpnColc.rpnv2. There's also version 1 where the entire code fits on one slide, but it is way more buggy than version 2. So we just skip that one. Very quick run down through the code. On the right, you have a little mini map to see where we are. So Calculator gets a config, Calculator has an empty stack. And the config isn't really important. It just stores the prompt that should be output. Then we have a get inputs helper function that prompts for inputs. can enter multiple things separated by spaces and it will just call split to split those. And finally a run function that is kind of like our main function as an endless loop that goes over the inputs returned by this helper function and will then call evaluate on it, except for two special inputs, if it gets a Q it will return and if it gets a P it will print the stack. Before we get to the main part, before we get to this evaluate function, we have an error helper function that just prints an error to standard error. In evaluate, we convert something to a float, or try at least, add it to the stack if it is a float, otherwise we check is it one of the known operators, and if not, we print an error message and return. And if it is a known input, we make sure that we actually have two operands on the stack, otherwise we error out again, we get those operands, we pop them off the stack, we do our calculation with the calc function you have briefly seen before, we make sure we don't crash if the user divides by zero, and then we append the result to the stack and print it. And then we have this config class as well, and this config class basically just holds a prompt. It has a bit more. You can load the config from a file, you can save it to a file and such. We won't really look at those in detail today, though. Now with that out of the way, let's take a look at fixtures, and we will start with some more questions from me to you. Who of you has used PyTest fixtures before? That's all of you, more or less. Who has used a ConfTest file? That is almost all of you. What about using yield inside a fixture? Still almost all, maybe two-thirds of you. What about TempPath or MonkeyPatch? That is, again, two-thirds or so. What about scope or auto use in a fixture? We're down to maybe half or a bit more. What about parameterising a fixture? And we are down to ten hands or so. Excellent. So before we look into those things, let me show you two things I consider best practices for fixtures. As I would say in Python code in general, I would really recommend that you use type annotations. I think especially for fixtures, they can help because fixtures tend to be a little bit magic, a little bit intransparent, so having a type annotation here helps a bit with having some context. What is this RPN thing? They should probably use mypy or as you have seen in other talks at this conference, maybe nowadays something like tie to validate them. And write the doc string again I would say good practice for Python code in general. With pytest you have this nice little feature that you can call it with dash dash fixtures and it will show you all fixtures it knows about with the doc string printed again which can help to get a little overview of what's happening behind the scenes. Now, normally, if we have a fixture function, this RPN fixture function here gets called every time a test uses an argument with the same name, an argument RPN, which means those two tests are completely independent from each other, but it also means If we do some sort of expensive calculation here, then this will be done twice, so those two tests now take four seconds in total. We can specify a so-called caching scope here, and sometimes the name here trips people up. This has nothing to do with visibility. The fixture is still visible and usable based on where it's defined, but this is the caching scope. Function is the default, so this doesn't actually change anything. But we can now set this scope to module, which means all tests in the same test module will reuse the same fixture value. So this function will only be called once, and the return value will be reused and cached for all tests. There is also class, all tests in a class. There There is session, all tests in the entire test session. There is package, but that's kind of a special case. So I will not go into that in more detail. And the nice thing is the tests are now faster. They only take two seconds. The less nice thing is that the second test now fails. Because we lost the isolation between test cases. So if here we append something to the stack, in the second test here, it is still on the the stack. There are various ways to avoid this kind of thing. Maybe you can make the return value immutable somehow. Maybe you can copy it somehow. Or maybe you can reset the state between tests using a second fixture. And that's an approach I will show on the next slide. If it's something like a database, you could even take a snapshot before the run your test case and then roll back to that snapshot. So what I like to do if I want caching but if I also want some sort of cleanup between tests is that I use two fixtures. I use a fixture like RPN instance which does whatever work work is expensive and uses caching. And I don't use that fixture in a test directly. I use another fixture, RPN, let's say, where, to the extent that's possible, I try to clean up things between tests without actually having to redo the entire set-up. So here, for example, I clear the stack. Or for end-to-end tests where I actually run my browser project and send commands to it, I don't restart the browser every time because that would take like two seconds or so or maybe one second for every test case. But instead I send a couple of commands where I say close all windows and tabs, reset all settings to their defaults, clear some caches and such, and then we are good to go for the next test. And with that approach, we have the best of both worlds, we have caching, it only takes two seconds here, but we also have proper isolation between test cases. Now you have seen marks as decorators with this PyTest.mark namespace. There are also imperative versions of those. So there's also a thing called PyTest.skip or PyTest.xfail or PyTest.fail where we can abort a test from inside the test case or inside the fixture as well. So internally this raises some sort of special exception, so we can call PyTest.Skip in this fixture if, for example, a server is not reachable, and all tests using that fixture will automatically be skipped once it's used. Then as for yield, the nice thing about yield in PyTest is that you don't actually need to know what yield does in Python to use it in PyTest, and often I think it's easier if people don't actually know what yield does. Because it's in the context of PyTest, it's essentially like returning this thing from a fixture, but in a way that the fixture function resumes after the test, and everything after it is clean-up code. So everything before the yield is setup code, where we have the yield, the test runs, and it gets the client as an argument just like if we did return client, and everything after the yield is teardown code. a quick word about autouse. With autouse, we can tell a fixture that it should automatically be run for every test that sees it, basically. So it should run for every test for which it is visible. So if, let's say, we have this fixture here defined inside the class, and and we say auto use equals true, every test in this class will automatically run this fixture code which sets an environment variable, even if like here we don't actually have the fixture as an argument, then it gets implicitly used anyways. If we need the return value, if we need to stamp off that return here, then it follows that we also need the argument, otherwise there will be no way for PyTest to get this value inside your test. But this is often useful if there is some sort of patching that should be done for every test case. A less implicit version is to use this mark, PyTest use fixtures, where we can pass a marker to this decorator, and to this mark, sorry, we can pass the name of a fixture to this mark, there's the schnitzelkoma again, and the fixture gets run automatically for every test that is marked like this. And as we have seen before, we can apply this mark to a class or even to an entire test file. It's stuck again, yes, thank you. That was my bad, I think. Not sure how that happened, but probably the schnitzel coma, sorry, overused joke at this point. So, let's get into a bit more advanced fixture territory with the request fixture. Via the request fixture, a test can get info or a fixture can get information about which test requested it. So if my test uses a fixture, my fix, like here, that fixture can use a request and we get all kinds of information about this test case, so some sort of introspection in a way. And among other things, we can access the collection node, the kind of internal representation PyTest has of this test case, or we can access the config object PyTest has internally. And there's a couple of fun things you can do with that. You can, for example, get fixtures dynamically as well by a request with a thing called request.getFixtureValue, which lets you essentially parametrize tests with different fixtures which is a question that sometimes comes up, how do I do that kind of thing? And that's one approach. You instead parametrize it with the names of those fixtures and then you call request.getFixtureValue to get the value inside the test. One thing that can be really useful that it can do with this node object is that you can check the markers on a test case, and you can change the behaviour of your fixture based on those markers. So we are back to this idea of markers being a way of attaching meta information to a test. And we can call get closest marker on this node object and pass the name of a marker. And then if there is no such marker, we get none back. So this test here will get a config object with greater than as a prompt. But if this marker exists, then we okay, then we return a config with a longer prompt object. Now not only can we check of course if a marker exists or not, that would be kind of boring, but we can also access the arguments passed to it. by using marker.args, which is a tuple of the arguments passed here, and marker.keywordArguments, which is a dictionary of keyword arguments passed to it. So we can access this marker object, and we can say we want to create a server config and pass all arguments and keyword arguments that we got to the server config object. Now there would be another exercise here. Let me think for a second. Let's maybe do a quick exercise to have a chance to drink a sip of coffee. Again for maybe just five minutes or so, even if it's a bit more advanced. You have seen this config fixture from the last slide where we just checked does it have a long prompt marker or not. adjust that code so that instead we can pause an RPN prompt as an argument and pytest uses that accordingly.

Speaker 1 [70:13]

Okay, one more minute for the exercise. Thank you. Okay, so if there is any question about the exercise, we move on to the last 10 minutes of the talk, then 10 minutes questions.

Speaker 2 [71:27]

So I will skip ahead a bit in the slides. I have an entire chapter on built-in fixtures there, but most of you said you are already using something like TempHalf or MonkeyPatch, so I would rather tell you a couple of tips and tricks all around debugging failing tests, because I think there is a lot of useful features there, useful command line arguments to PyTest, which people often don't know about. Starting with various arguments to customize its output. So you can tell it, for example, that the tracebacks, which usually show you an excerpt of the source code, should be a bit shorter by passing tb equals short. So you get shorter outputs closer to what Python normally would show you instead of the whole thing with source code and such. Quite useful if you update the library, update your Python version, you have 50 failing tests and you want to get this bird's eye view before you dig into the issues in more detail. And if that's still too much, there's also TB equals line where it shows you only one line of output, only the exception message for every failure. Now in such a situation, you upgrade a library, you upgrade Python, you have dozens of new failing tests, there are also various arguments to select tests in different ways. So if we start with this scenario, where we have a passing, a failing, a passing, a failing and the passing test, we can, we run PyTest, and it actually remembers what happens last time, what happened last time it was run. So that's part of this dot PyTest cache directory, among other things. And we can say only run the tests that failed last time. Or run all of them, but put the ones that failed last time, put those first. Or first run any tests that you haven't seen at all last time that I maybe just implemented and then run the rest of the tests to it. We can say exit on the first failing test or on the nth failing test. And as a kind of more sophisticated extension of that, there is a thing called stepwise Where we can tell PyTest run until the first failing test. And then we work on fixing that. And we rerun PyTest again and again with the stepwise argument. And it only runs this one failing test until that starts passing. And then it runs from there until the next failing test. And repeats that again until that starts passing. And so on. Of course at the very end I would recommend run the entire test suite again, there is a chance that while fixing four you broke two again maybe, but it can make it so much easier to have this kind of more interactive loop while fixing your tests. Then as I mentioned before, people sometimes tell me that fixtures can be a little bit bit intransparent, a little bit implicit, and I tend to agree, but there's a very nice setup show arguments to pytest where we can tell it to show how fixtures are set up, used, and torn down. So you run pytest with setup show, and it shows us here with an F for function scope that an RPN fixture is set up, that this test empty stack is run, and then at the end things are cleaned up again. If you have an assertion and if you want to print some additional information after it, you can add that information to the assertion with a comma and whatever you put there will be printed in addition to the normal PyTest output. Of course, PyTest tries to give you quite nice, quite useful output already, but sometimes there is just some extra detail you want to output there. Sometimes your tests are too slow, well, they are always too slow, it would always be nice if they were faster. And PyTest gives you a dash dash durations argument, where we can tell PyTest, show me the 20 slowest things in my test suite, for example. And it shows you split into three phases, set up, call, and tear down, which tests were slowest. If you want more granularity than that, I would recommend that you use a profiler, Python for example, has cprofile built in, and I'll link the nice blog post on the topic, a profile that will tell you on a function by function basis where is the bottleneck as far as performance goes in your code. Sometimes tests hang, and PyTest has a thing built in called fault handler, which I can really, really recommend. You can set a fault handler timeout, and that's another one of those kind of options I would set everywhere. You set it to some conservative value, some conservative duration, so if your slowest test takes ten seconds, you maybe set it to 30 seconds or a minute or so. And then after that timeout, pytest will print a trace back. So you can find out where your test is hanging, which can be very useful, especially if it's hanging somewhere on a CI system. In PyTest 9, there is a new option to say you want to actually exit PyTest on a timeout, and there is a PyTest timeout plug-in which gives you a timeout that can fail individual tests instead. As I mentioned earlier when talking about Xfail, sometimes tests can be flaky, and the PyTest rerun failures plugin can help with that. You can mark tests as flaky, and if a flaky test fails, it will be run a second time, and if it pauses then, it's considered okay. That should be a last resort, of course, if you can fix the flakiness instead, but sometimes if things are out of your control, this is like saying just don't write any bugs, then you don't need tests, and the real world things aren't quite as easy as that. I've briefly mentioned a couple of plug-ins, and there's actually a giant ecosystem around pytest. So there is a plug-in index in the pytest documentation, there is almost 2,000 plug-ins published there at this point. It's an auto-generated list of anything on the Python package index that starts with pytest dash, so there is undoubtedly a lot of garbage in there as well. But if you want to change some aspect of pytest, maybe take a look at that list, do a search or two, and maybe someone else did that already. And it's trivial to write your own plugins. In fact, all of you, not all of you, almost all of you did it before without realising. You said you used the ConfTest.py file, and the ConfTest.py file is actually a PyTest plugin. So you can put PyTest plugin hooks in there and customise PyTest behaviour, but unfortunately I am running out of time, so I will not be able to go into that in more detail. I want to close with a book recommendation, Brian wrote an amazing book, Python testing with PyTest, I'm a technical reviewer of that book, so I read it front to back and back to front again, I can really, really recommend it. And finally, if you want to hear more about PyTest, more than just one and a half hours, and are not part of the master class tomorrow, I will have a three-day in-depth PyTest training in collaboration with Python Academy in March 2027 either in Leipzig on-site or remote, so it will run on a hybrid format. And that's all I have.

Speaker 1 [80:45]

Thank you. Thank you very much. So if someone is more interested or you need more time to talk, you can reach her afterwards in the break time, or there are some links and you can send an email or anything. We have to go to the question. Okay, the first one. How do you envision the future of PyTest working with the new agenting framework?

Speaker 2 [81:28]

I mean, there have been people asking, like, hey, PyTest people, could you write, like, an integration for the MCP thing, the model control protocol, I think. So far, none of the core developers is interested, quite the opposite. So we basically closed it and said, no, do it yourself as a kind of PyTest plugin. I don't think it has happened so far, but it's also a topic I'm not terribly interested in, so I cannot really say more about that.

Speaker 1 [82:02]

Can you share any tips on using PyTest with MPI? Do you have a favorite plugin for this?

Speaker 2 [82:12]

MPI as in multiprocessor infrastructure, I suppose.

Speaker 1 [82:18]

Question is from...

Speaker 2 [82:24]

Oh, okay, I was way off message, pausing intervention, no idea, I'm afraid, sorry.

Speaker 1 [82:31]

What if I don't want to match a substring but test for the exact string?

Speaker 2 [82:40]

I think that was probably in the context of PyTest.match, sorry, PyTest.races, and its match keyword. So there are two things you can do there, either because this is a regular expression, You can just use the character, the little hat, and the dollar at the end to say you only want this regular expression to match at the beginning and at the end of the string. Or you can use this exception info object, sorry, it's my scroll wheel and for some reason it's incredibly sensitive. Where was I? So you can use this exception info object and then look at that compared to string and just do a normal equals matching.

Speaker 1 [83:44]

Thank you. The next one. I also like raising warnings as errors, but it gets incredibly noisy with warnings coming from transitive dependencies. Is there a config mode to only raise warnings coming from your direct dependencies? Also, how do you maintain the ignore list so it doesn't become absolute with outdated ignore rules? Once the, okay, rules.

Speaker 2 [84:18]

So, personally, for me, it was okay to maintain, because even transitively, I maybe have 20 dependencies or so, and I try to always keep my dependencies up to date, and then hopefully other people solve that problem for me, as I think if that's not the case, there might be like a deeper rooted problem or just a very complex project. And yeah, I've had issues with outdated warning ignores in there. So what I try is to really scope it to a particular place. I don't ignore this application warning everywhere. I only ignore it in that project, in that dependency. And then it's not that bad if it's outdated. It's just, I mean, it's useless, and you shouldn't have useless code. But it also doesn't hurt anyone. And then usually what I do, I add a comment above each line with a link to the GitHub issue where I reported this upstream. And then from time to time, when I look at my PyTest config file, oh, I have like ten warning filters, I wonder which ones are needed anymore. I click on those issue URLs and then I see, okay, it's long fixed, I can delete the line. Or ideally, I mean, I'm subscribed to the issue I opened, ideally I would delete it as soon as I see the issue is closed, but often I don't remember at that point anymore.

Speaker 1 [85:50]

Okay. The next question has two parts. Stacked parameterization can result in combinational explosion, which add a lot of overhead to the collection phase. Any performance tips? The second part, are inner for loops with pi test subtest better?

Speaker 2 [86:16]

I mean, for one, it's always the question, like, you cannot test everything you would like to test. Because you're limited, for one, by CPU time, but also by human time. So I think testing is always a trade-off. What do you want to test? Where does it make sense to test? What kind of thing is easy to test? What kind of thing is brittle in a way? What's maybe mission-critical code? And there I would say if you're at the point with like a stacked parameterize, you end up with like a thousand of test cases, is this really a useful test in the end, is the question I would ask. I have read a blog post about subtests just yesterday where someone also measured performance. I think they said it was slightly faster, but I don't really remember the details anymore. So it might be worth a try, but I'm not sure if it will help.

Speaker 1 [87:17]

How to make pi test output more suitable for CI or CD pipelines

Speaker 2 [87:26]

I mean, the question would be, what does make it unsuitable in the eye of the person asking the question? One thing I like is forcing colors on, because PyTest is used to disable colors if the output is not terminal. But with a more modern PyTest version, at least for the usual CI systems, it enables the colors again on CI. and I think that helps a lot. Other than that, there is a lot going on in terms of custom reporting. So, for example, you can generate JUnit XML from PyTest, which is a standardised format for test results, and then there are plug-ins for GitLab, or I think in GitLab it's even built-in. There is a GitHub action thingy. There is a checking thingy which can nicely show this in the UI. Or there is, for example, PyTest HTML, which gives you a nice HTML report, and especially for more custom CI infrastructure, you could host that somewhere view it in a nicer way.

Speaker 1 [88:44]

the last one what are the advantages of fixtures over simple function calls inside of the tests

Speaker 2 [88:57]

The idea behind fixtures is kind of this inversion of control so that it's not your tests anymore that are responsible for creating and managing those objects, but it's PyTest. So this idea of dependency injection, essentially. For very trivial things, and I guess my examples here with the RPN calculator are quite trivial. It would be a one-liner to create RPN calculator. There, I agree, you don't really see the payoff yet. But as soon as things get more complex, as soon as you maybe have three fixtures building on top of each other, or as soon as you use caching, as soon as you use cleanup with yield, All that functionality you are missing out on if you don't use fixtures

Speaker 1 [89:55]

Okay, so we highly appreciate her again. Thank you very much for the interesting topic. Thank you. Okay. And again, thank you very much for your attendance, for your attention, and enjoy the rest of the conference.