pytest tips and tricks for a better testsuite

Speaker 1 [00:11]

Hi, so I'm Florian, that you already heard, or known around the internet as The Compiler. Before we start, I want to go over a quick introduction of myself and then also get a little show of hands about you afterwards with various topics. I started programming originally with QBasic and Bash when I was 12 or 13. Then did some electronics, and during my apprenticeship in electrical engineering, I wanted to get back to programming again. Looked at Python for a weekend, looked at Ruby for a weekend, because I couldn't quite decide. Now that I'm standing here, I guess you know where the choice went. And the reason I started with Python was a project called Schreimaschine, it doesn't quite work in English, unfortunately, where we modified an electric typewriter, a friend of mine and I, to automatically type out tweets from Twitter. And that's where I kind of was at the end of my knowledge with QBasic with Bash, so I looked at Python, was quite happy with it, and that's where I first used it. Over a decade ago, time flies, I started developing Qt Browser, which is a web browser for power users focused on keyboard usage, written in Python, based on Chromium as an engine, so same as Google Chrome. And that project is also how I got into testing, because the project exploded quite fast, and Nowadays I would say there are some thousands of people using it. And people tended to find bugs before I found them. And so I discovered, hey, there's this thing called unit testing, there's this thing called CI, and that it's really worth putting some time into that. 2015 then, there was this project called adopt PyTest month by PyTest maintainers where they they helped open source projects to switch to PyTest. I took part there with my QBrowser project. Ended up switching to PyTest, of course, and as those things go, in the same year, ended up as one of the PyTest maintainers. I started contributing. At some point I was asked to help with the release and was like, yeah, wait a second, I don't even have the commit rights. And then I had them. Also in the same year, there was EuroPython 2015, where Holger Krekel, the original founder of PyTest, was looking for someone to do the training there for PyTest because on quite short notice he couldn't come. So I was like, yeah, I'm using PyTest for a couple months already, but quite intensely. I never was at the EuroPython or at any sort of Python conference, Bilbao, Spain, sounded beautiful, yeah, let's go, let's do this. Worked out beautifully, and since then I'm doing PyTest and Python trainings at conferences and especially also for companies. Out of that, I founded my own company, one of my company, Bruhin Software, and nowadays I'm teaching Python to Bachelor of Science students in autumn, part-time, and the rest of my time I work on open source and giving company trainings. So I get some donations from cube browser users, I get some donations via PyTest, I do those trainings, and that's pretty much what I do. So before I have a couple of questions for you, I have lots and lots of materials for I have the, I would say, both pain and luxury of having three days of materials for PyTest trainings and needing to cut that down to one and a half hours. And sometimes it feels like that's actually more effort than building it up from the ground. So what I usually do, though, is I bring materials for more topics than I can probably fit in and then try to get an idea of what people are interested in And so I can kind of focus on those things. So I can talk about PyTest fixtures and various topics, various tips and tricks around those. I have a whole chapter on mocking, which we probably, though, won't be able to look at in full, just time-wise. I have various things about plug-ins, also writing your own PyTest plug-ins from scratch. And I have a little bit about property-based testing with hypothesis, where I think there's a full tutorial about, not sure on which day offhand, might be tomorrow. So there I really only will look at the basics, but even if you're only really using the basics of hypothesis, it's still an amazing tool. And it integrates very well with pytest, of course. So first of all, the most boring part, the setup. Just due to time constraints, I decided to turn pretty much all exercises into live demos. Feel free to follow along, of course, though I dare to say I'm quite experienced in live coding at this point, so it might be rather fast, unless I make some mistake. But still, feel free to follow along, just time-wise I'm not quite sure how much time I will have for one-on-one support in case there are any issues. But feel free to ask questions, even like in the middle of exercises and such, and I'll do my best to answer them if you're stuck somewhere. Or if you just have a general question as well. So let's start with a quick show of hands. Could you raise your hand if you have set up PyTest so far? Good. Could you raise your hand if you have used Virtualenv before? That's the nice thing about doing this at a Python conference, usually in companies people tend to be, yeah, what's a virtual length? Did you clone the repository with the example code already? Okay, that's about half of you maybe. I'll give you the link in just a second. Do you know what an RPN calculator is? The mathematicians. very few of you, that's something I will use as a small case study for examples, but I'll explain what it is and why I chose to go for that. Probably kind of self-explanatory, but we will use one of the supported Python versions and a more or less recent PyTest version. We will mostly use PyTest on the command line, which, again, in company trainings is sometimes something very new for people. But I think it's good to also see how it really works behind the scenes, and there are various command line arguments which can make your life easier there. You will find the example codes for the exercises on GitHub. You will also find a link to that repository in the talk description on pre-talks, so on the conference schedule. You will unfortunately not find the slides there yet. I will upload them as soon as I actually get around to cutting things down to whatever we ended up including as topics. So shortly after the training you should find a PDF there with the slides as well. So I'll give you another minute or so for the link while I actually search for something here. And then I already have a couple more questions for you. Have you used PyTest Aprox before? Do you know what 0.1 plus that plus that equals according to most computers? I mean, you don't need to know the exact value. Not certainly. Have you used PyTest races before? Good. You're about the audience I imagine for this talk, so far so good. Have you used the match keyword for it before? That's less of you. Good. Do you think PyTest is around five years old? What about 10? 15? 20? It actually turns 20 in a couple of months, I think in July or so. So PyTest has a surprisingly old history, and it actually was part of PyPy, where PyPy started in 2002, and they had a focus on testing already quite early. around 2004, actually at I think EuroPython 2004, Holger announced the library originally called STD with the idea of having kind of a complimentary standard library between PyPy and CPython and whatever wants to use it. And there already there was an STD.U test with plain assertions so that you can just use the Python assert keyword to write your tests. Then shortly after your Python, STD got renamed to Py, with the explanation that it was kind of a confusing name. The first thing I asked Holger, how is Py a less confusing name for a library? He couldn't answer it. But that's how the name PyTest came along. it was just the test sub package of that Py library. Then over the years, not too much happened in terms of this Py library. In 2009, lots of things you know nowadays, plug-ins, for example, fixtures, already had their like basic foundations at least down. So that's about those 15 years, I suppose. And nowadays, PyTest is pretty much an independent project of that PyLibrary. There is a problem that it exposes Py.Pov, the kind of poflip-like component in its API, so it couldn't get fully rid of it, but we ended up vendoring that part and kind of cutting ties to this PyLibrary. So you said you mostly know about PyTest races already, so there's not really much more I I want to say there, except show you the much argument. So I have a file here where I have a calc function, which takes an operator as a string, plus, minus, times, or slash for divided by. And if we call this with divide three by zero, we will get a zero division error by Python. We can write a test for that using pytest raises that catches it. We could also, for example, write another test now that catches this value error, invalid operator. But there is actually kind of a gotcha there when you test for exceptions that are kind of ambiguous. For zero division error, there is really only one reason why it could happen. You divide it by zero. But for a value error, there are lots and lots of different underlying causes it could have. And let me show you what I mean with the first live demo for today. So I use a Jupyter notebook here and use an integration called IPyTest, where I can just run some PyTest tests inside a Jupyter cell. At the end of the training, I'll also save that notebook file with all the live demos I did and also push that to the Git repository. So you will get a transcript of everything there. So suppose I have a function that gets a string and parses a number. But it also ensures that that number is always positive. So I could say I call int to turn it into a string. And then if it's negative, I raise a value error. And of course I need to check the number and not the string. Wait, no. I got very confusing names there for my argument. That's what I get for coding and talking at the same time. So that makes more sense. You would probably agree. Now I can call my parse positive number here. Let's maybe show that real quick. I can pass a number to it and it works. If I pass, let's first look at the negative number, I get my value error. No negativity allowed. But if I parse a letter into it, I also get a value error from Python. Because value error is a quite common exception used for various kinds of invalid values. And now if I write the test for it, and say... I would, of course, write the test for, like, the happy case and say if I parse a string with a number in it, I get the number back. But let's say I write a test with a negative number. And call it with something like this. And then just say that should raise a value error. That test will work. Hopefully it does. But if I now accidentally have a typo here, for example, my test will still work. Because that just happens to raise a value error as well. And it still works even if my code doesn't actually do what it was supposed to do. So those cases are, if I have any kind of exception that's not by the type of exception already very clear when it's raised, what I like to do is to say here match equals, and this is a regular expression pattern that matches against the exception message. So now when I do this, I get an assertion error that tells me my exception message wasn't actually what I expected. And here it would have prevented a broken test with this typo here. In other cases, it might just make my tests much more useful as documentation. If I have a parse error, for example, and a test for that parse error, then it might be useful to also have the message in my test and check for that, just so I see what kind of parse error is this test actually intended to test. If you want to have more sophisticated checks against the exception, you can also use as with your width statement, and you get an exception info object back, and that lets you access the value, the exception itself, or the entire trace back if you need it for some reason. So if you want a little more exact check, and we don't want any regular expressions involved, we could do this kind of thing. Finally, there is also pytest-warns, pretty much the same API as pytest-raises, but checks for warnings. I would also recommend using that for deprecation warnings. There is a pytest-deprecated call, but it's actually not quite clear from that API whether it's checking for deprecation warning or pending deprecation warning. I believe it's checking for both. So I like to be explicit here and just use PyTest warms instead. Maybe at some point I will see if we can deprecate the deprecated call one. Any questions so far? Why wouldn't I write my own exception instead of raising a value error? That's possible here. Sometimes it doesn't make sense to do that just for testability. So my other example was if you have a parse error, and there it doesn't make sense that you have a subclass like missing close parenthesis parse error, missing comma parse error, but you still want to check the message also for documentation purposes when you see your test cases. One, two, three. Yeah, it's working now. There are a couple of online questions. So how would you detect a drift in your data? How would I? Detect drift in your data. Drift in my data? Yeah. I'm not sure if I understand the question. It appears like five minutes ago. So I was not sure that you're taking, like, yeah, you parse the stream to int. But yeah, there is a question about drift in the data. I don't understand it neither. We'll have to skip it then, I'm afraid. If the asker of the question hears this, at the very end, you will have my email address. Feel free to send me an email, and we can take it from there. Yeah, and the rest we can keep in the end, I think, because one is related to how you found PyTest, how you can give some funding. And another one that I see here, it's are there any setting points for PyTest 8.x? If you can briefly mention why people should use latest and greatest. I would say there are lots of setting points from 6.4 to 7. Also, like, deprecating all the API and such. which I think 7 to 8 isn't quite as exciting, really. I think I looked through the changelog and I was like, yeah, okay, fine, for my own project in the end. So it deprecates some things, again, or removes some deprecated things, that's why it's a major release, but I don't think there is the shiny new feature. At least from memory, I would need to dig through the changelog again to confirm that. But also, like, second to last slide or so for today, there will be a PyTest development sprint soon in July. And then maybe there will be a very interesting PyTest 9 sooner rather than later. So back to floating point numbers. If you you add 0.2 and 0.1, then perhaps somewhat surprisingly, you're not going to get 0.3, because floating point numbers have a limit precision. So normally you wouldn't compare them with equals. There is this very nice website here with exactly this value.com, Which gives you examples of this problem in various programming languages. So this is not limited to Python. Python has math.isclose to deal with this. Which you can use with pytest. But I would recommend using pytest.approx. Which is a helper function which wraps the value, the expected value. So you say, my value should be approximately 0.3 by something like the, I don't know, probably a decimal digit or so, not sure what the default tolerance is, and then the test will pass. The nice thing about this APROX helper is that you get nicer output, so if it fails, PyTest will print the expected value with the tolerance and everything. And you can also easily override this tolerance. So I have customers that test electronics with PyTest and actually control a climate chamber with PyTest fixtures. And there, of course, the temperature of it, if you measure it, it's probably going to be off by, I don't know. I mean, I don't know how exact those are, but usually temperatures have a little bit of a tolerance. And there you can also nicely use pytest-approx by just setting the tolerance to, for example, 2 degrees here. Either absolute or relative. Next up, let's look at pytest-marks. And, again, let's do a little stretching exercise and have a couple of questions for you. Have you used PyTest mark skip before? What about X fail? Have you used parameterize before? Okay. Lots of hands there. Good. What about PyTest param? Have you set test IDs with that? Have you used indirect parameterization where things get really funky? That's about five of you left. And have you used the tightest mark global variable? Good. Again, you're the perfect audience for this, hopefully. I'll start at the basics just very briefly and then build things up from there. So as you seem to mostly know already, you can use pytest marks as kind of tags or labels for your tests. Use this mark decorator and you can filter tests by this mark. One reason to use marks also is to parameterize things where you easily can run your test function with different sets of data. Now you might have been wondering why is parameterize a mark? It doesn't really have anything to do with marks in the sense of labeling tests. And the reason for that is that the mark itself doesn't really do anything behind the scenes. It just stores this metadata attached to the test. And then pytest internally reads this metadata and changes its behavior. And as we will see in a minute, you can do that yourself with a custom mark as well. And customize a fixture's behavior. So kind of pass things into a fixture, basically. But for a single test. Now for skip and X fail, you use those marks to skip a test if it cannot run on a certain platform or maybe you have some sort of optional dependency, some connection to the database server for end-to-end tests, whatever. And if that's not available, you want to skip the corresponding tests. And that's done via mark, PyTest mark skip if you pass a condition into it and a reason which gets shown when you run PyTest with the verbose flag. The result of that test case then is skipped instead of passed or failed because it wasn't run at all. X fail is somewhat similar, but I would say it's actually a different purpose. Test fail is something you use when your test implementation is lacking currently. Sorry, not your test, but your actual code is lacking. So this test should work in an ideal world, but for some reason it doesn't. Maybe it's a bug that you know about but you can't fix right now because of priorities. Maybe it's a bug in an upstream library. Maybe it's a bug in Python itself. But by the point you discovered that, you hopefully wrote a test already, because the first thing you do is write the test if you get a bug report. And now you don't want to just throw it away. So you keep your failing test case, but you mark it with expected to fail. And the result will be that if your test fails, it's now going to turn into this little X in the outcome. So expected to fail. What can happen now is that your test pauses unexpectedly, and by default this gets turned into an unexpected pause, denoted by a capital X. As we will see on the next slide, there is a way to improve that situation. You normally wouldn't pause the condition here for X fail, but it makes a lot of sense to pass a reason, and then I actually really like to link to some sort of bug ticket, so that I have this additional context what's going on here, I know I have a ticket to track this issue, so why not just link it in a way that I can see it when I run the respective test. So as mentioned, if the test passes, this results in an X-pass. By default, an X pause, an unexpected pause, just counts as a pausing test. This can be changed, and that's something I would actually really recommend. It's probably the first thing I do in the PyTest config in every project. I said X fails strict equals true. Which means, if I say a test is expected to fail, but that test now pauses, that's now a failure. Sounds complicated, but the end result is that if you have an X fail test for some sort of bug, and you accidentally fix your bug, and believe it or not, that happens, let's say with a dependency upgrade or with a refactoring, And you can use this to let pytest alert you about it. Because then you have a failing test case, and you see, oh, wow, I can actually now go to this issue I linked in my reason string, and I can close it without any additional work. Some people don't like this behavior of xfail being strict, because they use it for flaky tests. So for tests that sometimes pass but sometimes fail. For like external circumstances. Now it's easy to say don't have flaky tests, that would be ideal but then we wouldn't need testing because we could just say don't write bugs. If you can avoid it, fix your flaky tests of course. But sometimes they're just there and are somewhere like in the bigger picture and you can't really do anything about it. For my browser, for example, I've had flaky tests because of an assertion failure in the libc library. So the C standard library, essentially. And what am I going to do about that other than run my test again and hope it pauses the second time? And for exactly that, there is a pytest rerun failures plugin. For those cases where you can't fix test flakiness, You can mark your test with py.mark flaky, and then it reruns, and if it pauses the second time, then it counts as paused. Also configurable and such, but that's essentially the behavior. So I would recommend use that, and use xfailstrict pretty much everywhere. Now back to parametrize, let me show you a couple of tips and tricks around that. You can stack the parametrize decorator, and what happens is that you get all permutations of the arguments you pass. That can be quite useful if you have some sort of strict flag, for example, or some sort of whatever flag you have in a function, but for this particular test or set of tests, you want to ensure they still work the exact same, no matter if strict is given or not. Or maybe you have some sort of subtle change in behavior and you can handle that inside your test function, so you can just say, run all my parameterized tests once with strict true and once with strict false by putting a second parametrize on it so regarding the stacking of the parametrize mark wait I forgot my question time okay so i what i use what i usually do is i open the repl i use iter tools.projects to write everything out because i didn't know you could do this is this better than it's just shorter yeah sorry my question makes no sense i'm glad to have helped yes there is another question online regarding to parameterize what is the preferred way to name parameters in a parameterized test? One string or a list of strings? I guess the question is about the first argument here, I'm guessing, where you can pass a comma-separated string or a list of strings. Personally, I mean, normally I would say just pass the actual data structure in it. It doesn't make sense to pass a string and then parse it or split it again. But here I got kind of used to just passing one string with commas in it, just because it's shorter and nicer and less characters. But I mean, matter of taste in the end, doesn't really make a big difference. One reason against the string is if it gets really long, then if you for some reason have like ten arguments you want to parametrize, then it might make more sense to pass it as a list because you can more easily split the list across multiple lines, maybe. That's the only reason I can think of. But you probably shouldn't have, like, 10 or 20 arguments to parametrize. Hi. I would usually pass, like, input and output and then check if I get the right output, the expected output. How would you do this when stacking and the

Speaker 2 [36:08]

commutations

Speaker 1 [36:10]

Right, here you can't really assign an output to it because you don't have the pair in that list yourself. So either you use it in a sense that the output is always the same or you don't care about the output. It's just a test where you'll see if it doesn't explode. Or as in here, though this is somewhat dangerous, you recompute the output in your test. So here I did A plus B. to check for. If it's less trivial than this, I actually wouldn't recommend doing that because you run into the danger of doing the same bug in your test than you did in your code. The tests shouldn't be all too clever. And then, yeah, you're back to a single parameterize where you list the combinations by hand. Sometimes that's just the better solution.

Speaker 2 [37:08]

We do have more for questions up to you to do now or later

Speaker 1 [37:12]

Let's do them now, sure.

Speaker 2 [37:14]

So what should I test if my service uses an external service should I mock the response should I also mock failing responses?

Speaker 1 [37:25]

I mean, it's difficult to have a one-size-fits-all answer, but if you have some sort of client for some API, then probably you want to know how that client behaves in failing situations, so you should absolutely test those as well. Especially because any sort of error handling code is probably what gets the least manual testing. If the happy path breaks, you're going to see that yourself while developing, or if not, your users are going to notice very quickly. But if error handling fails, and you maybe forget to print the error message, or you forgot the F for the format string or something, then for one, it's only going to happen very rarely. And if you're especially unlucky, then you don't have any information about what failed, but also you can't make it fail a second time. So I would say it's almost more important to test failing cases.

Speaker 2 [38:26]

What is the best starting point for companies to get involved in PyTest development and give it back to the community?

Speaker 1 [38:35]

I would say various things. I mean, you can look at the GitHub repository and see there are, I think, good first issue label issues, for example. You can come to the PyTest sprint. More about that at the end in the hope I will still have time. Bonus points for paying for people's travel to the PyTest sprint instead of PyTest paying for that. And in the end, I mean, we have a tidelift. We have an open collective for donations or for sponsorship because donations is something companies don't like, which will get used for things like the PyTest sprint. So in July, we are actually planning to pay for people's hotel and pay for people's travel to get together and contribute to PyTest. We've last done this in 2016, and it was great for the community behind it, it was great for the project, and it really makes a difference if you're working with people year after year online, and then you actually meet them in person. It's an amazing experience, and it also helps with collaboration when you know how people work. So that's something I would like to do more often. Now I'm organizing this one, and maybe if we have the money, we will organize one, or I will organize one next year as well.

Speaker 2 [40:05]

What are these interesting points, worth points, to check out in the newer major PyTest versions?

Speaker 1 [40:14]

There is the changelog, which is, well, not exactly short. I've actually had this question before where someone asked, is there something like a nice summary, like release notes, basically? For example, for Python, there is a nice page by Ned Batchelder, what's in which Python, where he lists the short breakdown of what does every Python version have. And I would like to actually introduce something like that for PyTest as well. Because right now the change log is your best bet.

Speaker 2 [40:47]

Maybe one more to let you continue, but there's a bunch regarding Why should someone use pi test dot mark dot X fail instead of pi test dot raises?

Speaker 1 [41:00]

I actually have a slide on that, but I removed it, so I can show it. PyTest raises is for expected behavior. Your exception is part of your API of your contract, basically. PyTest X fail is for unexpected fail. That's a little confusing. Expected failures is for unexpected failures, oh well. For cases where in an ideal world your test would pass but it doesn't because something is broken. And not for like testing error handling, for testing does my code raise this exception if I do this. So that's the big difference. So back to marks we go. Another quite basic one, or quite simple one, you can, if you have a clause, which you can use in pytest to just group related tests, you can put a mark from, you can move it from a method to just the clause itself, because Python decorators also work on clauses. And pytest basically applies that mark to every test method on that clause. So that can be a nice way to share a mark across multiple tests. If you want to go one step further and apply some sort of mark to an entire test file, maybe you're not using test classes and you want to apply something to every test function, there is a kind of magic global variable called PyTestMark. You set that variable to a single mark or a list of marks. Let's say I want to skip all the tests in this file on Windows because they are Linux-specific. And PyTest will apply that mark or that list of marks to every test in that file. Usually there is a helper which helps you with importing some module, but if there is an import error, it will skip this test file. So if you have some sort of optional dependency, maybe also like platform-specific dependency, you can use import or skip, pass the module name as a string to it. If it can be imported, you get the module object back, so this is basically the same as import P expect here. But if the import fails, PyTest will skip all tests in this test file. Now we zoomed out to like an entire file. Now let's go the other direction and zoom in to how do we mark a single parameter in a parameterize. To do that, you can use PyTest param. PyTestParum is a little wrapper you use for a single entry in your parametrize. And on its own, it does the exact same thing as passing a tuple there. It's just a little wrapper that PyTest then unwraps again. So this does the exact same thing as if we didn't have PyTestParum. But now, because this is a function call, we can use keyword arguments. And one of those keyword arguments is marks, where again we can pass a single mark or list of marks, and that mark is applied to only this test in a parameterize. So this is how you can mark a single test in there if you have multiple data sets like that, as skipped or as X fail, for example. Another thing you can do with PyTestParam is overriding IDs of tests. By default, those are auto-generated based on the values it parametrizes. But if, for example, we test the minus for our calculator, we end up with something here that's maybe a bit awkward. And with PyTest-param, we can pass ID equals add, ID equals sub to make that a little nicer. There is a different, older API to do the same thing, and that's passing a list of IDs to the parameterize decorator. Personally, I like PyTest Powerarm much more because the ID is close to the actual values. Here you have to kind of do the mental mapping of first element here is first element in the parametrize, second element here is the second element in the parametrize. But you might see it in existing test suits because PyTest Powerarm only was added maybe a couple years ago. One thing you can do is pass a callable to ID as well, and PyTest will call that to get your IDs from the values. So if you have lists, for example, in your values, you can use comma.join to quite nicely turn those into test IDs. One more question to you before the next slide. you used data classes before? Okay. That's almost all of you. Quick introduction before I tell you why I'm asking this. Data classes let you do the thing on the right instead of the thing on the left. So lots and lots of code turn into four lines of code. There It was a very short introduction. The even shorter one would be data classes are amazing. They help with, like, classes that are typically data holders and generate things like a proper EQ implementation, a proper debug representation, and so on. But why am I telling you this in a PyTest talk other than me just really liking data classes. I'm telling you that because they can be useful for complex parametrizations as well. If you have a test case where you want to parametrize things and you want some more readability and maybe even use default values, so you only want to override the operator for certain tests here, but not all of them. You start with defining a small throwaway data class, like for this test function. And then you parameterize tests with your data class, with instances of your data class instead. So you get default arguments. You get names of your arguments here. And you can say, for example, my first argument here should be the ID. And then I use this idea from before where I can just pass a callable or a lambda here to my ID's argument where I say just use the name attribute as the test ID. So with that, I have my three tests here with proper names, and I don't have to do this mental mapping of, okay, what does the first, second, third thing in my tuple mean? I wouldn't do this for, like, trivial parameterization because it's a little bit more code. You need to define an additional data clause every time. But for something more complex, I found this useful. So you get default arguments, you get type safety, you get autocompletion from your IDE, and things turn a little more readable. Now, any questions before I proceed? Yes. Feel free to just ask. I'll repeat it. The question was for this specific use case, wouldn't it be leaner to have a named tuple? You don't need things like EQ and wrapper and such. Good point. about it. No, it should work, should work, I agree. Let me take a note. If names tuples are on my next training iteration, it's your fault.

Speaker 2 [50:22]

There's also an online question, someone got a little lost, I didn't quite understand why would we use IDs with pytest.param inside pytest.mark, parameterize.

Speaker 1 [50:37]

The question, let me find the slide, somewhere around here. So the idea here was that you want to override your test IDs where every line here in your parameterize is one single test. But the IDs automatically get generated here based on one, two, plus three, so you end up with this name. that's not too bad, but 3, 1, minus 5 gets turned into this, which is kind of unreadable because PyTest just happens to use dash or minus as a separator here. And then if we pass IDs here, we can override the IDs for this particular set of parameters. And we need pytestparam here, because otherwise it's a tuple, and there isn't really a nice syntax otherwise, basically. So pytestparam is just here as a means to get this ID keyword argument.

Speaker 2 [51:50]

Do you find helpful to categorize tests into unit tests, feature tests in the PyTest context?

Speaker 1 [52:00]

What I do personally is I have in my tests folder, I have a unit and the end-to-end folder, where end-to-end tests are tests where I really run the browser application and I send commands to it, and I have a web server running on localhost. And unit tests are everything else. So there might be some integration tests there. I think it's difficult to have a clear line between them. But I think this really depends on a project-by-project basis. I mean, in the end, the question is, do you have a use case to only run unit tests and only run end-to-end tests? In my case, the answer is yes, because unit tests usually are much faster. And if I am, depending on operating system, my end-to-end tests actually end up with windows popping up, so I can't continue working while they are running. So sometimes I only want to run one or another. But if your test suite only takes a minute or so, then it probably doesn't make sense to separate the two.

Speaker 2 [53:09]

Regarding the external service testing, wouldn't that be very undeterministic? What if the external service is down?

Speaker 1 [53:20]

I agree. It, again, depends on your use case. I was actually quite surprised by the customers I see booking PyTest trainings, because I would say less than half of them are actually companies developing Python code. Most of them are using PyTest to test some sort of electronics. One customer tests refrigerators and washing machines with PyTest. And there, what they want to test is the washing machine. And they actually run PyTest on a Raspberry Pi and use image recognition to see what the display of the washing machine displays. It's quite fancy. Or if you write some code that integrates very tightly with some sort of server component, then you might get more value from testing the entire system as a whole including the server component rather than replacing it by a mock thing. So another customer for example they have an application to alarm emergency organizations and they wrote it like 20 years ago using Tcl, or Tcl, and now they're rewriting it all in Python. But the server component is still Tcl, and they do actually a whole lot of things server-side, and for now, because they're refactoring the entire thing, it probably would make more sense for, I mean, right now they barely have any tests at all, but it would make more sense for them to have end-to-end tests, because they're still in the middle of refactoring everything, and the server component is very custom and is an integral part of what they do. But if you're just talking to an HTTP server, then yes, by all means, mock it out if it's reasonable to do so.

Speaker 2 [55:28]

How would you bypass the cost issue of testing an LLM app? For example, code is calling an LLM, and if you test it with EverPR or code, change it is going to be expensive.

Speaker 1 [55:44]

I'll have to admit that the whole PyData or Python for Science thing is just not something I do much. So I would find it interesting, but I've never really found my ways there. So as soon as I get a question in that direction, I'll have to pause.

Speaker 2 [56:03]

Let's change it then. How would you set up a test for a function that needs lots of input data to run? Many parameters, many possible output options.

Speaker 1 [56:17]

Different approaches there, I mean, there are certain plugins that can help, like PyTest Cases, which lets you do more sophisticated parameterization, also based on files, if I'm not mistaken. You could use Hypothesis, maybe, to, like, force things, to generate things. There are others, NameFactory, the typical tools where you can give it a Django model and generate some data for me matching that Django model, but again, it really depends on the exact circumstances, I would say.

Speaker 2 [56:58]

Is anyone here that would like to ask a question?

Speaker 1 [57:06]

I have a question about how to structure our test. Like I'm referring here how we pack it. As a namespace packages or a regular package? I would add Dunder init files to your test as well. Because then PyTest can import them internally using like the full path. And you don't run into the problem that you can't have two files named test utils.py even if they are in different directories. But I think the PyTest documentation as a page, good integration practices, which explains this in more detail.

Speaker 2 [57:45]

Can parameterize be used with pydentic classes instead of at data classes as well?

Speaker 1 [57:54]

I've never tried it, but I don't see a reason why it wouldn't work. So, it should work. So, I would actually have a whole bunch of slides for the example project. Which is kind of context for the next examples. But I'm in the unfortunate situation that the context alone would probably use half of the remaining time or so. So we will mostly have to skip that. The gist of it is that I wrote an RPN calculator in Python as an example. And an RPN calculator is just a means on how to enter terms into a calculator, mostly known from old HP calculators, where you first enter the operands and then enter the operator. So you can enter one, two, plus, and you have a little stack, or in Python just a list of of values, and we get the result back, and then we can continue doing calculations with that. So this would be five times one plus two. You'll see the code there in the RPN calc directory, and the first version of it actually It has a couple funny bugs, but that's the part of the training here which is least related to PyTest other than just the idea of thinking what to test or thinking what kind of issues some code could have. But instead of that, I would rather tell you a bit more about fixtures. another stretching exercise. Have you used PyTest fixtures before? Good. Almost all of you. Have you used the ConfTest Py file? What about yield inside a fixture? Temp path for monkey patch? What about scope or auto use for a fixture? Now that's less of you. Let's Have you parametrized a fixture before instead of parametrizing a test? Okay, less and less people. Before I dive into those features I just mentioned here, let me show you a couple of best practices around fixtures. I would recommend using type annotations there, because fixtures being resolved on a kind of implicit basis, means that it's difficult to tell whether, what an argument exactly means in a more complex code base. And there I find it helpful to have type annotations, especially for fixture arguments. It used to be that VS code otherwise had no idea what this RPN could be inside your test function. PyCharm for a longer time has known about PyTest fixtures, so it knows if you have a test function with an RPN argument, it will actually need to look at the fixture function RPN to figure out the type of that. VS Code recently gained that ability as well, so nowadays it can, I think, give you auto-completion and everything in here, but I still think it pays off to be explicit and that type annotations. I would also recommend adding docstrings, because there are certain arguments for pytests, for example, dash dash fixtures, to show all defined fixtures, and there that docstring gets printed again. And again, in more complex test bases, I find that quite useful to still have an overview of what fixtures exist and what they actually do, what they are here for. Fixtures can have a caching scope where we tell PyTest to reuse the value returned from the fixture function for multiple test functions. So if we have an RPN fixture here that for some reason takes two seconds to initialize things, Here, we just sleep for two seconds. And then I have two tests using it. Every test will get its own object, so the fixture function will run twice here, which is great because you want tests to be isolated from each other if possible. But sometimes you want for some reason you want PyTest to reuse things, so you can add a scope argument here, still being set to the default function, and then set the scope to module instead, to reuse it for all tests in this module. Scope here is just the caching scope that's not related to, not directly at least related to scope in terms of where the fixture is visible. Just in which scope things are reused. Now, both tests reuse the same R-PAN calculator object. But we have a problem here. Does anyone spot it, actually? You changed the value in the test A. Right. So my test A changes the stack of my R-PAN calculator. But because the tests aren't nicely isolated anymore, this change kind of bleeds through to the second test. So yeah, our tests run faster, but they also don't run anymore, which is not terribly useful. What can we do about it? Sometimes it's nice to combine fixtures with different scopes. So here I have an RPN instance fixture, which is the thing that actually gets cached, my RPN calculator object. But I don't use this fixture directly from tests. What I use instead is this fixture here, where I prepare my instance to be used by the next test. Here, for example, by cleaning the stack, by clearing the stack. And then tests can use that instead. And now we kind of got the best of both worlds. Our tests are still fast. But we have an additional fixture that takes care of cleaning things up. So for my cute browser project and for the end-to-end tests I run there, I have a fixture like this where I actually launch the browser as a process. But I don't want to kill and relaunch it between every test, because that just takes a couple of seconds. So instead, the fixture I actually use takes care of, like, trying to reset things as much as possible. So it resets everything to the default config, it closes all open tabs, and tries to get it in a clean state without restarting it. And then I still use this module caching scope. So for every test file, I restart it just to make sure that in between things, I have a clearly defined state. And that gives me kind of a compromise between performance and isolation between tests. Thank you. You can run or any questions before I proceed? Thank you. Would it be possible to do a deep copy of the fixture at the beginning of the test instead? Wouldn't that be, like, more simple? Depends on what it is. I mean, if my fixture launches an external process, it doesn't make sense to make a deep copy of that or if your fixture represents some data in a database for an end-to-end test again you can't really copy that but then maybe you can use this kind of thing and use a transaction and then say you roll back the transaction after every test but sometimes yeah if you have some sort of data structure then the deep copy would work

Speaker 2 [67:30]

Any recommendations for handling schema change during the integration tests with pytest?

Speaker 1 [67:37]

Schema change you lost me there

Speaker 2 [67:40]

But if you don't know, me neither.

Speaker 1 [67:43]

I'm also in the Discord, by the way. Feel free to hit me with PyTest questions. I'll answer that until I get bored.

Speaker 2 [67:54]

And when would you consider it a misuse of fixture in pytest?

Speaker 1 [68:00]

A misuse of fixtures. I mean, fixtures are great. I wouldn't say misuse of fixtures. I mean, that's asking when would you consider it a misuse of functions in Python. But I would say the danger is more around scoping or in the sense of visibility of fixtures. So you can run into danger of having lots and lots of global fixtures with maybe confusing short names because PyTest, sadly, doesn't use imports to declare fixtures, but everything is kind of auto-discovered. That can turn confusing. So that's probably the area I would be most cautious about.

Speaker 2 [68:51]

And how is PyTest tested?

Speaker 1 [68:54]

With PyTest. There is a fixture called PyTester, which lets you run kind of integration runs, integration tests against PyTest. It's also great for testing PyTest plugins and lets you do things like create a Python file, run PyTest over it, and then look at the results and such. So that's what PyTest itself does a lot, but also it has just unit tests of its own code using PyTest.

Speaker 2 [69:22]

Would you personally recommend using PyTest for contract tests to test API?

Speaker 1 [69:32]

I don't really have much experience in Python with contracts in the sense of pre- and post-conditions and such, I suppose. So not speaking from actual experience, yes.

Speaker 2 [69:49]

Will tests using the RPN fixture be run after another if the scope is functioned, or how else is it guaranteed that the RCP instance is not accessed concurrently?

Speaker 1 [70:09]

So by default, pytest runs tests just in series, not parallelized. If you use something like pytest-xdist to make them parallel, then that's something you would need to take care of in a more manual sense.

Speaker 2 [70:28]

Do you use fixtures to load test data file in your test?

Speaker 1 [70:36]

Sometimes, yes. Sometimes just parametrize. But I also do things like I have a folder of HTML files and I turn every file in there into a test case. And then I have a comment in the HTML file with the expected value. So I don't need to... If I want to add a new test, I just add a new HTML file in there without having to change any Python code. Okay, so I did ask the question of the misuse of fissures. So say, for example, you just have a test case that tests for just one thing. You don't need to use a fissure in other places of your test files. For example, you want to test for division by zero, for example. Does it make sense to create a fissure for zero? Right, I see, I see. I would say it makes sense as soon as you use something more than once. And if those are, like, the same thing, I mean, zero, you could use zero as a value in different tests, but they don't mean the same thing. But probably kind of difficult to give a rule of thumb here. I would like to continue with a couple of more advanced fixture topics. Just to find a nice end at some point. And to have something more for people who feel like, okay, I've not learned that many new things so far. So there's a request fixture in pytest which lets you introspect the currently running test from a fixture. So you can access things like what's the name of the test that runs. That's what tempd or tempov uses to create a temporary file with the same name. But you can also access the internal test nodes, the internal representation PyTest has of this test. And that lets you do a couple of interesting stuff. One of those is dynamically requesting a fixture based on its name, which is one possible solution for a question people ask a lot. How do I use fixtures inside parametrize? And one approach is that you parametrize the name of your fixtures. So here I have a default config and a long config. With different prompt attribute sets here. And using the request fixture, I say for this config thing here, I want the fixture either default config or long config. But it turns out, when people ask, like, how can I use fixers in parametrize, they actually mean different things. And there's an issue in the PyTest tracker where you scroll through it and you think, like, okay, there's not too many comments, and then you see the thing from GitHub, like, 150 hidden comments, or collapsed comments. So there's a whole lot of discussion there, if that's something you would be interested in. Another thing people ask me is how do I pass parameters into a fixture but from a test? Or how do I customize the behavior of a fixture for a certain test? And again, there are different approaches there. One of them being that you can access a marker on your test from inside a fixture. So you call request.node, which is the underlying PyTest node, .getclosestMarker. You pass the name of a marker, and then if that marker exists on your test, you get the mark object back. So here we, as a first step, we just say if the marker exists, I want a different prompt in my config. And then in my test here with the marker, I end up with that. In the test using the same fixture without the marker, I end up with the default value. Now you've seen already that it's possible to pass arguments to a marker. You've seen that with parameterize or with skip if. And that kind of meta information is saved in the marker object. And you can access that from your fixture as well. So you can access marker.args and marker.kwargs. And get arguments and keyword arguments that have been passed to the marker object. And with that, you can pass those into some class you're returning, for example, as arguments. And then you can have a marker like this where you have your arguments and keyword arguments. And depending on what marker is on the test, your fixture will return something different. One thing I would really recommend here is passing those into some sort of clause or function. Because if you're just accessing marker.args subscript 0, you don't have any argument validation. So you could have additional arguments there which aren't processed, which are just silently ignored. And by passing everything into a class here, we make sure that we only have arguments here that are actually supported. If we don't return a class here, one thing I've done before is just define a dummy function with a single argument that returns that single argument. So just something like Like that. And then inside my fixture, I can get my marker. And then instead of just doing this, which I wouldn't recommend because it doesn't validate what is passed in. It doesn't validate the arguments match up. What I do instead is I call my dummy function like that. And I get argument validation for free from Python. Because now if my marker has additional arguments, I will get a type error from Python because Because this function here only accepts one argument. Any questions at that point? I have a question about the scope. If I return a context manager in my fixture, so I have a yield, would the scope and kind of exit method have any interaction that I should care about? What happens is kind of what you would probably expect. They're always symmetric to each other. So if you have a function scoped fixture, then setup and teardown will be around the function. But if you have a cluster module scoped fixture, that will just be at the very beginning and the very end. So it works out brilliantly. Okay, thanks. In my mind, you get a similar behavior by using fixtures as a factory for fixtures. That's kind of a pattern I use, but what is your opinion on that? I've done that before as well, returning a function from your fixture function. I've also done that you just return some sort of utility object, and then you can call methods on that. And at the end, I've used all of those for, like, different use cases. I don't think there's, like, one clearly better than the other. So one last quick topic before I wrap things up. Debugging failing tests. My main takeaway here is that pytest gives you lots and lots of command line arguments to deal with kind of looking into things in more detail or only running a subset of tests. And those can be quite useful, and I think that's something people miss when they only use pytest via some sort of IDE integration. IDE integrations can be great for like day-to-day usage, but if you want to really dig into things that can be helpful. If you get lots and lots of failures and you're kind of overwhelmed by all the outputs, you can tell PyTest to be a little less verbose when it comes to trace backs, for example, and not print source code snippets and such. Or you can tell it, for example, to exit instantly on the first failure. What's even nicer than that is the step-wise argument, which is more recent, where PyTest runs everything until the first test failure, and then if you call it again and again, it will always run that failing test, so it remembers what test failed last time. And as soon as you fix that one, you can run it again, and it runs from there until the next failing test. You fix that, you run again with stepwise, it continues from there until the next one. So that can result in quite nice workflow for dealing with things. If you're confused by what's happening behind the scenes as far as fixtures go, there is a collection of arguments for that. And one One I use quite often is a special setup show, where PyTest shows you that additional output related to fixtures. So it shows you, you have a function scoped fixture here, hence the F, called RPN, which gets set up. Then your test runs and uses this RPN fixture. And then this open fixture got torn down. And finally, there is dash dash durations, which can give you a great first idea of me only having five minutes left, of how long your tests take, where you can run PyTest And dash dash durations equals 20, for example, and it shows you the 20 slowest things in your test suite. So here we can see the example from earlier where I did a time dot sleep two is actually the slowest test here, and a test I use for my mocking chapter, which we will unfortunately have to skip for today, is the slowest after that, because there I actually access a real web service. I know it runs the tests, but then at the end prints the duration. So I have just one slide on mocking I want to mention. There are, like, different approaches there. And I think people tend to default on an approach that is too low level sometimes. So you can use monkey patch with, like, a handwritten stop clause. And you can use unit test mock to almost auto generate or have your magic mock object. But in certain cases, I think it really pays off to use premade mocking libraries. And I would really wish there would be more of those. So if you use requests, for example, then you should use responses to mock things. And responses gives you a nice high-level API where you can say something like responses.get and if there is a request to this URL, answer with this JSON snippet. That lets you mock your things on a much higher level than on a Python API level and will continue working if you use requests differently. If you start using request.session, for example. There are other tools like VCR.py in that space. Where you can run your tests against the real server, but say, please record all requests and responses. And it records that into a so-called cassette file. And then if you run it again, it just replays the response from that cassette file. Where you don't need to do anything, you just need to add a PyTest mark VCR to your test, run it, you will end up with a YAML file like this, where it records all requests and responses, And then you can just let it replay that. So I would recommend using those if available rather than hand rolling things. Just shout. I'll repeat it. I imagine I have a test setup where I have like a real running server that provides the real answers. So this would be, okay, I want to set up my tests one time using a real server because it's quite tedious to do all that interaction stuff. So I do it one time, record, and I have that, this YAML content now in a GIF repository or somewhere where I can just put that in my repository. does it then work out of the bottle or do I still need to provide the information that this was recorded using VCR? I think the question was, does it suffice to just store these cassettes, store these YAML files in my repository or do I need to do anything else? Nothing else really. You run it against a real server, you get those cassettes, you store them in your Git repository and then everyone can run the tests with those cassettes. And you only need the real server if you want to rerecord them because maybe the remote side, the API changed somehow. So I will need to wrap up things. One last thing I have to say, PyTest has 1,400-somethingish plug-ins. There is a list of them auto-generated in the PyTest documentation. And it really pays off to take a look there if you want to change some behavior. Because there is a chance someone wrote a plug-in for it already. Things like recording benchmarks here or things like HTML reports including screen shots, Which is, by the way, what Mozilla uses to test their websites with pytest. There is a lot already in that space. There's also a lot of plug-ins there which are uploaded by people as a test or like company internal or whatever. The list is just everything that starts with pytest dash on the Python package index. And it's easy to also write your own PyTest plug-in, especially if it's just for things like some custom outputs being reported. So that's just like three lines or so in a Conf Test Py, and you can add your custom version numbers in to the PyTest output. So one minute left, that's perfect. I have a book recommendation, Titan Testing with PyTest by Brian Ocken, and I asked nicely and got a discount code for the ebook variant, PyCon DE Berlin. If you use that, you get 35% off on the ebook until my 10th. I'm a technical reviewer of the book, and I ended up with a sentence I'm really proud of. It says if there are any mistakes left in the book, it's because I didn't listen to Florian everywhere I should have. Because I read it back and forth and wrote them like ten long e-mails with suggestions. I will have a full three-day training where I go over things like mocking, writing your own PyTest plugins and much, much more. Remotely in collaboration with Python Academy in June. And another one hybrids onsite in Leipzig and also remotely in next March 2025. And also in June, as mentioned, we will have a PyTest sprint at a customer of mine who offered to offer meeting rooms and food for everyone, and ice cream, I think. They have free ice cream for their employees, so I'm sure we can get some as well. In Vorarlberg, which is in Austria, but almost in Switzerland, so you can get there quite easily from, for example, Zurich Airport, but also Vienna or from, what was it, somewhere in southern Germany. Or, yeah, Munich as well, true. I offer custom company trainings and coaching all around Python, PyTest, and GUI programming with Qt or best practices around Python or, for example, Git. Remotely, on-site, whatever, just shoot me an email and we can talk about it. And that's all I have. Thank you very much for your attention.