pytest - simple, rapid and fun testing with Python (3 hours)

Speaker 1 [00:04]

Yeah, welcome everyone. It's nice to see so many people here. When I arrived this morning to pick up my badge, they already told me that the training was, well, not told me, but told others, the training was sold out and that, like, while I'm fetching my badge, like, five people asking for the PyTest training. So, yeah, I'm humble. So I'll start with just a couple of words about me, then ask you for a couple of show of hands some questions and then we will proceed to kind of setting up PyTest and then into the kind of interesting parts. So I started programming in 2006 when I was 13 in QBasic and Bash for whatever reason. It wasn't a good decision but yeah. Five years later I finally discovered Python when I worked on a kind of art project with a friend where we modified an electric typewriter to automatically write tweets from Twitter and such. So he did the electronics, I did the software, and that was something I really couldn't do in like Bash or QBasic, like talking to web APIs. So I tested Ruby for a weekend, tested Python for a weekend, and ended up with Python, which is still my favourite language today. In 2013, I started what's now my main project, Qt Browser, which is a web browser focused on keyboard usage and on power users. And there I also kind of started writing tests, I started discovering that my users were finding bugs before I was because maybe they were using it on MacOS or on Windows and I was using it on a quite modern Linux distribution, others might have been using them with older libraries and such. There were a lot of different configurations and I started realizing I really need to write tests and have some kind of automated system running those tests to manage this complexity. In 2015, the people behind PyTest had an event in April, I believe, called adopt PyTest month, where people behind PyTest helped projects to switch to it. So for the month of April, I think every project had one or two PyTest core developers assigned and they helped to switch things over. And that's where I finally switched to PyTest and as those things sometimes go, I started contributing and ended up as one of the maintainers of PyTest in the same year. I also started giving PyTest courses and trainings originally at EuroPython 2015 as well. There Holger Kraeckel who originally started PyTest and also gave those trainings back then, he had a slot for a PyTest training at EuroPython, I've never been at a conference before, I think at that point, and he couldn't make it and asked on the mailing list, hey, who is available to take this training and nobody answered so I thought I'm only using it since a couple of months but I'm quite familiar with it at this point and there are slides and everything, why not take a trip to Spain and do this? And it worked out brilliantly actually and that was kind of the start of me giving those trainings at a lot of Python conferences by now and in a lot of companies as well. 2020 I made all this a little bit more official when my studies were finished and founded a small one-man company called Bruhin Software. And nowadays 60% I'm working in this Bruhin Software company basically doing a lot of open source, so also donation funded on Kube browser on PyTest, a little bit of freelancing, consulting, coaching, and 30% I'm employed at the Eastern Switzerland University of Applied Sciences, which is quite a mouthful. They just call themselves Ost or East, which is very creative but surely a lot shorter. There I'm employed and teaching Python to first semester students. So that was all about me. Now let's do a quick show of hands so I can learn a little bit more about you. So who has used a test framework other than PyTest in Python before? So unit test knows, okay, that's I would say three quarters or so. Then who has used PyTest before? Okay, almost all of you. Nice. Who has used pi test fixtures before? Okay, about half or so or three quarters maybe. Who has used pi test, Python decorators? Okay, almost all. Then context managers like with open, I'm guessing, yeah, almost all. Who has used yield before? Okay, half or so. Who has used virtual lamps before. Half and some people laughing. Who has set up PyTest successfully as a kind of preparation of this training. Okay, that's most of you. So I guess we can skip over most of the setup slides and such and jump right in. For people who have used PyTest before, I will start at the basics but then hopefully go into more advanced topics quite soon. So we have this first part of the training here until 12.50 with the 20-minute delay. Then we will have lunch, announcements, keynote, coffee, and then the second part will start or at least is planned for quarter to three and will be another one and a half hours. So it's kind of two parts. Now it's maybe a little bit more basic. The afternoon definitely will get more advanced. If you don't plan to attend part two for whatever reason, maybe you have something else to see or maybe you didn't like it, that's okay, then please remember to return your wristbands to the station you got it or gift it to someone else because then maybe someone who is interested still can go pick it up there. So we will start with just a little bit of, let's say, the more theoretical side of testing. Why do we test? How do we call different tests? But I won't really go into much detail there. I think there is a workshop tomorrow, if I remember correctly, about test-driven development. I'm imagining you would learn more about this there. Then I will tell you a couple of words just why you might want to use PyTest, since most of you already used PyTest, I guess I don't need to tell you why. Then we will go into how to configure it, how to use marks for grouping tests, skipping tests, then look at parameterisation, how to run tests with different sets of data, and then finally dig into a rather big topic of PyTest fixtures, which is a very central concept in PyTest with a lot of useful functionality exposed as well and they are an important building block for your own test suits. Then in the second part we will talk about more advanced fixture features, have a couple of slides about migrating to PyTest, though from what I saw most of you already use PyTest so we might want to skip those. And then those bullet points here which are grayed out, so the last two and mocking, that's something I do have material for if interested, but I don't have it in this slide deck. What I do have is a short tour of useful PyTest plug-ins and then just a small introduction property-based testing using hypothesis and writing your own PyTest plugins. So let's start with a couple of words about testing. We usually write tests to, you could say, to raise our confidence that the code works so that we can do changes in the code without the fear of breaking stuff. tests can also be useful to document behaviour or specify behaviour and in the end you should end up with faster development cycles because you don't end up in this state where you are afraid of making changes to your code because if you do end up in that state, things are just going to get worse and worse. We usually talk about more or less three different sizes or types of tests. That's unit tests where we test a single unit, so usually a function or a clause. Then integration tests where we test the collaboration between different units, how they work together. And functional tests or end-to-end tests, system tests, depending on who you ask, those terms might not exactly mean the same, but I'm just throwing them into the same bin here, where you test that the full code works more similar to what a user actually would do. So if you maybe write a web application, you actually automate the browser to use your website. Or with a GUI application, you automate this GUI, or with a command line tool, you actually run the tool with some arguments and check its output. Now there is this kind of common wisdom of the testing pyramid, which says you should have a lot of unit tests, a couple of integration tests and only very few end-to-end tests. And I tend to not really agree with it. As usual, I tend to say, well, it really depends. If you use PyTest to maybe even test something which isn't written in Python, I've been at companies which use PyTest to test washing machines and refrigerators, then pretty much the only choice you have is end-to-end test. You're not going to write unit tests for it. And PyTest is a perfectly suitable tool for that. And the benefit of end-to-end tests also even for Python code is that they really test what the user actually does and how the user actually uses your application. If you write a lot of unit tests, then they might be very nice in theory, they might be small and fast and very focused, but they also might be very far from reality, and in the end, if you're unlucky, you're testing something in a kind of different environment that your real application really works in. So maybe your test works, but your application doesn't. The other end of the spectrum, the end-to-end tests they might be rather slow, they might be a bit brittle, so sometimes they tend to be flaky, so tests they fail even if your code is totally okay for reasons which might not be quite obvious. And if they fail, it's not immediately clear what the issue is. You need to kind of dig in, maybe read the logs, maybe try to reproduce by hand, while usually with unit tests, it's quite immediately obvious. So both are kind of a good thing to have. So what's the job of an automated test tool like PyTest in this whole picture? I would say, well, it obviously needs to help you verify that your code works, that's kind of what it's here for, but it also should be helpful when your tests fail, and if that's not the case, then you should take care of making it helpful, maybe by improving the test tool, by, for example, extending PyTest in some way, or by writing more or different tests. So if you have some kind of end-to-end tests which fail a lot, because maybe it's testing some very tricky code or whatever, then it might be a good idea to also write a couple of unit tests for the same code to have kind of more focused error output when they fail. So let's go on to PyTest. I have already shared some exercise sheets with you where you can find the exercises. We will also have the URL for this again in a second. We will use Python 3.7 or newer, everything older is dead by now and you're hopefully not using Python 2 anymore and also not using Python 3.6 anymore, though I should be careful because I only dropped Python 3.6 in Kube browser like a week ago despite it being end of life in December. And we are using PyTest 7.1, that's quite recent, depending on your environment you might get 7.0 or even 6.4 or so. That shouldn't be a problem. Most of the things we are looking at here won't change, but for the demos and such, I will be using PyTest 7. You can use whatever editor or IDE you like. Usually I would recommend PyCharm, community edition is totally fine, or VSCode, they both have a very good PyTest integration, but for the training here we will also explore PyTest on the command line to really see how it works and also play around with different options and such. Now, on this URL here, you will find the example code for the exercises. Now, it's kind of planned as a hands-on training, But also I'm usually giving this in company trainings with maybe 10 people or so. So obviously with over 100 people, I won't be able to help everyone with their problems as much as I'd like to. So we will see how this goes. I will give you some time to solve exercises. But also please help your neighbours, ask your neighbours for help if you need to. Don't be shy. So this kind of load is a little bit distributed. Now if you take this URL and replace zip with PDF, then you will also find the exercise street to follow along. Or in Discord, I've also posted the link if you're in the Discord channel. I think the first message in this room's channel. If you are on Windows, you might need to add the Python scripts directory to your path environment variable. At least if you want to just invoke PyTests directly from the command line. If you would rather not mess with those settings, you can usually with normal Python install run pi dash m pi test instead. So whenever I say pi test in my slides or in the exercises, you just run pi dash m pi test instead and it should just work out. Here is a couple of examples how you would add it, but you also have all this information copied in the exercise sheet. So I always recommend that people create a so-called virtual length, and I've already heard someone here in the audience before, yeah, I've set things up but I was totally confused by this virtual length thing. Let me still just say a couple of words because I think it's a very useful tool even outside of this training. So what a virtualenv is, or does, is just an isolated Python installation in a folder. So you create a virtualenv, usually in the project directory as a .vnv folder, and then whatever libraries, whatever dependencies you install in that virtualenv will be there and not mess with your system-wide Python install. And as soon as you work on more than one or two projects, that can be very useful to have because then you can just install something to play around a bit, like here maybe with the training, and don't need to install everything system-wide. Also on Linux and macOS, you can avoid running sudo pip install, or whatever the equivalent of macOS is, which sometimes can mess up your system install. So if you have, for example, Fedora I think has its package manager written in Python but usually in a stable or more or less stable distribution, ships an older version of libraries and then when you accidentally upgrade them with sort of pip install, you might end up messing up your package manager which isn't fun. So it's much easier really to use virtual arms. There are two steps to doing so. You create a virtual env, using, for example, Python-mv-env, which is built into Python since 3.4, and then specify a folder, .vn, for example. The dot is just to make it hidden on Linux. You might as well leave it off. If you are on Ubuntu, I think it prompts you to install Python 3 vn for something, so just do this with opt and then it should work. Then the second step, and some people find this a little bit magic, is activating the virtual length using this VNF bin activate with source. Now all activating actually does, that's really all the magic of virtual length is, is changing your path variable to point into the virtual length. So if you run PyTest, you don't run your system-wide PyTest anymore, or pip, you just run the pip or PyTest in this .vn bin directory. So if you prefer or if you just want to try this out, you can also just run .vn bin PyTest instead of activating anything, and you will be working within the virtualenv. So let me just show this. Here for the notes, by the way, I will be using a Jupyter notebook, so I'm just doing live demos in here, kind of like a terminal basically, and then at the end I can upload an HTML file with all the live demos I did in case you want to take another look at it. So here I have a virtualenv with pytest installed, and if I do, let's say, vnf bin pip, then I see it's using this pip in this virtualenv. So if I would activate it, that's all it does, it sets the path to this bin directory in the virtualenv. might as well just invoke things with their full path instead. Then finally, you should install PyTest and the dependencies. If you use this requirements.txt file, then you will get some more dependencies intended for the full version of this training, but I think you should also be able to follow almost everything along if you just install PyTest if you don't want to install the whole bunch. And finally you should be able to run PyTest-h, like help, to see that it works. So could you raise your hands again if you have PyTest working on your system? Could Would you raise your hand if you don't have PyTest working on your system yet? Okay, just one or two hands. I will proceed for now just so that we can go on to the more interesting parts and then in the first exercise I will go check if I can help you somehow. So why should you use PyTest? It discovers your code, your test automatically, which leads to test code without boilerplate. Boilerplate is what you call kind of all that code which you need to write, but doesn't really do anything. If you have written in Hello World in Java before and then in Hello World in Python, you probably got an idea what I mean. PyTest gives you useful information when a test fails, with coloring, syntax highlighting, and I will also show you a couple more demos in a second. It lets you parameterize tests, so lets you run the same tests across different data sets, something we will look at as well. It lets you modularize, set up, and tear down via fixtures, as I said, are a rather central concept in PyTest. And it's very customizable. So it has many options you can configure and hundreds, if not over a thousand by now, plugins. And last but not least, it's also the most popular test framework in Python by far. So JetBrains together with the Python Software Foundation does a yearly developer survey. The one in 2020 had around 28,000 people who participated. The one for 2021 isn't out yet. They said it's almost finished but not public yet. And there, around half of people said they're using PyTest. Also, 37% said they are not using any test framework at all, which is a bummer. I'm really hoping, of course, in 2021 that we are seeing this non-number go down and the PyTest number go up. But, yeah, we'll see. And finally before we start, I want to give you a very quick history lesson, because I kind of digged into this a bit and found it quite interesting. PyTest is actually a very old project for Python compared to other Python projects and it actually was born as a part of PyPy which is an alternative Python implementation instead of the CPython implementation people normally use. So PyPy was born in 2002 and very early on it had a focus on testing because they of course wanted to test that it really behaves like the normal standard, if you will, Python implementation. In 2004, they already started a test framework in there with plain assertions so they can only write assert A equals B, more about that later. This std library was kind of intended as a complementary standard library instead of or in addition to Python standard library. Then they somehow decided this wasn't a very good name and renamed std to pi or pi library. It's beyond me how this is a better name or less confusing name, but oh well. And the test framework was a sub package of this called test. So that's where the name pi.test came from then. Then not too much happened. There was a couple of releases in the next five years or so. In 2009 we already had kind of most of the fundamental features of PyTest in place. We had plugins, we had the concept of fixtures, even if it was named a little bit differently. So most of the concept in PyTest are some 12, 13 years old by now and still working out quite well. 2010 then PyTest split into an independent project of this pi library but still being called pi.test and in 2016 then efforts started to kind of deprecate the old pi.test name and just call it pi test instead. So you can still I think call pi.test on the command line, it will still work, but most people nowadays kind of dropped this dot. That's the logo, by the way, of PyLab, looking very 2004ish. So you run tests by running this PyTest command line tool, and other than the test files, it looks for two different files. That's ConfTest.py, project-specific code, things like fixtures, we'll look at that later, the PyTest.ini file for configuration, we will also get to that in a second. If you just run PyTest, it will use its autodiscovery, more about that in a second, or you can pass a test file or a folder of tests to it, as you would expect. By default, PyTest walks over your file tree and discovers any files starting with test underline something.py. There it discovers functions starting with the same test underline prefix and clauses starting with test. It also discovers clauses deriving from unitest test case so it can run existing unitest test suits. This kind of autodiscovery avoids boilerplate So it avoids that you need to kind of register your tests somewhere where then I'm sure you would write the test and forget to register it and it would get wrong. So that's why this kind of autodiscovery is there. Yes? Do the folders need to be packages and do they also need to start with test underscore? So the question was do the subfolders need to be packages and do they also need to start with test underscore? no to both. They can be packages, it changes things a little bit about how PyTest imports things and it is recommended to make them packages, so to create a double underscore init, .py in there, but more about the file layout in a minute. So here at the right you see an example how you would write a test with the usual unit test library coming with Python, there you would subclass unit test test case, implement your test method and then call some special methods here like self.assert equal. Now I'm trying to not hate against Java too much, but this looks very much like Java, and that's not by chance, it's actually inspired by the Java JUnit API, and there's a good reason for it. when you have the assert keyword in Python and the assertion fails, it will just throw an assertion error at you without any details. PyTest instead takes a different approach. It has plain test functions, it uses this assert keyword, but it does a lot of weird magic under the hood, but this magic works very well, usually, to give you information when the test fails. You can put your tests into classes. So here, I'm just doing a test utils class, for example, to kind of group related tests together. Those classes don't need to subclass anything. And usually it's just fine to have plain test functions. But maybe you have some related tests, maybe tests for one function and you want multiple tests for it, then you use a clause to group them. With unit tests, often clauses also define behaviour for which we use fixtures in PyTest, with things like the set up and teardown methods, but more about fixtures later. So, as you already saw on the previous slide, you can use the plain assert keyword instead of those assert true, equal, not equal, false methods. And you can even combine them. Basically any expression you could use with an if you can also use with an assert. Now if something goes wrong, then pytest will give you a lot of information about what is going on. Here this file doesn't have this test underline prefix, just because there are a lot of failures in it and I don't want to run it every time PyTest discovers the tests in here. But if I pass it to PyTest as an argument, then PyTest will happily do what I ask and run the tests in this file. And here we can see that pytest kind of specializes its output depending on the types of the arguments. So if you have two strings, it will give us a diff. If there's just one character difference or one or two, it will highlight the difference. If you have long strings, it will skip over the parts which are equal. things like lists or dictionaries, it will try to point out which item in the list or in the dictionary is different. Same for sets as well. Same for lists of different length. If you for example have a string and use not in, then it will highlight where this is contained if the assertion fails. So there are a lot of kind of small specializations of this output, also for things like data classes or named tuples, it will print the field names and such, to kind of help you diagnose the issue. And usually or hopefully you would be able to instantly tell what the problem is instead of having to fire up a debugger and start digging around. Now if there is an exception, for example, PyTest will also show us a traceback, of course, and also try to enrich this with additional information. So here we have a divide function, which just divides x by y, and a test for it with a divisor of zero, so this will raise a zero division error. And if I'm running PyTest again over this, then we can see that PyTest gives us the different parts of the stack trace, but also gives us a kind of a part of the source code there. If there were more frames in between then it wouldn't print the full source code there by default. But at least for the last frame where the exception actually happens it will show us the code which again should make it a little bit more easy to diagnose the issue. There are a lot of command line options for PyTest. I'm just showing a couple here. If you run PyTest dash H, then it will print you a full list. Many options are around kind of changing its output. So for example, we could use dash V to get more verbose output where it prints the test names, dash S to disable output capturing, you will learn about what it is in this exercise just in a second. Dash X to exit on the first failure, so if you have maybe 20 failing tests after a refactoring or so and you want to kind of take things a little bit slower and just start looking at one of them. You can filter tests using dash K or dash M by name or by markers. We will get to that in a minute. You can tell it to run the last failed tests again. So if some test fails and you are rerunning PyTest again with dash dash LF, it will only run the tests which failed last run. And to collect only to only show which tests were collected without actually running anything, can sometimes be useful to debug some issues where PyTest is not finding a test, for example. As for the traceback, we can kind of influence how that shows up. Again, when we have, like, dozens of failing tests, maybe we just want a small overview before we start digging in. We can use TB equals line and then it will print one line per failure which is quite nice to get a short overview. With TB equals short, we still get a trace back but it's a little bit closer to the normal Python output. We just see one line of the code instead of seeing entire functions. So that can be helpful if the output is a little bit overwhelming. Yeah? So the question was if there is an equivalent to assert raises and yes there is and I believe it will actually be the next exercise after this one. So just to get you started, write a quick test function, play around a little bit with those options and help your neighbor in case of any trouble. And then you should also take a look at PyTest's output capturing. So insert the print call in a passing and in a failing test and see what happens with it. could you raise your hand if this has worked for you okay that's most of you so let's go on Let me show a quick demo of this output capturing. So I'm using a kind of plug-in here for this stupid notebook so that I can just run PyTest from here directly to demonstrate some things. So if we have a test and if I print something there, now the output is a little bit shorter. That's just something this plug-in does. And what you can see there is that pytest doesn't print this hello world output. And that's deliberate because usually you aren't interested in the output of pausing tests. This kind of information is only really relevant if a test fails. So if we have just an assert false here, let's actually go for a passing and a failing test. and the failing test both printing something, and what we can see here is that we see this kind of captured standard out call. So if there is an output in a failing test, then PyTest will show it to us, but if you are just running the passing test, then we won't have any output. If you add dash S as an argument here, we disable this kind of capturing. So PyTest doesn't mess with output at all, and if you run PyTest normally, then you would have kind of the test output and PyTest output kind of intermixed, but sometimes it can be quite useful for debugging to just say, hey, PyTest, please, just leave my output alone. Now as for project structure, normally you would have a structure somewhat similar to this where you have your project folder and then depending on how you like to set things up but I would usually recommend the source layout where you have a source folder with your packages in it and then there you might have something like utils.py, maybe an export sub package with JSON export, YAML export and then you have your test folder which more or less mirrors the same structure here. So if you have a utilspy you would have a test utilspy, you would have the same export folder here, and basically for every Python module have a one-to-one test module. You don't need to do this, sometimes there is a good reason, maybe tests for a module get very long, maybe even for a utilspy and you want to split the tests but not your implementation or maybe for some reason your tests are very short and you would want to maybe have both export tests here for both export formats in one test file. Maybe because you can test both together or something. But usually as a kind of rule of thumb, I would have the same structure here. Now, pytest also has a couple of files for itself. Usually we would have multiple of them for things like project-specific fixtures or plug-in hooks. We will get to those later. And the pytest.ini normally in the root of your project for more declarative configurations such as key equals value stuff. You can also put that in some other files like in a PyProject Tomo or in a TOXINI if you don't like to have many files around. Personally I like to keep it in a PyTest.ini because then you have one configuration file per tool and not one giant file where you configure everything. But that's a matter of taste. This part here is based on the source layout. I found or am aware of two good blog posts about this. Packaging a Python library where I think this layout was originally proposed in 2000 something, a couple of years ago already. But it's also very good for applications and there is a more recent article where testing and packaging originally I think was kind of against the source layout but now it has an update which kind of admits that the source layout is a good idea and everyone uses it nowadays, many big projects use it nowadays, including PyTest itself by the way. So as I said, there is this INI file with kind of key false value configuration options. It needs to have this pytest header in square brackets because that's just the INI format. And then different options in there. Again pytest.h will print you a giant list of all available options. One very useful option is addOpt which lets you persist a command line option so it gets always used. So if you for example really like the verbose output of pytest then you can configure as this addOpt as a value for the addOpt key and pytest will act as if you would always have given this dash V option. Instead of making this an exercise because we are so many people, let me just demonstrate how this looks. So if I'm just running PyTest now as a sub-process with this exclamation mark, so it runs all my tests here, some of them are deliberately failing of course, then normally we would just see green dots for passing tests, F for test failures, E if there is an issue in a fixture, and X and S for things like skip and X failed just to get a kind of overview of what's happening but maybe we would like this verbose behavior here where pytest prints one line per test including the test name and if we say yeah I really like this output I always want it then I create a pytest.ini here this command just writes this content here I'm writing here into this file and say adopt equal dash V and now if I run pi test normally then it will still use this verbose output. So that's how I can turn command line options into kind of persistent options in this config file. Now, back to a question from somewhere around here. If there is an equivalent of assert races, yes, there is. So far we have not really used any PyTest API, we have only used its auto discovery and the assert keyword built into Python. But now we will need to start to import PyTest. So we can actually use the PyTest module. And there, there is a function normally used as context manager, so with this with keyword, called PyTest.races. Now normally you would use with with something like with open some file name as f, but Python kind of generalizes this concept, calls it the context manager, and any function or any object can expose itself as such a context manager and then define what should happen before this block of code in there and what should happen after it. And in the case of of pytest it uses this kind of mechanism so we can define a block here with pytest raises to say this code inside this block inside with pytest raises needs to raise this exception. In this case, we expect this to raise a zero division error. Now, if this exception doesn't get raised, then this would result in a failing test. And that's also the difference to a try, accept. You could say try your code, accept zero division error, pause, else fail my test somehow. That's basically what this helper does. There is also an additional keyword argument you can pause here called match where you can match text on the exception message. If you want to be a little bit more exact, maybe whatever you're testing just raises a value error, but there are lots of reasons a value error could happen. So you could use match to match on the message. And this here is a RegEx pattern, so you could use things like .star, for example, as a wildcard. Let's do another exercise here. And as usual I will walk around and help people. Let's go on there so I can still cover the last kind of basic topic and we can get into more advanced topics in the afternoon. So let me just make a quick demo here again and also answer a question I've heard a couple of times now. So we have the divide function here again, which just divides A by B, and a test function for it where we say we expect a zero division error to be raised when we divide one by zero. And that works. Now, this match argument, actually, this wasn't the best example because there is really one reason why a zero division error could be raised. Maybe we have something more generic, we have a value error, or maybe we are writing some kind of parser which has a parse error and doesn't really have different subtypes of exceptions and we want to check why exactly the parsing failed, not just that it was a parse error of some sort. So if we had, say, for whatever reason this divide thing takes two strings and converts them to an int instead, and then we use divide like this, now this would raise a value error, but maybe we want to be more exact in what we expect here. What I like to do is just say match equals blah blah because then I see the test failure and see here rig X pattern blah blah does not match invalid literal for int with base 10. So maybe here I just want to copy this and say I expect invalid literal for int to be in this exception message. And then my test is a little bit more descriptive because there could be many other ways this test fails within the value error. Maybe not here but in a real world thing. So for the last topic I'm guessing before the lunch break, let's take a quick look at marks. Marks let us mark, as the name says, sorry, let us mark functions and clauses. So here, for example, we use this PyTest mark decorator with PyTest.mark.slow and .webtest. We can use whatever names we want here. It's kind of a namespace with everything available, so we can just invent some tags or labels here. And then we have another test with just webtest. As you can see here, if we want multiple marks, we just stack those decorators. And this would look like this. We have to test slow API with two labels, test API with one label, and test fast with no labels or no marks. Additionally, we need to register the marks in the pytest config file where we say markers equals and then every line kind of describes a mark with the name and optionally a description as well, which then gets shown if we ask pytest about the available marks. This mechanism is here to avoid typos. Otherwise, if I accidentally had a typo here with slow, then the test would have a different marker. And if I want to run slow tests, then this test would never run and I would never discover. So that's why there is this registration mechanism where we have to have a list of all markers which are available here in the PyTest. Otherwise, you would get a warning by pytest telling you this mark isn't registered, maybe it's a typo. Then when running pytest, we can use dash m and give an expression to filter by those marks. So say, only run the slow test or with a Python-like expression, not slow, web test or slow. We can also use and of course, We can use parentheses, not just a kind of logical Python-like expression. Those marks are kind of intended at a very basic level to group tests across your test suite. Maybe all GUI tests or all tests somehow testing whatever or all slow tests, things like that. So a kind of organisation which doesn't fit into the file structure, I would say. Those marks can also be used to store additional information belonging to a test. And that's also what pytest does with its built-in skip and xfail marks, which you see quite a lot in pytest test suits. Skip is kind of self-explanatory, I'm guessing. You would skip a test if it can't run at all, usually on a certain platform. Maybe you write a test for Linux which doesn't make any sense on Windows, so you want to skip it there. Or maybe you have some kind of optional dependency in your project, and if it's not installed, you just want to skip the affected tests instead of making them fail. Or for end-to-end tests, maybe you have some tests accessing a real test database server for whatever reason. And if you are in the train to work and want to work there, you just want those tests to be skipped if the database server isn't reachable. Xfail is a little bit different. You would use Xfail if the implementation is currently lacking. So if your test should work, but it does not. I use this a lot when I discover a bug and I start to write a test for it so I don't have to always reproduce things by hand while fixing the bug. And then for some reason I discover that I can't fix this bug right now. Maybe it's just too complex and I have other priorities or maybe it's a bug in some kind of upstream library but for some reason I can't do anything about it. Now why should I throw this test away if I've already written it? So I instead add it to my project with an X fail decorator, or with an X fail mark, and this tells PyTest that I expect the test to fail. And when it then starts pausing later, maybe if I updated some dependency causing the bug to be fixed, then PyTest can alert me about that. By failing the test, for example. So I expect a failure, but it pauses, so that turns into a test failure. that can be quite cool to find out about bugs being fixed. Or sometimes even if I did a refactoring maybe and the refactoring fixed some bug without me noticing. There are two different ways to do this kind of skipping and X failing. One of them is declarative using decorators where we say PyTest mark skip if, then we we have some kind of condition as the first argument, here for example if we are in Windows, and then a reason as a keyword argument that's just being printed by PyTest when we run it in verbose mode. Similarly for xfail, we could pass a condition but normally we don't because if it's a bug, normally it's probably there everywhere, and we also pass a reason of some As you can see here, we can apply those marks on both functions but also on test classes and it will just apply on every test in this class. On the other hand, there is imperative skipping where we call pytest.skip or pytest.xfail inside the test function as a function call. That's normally done if you have some kind of more Elaborate logic, maybe we want to try to reach a server and if it's not reachable we want to skip the test So that's not something we could Do in a good way as part of this condition in the marker in here. So that's where we would use this imperative way Yep Yes, at import time, basically, yes. Exactly. But the kind of drawback is that PyTest, of course, only knows that your test will be skipped after it already started running it. Exactly. So this is kind of a little bit more lightweight and better from PyTest's point of view because it doesn't need to run your test at all. But sometimes it is necessary to do things this way. I will skip over this exercise for just a second and get back to it because in ten minutes we will go for a lunch break. So let me just show you the final topic I want to cover right now. And then we will go back to the exercises and you can go for a lunch when you are Fed up with the exercises and would rather have food instead The final built-in way pi test uses marks in a very powerful manner is parametrizing so this is always a little bit of cause for discussions because English actually has four valid spellings of parametrize, two British and two US spellings. PyTest uses this one, parametrize, not parametrize, and then of course Z or S, British or American or British. PyTest uses this spelling, personally I don't mind because I've never used the word parametrize outside of PyTest, so that's what I got used to, but there is a plug-in, and I'm not saying you should use it, called PyTest Para WTF, which lets you use all four spellings. But, yeah, back to topic. What Parameterise does is it runs your tests with different sets of data. So we can add an argument to the test here, for example, num, then instead of having a a hard-coded value here, we use the argument, and then we add this PyTestMarkedParametrized decorator. As a first argument, we have a string saying what should be parametrized. Note if there are multiple arguments, this is just one string here, not multiple arguments in the decorator. And then we pass a list of values. So here, test is even, would basically would basically turn into three tests. One with two, one with four, and one with seven, which would then fail. If we want to parameterize multiple arguments, we instead pass a list of tuples. So we say, if you put in one, then we should get out one. If you put in two, three, three, four. And there, of course, again, the first test would fail here. here. Now, you might be asking why shouldn't I just write a for loop inside my test function? Basically do the same thing. This will help you easily see which test is failing. With a for loop, you wouldn't see what the actual data is coming from the for loop. And those three tests will really be independent. So if the first one fails here, then the others will still run normally. So let's move back to the exercises for the last ten minutes and then after lunch break we will look at fixtures, a little bit more advanced and very central concept in PyTest. Again as a reminder, if you don't plan to come back after lunch, then I won't be mad at you but I will be if you don't return your wristband because then you can give someone else the chance to go. So first for markers, write a test which is skipped declaratively using a marker, write an X fail mark test to see the difference, maybe also make it fail, make it pass, play around a little bit and run with dash V in verbose mode to see the reasons why those tests are skipped or X failed. If you have some more time and are bored and don't want to go for lunch yet because there will probably be a queue anyways right now, you could also try using PyTest skip, PyTest X fail in the imperative way, so inside a test function as a function call. And for parameterise, find the test, a function to test which uses arguments, you could use the divide example again, which we've used before, write the test for it, just testing a single value, and then parameterize it for multiple inputs and expected outputs. If you don't see me here on the screen, you're probably not missing anything, it's just that I'm walking around during the exercises helping the people here in the room with any problems they might have. But as long as I'm talking, I will be here in the front and you should be able to see me on camera. So we stopped here at fixtures in the morning and I told In the morning we had the more basic features of PyTest and now we are going a bit more into depth. So a fixture is a concept in PyTest which isolates set up and tear down from the tests themselves. So a fixture is responsible for setting up some kind of object you want to test to provide your test code with those kind of base objects and to avoid repetitive test code. If you maybe have a config class or whatever and you want to write tests for it and then you need to set up some basic config class with a couple of settings and such, then instead of having to copy-paste this kind of setup step into every test, you would move it into a fixture and your tests would then use this fixture. And this is realised via dependency injection, so we have a fixture function, a fixture function creates and returns a fixture value and test functions can request and use a fixture. So what do I mean when I say dependency injection on a very basic level outside of PyTest even? If you have a mail sender object, and that mail sender object needs an SMTP connection, SMTP is just a protocol usually used to send mails, then it could create a server object here in its constructor. So the mail sender object is responsible for creating its server connection, its dependency. On the other hand, with dependency injection, it would get the mail server as an argument here and just save it, and now the mail sender isn't responsible anymore for creating its dependencies. Instead, the caller is responsible for creating an SMTP object, creating the server connection here and passing this in to the mail sender. This is important for two reasons. For one, that's how PyTest fixtures work. On the other hand, that's unfortunately not something I will show you today, but it can be a way to avoid having to patch objects. If you now want to write a test for this mail sender and just testing the kind of sending logic without actually sending out emails, then one approach would be to create a fake SMTP clause which kind of fakes a connection to a mail server and acts accordingly. And here on the left we need to use monkey patch or mock.patch to actually patch this clause while here on the right we could just pass in a fake SMTP object here. So it makes the code a little bit cleaner often and more testable. But back to fixtures. And now it gets a little bit strange. This is something which, at least for me, it took some while to get used to and then at some point you understand how it works and then it should be easy. So a fixture is a function with a given name. Here we have a very stupid example, just an answer fixture returning 42. Now if you have a test and want to use this fixture, we request it by the test having an argument of the same name as the fixture. So if you have a test with an answer fixture and a fixture function, so an answer function decorated with pytest.fixture, then what will happen is that pytest will call the test and pass the return value of this fixture function into the test. So inside this test, the answer will be 42. You could imagine it like this, that Python kind of looks at the arguments of your test function and then fills the answer argument as a keyword argument with the return value of the answer function. So what you get inside the test is not the function itself, it's the return value of it. And we will already move on to a small exercise about fixtures. You will find this code here in the fixtures subfolder. Now run PyTest on this file with a setup show argument. That's an argument which tells PyTest to show you how fixtures are set up and turned down. But for now you can ignore this tear down part, we will get to that later. Then write and use another fixture function in the same test or you could also write another test using it just to get some experience, especially if you don't have used fixtures before. And finally if you have some leftover time you could add PyTest skip the imperative variant, using a function call to the fixture function and see what happens when you do so. Again I will give you a couple of minutes and walk around here and also take a look at Discord and such. I'm seeing most people are finished by now. So let me just demonstrate real quick how this could look. So I'm loading the original file here. And if I want another fixture, I just write another function, decorate it with pytest.fixture and use it as a second argument in my test. There we go. And if I use PyTest.Skip in a fixture function, what will happen is that all tests using that fixture will be skipped. So if I maybe have a fixture representing some kind of database or server connection to a real test database for some kind of end to end test or whatever, then if that database isn't available, I could call PyTest.Skip inside the fixture and all tests using the fixture will automatically be skipped. Now, you might have tried using PyTest mark skip here. This, however, won't work because marks are a kind of more general concept used for more than just skipping and marks only apply to test functions. They don't apply to fixtures. You might also have tried this but that won't work either because this would skip the entire test module and I think by default it will complain. Perhaps you meant to use a mark. So what you should do is call PyTestSkip here without an at now because it's not a decorator, it's just a normal function call. And here you would say, for example, server not available if this was some kind of connection to a server. And then if we run again maybe in verbose mode, we will see that the test using this fixture now automatically got skipped from the fixture. And any other tests using it would then be skipped accordingly as well. PyTest also has a lot of useful command line arguments to kind of manage fixtures or to tell you more about fixtures. There is dash dash fixtures and fixtures per test, which just gives you a list of available fixtures or a list of fixtures a given test is using with their documentation. And then there's this kind of three different setup arguments. Setup show, which I believe you have already used, which shows you how fixtures are set up. Setup only, which only sets up the fixtures but doesn't run the tests. set up a plan which is a dry run which tells you what would be done but doesn't do anything. Usually if you implement a fixture, you should always write a Python doc string for it, so just a normally triple quoted string at the start of the function where you quickly describe what this fixture does because then PyTest will also show it in the dash dash fixtures output. Maybe I can show you how this will look then. So if I run pytest dash dash fixtures here, it will show me all the built-in fixtures including their documentation and such. Now I said fixtures help you make things a little bit more modular. If you have used unitest.py before, at least this was my experience, then you have this kind of problem that you have kind of setup and teardown methods but you want to reuse them across different test classes. And then you start with mixing and multiple inheritance and whatnot and quickly lose track of what works how. With those fixtures, setup and teardown is kind of isolated Isolated in one of those fixture functions and you can then plug them in however you want And one thing you can do is using fixtures. Oops. Sorry. It's using fixtures from fixtures So again, just just kind of theoretical example we have this answer fixture Then another half the answer fixture using answer Then a test for answer and the test for half the answer So if you will kind of draw the dependencies here that would look like this We have a test using half the answer. We have a test using answer and One fixture is also using the other one Now here this this might make not too much sense with this example But consider for example a test for a web Web application may be written in jungle where you could have a fixture using the DB fixture from the PyTest Django plugin, we won't really look at this in detail, a user fixture which creates a user with your web application just with some test user username and whatever password and returns it. Then we have this client fixture again from the PyTest Django plugin which gives us a Django test client object. This is basically just a thing we can use to request websites from our web application. But now we want to write some tests where we need to be authorized to access those sites. So we always need to create a user, then log in with that user's username and password, and then maybe also have an assertion here with log in so that we can be sure that logging in actually worked because it returns boolean and we would need to kind of copy paste those two lines into every test which needs an authorized user. So what we do instead is we write a new fixture, here I called it authClient, which takes the existing client fixture and this user fixture above, calls login and then returns the client fixture for tests to use it. Now in the test we can use this off client fixture. We already have a client where a user is logged in now and we could for example test a user profile URL or whatever and make sure that we actually could access it. Maybe if we use the normal client there without being logged in, this would give us a forbidden or something. That's how we can use fixtures as kind of building blocks to simplify, to kind of use another fixture and extend it in a way that we don't need to copy-paste things in our test functions. PyTest also has lots of built-in functionality which it exposes via fixtures. We will only look at two or three of them here. There are some more, but those are probably the most interesting ones. There is temp-pov or temp-dir, I will explain the difference in a second, which gives us a fresh empty directory for a test to store temporary files and such. There is monkey patch which lets us modify state for test duration, so patch things, set environment variables temporarily, that kind of thing. There is this kind of three factor of capturing fixtures to capture standard outputs from the tests or capture logging messages and then we could run assertions on whatever outputs our tests generated or on whatever logging was done. Maybe check that a warning message was logged to the Python log. And there is a request where a fixture can get information about the test which requested it. So basically things like the name of the currently running test. Normally you wouldn't use this directly, but it's a useful thing if you want to kind of start extending PyTest with custom functionality. So let's start with temp path. Temp path gives us a temporary empty directory for every test. So if we here have the test things and the test other, they They both use the input file fixture and this input file fixture uses the temp puff fixture kind of wrapping it like in the previous example with the off client. Now temp puff is a puff flip object. Could you raise your hand if you have used puff flip before? Okay, that's all over maybe three quarters. So pathlib essentially gives you an object representing a file path instead of using strings for paths. One thing again a little bit strange but you can get used to it is that it uses the division operator or a slash to concatenate parts of a path because that would use slashes on most operating systems and backslashes on Windows. And the slash operator takes care of always using the right thing then. It also provides, so this would generate a new path representing this input.txt. And then we can call write text on it to write some text into the file without having to take care of opening and closing it and that. And then we return this path so that the test can do things with it. Here we just call assert false, and when we have an assertion we can use a comma to add additional information which will get printed by pytest when the test fails. So here we use this to output the file path to the terminal. Now what will happen if we run this two times is that we will get this kind of directory structure here. In my temp directory, I will have a PyTest of Florian folder, just one folder per user. You don't get into each other's way if you are using the same machine with multiple users. Then in there, one directory for every run of PyTest and then finally one directory for every test in that run. Here we would see the input.txt in this testing0 folder. By the way, under the hoods, the temp puff fixture will use this request fixture to find out the name of the test currently running. And what we see here as well is that pytest keeps around those files, and that can be very useful if you need to investigate things by hand. So maybe you have some kind of data files generated you want to store there. Or maybe if you test a GUI or a website, you would even want to take screenshots or something and store them in this temp tier. And PyTest keeps around the last, I believe, three or five directories here, if you run PyTest multiple times. And then only on the fourth run or so, it will start cleaning up the oldest one. So then you can still kind of investigate and take a look, maybe rerun it and look at those files by hand, which can be quite useful. There is also a temp tier fixture you might see in older code using pytest. It's essentially the very same thing as temp path, but not using path flip, instead using py.path.local from this py library where pytest also came from. It's just here for historical reasons because back then PovFlip wasn't a thing that only exists since Python 3.4 so this PyLibrary exposed its own similar API. It's quite the same but of course there are some subtle differences. Nowadays just use tempPov for new code but if you see temp year in older code then that's what it is. Another built-in fixture is monkey patch, which allows you to temporarily change some states for one test. So modify some kind of attributes, kind of like unit test mock patch, if you have used that before, modify dictionaries, environment variables, the current directory, stuff like that. Now, you might be wondering, or at least I was, where is this monkey patch name coming from? this thing about monkeys? So there's actually a wonderful story behind this. It's coming from the SOAP project, Z-O-P-E, this term. They originally called this kind of runtime patching, they called it gorilla patching. And then due to some misunderstandings, people changed that to gorilla patching, but to make it seem a little bit nicer and a little bit more controlled and they changed gorilla patching to monkey patching or at least that's what Wikipedia says. Gorillas are apes, not monkeys, wow. So let's look at an example for monkey patch. We have a function here, which gets the current path environment variable, gets this.platform and prints both. And then we have two tests. The second one just calls this function without any patching. The first one first uses PyTest's monkey patch fixture, again by just having it as an argument to this test. need to import this from anywhere, you just have a test with the monkey patch argument and then call setOtter and setEnv to patch this.platform to monkeyOS and path to slash zoo. What will happen if you look at the timeline, monkey patch will set up things and do those kind of patchings you requested. Then the test will run with those patches and see those values. Then after the test, and later we will see how you can do this yourself as well, the monkey patch fixture runs again and takes care of undoing all this patching. So that for the second test, everything is in a clean state again. And that's very important. You could just say here, why am I doing this with this weird monkey patch thing? Let's just do sys.platform equals monkeyOS. But then that's global state you are modifying. And for all tests following this one, they will still see the changed value. And that can be quite a hell to debug if you have lots of tests and one of them changes something in a subtle way and you don't figure out which one it is. There's a lot of time to be lost there. Those kind of modifications should always be done via something like monkey patch which takes care of undoing them as well. Let me show a quick demo of how this will look when And we run this code. So here I did assert false just so that we can see the output easily. And we see the first test which used this monkey patch calls, which did them, has those changed values, while the second test which doesn't use monkey patch has the original values from my system, and I should probably clean up those things in my path one day, but so far that's what it is. So we can see that we still have this kind of isolation between tests which is always a thing you want to have. So we will move on to exercises about those two topics. For tempPath, use the provided input file fixture from another function and read the text from the file which the fixture has written to it. And for monkey patch, based on this kind of example I did before, write the function reading a password from the terminal using the get path module in the Python standard library. Then use monkey patch to overwrite it because you wouldn't want to have to always enter a password in the terminal. And that takes three arguments. The first one is the module where you want to patch things, or the object in general, but here it will be the module. The second is the attribute you want to patch away as a string. And the final one is the value you want to patch in. Now here we are replacing a function, so we will need to patch not just a value, we can't just replace the function by a string. That would give us a type error because this isn't callable anymore. we need to replace the function by another function. What we are doing here is using a lambda. Could you raise your hands if you have used lambdas before? Wow, that's a lot of people. I didn't expect that. So lambda is basically just a function without the name. If you prefer, you can write your own function instead, call it fake get pos or whatever, and then here path in fake get path instead of using a lambda. So the question was around using mock objects with monkey patch. There's this unites.mock library which provides kind of magic objects called mocks which, For example, record all actions done by them and then later you can ask them how have you been called. What I would recommend when using those is to use the PyTest mock plugin, which is a quite thin wrapper around it which just gives you a mocker fixture where it can then also do this patching and stuff and it will take care of cleaning things up. Many patches have a simpler approach which can be useful if you want to replace a value by another one but without all this magic around it. So let's go on, I would say. Then I will show two quick live demos of how a solution for this could look like. So first for TempPath, we had a kind of template here already. there I can actually just edit this test here or let me first show the puffs again how this will look so I have a dev test second and again here I'm just using assert false so that I can easily show this path and you will see the first test gets a path with test things the second test gets a path with test second. So every test gets its own directory. Now here I could read this text out again using read text from the PowerFlip API and then use it somehow. Let's maybe just comment this out for now and that works. Now, there are some questions around especially auto completion and such in IDEs because depending on your IDE, it doesn't really know what this input file thing is. I think PyCharm has support by now for PyTest fixtures and will kind of figure it out. But other IDEs might not and you as a kind of human reader of the code might also want some more context around there. So what I can do there is use Python type annotations. So here I could say I also want to import pathlib or from pathlib import path and then I can add a type annotation saying this is a path, this fixture returns a path and this argument here is a path as well. And that will kind of help me know especially if I have many fixtures to know what I'm dealing with here. For objects kind of internal to PyTest, things like the monkey patch fixture for example, PyTest exposes those types as part of its API. So if we take a look at PyTest.org, There is this reference documentation here where we see that we have some kind of clauses exposed here like monkey patch or basically all those kind of somewhat internal PyTest objects which are exposed in some way to the public are available here in the PyTest module so you can use them for type annotations. Most of them aren't really for fixtures, most of them are actually for plug-in hooks which we will only take a very quick look at. Yeah? When you write a custom fixture that's going to be a function, how can you use type annotation without your content guess, is there a workaround? So the question was if you write your own fixture, how are you going to use type annotations with that? I mean, you can, depending on what it returns as a normal function. If you have some kind of utility thing like monkey patch, then you would write usually a clause for it and return an instance of it in your fixture function and then you could annotate that as well. actually see a couple of examples later I think. So for the second exercise we said we had a function which reads a password from the user. Let's just do something like this, or let's actually say... Now I want to write a test for this, and I could say assert validate password super secure, this shouldn't be assert, this should be return here. Let's say maybe even have a good test and a bad test. Let's expand it a little bit because then I can show you something more. So that's how I kind of write two very simple tests for this. Now the problem is if I run this, then it won't work. That's a problem. Yeah, well, I would actually need to write it like this, of course, and now this doesn't make much sense. But still, if I want to run my test, I'm now getting asked to input a password, and if I type the wrong password, the test will fail, so that's not quite what tests are for. For some reason, they are automated. So now I want to patch that out. So I use this monkey patch fixer here. If I want to use type annotations, I could say this is a PyTest.monkey patch, I believe. And now I say monkey patch.setAutor get pass is the kind of module or object where I want to patch something. get pass, here it is a little bit confusing because both names are the same, but this get pass would be the get pass module and this would refer to the function, so that's what I want to patch. And I want to replace it by a function returning super secure. And in this test I would want to replace it by a function returning something else. Now this is still running here. Might be a capital P, yeah. Now of course here I need to have monkey patches and arguments as well for this to work. And now we see that the tests pass. So monkeypatch took care of replacing this get pass method by a kind of static function well, not static in a program sense, but by predefined function returning a password. And then we can test this in an automated way despite it being something which requests as user input. We are just asserting the return value that it is true or false here. Because we know... We are assuming that this returns a Boolean correctly. We could also say here is true or is false if we wanted to be more no, because this wouldn't return, this doesn't return the password. So we yes, if we notice wrong still and if we say here we change something, then this test would fail because to validate password, the function we are testing expects the password to be this. Now here we have hardcoded the password, basically the correct one. So the question was how would we do the same thing if we say from get pass import get pass. Now here it's tricky because we are in the same file so there might be a way to do this but I'm not sure offhand how. Usually you would have your code of course in a different file than your tests and then you could just say maybe validators or whatever then your model under test is. Yes, so that's in general there are some pitfalls with patching. It does increase your test maintenance costs. Someone asked earlier, yeah, but then if I'm going to refactor things and rename something then my IDE isn't going to change this string here. And yeah, that's true. Patching in general gives you a little bit more work. That's why sometimes it's better to adjust your code in a way that you don't need to patch things, that you can instead, let's say, pause in a, I mean, this is a kind of artificial example, but here we could pause in a getter, a password getter. So we could say get pause func or whatever equals get pause get pause. default value, and then call this here, and now we wouldn't need to patch anymore, we could just pause in the fake function as an argument. As usual, I mean, there are some drawbacks with that as well, we kind of need to have an SK patch which we only need to use in tests, but sometimes it makes sense to do this kind of thing. Or like with the The example I did this morning, that would maybe be a better example of this kind of thing where I have shown dependency injection, there we would pass a server object in instead of creating it here in the mail sender and that means we don't need to patch anything anymore, we don't need to take care how we imported things here, instead we just pass in a fake SNTP server object as an argument here. I think for things like this, this makes perfect sense. Okay. Let's go on, I would say. Let's recap what we have seen so far about fixtures. We've seen how fixture values are returned from a fixture function, often we just call that the fixture. Each fixture has a name and tests get value injected as an argument with the same name. We've seen how fixtures can use each other and how PyTest provides various built-in functionality via fixtures. One question which comes up a lot and I've already seen today as well is why or how do I import things? And the answer is sadly you don't. this kind of auto-discovery mechanism by pytest. In hindsight, it's questionable whether that's a good idea, kind of throwing all fixture in a global namespace, if you will. But also very difficult to change now without breaking thousands, if not millions, of test suits. So yeah, maybe nowadays you could use something like type annotations to define which fixture to go in which arguments and such, but nobody so far dared to cook something up based on those ideas. Next we'll see how fixture results can be cached with the defined scope. We'll see how to do clean-up from fixtures and how to use fixtures implicitly using auto-use. So we can use parentheses here to pause additional arguments to this PyTest fixture decorator. And here we are doing so to define a caching scope. We have a fixture which takes two seconds, say, to generate some kind of data or whatever, and then returns something. And then two tests using that fixture. Now by default with the function cachingScope, this fixture function will be rerun for every test and every test will get its own new value. So this would look like this, but also this will take four seconds. If we now change this scope to module, that would mean that every test in the same test module, so in the same test something.py file, would share the same value. This will only be called once. The value will only be generated once, only taking two seconds then. And both tests will get the same value. However, this can be kind of dangerous. So, here, if I'm running this again, We see how it takes four seconds because every test gets its own fixture value. We can also run PyTest with durations equals five, for example, telling it give me the five slowest things in my test suit. And it will tell us that the setup phase, so initialising the fixtures for those two tests is what took two seconds each. Now if we do the same thing but change the scope to module here, we can see that it is much faster. We only need to spend those two seconds generating whatever data we have once. However, we need to be more careful about what we do with the return value so if we here for example say we want to append some value before the test this will get us into a problem because this value is now shared because between the two tests because it's cached for the second test will again get the value get the list with the additional value here so that's That's kind of dangerous and by default I would really suggest not using this caching if you can do without and only turning it on if you really need to for performance reasons. I think I will show an example as well where what you can do if you have something like a database and want to have both caching but also isolation between tests. The question was maybe it's possible to make the value here immutable. Yes, I mean, you could here maybe return a tuple even or something like that. That's certainly a good way if you need to share things or you just say, yeah, you need to be careful what you do in your tests. sometimes that's okay. Let's skip the exercise here. Basically I have already shown you what you would see there, that it just is faster. There are five scopes available, but you will probably only need four of them. That's function, the default scope, every test function gets the value. Class, if you group tests in a class and use this fixture in different classes then every class will get its own value, but tests inside the class will share it. Module, so everything in the same file. Then package, which is kind of a special case for projects using monorepos and using different subfolders to test different parts. Personally I've never used it, and session, which means share the value for the entire test session for all tests. Now let's go back to how to do clean up from Fixture, I have already told you monkey patch does something so that it can clean things up after the fixture has finished running. And that's done by using the yield keyword. Now, if you haven't used yield before, that's perfectly fine. You don't really need to know how it works in Python. Basically, it allows pytest to do something so you would just use yield here instead of return in your fixture function and it allows pytest to run this part of the code here of the fixture before your test, then here where the yield is your test will run and then after the test your fixture function will be resumed and everything after the yield will run as a clean up step after the test. So if we had some kind of client clause here, we could call connect before the test, then run the test, and then call disconnect at the end when the test is finished. Just by using yield here instead of return. Let me take a look at the time. I'm guessing that's probably another exercise we would skip at this point, yes, definitely. So what you would see here is that PyTest runs this connect and disconnect steps around every test. And if you modify the scope, those will always be symmetric. So if you say module scope, then connect would run before the tests, then all the tests run and at the very end disconnect would be run. Now what happens if you want to share fixtures between different test files? That's where the conf test.py comes into play. In a ConfTest.py file, we can put in fixtures or plug-in hooks which are visible by all test files in the same directory and subdirectories. So if you have a ConfTest.py in the test folder, we can define project global fixtures in there which will be available for all our tests. And if we had the export subfolder and maybe had some fixtures we need just for export tests, then we could also put conf test.py in there. And again the fixtures here would be visible for test files in this export folder. So if we defined an exporter fixture in this test export conf test.py, then test YAML export in the same directory would have access to this fixture, while a test format maybe in the test utils file outside of this export folder would raise an error because the fixture wasn't found. So that's a good way to share fixtures between tests but still keep them Someone in control instead of my making all fixtures global in your in your tests contest pie So to wrap this kind of thing up This is something you would usually call scope as in the scope of a variable I'm calling it visibility here just to not mess and not Not mix it up mix it up with the caching scope. So if we define a fixture in a test clause, it will only be available for tests in that clause. It can be a good way to group related tests together and add some fixtures for them. If we define them in a test module, so in a test something.py, it will be local to that module. And as I said, in the conf test.py, it is wherever the ConfTestPy file is and any subdirectories. Finally, if we define fixtures in a plugin and that plugin is installed, then those are globally available. Again, as a note, we don't import fixtures from anywhere. In fact, if you do that, you can confuse PyTest because it sees the same fixture defined in two different places. Instead, you would rely on the autodiscovery of PyTest for those fixtures. Finally, I want to show a quick example of how to use fixtures implicitly. So sometimes we want to use a fixture but really only for the side effects of it. Maybe we have a fixture calling monkey patch to patch some things away and want to do this patching for all tests in the module. What you could do is marking this fixture as auto use, auto use equals true. We still return the value here, this temp path we set as a temporary home directory just in case a test wants to use it like here. But we don't need to. So any test which sees this fixture, in this case here, when it's in a conf-tester pi, any test in this auto use subdirectory will automatically run this fixture before it. Or again here, if we look at this auto use subdirectory, the first test really requested the fixture as an argument and gets access to value. The second test uses the fixture implicitly. As I said, typically to do some kind of patching for all tests in a subfolder, for example. And now we will have a quick live demo of how we can kind of combine all those building blocks to get more advanced behaviour from fixtures. I'll go over it quite quick because we are running out of time. I will upload those live demos later so you can take a look at the code again. Let's imagine we have a database class and for some reason it needs some time to generate some test data for the tests. Now, we don't want to spend those five seconds for every test we run. But if you use a database it usually has some kind of transaction feature where we can take a snapshot, run the tests and then restore the database to the snapshot we took, usually called transactions. So let's imagine we have a begin and end method here, or begin and finish, and we are taking a snapshot and restoring the data. So now if we want to write tests, we only need to generate the data once, but we still want to use the snapshot feature to get tests isolated from each other. So let's start with a database fixture and two tests using it. And when we now run this, let's maybe bump it down to two seconds again, we'll see that this takes four seconds because right now every test regenerates the test data from scratch which is slow for whatever reason. Now we could say scope equals session here, we want to share the test database for all tests and it's much faster, it will only take those two seconds once, but now tests aren't isolated anymore. So what we do is that we have a second fixture marked as auto use, where we get the database fixture, call begin, run the test, and then call finish. So this will be done automatically for us around every test. Let's also print something here. And now I will rerun PyTest with dash V and S so we can see this output. What we can see is that the data got generated once, but around every test we automatically have this kind of transaction feature so that tests are still isolated from each other. That's how we really can use all those fixture features as kind of basic building blocks to do more complex stuff. So let's finally wrap up this fixture topic and cover a last couple of things I want to tell you about. That's mainly about plug-ins. So as I said, there are over 1,000 PyTest plug-ins. One of them I would really recommend for a lot of projects is coverage. For one, because it has a very cute logo, but also because it's really very useful. Could you raise your hands if you use test coverage before? That's less than half, I would say, maybe a quarter or a third. So coverage runs your tests, or you run your tests under coverage, and it tells you which lines of your code have been run or haven't been run. This doesn't tell you if your tests are good. It only tells you your code has been run. The only thing it can tell you for sure is what parts of your code isn't tested yet because it was never run during the tests. It will give you an HTML report, for example, or different kinds of reports, but one of them as an HTML report. Let me see if I can just find one. It gives you an HTML report where it tells you, and it's probably not the best example because I probably was just running a subset of my tests. Let me find something better. It tells me I had tests for this parse method here, it was called at some point. But I don't have any tests which checks this value error here, so maybe it would be good to write a test for this kind of thing. It also shows me cases like here, where I have tests covering this code here, but I don't have a test covering the jump from this line to line 254. So I don't have a test where this branch here wasn't taken, that's called branch coverage. So when I start writing tests for a new project, or maybe writing tests for a project which doesn't have tests so far, this can be a useful tool to tell me what to focus on. There are, I believe, two more plug-ins I want to show you. This is PyTest BDD which lets you write tests in a more human readable way, kind of a specification. And then implement some Python code behind it to say what those steps here do. So we have a when decorator from this PyTest BDD project and can say when I go to the article page in my test specification, then this code here should run. And this, depending on your project and also your contributors, this can be quite nice because it allows people who maybe are a little bit less technical or less used to writing tests in Python to still read and write your tests. And finally, and I would love to talk in more detail about this but I'm running out of time, is Hypothesis which is a project as well as a PyTest plugin which is kind of like a fuzzer. So we say here we have a test which takes any kind of string here, and then instead of really testing values, we test some kind of invariance, so certain conditions which always hold true. In this case, if we encode this text with a given encoding and decode it again, we would expect the same thing to fall out as we put in. Those kind of invariants, for certain kinds of codes, they can be very powerful, especially for things like here, encode, decode, compress, decompress, whatever, if you have a way to convert something and back again. And then you can say this conversion should work for whatever output, whatever input I throw at it. And hypothesis runs your tests with like a couple of hundred examples. And when it finds a failure, it tries to minimize the example to give you the most simple data failing your test. So here it says with the OSCE codec configured, this character here Which is exactly one position over off-key. So the first character in Unicode, which isn't representable in the off-key encoding, which is basically just US characters. It gives me this character and I already have an idea of what my problem is without needing to know more. So this can again be a very powerful tool. even if I just use it and throw some data at some kind of parsing function and see if it explodes. That's a way I found something like five or ten bucks by now in my project, by just throwing strings at my config parsing code. There are countless more plugins, things like for running benchmarks with pytest, For getting an HTML output, that's something that Mozilla uses as well, with screenshots of their websites they're testing, and so on and so on. The PyTest documentation has a list which is updated weekly, I believe, by automatically pulling available plug-ins. You can also write your own plug-ins, and usually you would say it sounds very complicated to implement my plug-in, implement a plug-in, it sounds like a lot of work is involved. But if you have written a ConfTest Py file, you have already written a plug-in, because you cannot just put fixtures in there, you can also put in plug-in hooks to customise how PyTest works. You could, for example, implement PyTest report header to add some extra information to the version output at the very beginning. So before I wrap up, I want to give you a recommendation for a book if you want to learn more about PyTest. That's Python Testing with PyTest by Brian Ocken of the Pragmatic Bookshelf. I've been a technical reviewer of that book so I asked them nicely, hey, I'm presenting about PyTest, could you give me a discount code and they said, yeah, sure, here's a discount code for 30% off the ebook, so there you go. But even without the discount code and even with me being a reviewer, I really enjoyed how it explains things and I would really recommend giving it a read if you want to know more about PyTest. If you would, this is also, by the way, in the exercise PDF with the code and the ISBN. If you would like to learn more about PyTest, I have a full three-day version of this training planned with Python Academy, which is what he is doing, in Leipzig and remote in March 2023. Yes, that's still a while, but that's how it is. Or I also do custom trainings, coaching, consulting all around Python, PyTest, GUI programming, best practices. Feel free to reach out if you want to know more about that. I also do have some Some business cards as well as PyTest stickers here if you want to fetch some. I'll try to distribute them in some nice way, I guess. And then that's all I have with only 30 seconds late. Okay. Thank you very much.