They are not unit tests: a survey of unit-testing anti-patterns

Speaker 1 [00:06]

Hello, hello, PyCon. My name is Stanislav Zmiyev, and today I'm going to be talking about the worst ways to version and a little bit better ways to version. I work at SEMA. At SEMA, we provide the product teams and the development teams with a clarity on where they are on the roadmap and how to stay on it by analyzing the entirety of Git commits and Jira tickets and providing transparency for the entire organization. So, before we start, I'd like to ask you, and please just raise your hand and feel free to shout out. In unit tests, like everybody talks about, you know, unit tests or tests that are testing units. What are units? Functions. I hear functions. Anything else? Classes. OK. Methods. All right. That's good. Well, right, this slide, right? So we have unit tests. They're testing the functions. Then we have the component tests. They're testing either classes or modules. Then the integration tests are testing, like, integration between the modules, I guess. And then the system tests are testing the end-to-end, you know, the entire application. Well, this is a useless definition. Absolutely useless. The entire thing about unit tests being, you know, for functions is a little bit misplaced because now when I'm trying to test a big application and I'm going into the internal details, I am getting lost in testing the things that are not necessary. How do you pick a function for testing? How do you test it? Nobody is going to answer that, especially with this model. And people oftentimes disagree on what integration tests and Component tests even are, because everybody has a little Bit of a different definition. So to answer that, let us go Back a little bit and ask ourselves, why do we write Tests? and let's do the shouting again. Why do you write tests? to test? confidence? Refactoring. Okay, i've heard refactoring. Good. I agree with the person. I write tests because I need to be able to refactor really quickly with confidence, as someone else there said. So for me, testing is a tool to know when my public interfaces are going to change. I want to cover my public interfaces with checks that will tell me if something's broken. That's it. Sometimes people also use it for exploration, such as with TDD, which also lost all of its meaning in semantic diffusion. But the general gist is the right tests, the tests for the right interfaces are supposed to speed us up, are supposed to, as we said, help us refactor, help us deliver features quickly, help us iterate quickly. Problem is, if we go to the previous definition, they don't. And so to answer the question that I asked in the very beginning, if unit tests are tests for public interfaces, then unit is a piece of a public interface. Probably a small piece, but it really depends. And so if you're testing a library, for example, if I'm testing OS walk, my test is going to be testing a function. Yes, because the function is the public interface. But if I'm testing a graphical user interface, it's not going to be a function. It's not going to be an internal function somewhere out there. And if I'm testing an API, it's not going to be a calculate tax function. No, it's going to be an HTTP endpoint with its method and with its response and with its headers. That's my public interface. That's what I'm afraid of breaking because that is going to break my user's code. I'm not afraid of breaking some internal method. I'm afraid of breaking my endpoints. And so the problem with public interfaces is that they're really hard to define. As per Hyrum's law, the space bar heating up and then us using a macro to navigate it and to do something on it is also public interface. Because after infinite time and infinite amount of users, all visible behaviors are going to become public interfaces. So does that mean that we should cover them all? Does that mean that we should have the 100% coverage and then we should have multiple layers of coverage and we should try to make sure that the entirety of our code base is static and impossible to change? Or at least we will know about any change? Of course not. And I mean, you would probably say that some people, such as Kent Beck, would argue with me because TDD, because 100%, you know, we need that. No, actually, in 2008, even Ken Beck said, yeah, we're not getting paid for tests. We're getting paid for the products not getting broken and for, you know, the tests are the payment for this confidence. And in 2008, he mentioned that maybe someday there is going to be some sort of universal theory on testing. And he actually came up with it. Oh, that's laggy. He called it Test Desiderata. The purpose of the Test Desiderata, and I'm not going to go into all of the details because it's a huge concept. I advise you to read it. It's amazing, and it deserves a separate presentation probably of the same length. But the purpose of it is to tell you tests are not perfect, and no tests are perfect. No tests can satisfy the entire Desiderata. But all of the points in the desiderata are desirable. So all tests should be inspiring. All tests should be deterministic. All tests should be fast. We want that for all tests. But desiderata tells you that these properties are actually conflicting with each other. If your tests are really fast, then they're not as predictive of breaking changes. It will be harder to notice a breaking change if the test is really fast because whatever we do to make it fast is The same thing that makes it less predictive but saying that something is a trade-off and you know, it's all about trade-offs I think it's a cop-out answer. It's you know, it's just same thing as saying well I don't know what it is and I don't know what the trade-offs are Well, thanks to this is the rather we kind of do we kind of understand what trade-offs there are and how to pick them And so, based on this, let us continue the discussion about units. So, what about the integration and component tests? Some people say that integration tests are just another form of unit tests that are just touching more of the internal interfaces. I think all of that discussion is absolutely useless because look to your right, look to your left. These people probably have a very different definition about what integration tests are than you. I promise you everybody has a very different definition, which is why I decided to return to this testing pyramid. It's a lot less precise, but I think that it serves a much better value because it doesn't communicate any extra information. It just tells you, okay, you've got some automatic unit tests on the bottom. There are a lot of them. And there are some, like, end-to-end tests in the middle. There are a few of them. And there are some UI tests at the top. There are very, very few of them. And each of them is more predictive and, as a result, slower, each top layer. But even this model is not really accurate because not every interface has a visual component. Which is why I don't really want to talk about integration tests. I want to talk about tests in general. I don't really distinguish. And so, in my opinion, tests should speed up your work most of the time. The unit size depends on what we're doing, on the product that you're developing, on the public interface for your clients. And there is no right way to do tests. You got to optimize for the tradeoffs that you would prefer, and you should use Desiderata as a guide. However, the fact that there are no perfect ways to write tests does not mean that there is no horrible way to write tests. Let's get to that. Let's build an e-commerce API, and talking about horrible, this is bad API design. Who can spot the problem here? I'm just kidding. I'm not really asking. But let's start with the first problem. As you can see here, we've got our order, which is our kind of ORM model. We're importing it from our tests. Then we're importing our service and the function that calculates order total without tax. And so the purpose of this is to, so like, order service contains all business logic. And then what we do in our tests is we create some database models, and then we test our function and see that it's 18. It looks just like every guide. Every guide that you've read on unit tests looks like this. Problem is, if anything changes about these arguments, our users might not notice it. But this test will. This test will break, and you'll have to fix it. If you have 100 tests like this, well, sad for you. Same goes for this. If anything about this function changes, you'll have to rewrite the test. And the biggest problem is if we stop using it completely in our code, we won't notice. We'll just have a dead test, spending time and wasting our money and wasting our, you know, brain resources. So the easiest fix to do this would be to be in a really high-performing team that knows how to write unit tests and that knows how to fix such things. Or if you don't have such a team, like almost 100% of us, you could lint the architecture. Essentially, you could disallow imports from tests to your business logic. And here, we just, in our PyProjectOML, we just decide that the tests cannot import from services, utils, or models. And then if we run it, we'll see broken contracts, and we see that tests, test orders, imports It's that simple. And you can use import linter To test for any kind of architecture. It's really good. So i advise you to use this tool if you want to keep the Architecture without micromanaging every code review. Moving on, if we apply this, then we'll have to have another Way to test the tax calculation. What i propose is to use the Default fast api test client and then to use the post interface. interface, the interface that our users are going to use, the public interface. Then we just check that the status code is what the user would expect, and then we just check that the total and tax are as expected. That's it. It's very simple. We're not touching any database models. This will only break if the actual interface breaks, if for one of our clients the actual, you know, the actual integration breaks. Moving on. So there also can be units that are too big. If, for example, you use pytest subtests, I'm not sure which one of you uses it, but I am looking at you, please stop. Most of the time, most of the time, it's a sign that you're trying to mash all of the multiple tests into one test. So here, for example, we have a for loop which just runs twice, and then Then we have four subtests. This is actual production code, although anonymized. And the original one was about 200 lines long, 200 lines of tests. And then the fix here is simple. Just split the test. And as you see, the previous one, the previous slide, it had some preparation. It had some logic that is made for every run of the subtests. So you just put this logic into the fixtures. Simple enough. So moving on, okay. But the fact that I just criticized that test doesn't mean that all tests must be slow. In fact, as I said, it's a tradeoff. And so if your flow is large, and if you really, really want to test it, and if it's important for you that the entire flow works, if it's worth the debugging effort when it breaks to fix that test, then it's fine. Write the long test. As long as it's not testing multiple flows. Because this one, this one tested multiple flows. But if it's like a single flow of, like, create something, push it into the status, push it into another status, pay for it, and then, I don't know, expect that the web hook was sent, that's fine. Except that you don't want to have many tests like this one. Moving on, if you take a look here, this is a typer command. So this is like a CLI tool written by Sebastian Ramirez, which does really simple CLIs in the style of FastAPI. And testing it is a pain, because if we try to test it, the default way is to run the actual command line application as if we're running it from terminal. This is super painful. It's super hard to debug. So in such cases, what I advise is to make the thinnest testing layer, which essentially means that instead of calling the entire application, which is slow and really hard to debug, in such cases where it's, like, really hard, where the actual public interface is really inconvenient, I advise you to just make a function that has no logic, well, not like this one, but similar, that has no logic, that you can call from your tests to validate 90% of correctness. Of course, it's not going to cover the CLI things, but those can be delegated to a single small test. Similar to in a testing pyramid, that would be like a system test or like a UI test. Moving on, that problem doesn't just apply to CLI applications, it also applies to graphical user interfaces. If all of your tests are UI tests, your test suite is going to be really hard to debug and it's going to be really slow. So you want to have most of your tests really fast under this thinnest testing layer, very close to public interface, but not exactly the public interface. And then you want to have a few UI tests just to make sure that everything is fine, as long as you need them, as long as you want the graphical user interface to stay stable. Because it's not always the case. The third anti-pattern is asserting too much. If you take a look here, we're asserting a lot of things. A lot of things. And problem is, this test is only supposed to check that we can mark our order is done. That's it. So that state is done is the only thing that we need, but we're searching everything. I've seen so many tests like this. So once this JSON gets another key, this test is going to break. Imagine if you had 100 tests like this. That would be horrible. And I've actually seen a situation like this. If any of these fields are going to change, this test is going to break, even though the actual public interface of moving from status to status didn't break. For which the easiest fix is to just assert less. Just assert, well, status. You don't need much more. But then there are also other tools that allow you to assert a lot, but at the same time they allow you to not spend a lot of time fixing your tests. And now, ladies and gentlemen, let me move to inline snapshot and 30 equals. Can you see everything? I hope you can. Okay. So let's okay. So we have our app right here. It has the order resource. It has these fields. And it just returns something. Assume it does something in the database. And so we have a situation very similar to what we had before. So if I run these tests, if I run this test, boom, okay, it runs, good. Now if we add the date, then what's going to happen is it's just going to break, and then if you add 100 tests like this, you just have to fix them by hand, horrible. What you could do is to write the tests, instead of writing it like this, we could write it like this. Let me show you. Let's just use two. Just for the demo. And then let's mark the date. Okay. If you were writing this test from scratch, then it would look like this. You would say snapshot from inline snapshot. Right here. Now let's run it. Oh. That's clear. Okay. It still broke. It said failed. Your snapshot is missing one value. And look at what it did. It inserted everything. I ran the tests. I just said, okay, snapshot, do something for me. And then it inserted the value for me, which is amazing. Imagine you writing 50 tests like this. Imagine you being able to cover your product where your tests are, like, three lines long. And all of the asserts are essentially written by you. Well, for you, I mean. Not by you. The opposite. And so now let's take a look at what happens if we change our API. Okay, so date. We add the date. And now let's run it again. Look at what it shows. So it shows that something changed. The date was added. And it asks me, hey, do you want to apply it, or is that wrong? So this would allow you to go through, like, 50 tests in the matter of, I don't know, a A minute and a half, two minutes, whatever, I would spend, like, 30, 40, 50 seconds just fixing this one test if it was by hand. And now, okay. It added the date. It can even format it for you, and it has so many modes of functioning, it has a lot of features, and I really, really, if, like, you forget everything from this talk, every single thing, I want you to remember, public interfaces and inline snapshot. It's amazing. Now, let's move on to the other one. So this situation is also common when you're testing endpoints. Let's say that the user ID is auto-generated, and we can't really assert it. We don't really know what to do with it. And then let's say that created ads and updated ads are both populated by the date time now. So you just get some ISO strings in the response which are, like, this long. What do we do with them? Well, i've seen people just not assert anything about them. I've seen people assert that they're strings, assert that They're not none. I've seen people write a lot Of logic for making sure that it's actually none. Well, there is another library that allows you to do this Really, really easily. It was written by samuel colvin And the author of pydantic. And it allows you to To essentially assert things like these is UUID. Let me show you. Equals is UUID, and I want it of the version 4. Oh, I thought that I disabled Copilot. Well, whatever. And isostring true. Okay. Now, when I run this test, I guess there was oh, we forgot that this one, this test was not modified because it doesn't have an inline snapshot. So as you see, the test passes. Okay. That's good. And now this test is still ugly. We still, like, do indexing every field, mentioning the response JSON. We could, like, put it into a body. But I think the better way to do it is, oh, this is great. Thank you, Copilot. I mean, it's not Copilot. It's Cursor. But, you know. Okay. Name snapshot is not defined. We forgot about it. Booms. Okay. It did it for us. Soon or, okay, I wouldn't say soon. Someday, hopefully soon, inline snapshot will have the Functionality to do this automatically, but now, thanks To LLMs, you know, it's not hard anyway. So that's it. You got the snapshot, and if anything changes, let me show You how it's going to look like. Let's say that from here, the user name, instead of being the It's going to become just some random string. Boom. We know about it. That's it. It's that simple. And this test is going to pass now. But the other test is going to fail because we need to fix it by hand and everything. So, yeah, that's it with the snapshot on dirty equals. Now let's move on. I forgot my clicker. Okay. Right here. Is it going to work? Okay. There is another anti-pattern. I think this one is one of the most prevalent ones. Testing volatile code. For example, if you're writing a prototype, or if you're a pre-seed startup, or if you are writing something that is expected to change a lot, or if you are testing your internal logic where, like, you already know if you're changing it, you shouldn't test it. Just don't test it. But I know that some people still are going to try to test it because coverage. Because we've got to have 100% coverage, because if I wrote a Wrapper method or a string method, now all of a sudden we Need to test it. Why? Especially with the wrapper method. If I'm just writing it for my own debugging and I test it, Then every time I change it, I'll have to change the test. I might as well not just write it at all. Which is why coverage, by default, recommends to have To have this in your project, which allows you to ignore all Such things, and you can add any that match your use case there. Sadly, it doesn't include them by default, but it has them in Documentation, and it recommends you to use them. And i also like to omit things like my plug-in from coverage, Which is my docs plug-in, which is like run in my ci to deploy My documentation for my framework, which i'm going to be Tomorrow, by the way. And so I'm omitting this because if it breaks, I'll find out. I'll find out on CI. So if something is a short iteration anyway, you might as well not test it. You might as well exclude it from coverage. And actually, thanks to Marcelo, I now know that this approach of, like, excluding everything that you're not testing but still keeping your 100% coverage is called clean coverage. I won't have time to talk about it, but I have a link to it in my slides. I advise you to read it. Really nice approach. And apparently, for some reason, every single open source developer that I meet is using it but doesn't know that they're using it. So, moving on. Now, another anti-pattern is non-paralyzable tests. Well, if your tests are using common things between each other, then you're in trouble. Because whenever you're using common session-level fixtures or if you're using some random storage shared data like this dict, this is horrible. This is going to, first of all, be really hard to debug if it breaks, but then it also prevents you from parallelizing your tests. And parallelizing tests is one of the best things that you could do. Because once your product is mature and you have hundreds of tests or, like, thousands of tests, without parallelization, you're going to suffer. Your releases are going to become so slow. And the main purpose of tests, for me, the first thing that I want is to make my releases fast. Because I want to omit steps, not make testing the biggest step. This is stupid. So just don't use it. Isolate everything. Use scope function in your fixtures and try to not have any common data. It's okay to have something common. For example, if you have a common event loop or if you have a common database or something else. Something that is either stateless or stateful, but it cannot affect the parallelization of tests. They still stay independent. Moving on. This one is a big one. Mocking. Some people believe that it's okay to mock your internals, that it's okay to just, you know, mock your repository so that you're not calling the database, mock some other part of your code so that you can test one small portion of your code base in isolation. Isolation sometimes maybe most of the time no no this this is going to break a lot because Whenever we change the interface of our repository the mock is going to break the mock might Not be so great my our logic might not expect the mock to start returning what it's returning especially like for return value here, it also makes our tests need to change if the structure of our services changes, like the actual directory structure or the names of the methods. It's horrible. Please don't do this. And so the easiest way to not do this is to have your own database and to, you know, running while you're running the tests. And I advise you to, if you want speed but a little bit less convenient, I advise you to do it with Docker Compose, which all of us know. And if you want it to be automatic, see, tradeoff between speed and something being automatic. If you want it to be automatic, use test containers. They allow you programmatically from Python to pull the container of Postgres or any other Redis, whatever you want, and then run it, provide it to your tests, and then kill it at the end. Which would make it really easy to just run your tests anywhere, because they take care of their own infrastructure. And it's really simple to set up, as you see. But there is also the case where we're not calling, like, we're not mocking the internal things, we're mocking the external things. And whenever we're mocking the external things, it's really hard to support these mocks. You mock them once, then their API changes, and new features are added, you have to update your mocks, or breaking changes are added, and you have to update your mocks all the time. And how do you know that the mocks are still relevant, that the mocks are still valid? You don't. Which is why I think this came from the Ruby world. It's called VCRPy. And the main purpose of this thing is to have, essentially, when you first run your tests, this is going to record all of the actual network responses from external services. But then it's going to save them. Similar to inline snapshot, but inline snapshot most of the time saves things within your Code like in line, while this thing by default saves them like in a separate directory. And the thing about it is every next run of your tests, it's going to use the recorded value. So your tests are not going to be calling the Internet every time. If you have a lot of integrations with external services, I still advise you to probably have one or two tests, just one or two, validating that the, you know, the actual interfaces Does network interfaces work? but, again, it's optional. The main thing about vcrpy is it allows you to not just record The values and make your tests really fast while not writing The mocks by hand, it also, like inline snapshot, allows you to Update the mocks by hand, which is really amazing. Moving on. Wait. I said it allows you to update the mocks by hand? It can update the mocks for you. That's the purpose. And you can run it without the mocks, too, so it's really great. The next one is not enough assertions. Especially in environments that are heavy in coverage, you meet this beast where people essentially call the code, and then they're like, okay, I just want the status to be 200. Or I have seen tests that were just calling the code and are like, good, no exceptions. Amazing. So the good way to solve this problem is to assert more. And if you remember, the other anti-pattern was about asserting less. But we got to assert exactly our public interface. So like we got to assert the things that we want to check. And so if we expect that we need to, you know, that an order needs to be returned and we don't have any other tests that are validating the order body that is returned from this Then we might as well validate it here. But the biggest problem with such Checks is that it's almost impossible to know how good your Assertions are. You can check coverage. But coverage on its own is absolutely useless. It's not because you can cover everything without asserting Anything with zero assertions. That would be useless. Which is why there is a tool called MuteMute. MuteMute? I'm not sure. It's mutation testing. Really cool concept. I think it also came through from Ruby, or it was popularized by Ruby, something like that. But the basic purpose of mutation testing, please look at the bottom. The basic purpose is, as you see, it takes your code base, it takes all of the tokens that it can mess up. So, for example, here we're returning an empty string, and it makes us return the five Xs. And so it modifies your code base with some invalid constant, or it changes the more than or equal to to just more than or to less than to make sure that you're checking your boundary conditions or to make sure that your code actually cares about what happens here. And so as a result, it does this operation. It modifies your code. Then it runs all of your tests, the entire test suite, and then if it breaks, if even one test breaks, it says, great, your tests have not allowed this mutation to pass. This means that this piece of code is not just covered, but it's actually tested. You actually care about what happens there. And it does that like a million times. Or thousands of times. It depends on you. And so as a result, yeah, of course, it takes really long to run. But as a result, you will have a really nice list of like, hey, here's the mutant that you missed. Here is the place of code that if you modify it, even though it's covered, nothing is going to happen. No tests are going to fail. And it gives you all of that information. So combined with coverage, mutation testing is an amazing tool to provide you with information on, like, the quality of your tests. But does that mean that we should have zero mutants at the end? Zero things that passed all of the tests? No. Remember, we care about our public interfaces. So there should be balance in your mutants. There should be balance in the things that are passing and that are not passing. So it's useful as a tool to look at the things and see, oh, okay, this piece of code cannot break here. This piece of code must be tested, and it's clearly not. And this piece of code, no matter what we do with it, for example, the wrapper, you know, the dunder wrapper method, if something is replaced in that string, we don't care. That's fine. So it's a good tool. Just don't be dogmatic about it. Like with all tests, really hard to make them perfect. There is another kind of problem where your tests are really hard to write, so you don't write them. Have you ever had this situation? So, for example, in here, in order to write a simple test of just checking that the order done is done, we got to prepare the customer. We got to prepare the order. We've got to do all of these steps, and we do this not for one, but for two tests. Okay, yeah, it looks like it's the same test. It's a different test. Because one is paid, one is unpaid, and the responses should be different. But the problem is we're writing a lot of extra code for this. Well, you could make this into a fixture. The problem with fixtures is that here, as you see, the status is different. You can't just write a fixture for this. You have to write two or three fixtures. So you have to write a customer fixture, then you have to write The fixture for unpaid and then you have to write a fixture for Paid. This is solved with factory Fixtures. They're a default pattern in Py test. The problem with factory Fixtures is that they are not type hintable. So if you mess up, if you mess up anything in them while Calling them, nobody is going to tell you. You'll have no chance. You just can't do anything. Which is why I created the library called bytest fixture classes. And by the way, this thing, this anti-pattern can also be significantly improved by inline snapshot and dirty equals. But we would stand here for the entire day if I talked about them everywhere. They're useful. And so if you take a look, now we just create the order in status paid. And then we create the order in status unpaid. That's it. These tests are super simple to write and to read while not losing anything. We didn't lost almost anything. Of course, we lost the simplicity because now we have two fixtures. This one is for default customer. The worst anti-pattern with PyTest fixture classes is to use it everywhere. Don't use any tool everywhere except for maybe inline snapshot. Yeah, you could use inline snapshot anywhere. But the thing about PyTest fixture classes is they are a very specialized tool for such situations where your fixtures are just a little bit different. You don't want to write two fixtures, but you also can't keep them in one fixture, which is why here we have the default product, the default order, but then you can customize any field that you want, any single field, making your tests really simple to write. But since we don't need to customize customers, we just use the default one because it's always the same. Moving on. Actually, I don't even remember what's next. Okay. Oh, yes. This is, I think, the last one. If you are not calculating coverage over your tests, then you're going to have a lot of dead testing code. You should still calculate. So, like, remember when I talked about omitting things from coverage? Well, tests are the exception. Because everything in your test Directory should be covered. Because if it's not covered, Then why is it there? these are tests. They must run when your tests run. Same goes for fixtures. Same goes for, i don't know, the Extra functions that you have in your test. Anything. Anything that is in your test Must be covered. So please enable coverage for Your tests. Don't omit them. Because otherwise you're going to have a lot of extra code and And it piles up. It's a lot of piling up. And that's it.

Speaker 2 [37:32]

That's really nice you finished early because there are lots of questions for you waiting here. Thank you. It was an amazing talk. It was a very amazing audience with the engagement. And I have to start with this comment and question. There are many people out here wanting you to be a test advocate in their companies, it seems. So they are asking how they can infect their colleagues with your passion about tests. Thank you.

Speaker 1 [38:02]

Oh that's a good question of course I think that you should start with really good tooling and before you explain what to do with tests you should explain what not to do the first thing that I want to know when I'm using any technology and any approach is we're not to apply it because I because. Because if you have a few really good tests written using really good infrastructure, then it will be much easier for your colleagues to start loving them because they don't have to maintain them because these tests are not ugly because these tests, everything, you know, is automatic because, you know, they're pleasant to write, they're pleasant to read, and they're pleasant to maintain. And that's why I keep mentioning the tools such as inline snapshot, because these are doing the most important thing, making tests pleasant to maintain, because that's the biggest problem, I think.

Speaker 2 [38:58]

We need to get regularly those of Stanislav's love for tests to stay motivated. Good. So the next one is, would you recommend multiple, such as six asserts in one test, or rather split such tests up into multiple tests with only one assert each? And why? With Smiley.

Speaker 1 [39:18]

It doesn't depend on asserts. Your test can have a thousand asserts. It doesn't matter. It's still going to be a good test as long as it is testing the public. Actually, no, forget about the public interface. As long as it is speeding up your work and making you more confident and is not a pain in the ass to support. That's the main thing. As I said, when you're asking questions such as how many assertions do I need, look at Desiderata. Desiderata is going to show you a lot of, it's like 12 qualities. Pick the ones that you want and implement tests that adhere to these. So the more assertions we have, the harder it is to debug at the end. But at the same time, the easier the test is to read and it's probably going to be faster to run.

Speaker 2 [40:06]

The questions are coming up continuously. In the case the unit is too big, why not use mocking instead of creating an extra superfluous function?

Speaker 1 [40:22]

So the extra function does not actually conflict with mocking. You can mock and use the superfluous function. The function is there to make it really easy for you to be close to testing your public interface without actually going through the trouble of testing it.

Speaker 2 [40:46]

Why not use parameterized tests instead of PyTestFicture classes? Lots of votes for it.

Speaker 1 [40:53]

Oh, yes. That is very true. If you have a single test like that, then that's true. If you have a bunch of tests that are just validating the same thing, then PyTest fixture classes are going to be amazing. But let me show you. Problem is, in this example, the responses of the tests and the status codes of the tests are going to be different. And if that's okay for you, if you can parametrize those things as well, then all right. But if you have tests like these throughout your code base and you need to use the order object in multiple places, then PyTest fixture classes is going to be much better. But again, if it's just one test or one group of tests, then yeah, parametrize is much better. Please don't bring extra infrastructure just to cover for one test. So I agree.

Speaker 2 [41:48]

I'm using my session chair privileges to mention this before we finish. How can I avoid depression when returning to my company's code base and looking at the tests?

Speaker 1 [42:05]

So you mentioned excitement from my side. It's not excitement. It's anger

Speaker 2 [42:12]

Nicely directed, channeled.

Speaker 1 [42:16]

So it comes from, the fact that I'm excited about these things comes from the fact that I suffered so much with bad tests. So I don't have an answer. Solve it. Fight it. Tatakae.

Speaker 2 [42:33]

How easy is it to merge changes to snapshot output files in a complex project with multiple developers?

Speaker 1 [42:40]

really easy because they are not the snapshots how do I describe it one second the snapshots are not stored in separate files they are stored in your test file in within your test thing is if your test changes in two different pull requests and it changes in it in different ways then well you will need Merge it anyway it doesn't depend on inline snapshot in fact inline snapshot Within it will just have the regular response same as what you would do with Your hands it just automates manual work

Speaker 2 [43:20]

What do you think about property testing with, for example, hypothesis? It seems like a good alternative to snapshots or mutation testing.

Speaker 1 [43:28]

Hypothesis is not a good alternative to mutation testing or snapshots property based testing is amazing if you have a clear need to test all sorts of input however if you need to test all sorts of behaviors all sorts of workflows it's not going to help you and also comparing it to mutation testing mutation testing is there to validate that your assertions are good hypothesis does not. Hypothesis just runs your functions, your code through a Billion, like, a billion types of data.

Speaker 2 [44:08]

There are still lots of questions and also coming up. I would suggest to find Stanislav during the coffee break. I'm sure he's going to be happy to share his anger and passion and happiness at the same time. And come back here for PyLadies panel. Let's give Stanislav one more round of applause. Thank you.