Fear the mutants. Love the mutants.

Speaker 1 [00:03]

But today I am here to do a lot of things, mostly to educate and inspire, hopefully, that's the plan, we'll see how it goes. What I'd mostly like to do is point out how cute this penguin is. How do we all feel about this penguin? Open question. Good penguin? Awesome, cool, thank you. Yeah, clapping is fine, whatever audience noise you would prefer. I've just knocked over my water bottle. Right, so, hopefully you understand me. I have come from England and I have a terrible accent, so we'll have to deal with that. But what I'd like to do is talk to you today about mutation testing. in a talk that I'd like to call Fear the Mutants, Love the Mutants. So before I get started, I do want to say this company has paid for me to come here. They've paid for my flight from England, so I am going to mention them. Thank you. So we do communications APIs. We've got a booth. We're going to raffle off an Xbox in about an hour's time. If you want to go see us and do that, that's fine. That's all I have to say. More importantly, though, the reason I'm mentioning it is because I actually learned about mutation testing last year because I added it to an SDK that I maintained for this company. So I actually learned about this last year and what I want to do is share with you what I've learned and share with you the kind of journey I took. And so it's so weird not to be able to walk. I usually walk around. I'm used to getting my steps in. I have to stay here apparently. So I'll try and do that a little bit. You'll see me moving. So I'd just like to say that I may have the loudest voice in the room. I definitely do. But what I'd like to say is that I'm not the real protagonist here because the real protagonist is Henry. Now, this is Henry. How do we feel about Henry? Cute penguin? Awesome, awesome. I know most people just came here for the cute penguin, so don't worry, we've got you. We've got a lot more of those, so don't worry about that. Right, so Henry is awesome. Henry's going to be very useful soon. But right now, we're not going to talk about him too much, but just keep him in your mind because he's going to really tie everything together later on. Before we get there, though, this is going to be quite audience participation heavy, so if that's okay, I will now ask an audience participation question. Hands up if you've heard of testing. I am in the testing track at PyCon. That's a good sign. I found the room correctly. Great. Who's heard of code coverage? Most people. If you haven't, that's fine. If you have, awesome. We are going to talk a little bit about that too. Okay, other than through the talk and knowing that it exists, who's heard of mutation testing? Okay, keep your hand up if you've used mutation testing. Ah, gets everyone every time. So I'm seeing some very confident hands in the back like, no, I have. I'm here to see how you do it. I like that, I like the energy, I like that. Okay, so, that's awesome. If you have, if you haven't, that's fine, by the end you will know what it is and hopefully you'll feel inspired to use it as well. So before we start, let's just kind of get a bit of a common ground. Let's talk about testing. So this is an open question. Anyone here is welcome to shout out. I don't think the slider will sync fast enough, but why do we write unit tests? This is a question for anyone in the audience who'd like to shout out. Why do we write unit tests? To stay safe. To stay safe, or sane. Anything else? Confidence. Inspire confidence. I like that. Compliance. That's a good one. People don't usually get that one. Yes? We can teach with unit tests our interest. Oh, education. I would say go further. Documentation. Okay. Great thoughts here. Here's what I've got. So preview code actually works. Documentation. Get that confidence. we've said all of these almost. Regression testing, if you do some refactoring, and also for compliance, which people, again, don't usually get good work who ever said that. But there's a problem with this, right? Because projects grow, they get bigger. It starts off small, it gets bigger, it evolves over time, and as you're doing this, you might be refactoring, adding new features, and you might find that your code gets missed. You're not actually testing stuff anymore. You also might find that you often don't monitor your tests, and if you don't monitor your tests, you're going to have a big problem, because in that situation, you are going to be a bit stuck when you need to understand what they're actually doing and how much you can trust them. Luckily, there is a metric that we can use here, and that is code coverage. So I know most people have heard of that, but I'll just give you a quick summary. This is my opinion. This is how I say it. How much of the source code of a program is executed when the test suite is run, okay? I'm just going to drink some water. Talk amongst yourselves. Thanks. I did this talk in Canada last week, and they all started talking at that point. I don't know I didn't know how to feel about it so let's give you a real example so this is um this is basically an SDK it's one of the SDKs I maintain for my company and it's the one I applied mutation testing to but we've also got code coverage so here's an API example here's just a file and how code coverage works is basically you run it and it'll show you that the green stuff here is what we've tested but you can see there's a red line here for something that isn't tested so what this tells me is like oh shoot I'm not testing that authentication type I better check that so then you know I can go away and I can I can actually make that happen so I can test that, and this is useful because code coverage has told me something useful. What we can also do is have a view of our overall infrastructure, all of our code base, and we can see what we're testing and what we're not. And this is really useful, right? This tells us how we can get an idea of what our tests are doing. So what's good about this? Well, I'll tell you, more and better tests, right? It can also be a lot easier or cheap to measure this stuff. It's really easy just to evaluate and go, okay, so what am I covering? That's a really easy operation to do. It also shows you what you didn't test, And so actually, code coverage is pretty great, right? Yeah, let's hear it for code coverage, yeah. So code coverage is important, and it's pretty great, right? So I think actually, you know what, yeah, I think that's fine. Use code coverage. I'll see you later. Except actually, actually, wait, wait. So actually, there is some stuff that's not so good. It can be misleading, right? And it also doesn't guarantee the quality of your tests. And so at this point, I would like to ask for a little bit of sincerity. This is a safe space. You can answer this question honestly. I will be answering this question honestly. Put your hand up if you've ever written a test, not because it was useful, but because you had a code coverage score that you needed to make higher. Safe space. It's a safe space. Half of you are lying. Half of you, I respect your honesty. So yeah, you can see this is a real problem. This is the real deal. And so we see coverage is great for a lot of things, but it doesn't do everything. So what can we do here? I'll show you an example. This piece of code, right? It's very simple. I'll just explain. We're importing some requests. We're doing one HTTP call. We're doing a get request, and then we're returning the JSON of that as, you know, as we're returning the JSON of that response. And so I'm going to show you a, on the next slide, this test that I'm about to show you will give this 100% coverage. Now, this isn't ideal, because all this does is test that this function basically is stepped through. What we don't do is check the actual payload that comes back. We don't know if we got a 404. We don't know if we got a 200. We don't know what happened here. Right? So this isn't... It's not useful. And that's what I'm trying to say. And actually, why would I say it when I have someone smarter than me who can? So here's Goodhart's law. So this is a statement that really summarizes this well, this problem, where we end up with code coverage where we just try and make the number bigger. And it's this, it's when a metric becomes a target, it ceases to be a good metric. I'll say that again, when a metric becomes a target, it stops being a good metric. So if we just think about the numbers, we're not thinking about what the code coverage is supposed to actually tell us. Does that make sense? Awesome, okay, we see the challenge. So we've got a question here, an open question for the audience to answer. Think along at home if you're watching on the stream. How can we understand what our tests are really doing? Open question, anybody? Read them. Good answer. Break them. Even better. I've found mixed results with that, but maybe the V4 is better. I haven't paid for it yet. Those are really fair points. I think it leads to a further question. I won't answer these just because of time, but how do we know if our tests are trustworthy is really the question. I think the real summary here was provided in 100 AD by the Roman thinker Juvenal after a fourth glass of wine, he came up with the best statement possible for this, which is, who watches the watches? The test is supposed to keep your code safe in production, right? But who's watching your tests? I have a certain answer for you today. Mutation testing. Spot the odd one out. All righty. You may, if you're sharp-eyed, have noticed that Henry's made a comeback. So why do I keep putting penguins in these slides? You know, other than the fact I think penguins are cute. That's the real answer. But the other answer that I'd like to give is that so the code I work on deals with messaging and APIs. And so what I was doing when I was writing this, the way I was learning this myself and the way I was trying to understand it myself was that I was thinking about my code that works is like a pigeon or a dove. It's something where I can tie a message to its leg and I can send it off and it will carry that message for me. So it will fly away and do the job. So this in production is like a bird that can fly away and deliver a message. Now, penguins are birds, that is true, but if you may have noticed, it's a reasonably well-known fact that penguins do not fly. It makes them very angry, as you can see. So the issue here, yeah, it is very sad. But I wanna show you why Henry is gonna be really useful to us today, and I wanna show you why you should love Henry, so this is what we're gonna do. So mutation testing works like this. We start with some production code. We apply some kind of mutation operation to it. And we end up with a mutant version of the code. So we've still got a bird, but this code is not the same, right? So the production code works. This code may not work. It does something different. So how does it look in practice? Well, here's an example. Hit this function, adds two numbers. A mutant version of this might, for example, subtract two numbers. Or it might, it's just playing with this thing. It might add an extra thing, an extra constant. It might return the string versions of these things. It might return nothing. These are all valid. They're all cute-ass penguins. so that's good. So what we do with these, how this works, is when we do a mutation test, we generate all of these different things. I call these the fab four. But we've got these different penguins that all look a little different. They don't fly, right? They don't do what they're supposed to do. But this is the situation where we've got Henry here, and Henry is going to try to fly. And what I mean by that is that Henry here is going to actually, we're going to run our test suite with him, and we're going to see if those tests fail. Now, if those tests fail, that's good, because he's a penguin, and penguins cannot, as we know, fly. But if the tests pass, he's too convincing. This mutant's too good. Look at these wings. This penguin could carry a bus. That's bad, because it means that this might have made it to production, right? So the question I ask of you is, what does this give us? And I'm not actually asking this of you, because time. I'll like to tell you, I assert that this gives us a way to evaluate the quality of our tests. How do we feel about this? Awesome. We don't have to clap, but you can. Sure. Thank you, Sarah. I put parts in with audience participation so I can drink water. I'm a millennial. Let's actually go into something a bit more nitty-gritty. Let's talk about frameworks. In Python, we don't have big enterprise products for doing mutation testing, really. In JavaScript or Java, we have things. But in Python, we've really just got community-supported stuff. These two projects are community-supported. These are open source. I'm not affiliated with these. There's Mutmuta Cosmic Ray. I am not a doctor. I'm not a lawyer. I'm not a professional financial advisor. The value of your investment may go down as well as up. But I personally have been using MUTMUT because it just has defaults that work for me. So let's go all the way back. I've applied this to this code base, this SDK that I own. This is what I run. And I've added this in. So let me show you how I actually did that because what I want is by the end of this is that everyone here can go home and do this themselves, right? So pip install it. Super easy. It's on PyPy. It's all good. Run it. And if you've got sensible defaults, this will work. If you don't, you're going to have to do a little bit more with config. But for this, what happened when I ran this was I get a lot of explanation, basically. It will run my test suite, then it will generate mutants, and it will try and run my test suite with those mutants. By the way, am I talking too fast? I'm aware I talk pretty quickly. Is this okay? Awesome. Thank you. I appreciate that most people here are having this as a second language, and I really appreciate your patience with me, so thank you. So in the situation here, there's a few things that can happen. We can catch these mutants. They can time out. They can be a bit suspicious, and we're not sure of them, and they can also survive. When I ran this with my code base, we generated nearly 700 mutants, of which we caught 500 and we didn't catch 170. Is that good or bad? I'll tell you later. Let's look at some, right? Let's look at some mutants that we did catch, first of all. So here's one. This is just something in a class that I've got. These are just valid channels that you can send a message with, right? So a mutant version of this, for example, maybe played with this string. so now SMS is not a valid channel anymore. And that basically, thank you, that means we now have a bit of a challenge because we can't send a message with this, and so our tests, obviously, that try and do that are going to fail. So that's good. We catch that mutant. Here's another one. Does anyone use Pydantic? A few people. Awesome. I went to a talk with Pydantic yesterday. Great topic. So here's a validator. This is from Pydantic, a model that we use in the SDK. Removing the actual decorator here, this will never be run. That's a problem. And so in this case, we actually catch this one too. You can see mutants that you didn't catch with mutmuch show. You can have a list of them like this. You can see the range of them you didn't catch, and if you want to see a specific one, you will type mutmuch show and then a number, and that will give you... Oh, dear me. That will give you... There we go. That will give you a mutant that you didn't catch, or that you did. So this one, you can see, the authentication type was removed, and so this one failed, which is good. You can also see the HTML output, So you can see, for example, every file, and inside of each one, you can see all the mutants you didn't catch. So let's look at some that we didn't catch, because this is what we want to see, right? We're not here to see a number. We're here to see a mutant that we want to do something about. So I'm going to show you three, two of which we maybe don't care about, one of which we do. First one here, mutant 58. Can we see this? The mutant renames the logger. Do I care about this? I think no. I think logging is out of scope of what I want to test, so I say, nope, I'm fine with this. Mutant 62. What have we done? All we've done is we've changed the value of a constant in a method signature. Again, I don't care too much about this. So for me, this is fine. I can comment these lines out, but unless I go through and manually do that, I'm fine just to ignore these mutants, right? These aren't problems in production, but let's look at this one. This one is Mutant 112, and it is actually a bit more interesting. Because this one here, we're instantiating something from our client class, which is actually how our users will use that API. And in this case, if I remove my entire voice API class, it's fine. All the tests still pass. So how is that okay? Right? Hey, I'm going to remove my voice API. That's fine. It's not. But we actually test this directly, right? We test it directly. And that basically means that actually maybe we should test this the way that, like, use it the way that the client is using it. And maybe there is a case to write a valid test there. And so now I'm actually getting real insights about my code base that coverage was not telling me. Does that make sense? Thank you for the solemn nodding, I appreciate that. Yes, yes, it does make sense, it does, it makes a lot of sense. So I don't take these that seriously, I'm sorry. I care about the code. Right. So we go back to this number here. We didn't catch all the mutants. We caught about 75% of them. I ask you a question. Is this a good score? Yes, this is the correct answer. Confused shrugging is actually correct. 100% does not make sense here, because I'm not looking for a number. I'm looking for insight, and I've gotten that insight from these mutation tests, right? So let's talk about how we get started with these things. What I did was I started locally, and I started small. If you have a big project, you might want to start with a subset. You might also want to start tweaking for performance. So, for example, exclude tests that maybe you're not so interested in. For example, tests that deal with integrations, things like databases. Maybe you don't care about that. and exclude code if you don't need to run that. For example, you might want to exclude auto-generated code because, again, you're probably not going to want to write tests for that. So that's the first part of this. Hopefully we understand what mutation testing is and how we want to use it. Talk amongst yourselves. I feel like if I got a water company to sponsor me, that would be pretty cool because then I could get paid to do it. Anyway, I'd like to show you the next slide. I don't know what I typed into DALI to get this image, but I'm really happy it exists. Thank you. So you're very kind. So I'm going to tell you about how you run this off of your machine. Why do you want to do that? Well, this takes some time. Like to generate all these mutants, 700 versions of your code, different changes, and then run parts of your test suite against those takes some time. For me, that took me like 15 to 20 minutes. If you do this in the cloud, you know, if you do this with CI, you're using resources that you don't have to manage. And you can also specify the platforms and versions that you want. So, for example, different OSs, different Python versions, all this stuff. So if your code is expected to vary or have different behavior, you may wish to test that too. So I'll show you an example. Stop talking, you say. Sorry, I can't. I'm the only one with the microphone. I'll give one to Matthias in a little bit. So what did I do? I implemented this in my SDK in GitHub Actions because that's the CI system that I use. So we have this thing here where you can run this workflow. And I've chosen to have that so it's manually run. I don't want this running on push or PR. I'll tell you why later. But we run it. It does a job, and inside of that, it gives us a mutation test report, which is actually the HTML report of all those mutants that we saw earlier, right? So, thank you. How did I do it? Great question. It's not actually that impressive, and I'd like to actually quickly ask, right, who's looking at this right now and thinking, I want to maybe try this. This looks pretty cool. I want to maybe have a go at this. Whoa! This is the most roaring, like, you know, I've never seen so many enthusiastic people. This is amazing. Wow. Awesome. I feel like I'm doing my job right. Awesome. So if you didn't put your hand up, that's fine. Because what I'm hoping is that when I show you how easy it actually was to make that work, you'll be like, oh, this guy could do it. I could do it. That's what I want. OK. So I'll show you with GitHub Actions because that's what I used. But it will be the same pretty much everywhere because it's just a bit of scripting. So what have we got? I'm going to show you the entire YAML file. It's 35 lines. I think we can deal with that. I'll talk us through it. So, first of all, this mutation test, we have this job. I'm running on Ubuntu, because I'm basic, and I am using Python version 3.10, because I don't expect my code, it doesn't vary too much from Python versions, so I'm okay. So what do we do? We check out the code, we set up Python, we install our dependencies, now including MUTMUT, now with added MUTMUT, we then run the mutation test. So here we're running MUTMUT run, we've got two new flags, no progress, because we don't want the output in our logs to get all disgusting, which is what happens. And the CI flag here, which basically gives us a sensible error code. And I'm gonna shout that out because I added that to MuttMutt. It was my only contribution, but I'm still proud of it. But it means you can actually run this in CI without it giving you a horrible error or hacking it with bash, which is what I used to do. So we've got that. We then run the HTML output and we upload that so you can get hold of it. And that's it. That's literally it. And you can see this. You can go to my SDK, look at it yourself, and just copy paste it. Then you'll have it too, right? So that's what I would recommend you do, plagiarism. So, or ask ChatGPT and don't call it plagiarism, who knows. So, we're nearly done here, so I just wanna talk about other things you might wanna think about with CI. So, manual versus automatic triggering. I said that I run this on push, or I don't run this on push or PR, I only run this manually when I choose to run it. Now why is that? Any thoughts while I drink this water, yes? That's good. You get the tilted head nod of approval. That's a really good point. I don't want to turn this into, oh, we can't merge this PR until the mutation score is above, you know, 35, whatever, you know. That's just, we've just abstracted Goodhart's law again. We've got the next level of Goodhart's law, which is not helpful, right? What I care about is every now and then, I run these just to see how my code base is doing, right? I don't want this to be something I do all the time. It's going to take ages as well. If I want to quickly merge a PR, I don't want to wait for three hours for these things to build, right? So I would just say, consider what you're using it for. For me, it did not make sense to do it in that way. So you might also want to think about multiple operating systems as well. You know, you can in CIs, you can choose what you're using. Thank you. You can choose what you're using, which is cool too. Again, I didn't need to for me. So you might want to for your code. You can also use multiple versions of your dependencies or of other things that you might need as well. So just think about that when you're considering CI, and that's kind of why you may wish to consider CI. I will say as well, GitHub Actions, not sponsored, but my thing's open source, so I don't have to pay for this. So if you want to try out, do not care. Do what you want. But what I'd like to do now is just quickly summarize, because we aren't nearly there. We aren't nearly there. So I'd just like to say, we've been on a little bit of a wild ride together. Usually I'd be running across the stage. There's so much space. I'd like to pretend I can cartwheel, and have a go, but again, the camera won't focus on me. So, mutation testing, let's summarize. It tests your tests, okay? It helps you to beat that good-hearts-law situation where you're turning code coverage into just a metric that you need to get above a certain number, right? It helps you actually think about why you're testing your tests and how you get that insight. If you're going to do this, I would suggest starting small and starting local. Start on your machine and go from there. I would also say when you're ready, if you do have your stuff up in CI, you might want to consider running in a CI system yourself for asynchronous reporting. Finally, I would like to say that mutants themselves, right, mutants are valuable, and they are wonderful, right? Because Henry, right, he's a great penguin. Right, maybe he can't fly, but look at how much we've gained out of using Henry to learn about our code base, and he's helped us so much, right? So what I'd like to say is that you shouldn't fear mutants because you should love them. Thank you very much. Thank you. Thank you. I'd just also like to put on this, if you have any feedback or questions, you can tweet me if we don't get to you today. Also, if you want to steal my code, it's there. We're raffling off an Xbox if you scan the bottom one, so if you want to try and win that, we'll announce the draw in like an hour. Up to you. Those will be on. We'll now do some Q&A. Thank you. Yes.

Speaker 2 [23:22]

Thank you very much for your talk. Thank you. I think it feels like you could talk some more minutes about it So unfortunately we haven't any questions so far while slido So is there any question from the audience just raise your hand I will go to you and you can ask your question directly

Speaker 1 [23:46]

Oh, so many. Wonderful presentation. So how does mutt-mutt perform reasonable mutations versus unreasonable? Is that configurable? Is that automatic? Yeah, so you can't answer the question. Oh, I don't need to repeat it, because we have the mic, right? So I'll repeat it anyway for me. So the question is, how do we choose what mutants I think are configured? Is that the question? How do we choose what mutations we do to our codebase? Yeah, there's also unreasonable mutations such as importing some random package that's not installed. Sure. Right? Like, that's unreasonable. Ah, gotcha. Okay. So, no, that's a really good question. So, inside of Mootmoot, you can choose the types of mutations that you do do. And there are, there's actually, like, it's, again, it's pretty community supported. It's not like a nice thing. I mean, you're kind of hacking stuff. There's a, you know, you'll have a setup.config file which will have your, you know, defaults in that you set. types of mutations that way and also you can I mean MUTMUT by default only kind of has a certain set of those mutations as well so it's not going to like do everything possible it can as long as the code compiles there'll only be a set of valid ones that are done and you can add more but yeah as it is typically we don't see that kind of issue with it hopefully that helps with the question any other questions?

Speaker 2 [25:03]

Yes, another question would be, somehow I lost it, now they're coming in, how do OK mutants that were not caught, I think I don't know, what percentage of living mutants are false positives on average?

Speaker 1 [25:28]

What percentage of mutants are false positives on average? So I think the question, I think what it's getting at is sometimes we don't care about that mutation. You know, like if, for example, renaming a logger, we don't care about that. I think, I don't know, because again, this is something that I learned about last year, and I've started to use myself. This has really been kind of my journey with it. Again, I don't work for Mootmoot or anything. It's not something that I do myself. But what I would say is if that's a concern that you have, what I'd say think about is you can specify parts of your code not to mutate if you know that you're not going to want to touch that and you can also specify entire files and things so you can basically choose a subset of what you want to mutate so hopefully that answers your question I think that's what it was getting at

Speaker 2 [26:09]

Thank you for the answer. I think this one is also quite interesting. So are the mutations reproducible or is there some kind of or some level of randomness involved?

Speaker 1 [26:18]

Sure. No, these are deterministic. Yeah, what you set up, you'll get the same number of mutations every time, and it will basically go through a process to work out what is a sensible mutation, and then it will apply those sensible mutations, it will generate a type of mutant for each one.

Speaker 2 [26:33]

Maybe this is a little bit of a tough one. What was the biggest hurdle adopting mutation testing?

Speaker 1 [26:38]

What was the biggest hurdle? What's the biggest hurdle in adopting mutation tech for me personally? Good question, I mean I had to spend half a day It's really easy go have a go No, I mean honestly like What would it be probably with the cloud stuff? so the reason I actually ended up writing a new flag for MuttMutt was because I was trying to put this into my CI and GitHub actions only accepts every exit code of zero as a success right so if it returns with a zero good otherwise it fails but how much more returns by default is it will give you basically it uses a bitwise or to give you like your two four six eight well it will basically give you a different exit code depending on the types of mutants that were left so you don't ever get a zero if you never had any mutants that that were not caught but that's again that's not how I use movement it's not how you should use movement either or should use these kinds of tools and so yeah that was the The annoying thing there was me basically spending half a day hacking the bash to make it work and then being like, I'll just do it. It was actually easier to just do a PR than it was to just hack the bash. So probably that, honestly.

Speaker 2 [27:44]

I think we have time for one last short question. The question is, 200 false positives to find one bug, do you think is it worth the effort?

Speaker 1 [27:56]

That's fair. I think if we take those numbers at face value of we have to read 200 things and evaluate each one using the same amount of time to find a bug, yeah, you're right, that's very much like I wouldn't do that. That's not worth it, right? But that's not what we're actually doing. Because what we're actually doing is looking at the problem areas and we're identifying the things that we think maybe need a little bit more testing. So we don't give the same weighting to every mutant. If we look and go, oh, this is logging, we scroll through, oh, these are related to the part where I'm logging. I can ignore them. This is related to a function signature. I can ignore those too. I could just comment those out, I'm too lazy. So this works for me. So I would go and say, okay, it depends what you're looking for. I would say, as you do this incrementally, maybe if you start small and add parts of your code if you had a new feature, what I would say to do in that case would be to essentially, yeah, as you do each new feature or thing, this is how I use it, is to then run and see, oh, what's changed since I last did this? Because it does cache your mutants as well. So really, it's a bit of an outlet to go, okay, let's sort through, work out what these are telling me, and then it's just a little bit each time when you actually continue. Hopefully that answers the question.

Speaker 2 [29:00]

Yes, I think so. So I think due to time constraints, we unfortunately have to close the Q&A session, but I think Max is open and happy to answer some of your further questions. And yes, let's thank Max again for his talk.

Speaker 1 [29:15]

Before we do, I have a limited run of Python stickers. If anyone would like one, come and grab one, because we just got them for this conference, so please come and take one. Thank you very much for having me. It's been really cool to be here.