Mock Hell

Speaker 1 [00:02]

Thank you everybody for coming to my talk. I've actually been talking about this topic at various other smaller PyCons and meetups, and it turns out a little bit different every time. So even if you've seen one of those, this one is going to have a little bit of new content. So just a little bit about me. I'm a backend engineer at Quid, and we are based out of San Francisco. Quid provides tools that help users quickly understand large volumes of unstructured text. So what you see up here is an image that shows a topic clustering based on a corpus of several thousand public news articles and blogs. Python is actually my first dynamic language, and before I programmed in Python, I actually did a lot of development in C++ and also some Java and C Sharp, and that's relevant here, I think, because mocking as a technique is actually something that comes out of Java and the context of TDD practices. So one of the things that I think happens with mocking in Python is that a lot of ideas and concepts were mistranslated or lost as this idea of mocking was added into the Python library. And that's one of the things that causes lots of problems with mocking and what I'm going to call mock hell here. um so what are we talking about or what am i talking about when i talk about mock hell um and just like inferno which has multiple levels of progressive torture uh there's also different levels of problems with mocking and mock hell so at the entry level instead of limbo you have things like confusing patch targets everybody deals with that the first time they start learning about mocking eventually you figure it out and it's not such a serious problem as you get deeper into mocking and and use it more you'll start experiencing more of these other problems so there's things like tests that test nothing because they're so over marked problems where you end up reverse engineering mocks with a debugger it's just to get your get somebody else's code to pass or maybe your own code to pass other things I've I've had to deal with or seen is that there's mocks that prevent refactoring because they're just so difficult to change. And that's probably the worst case. So one of the things I started asking myself after seeing this kind of thing over and over is why is this so common, right? And I began to do some research and look around and this is sort of what I concluded. So I think what happens is when people first start learning about mocking they find some mock tutorial right and that mock tutorial looks something like this so there's some business logic core business logic that somebody's written something like total value which is an invoicing application which is summing up some list of purchased items and applying a discount that happens to make a call out to some database like dbread. And when it comes time to test this, you have a problem because you don't want to actually hit a real database. So what you do with mock and patch is you use that to decouple your total value function from dbread. So what this is doing is applying a patch here to decouple total value from the dbread. What that patch decorator, or sorry, that patch context manager is going to do is it's going to intercept all calls to db read and redirect them to a mock object called mock read here. Now, to get this test to pass, you have to set up mock read to actually return data. So at this point, this mock is always just going to return 100 and 200. So you do your assertion on that like normal, and then you also have to add this second assertion to make sure that you actually called the mock with the correct set of values, which in this case is an email address and an invoice number. So pretty straightforward. If I draw that in picture form, it's going to look something like this. So here's total value DB read in the DB. I have my test that's making the call and doing an assertion. I apply the patch to decouple my code from an actual database, substitute in this mock object with mock read, and then when I do the call, I have to assert on the interaction between total value and the mock. So I'm verifying that interaction or side effect. So that's pretty straightforward, and it seems really handy, but some of those problems that happen with mocking don't come out until you start applying that to more realistic or real-world examples and so I have an example here called news check and what news check is the scenario here is suppose you have a large corpus of script public news articles that are stored in s3 for use by your data scientists that corpus is going to contain some bad articles and here's an example so this is just a an article that's the list of the top movers in a stock market from a random day and the way you can tell it's bad is by inspecting the text and if more than 50% of the lines start with a stock ticker symbol then you can tell that it's bad so the test that you have to do is write a script that will scan your news archive to check for those stock quotes articles just in case you're not familiar with s3 I just wanted to talk about some vocabulary. S3 is basically just like a big key value store, and it uses URLs to identify objects that are stored in S3, except instead of hostnames, file paths, and parent directories, they use the terms bucket, key, and key prefixes. So you're just going to see some of those terms come up in this example. The other thing to know about S3 is there might be hundreds of thousands or millions of objects within a bucket. So if you need to scan a bucket, what you actually have to do is get a paginated list of keys and then parse those pages to get the keys out, then download each object by key individually. So here's an example of what one of those pages looks like. Basically, what's highlighted in yellow is what we need to parse out. So we just need to get contents and key, and there's only two keys in this example. So right here, this is what the package structure for our scenario is going to look like. init, config, and stock are not that interesting. They just contain some hard-coded stock ticker symbols, some credentials, and configuration like that 50% threshold. So the first thing we want to look at is this script.py. That's basically just the main. So all that's happening here is we're calling this findBadArticles function with our bucket name. Next thing is core, which is going to attain our core business logic that has the find bad articles function so this is just doing that iteration I described we're going to use the bottom three 80 of s library to create a client and then you have to use that client to get all the pages or the paginated list of keys for every page we got to parse all the keys out of it then iterate over those keys to get the files then once we have each file we can actually check the article and you see some things going on with config and the stock ticker symbols there. Then there's finally the util function which contains all the small helper functions. I'm going to go through this real fast but get pages AWS makes it really easy to get a page iterator over keys. Keys from does that parsing that I just described so this is just accessing the dict. Get file all this does is create a temporary file buffer in memory, downloads the file, does some splitting into lines in whitespace cleanup. And then finally check article is that checking logic with the 50% threshold. So all this does is iterate over those lines in a file and decrement a counter until it hits a limit and that's how you can detect when something is bad. So that's that in a nutshell. The really interesting part, though, is test core. And so what we're going to do is use mock and patch to decouple ourselves from Botto 3 and S3. So we're going to need three patches and three mocks, at least. Get file, get pages in Botto 3, client, dot client, all potentially are going to make network calls, and we don't want that to happen. So we patch those and get mocks. Then we have to set up the get pages mock with a can data to simulate a page and we're going to do that with just a good dot txt and bad dot txt key. Then we have to set up the fake data or the can data for get file mock. So we're just going to simulate a good article and a bad article here. And then finally we do this call to the code that's actually under test, assert on the return value, and then you have to also assert on the mocks to verify that the interaction happened the way you expected it to. So we're just going to verify on get file mock calls here. So it's pretty straightforward but even for something this small there's some pitfalls here. So if you look in the core module and find bad articles, if you do something as simple as change this import statement from bado3 to to import baudo3.client as S3 client, that will actually break your test. And what it's going to do, it's going to break this particular patch. There's a similar thing that will happen here is if you change this function call to get file from using args to keyword args, that's going to break this entire set of assertions here because that call function is not just checking parameters, it's checking exactly how you happen to call that function. There's a lot of other small things here so things like we forgot to do an assertion on this get pages mark that's one of the ways you end up with tests that actually don't test anything so this could have a false positive pass. This particular dictionary structure is coupling this test to S3 because that structure is defined by AWS and for something like this it's very likely that we would want to use the same code and same logic for checking something like a tarball there's a similar problem down here with the file stubs because there's going to be other kinds of bad articles besides stock tickers there might be things like like football games scores for that. So what went wrong here? This is a very simple example, and we followed the same process that we used for the tutorial. And the answer here is, I think, actually pretty simple. The problem is this, is that mocking tutorials in Python are typically going to tell you a story like this. Mocking is for test isolation. for simulating dependencies and avoiding side effects and that's pretty much the end of the story sometimes you're going to see a warning like this be careful about over mocking but it's usually buried in very deep and it's easy to overlook and the problem is that this is actually a misunderstanding of what mocking is supposed to be for mocking there's really quite a bit more to mocking than this. And if this is where your understanding of mocking stops, I feel pretty certain that this is the thing that is leading people to have all kinds of problems with mocking and patching. And the easy way to demonstrate how much more there is to mocking is to actually go grab a copy of this book, which is very well known by Freeman and Price. These are the guys who basically invented mocking as a technique and so they have an entire appendix where they talk about the history of where it came from and what they were doing and what they were thinking while they were doing it they also wrote this paper called mock roles not objects before they published that book which covers most of the same content and for our purposes the abstracts really says a lot right so what they say is mock objects is an extension to test-driven development that supports good object-oriented design. They also say it's less interesting as a technique for isolating tests than is widely thought. So that's exactly the opposite of what you'll find being said in a lot of tutorials. So when I look at mocking problems, what I see is typically something around this, right? Mocking is supposed to be a technique for object-oriented design. You're supposed to be doing it in the context of TDD and it's supposed to be a design aid and it's not really supposed to be just a tool for isolation. So what happened with the news check example is that we did the exact opposite of all three of those things and that leads to problems if you go in a little bit further they say dependency injection is a virtue so if you're not familiar with dependency injection we'll see some examples of this but for now all you have to know is that that's what you do instead of patching in Java because patching is not really a technique that's possible in Java. Another thing to know here is that there's actually different viewpoints on whether mocking is even a good idea at all, right? So this book is also really quite good, but it takes a contrary viewpoint and it says, you know, test using mocks creates more trouble than it's worth, so you really shouldn't do it. And the style that we looked at from MockRolesNotObjects is sometimes referred to is the London School of TE. So the point here now is that there's actually a lot more to mocking than just this idea of, I'm going to use this thing to isolate my tests or isolate my code for tests. And there's a lot more nuance to how you use mock and what you can do with it. So I think the best resource for understanding a lot of those issues, These ideas are on what's this London school, how do you use stubs, how to use mocks. There's a blog post called Mocks Aren't Stubs by Martin Fowler. So I'm going to try to do my best to summarize it here in three sentences. But essentially what it's saying is that a mock is just a kind of test double. And other kinds of test doubles include stubs, fakes, spies, and dummies. I'll try to demonstrate each of these by example directly. The difference between different styles or schools of TDD comes down to how you use these test doubles and how you think about testing. So in some cases you might not even want to use test doubles at all. So if we go back to this original example where we left our tutorial, what we did is patch in a mock. line here is a stub and the reason it's a stub is because it's returning a can value that will always be the same no matter what you call this mock with this particular line is is the mocking part and the reason it's a mock is because it's validating the interaction or it's the other way to think about it is validating the side effect or the call between two functions so earlier I had mentioned this dependency injection. And so this is an alternative to patching. So this is one way to start working your way out of mock hell, or at least patch hell. So if you look at total value, what we want to do is decouple it from db read. And there's no reason for us to really hard code this dependency on db read into this function. All db read needs is some collaborator to fill the role of a data provider. So what we can do is remove that important hard coding and parameterize it with the read function. When it comes to the test, we've now got to provide the mock. So we have to construct it explicitly ourselves, still set up the sub, and then the rest of the test looks mostly the same, except you're injecting it as mock. Now the other kind of test level which is really handy, I find really handy, is a fake. And so we're going to bring it back to this example. Still use dependency injection, but define a fake. And so what a fake is, in contrast to a mock, is that it has some actual implementation and logic that emulates what a production system is going to do. So in this example, I'm just going to use a file system instead of a database. And then when it comes time to test, I just inject the fake. And I don't need to do any assertions on side effects or verify interactions because that fake is designed to fail if it's called incorrectly. And so those are those two styles side by side. Once you get familiar with the difference between using patching and dependency injection and the different kinds of test doubles, you can start to mix and match them in different ways to achieve different effects in your tests. The style on top is mockist because of that mock verification. Sometimes you'll see that referred to as London style and the bottom referred to as Detroit style. The key thing is this side effect verification. So if So I draw that as a picture, it's going to look like this. Change read to expect a collaborator to be provided. For the test, I define my fake read, inject it, and do the verification. For the actual application, I just create a real DB adapter, inject it, and then call it. If I bring all those pictures together on one slide, it looks like this. Sometimes you'll see this term, ports and adapters, and this is a really simple example of that style of architecture. You have some abstract port and you can plug different adapters into it. That also goes under the name hexagonal. Sometimes you'll hear that referred to as clean architecture or onion architecture. The important point here is that dependency injection is a stepping stone to understanding how to build systems in this composable, sort of pluggable manner. And if you pick up a lot of bad habits around using patching, it acts as an obstacle to ever getting to this point and being able to build these kinds of abstractions. So the next thing we're going to look at is that news check example. And so this phrase, listening to your tests, also comes out of TDD. and the basic idea behind this is that if your tests are ugly, that's a sign there's a problem with the design of your code and you need to take a step back, go re-evaluate your design, do some refactoring, then revisit your tests and clean things up. So that's what we're going to do with the news check. So first thing we want to decouple is this article checking logic because it's very likely that we're going to want to swap that out. So this looks just like some of the examples we looked at. We need to change things so that we can inject a predicate function into here. In the actual application, what I'm going to do is use a closure. So I'm going to reuse the check article function we have and use the closure to bind the configuration to that function and then inject it. So this is a little bit of a functional style instead of object-oriented, but it's basically the same effect. So one of the things that happens when you see this for the first time, I think, is they assume that dependency injection is just passing things in. And in really simple cases, that's true. But in more complex scenarios, this can lead you off into a bad path and that's also I think pretty easy to show by direct example here yeah so this there's more to it than than just passing things in right so if you look at find bad articles we've already decoupled to check logic how do you decouple from bottle three or s3 you know what are you supposed to inject all right so I've seen some suggestion that that maybe, like, BOTO3 is something you can inject to decouple it. That will help you get rid of patches in your tests, but it doesn't actually decouple this function from the details of S3 at all. Same thing for client. That does help you get rid of some patches, and it's a little bit better than passing in BOTO3, but it still doesn't decouple you from AWS. So then you might look at something like, well, maybe I need to inject getPages and getFile. But even if you do that, keys from, that implementation is still coupled to details of the data structure, the page data structure that S3 is returning. So this is where you have to start leaning a little bit more on object-oriented design knowledge, right? So dependency injection and mocking are object-oriented design. So that means you have to try to fall back on some of those principles. So who here knows about solid? Okay, a good number. So this is just a set of principles for good design. If you're having problems with mocking, this is something that's good to become familiar with. We're not going to look at all these, but maybe one or two. So if I draw a picture of what our find bad articles function looks like, it's something like this. It's just a graph of function calls. And what we want to do is encapsulate all the things that have to do with parsing S3 or pulling data from S3, right? and then extract that, because I need some abstraction to hide that detail. So what I'm going to do is introduce this archive abstraction and an S3 archive implementation. So one way to think about this is in terms of roles and collaborations. So the title mock roles, not objects, actually comes out of this idea of collaboration-based design, where a program is a graph of collaborating objects and objects are filling roles for each other. So in this case the role is an archive and S3 archive is an object that's fulfilling that collaboration. Another way to look at this picture is to think about it in terms of dependency inversion. So archive is some abstract interface that S3 archive must adhere to, so the detail here depends on the abstraction. So looking at the actual source code, this is the stuff we want to extract. These are all the details that relate to S3. So we pull those out, and then we're going to inject archive. I'm going to take that extracted code, encapsulate it into a class, reuse some of the other utility functions I defined earlier. Now when it comes time for the app, I just construct the archive instance and inject it. So that's a little bit better, but if you look at this find bad articles function, it's still implicitly decoupled to S3 and the reason is because get pages is actually just a detail of interacting with S3. So if I wanted to do something like swap out this code with a tarball it would be kind of a strange interface to code to right if I wanted to create a tarball archive so what we need to do here is extract that responsibility and delegate it in the archive like this right so now archive has just this get keys function and this is what fun bad articles look like just get key get file and you iterate over them and so that will work with s3 or it'll work with a tarball or a file system and so you can freely substitute those things so this is a much better place than we were the testing should be easier if you look at this a little while though there's actually one more improvement you can make and the way you can see that is by asking something like, what would happen if I wanted to use this code with Kafka, for example, instead of F3, which is a totally reasonable thing to do. And the thing about Kafka is that it models data as an append-only file log. So basically, when you're reading from Kafka, what you're getting is a stream of tuples, which is basically nothing but a bunch of key value pairs. So you can't really random access into Kafka the same way you could with a file system or S3. So if you tried to do that here, you actually could make something that is logically correct, but it would have terrible performance because of all the scanning you would have to do between get keys and get files. So the principal code you could lean on here, if you want to describe this, is actually list-golf substitution. So if that's not something you've heard of before, it's this basic idea, it's a fancy way of saying you should be able to freely substitute different instances of the same abstraction. So in order to make this follow list-golf substitution, what I want to do is be able to freely substitute an S3 archive, a Tarball archive, a Kafka archive, or any other kind of archive. So what I need to do is relocate this responsibility for doing get keys and get files separately down into the archive again, right? And so I just add this items method here, and it's going to take care of that sort of dance that needs to happen between the client, Botto, and S3. what then happens is find bad articles becomes this so the real role of the archive is actually just as a tuple stream and this code becomes very straightforward to test I can just pass in a data dictionary because everything is faked I can use a simplified check logic and fake data do the test, don't have to worry about patching or mocking there's no side effects. So in closing, I just want to look at this tweet. So Hunnic is a Python Software Foundation fellow. Also happens to be a Berliner, which I didn't know at the time, but he wrote this tweet about four weeks ago. And I think we covered most of the ideas on here. So monkey patching, SRP, which is the essence solid, talks a little bit about interface segregation. If you go on in supply, there's mocks and mocks and different kinds of fakes and test doubles. But the thing I want to focus on is this idea of the shortcut that can cost you years. There's a couple of different ways to interpret that, but I think the one that's most important is in terms of personal development because if you spend up a lot of time building up habits around, or dysfunctional habits around mocking and patching, it can actually prevent you from developing a lot of intuition and skills for building modular software and more complex systems. So if you're having a lot of problem with patches, just try to stop patching, and it's going to be a little bit painful, but you'll probably learn some new things. If you're having a lot of problems with mocking, mock less. Consider using other kinds of test doubles. Maybe use this pretend library that's being referred to up here. So I'll just put my contact info up here and some resources that I refer to and should be useful. Do we have time for questions? Sorry, we do not have much time for questions because we are already over time. if you don't have anything else I'd like to thank you for this great talk