Achieving Resilient Code with Integration Tests

Speaker 1 [00:02]

I'm Alexandre Figuera and I will speak to you about how to achieve resilient code with integration tests. I changed the title of the talk like some weeks ago and so on. On the paper it's not the same title, but it's the same talk that I gave at your Python in July, but even better. So don't be afraid if it's not the same title. So I'm using Python since four years now. I'm French, but I moved in Berlin since two years, and Berlin is very nice. And I work in a company, we are making web hosting, and I work in a department where we are managing OpenStack Cloud. And so basically we operate the cloud and develop some program to automate our cloud. But I'm here to speak about integration tests. And when we hear about integration tests, most of the time, people think about mocks. Because mocks are really easy to write. So let's say I have an application, a web app, with a payment system. And of course, when I run the test, I don't want to make real operations on my payment system. So I will mock it. And it will be easy. It will be fast. But there are many disadvantages of using mocks, and the main drawbacks of using mocks are that your test will become dependent on your implementation, refactoring will be harder over the long term, and every time you will change the real code, the mock will hide these changes. So it's very theoretical point of view, and the purpose of this talk is to see exactly in the details why mocks are bad. And there is a lot of books and articles online telling you, yeah, you have to write a lot of unit tests, so you write a lot of unit tests, and you have to decouple your code, and you have to not have side effects, So you write a lot of mocks, and then there is a Chicago school or London school, TDD, and so the Chicago school don't use mocks, the London school use mocks, and there's a lot of kind of dogmatic advisors telling you use mocks. And my talk is about why you shouldn't use mocks. And so let's have a look with some real code. So, I wrote a small Jupyter notebook, and it's based on a hypothetical scenario. Let's say I wrote a library doing very complicated stuff. So, you can say hello, you can say world. And you saw this library, it's so nice, you decided to use it, of course. And so, you want to say hello world, so you call my library, say hello world, and okay, no thing fancy. Of course, you want to write a test, so you write a very basic test, you execute the function, you check the result, if there is an error, you display it, and if it's too successful, you also display it. So you run the test, and it's good. But you want to use mocks, because mocks are really more cool. You implement your own mock system, so how a mock behaves, basically. Let's say you want to mock a function and return a value, so you will get the original function, this is your mock returning a value, you modify the function itself, here you execute your test, and after you clean your mess and you set back the original function. So in your test, before it was looking like that, very simple stuff, now you mock it. So I decided to use a context manager here, so I use it like that, and now I don't check only the result. So before I was checking that I was returning hello world, but now I am also checking that my function, this one, is using the sayHello function and the sayWorld function, so I am checking the behavior and the implementation. That's why when we use mock, we say that your tests become dependent on your code. And the purpose of a test normally is to check the behavior of a function, not the details. So in this case, there is a new test, it's still good, Perfect, but one day There is so much success about my library. So I decided to improve it So before it was just saying hello world and now it's saying hello world or bonjour world and by default He's saying bonjour world now problem is that You saw I publish update. Okay, you update Your requirements in your project you run the test Everything is fine. Tests are still successful. You push in prod, then someone is calling you during the night on call. There is a problem, then your project manager comes to you. Why there was a problem, then your boss comes to you. You are the culprit, then you will get problems to negotiate salary increase, all this stuff. Because you used a mock in your tests. Your tests are green, but your code is kaput. Because you use a mock, and if you don't use a mock, you will have seen an exception. In this case, you will have seen the real result, Bonjour World. This is one of the main disadvantages of using mock, and that's what I said before. You are hiding the errors and the changes in the original code when you use mock, and also your tests are now dependent on the implementation. So your implementation is not just a black box, you know exactly in your tests what's happening in your code. And that's why I really advise to not use mocks most of the time, unless you have a very good reason, like let's say you want to simulate a network outage, you will not go in the data center to unplug a cable, run the test, and go back to the data center. So in this case, yeah, it makes sense to use a mock, there is no other alternative. But in other cases, there are other alternatives, and we'll see them later. But why do we keep still using mocks, mocks, mocks, mocks? So as I said, there is a lot of articles about mocks, a lot of libraries, and most of the time we keep to use mocks because it's very fast. Like I write my feature, I write a test, unit test, I mock everything. It's very fast. feature gets merged very fast. I get done a lot of tickets very fast. So at the end of the year, my boss say, it's very nice. You are very efficient. I give you more money. You are happy. So you are in this loop of working fast, getting money, and it's very nice. And you do not have to think about the code design. It's for other people who will replace you two years or three years later when you will leave the company. It's not your problem. Refactoring the codebase for long-term maintenance, not your problem. You will not be there in the long term. Replacing outdated libraries, not your problem. Fixing tests is a problem of someone else. So that's why we keep using mocks and mocks and mocks. But we will see that if you want to have a really clean codebase and help people who will come after you in the company to help them to have a more good time refactoring the codebase, there is a proper way to write integration tests, so please, really please stop to use Mock if you can, and it's really bad. So that's what my point, but okay, is there an alternative for Mock? Yes there is. Integration tests are built around two main ideas, so dependency, injection and interface testing. So it's a very complicated word, it could look afraid at first, but it's really really easy. So in practice, how does it look like? You have your test, and your test runs to check your application, And maybe you will need to depend on external systems, like a payment system. So what you want to do is that you don't want to, in your test, to communicate with the real system to not have side effects. So you will use a fake object. That's what we call, you will implement a fake simulation of the real system. And you will use it in most of your tests. But from time to time, the real system, which is probably shipped into an external library, this library will get updated, and you will have to check that your fake implementation is still up-to-date with the real implementation. So that's why you need to run interface tests to check that the fake and the real ones are still behaving the same. So, again, let's look at some code examples. So, let's say I still created a super cool library, I have so much time during my weekend, I make a lot of cool libraries. One is to manage your bank account, so you can withdraw some money and from time to time check the balance. Okay, you want to buy something, so you initialize your bank account to connect to your bank account. you take some money you give the money to someone or to some online shop and in exchange you get something okay so now you want to test it of course so in your test you initialize a bank account you check the balance at first you buy something something at 25 euros and you check the final balance well the problem is that you run this test during one month and at the end of months you saw that on your bank account there was no money anymore. So big problem. You decide to write a fake bank account. So to use this fake in your test and not the real one. So you know how the real bank account behaves. So if we look here, it was like that. There was a withdrawal method and a get balance method, but we don't know what is inside. And it's not really our It could do very complex stuff, we don't care. The logic is very simple. Basically, I can take some money, and so I will make some withdrawals, and I can check the balance. So I take the balance, the initial balance, and I can subtract the sum of withdrawals. And it's how the real bank account library that I gave to you is supposed to work. So, in your test before you were initializing a real bank account, well now you will give instead the fake bank account. This is the main point here that in your real code, if we go back a bit at the beginning, this was the real code, your function, you initialize a specific class, the bank account. But in your test, now you want to use a fake. So if you were using mocks, you could have mocked this class. But now you want to use a fake. And that's why you will inject the fake into your code. And it's just to pass as a parameter. So now your function will look like that. You want to use a fake in your test. So by default, if you give a parameter here, it will use a fake. otherwise it will use a real bank account and then the code is exactly the same and by making dependency injections i can easily give fake objects from my test into my code this is a global concept and this apply for test but this could apply so for my production system i can have a real bank account for my test another one for my staging again another one so it's very flexible A problem now is that you have a fake implementation, which is used in your test. But if I release an update of my library, you need to check that your fake still behaves the same. Otherwise, you have the same problems with mocks. So you write a basic test to check, for example, the withdrawal money method, and you just check the balance, withdraw some money, check again the balance. And at the end, you will run the interface test like that. You will run this method with the fake account and the real one. And the two tests should be green. If one is red, that means the two systems behave differently. So your test will fail. You fix them and then you can deploy your fix in production. So you don't have any more problems in production in this way. So as we saw, tests are not dependent on the implementation anymore. I really want to highlight this fact. So if we go back to the mocks test, we could see that we are here to say, I want to mock this function and I mock to this one. So that's what I mean by test dependent on implementation. That's what we know that what is running inside the code. So now this is not the case anymore with the new integration test because now I just, in my test, this is a new one, I just inject a dependency here, and then I don't know what my code will do with that. It's a black box for me. So now my text, it's a black box. i'm not dependent anymore i can change the dependencies as i want between environments and refactoring will be easier on the long term we'll see later why with more examples of course the drawback as we saw it's a lot of work at first and we have to choose in fact between more work at first or more work later during refactoring This is the main battle between mugs and proper integration tests in fact. So we saw some theory, it's nice, but let's look at a real example. So let's say I have a static website, I want to upload it to the cloud and I am using OpenStack for that, the object store API from OpenStack. So what is the object store API? Basically you you can create a container in the cloud which is like a directory and in this directory you can upload some objects which can be HTML pages or CSS still sheets and this is a container in the OpenStack vocabulary and let's have look to the official OpenStack API. So this is how it looks like. I can create containers for example I have OpenStack object store object and inside I can create a container I can delete a container I can get the container list okay fine so I don't want to run my tests on a real cloud so I will write some fakes. So this is a fake. So this is my object store. It's called a store but we are not enter into the details but it's like a fake. And I can get the container list, I can create a container and I can delete a container. We don't care about the exact implementation but basically when I initialize my object I just set up some dict and then I will add some if I create a container I will just add a container to the dict So on the real library OpenStack OpenStack library, I don't know how it's working behind but in my fake It's very simple. I just use built-in built-in Python objects How Now look my interface test. Well, for example, I I want to check that when I create a container on a real cloud and with my effects, I have the same result, so I will just give a name to a container, create the container from my object store, and I check something. And I like PyTest, so in PyTest you can modify some hooks to modify the behavior of PyTest. And for example, I say here that in the test cloud module, which is where all my interface tests are defined, for every test, I want to run the test with my fake object store and the real one. And for all other tests, I just want to use the fake one. So this is for the code part. Now, let's see how does it look like in live. So this is my website repository and let's say I will run the test with PyTest. So I will run my interface test. Let's run them. So tests are green, it's fine. And here we can say some tests have been skipped. And in fact, these tests are run with my fake. These tests are run with a real implementation. Why PyTest skips them is because I have to give some credentials on the command line to connect on my real cloud. And I didn't give them here. And because I don't want to give them to you neither, I created a bash alias to run the test so now it will run on the with my fake and with my real cloud and as you can see it's taking a lot of time too much maybe I hope network is good let's see maybe 10 seconds It doesn't seem to work. I will try again. But basically, now it's working. As you can see, it's very slow. The test with the fake implementation was very fast, and the tests with the real one are very slow. That's why we don't want to use the real implementation in the unit test, because it's too slow. I'm not sure that you believe me really when I tell you that I am running on my real cloud. So I will insert a breakpoint in my test. And I will run the test again. Okay, so now I am in my test. This is my test function. I have here a cloud object. I can have a look to what it is. It's a connections tab. So it's my fake object store. I run the test, tests are green, and now it's running with my real connection, so I am using Rackspace in this case, and this is running with the real one. If I go on my Rackspace account, we will see that a container has been created in my test. It should be displayed online, but it's a bit slow. Maybe I will show you later. but basically now the tests are in my most of my tests I am using my fake implementation and in my interface test I am using both implementation so it's too slow I will skip this part so it's a lot of work really are there some alternatives yeah there are in fact so the first one if you do something very clean, you write the fake and you make it fast testing. If you want to go fast you write full mocking at the bottom and in between you can make a fake injection with interface testing but a minimalist interface testing or you can also inject mocks and the higher you are in the pyramid the more work you will have to do at first but the lower you are and the more work you will will have to make coverage. And the lower you are going, the less coverage you get. So it's another use case. I have some movies on Blu-ray, and I want to convert them to MKV. The problem is that Blu-ray is up to 50 gigabytes large, and I cannot have an open source library that everyone can download and if I want people to run my tests in my library, I cannot upload on GitHub a real Blu-ray like 50 gigabytes large. So it doesn't make sense. So I cannot check on my computer. On my computer, I can run the test against one of my Blu-ray and someone on the other part of the world can run the test with one of his Blu-ray. But we will not have have the same result in the test. So we cannot check exactly what is the result in the test. So an alternative is to run the test, but not checking exactly the result, only to check that everything is running and there is no syntax errors. And how does it look like? So again, I wrote some fakes. I wrote some interface tests. But when I run my test, now I don't check. For example, if I want to get the number of playlists in a Blu-ray, I don't check the exact number. I check only that there is a result. So as we can see, it's not very detailed, the test itself. And that's why I say when you go very down to the pyramid, you get less coverage. Because you don't check the exact result, you just know it's working. And it's still better than not writing tests at all. But otherwise, the principle is the same than with proper integration tests. Now, let's see something requiring even less work. So it's a lot of work to write fakes and interface tests. And if we want to test complex systems, I don't know, let's say you are making requests to Google and want to check what pages are written. when you make a search, well, you cannot simulate in a fake the PageRank algorithm. You don't know how it works, and it will change between computers. So for that, you will mock the result. But you can still inject a mock in your code to make your test still independent of your code. So again, let's have a look to some real code. So, I have some functions doing some requests online, three different functions, nothing fancy, just get some text. I write some tests for that, so I have to mock, I am using requests as a request library, so I mock it in my tests, so I mock the get method, and then I retrieve some text, so I have also to mock text return value. So, again, I am checking the behavior here, check the result, but also the implementation. But then I go to a conference, someone tells me, you don't have to use unit test mock here, you can use response source, which is a library to mock a request. So you decide to use it, you refactor your test, it's still the same stuff, it's just two lines instead of three. But well, now the buzzword is async, so you have to use async to put on your resume, so it's so cool. And if you are cool, you will get more money. So you use async, so you change request and use request, so you modify your function, so you use request instead of request, it's the same API. And because responses library doesn't work with request, well, you have again to refactor your tests, because your tests are dependent on the implementation of your code. So you're right again, refactoring is a lot of work, as you can see. And what if you have done something differently, like injecting a request as a parameter to your function? So at the beginning, we were using requests directly like that, but now maybe you can do something different. Instead, you inject request as a parameter to your function. And then you start to write a fake, to not make a real request online. So it's just a fake HTTP client simulating the request library, nothing fancy. And in your test, you can say for this URL, I want to return this result. And here, I'm just checking the behavior of my function. If we look back in my test, previously I was checking the behavior, like here, but also the implementation. Now in my new test, I don't check the implementation anymore, it's just a black box. I just give a client and I give a fake response for this URL, but I don't know what my code will do with that. And so I was using request, let's say I want again to switch back to request, I just have to modify it here. Implementation stays the same. And if I want to use request future, which is something even more cool, let's say refactor just a bit to make my code cleaner. And it's the same APIs and request basically, but now because it's async, I have to call the result method before. And my tests don't change. I refactor three times here my code, and because I use dependency injections like here, I don't have to change my tests, because my tests are completely independent of the implementation details. Here, I just have to change my fake object, because now my code is running that. So I add a small method here. So I just change my test configuration in one place, but my hundreds of tests, I don't touch them at all. That's why when you don't use mocks, refactoring is easier. I wanted to speak about running tests with Docker. I don't have time for that, but I made a workshop about that at EuroPython. And basically, you can go on my slides on GitHub. I put the repository online, and basically what I mean by having resilient tests is just having tests independent of the running platform. So I run my source code and my tests in an application container, and I can put breakpoints in my code to stop my tests and debug them. nothing fancy, but that testing architecture, there are many ways to do it with Docker, and there is no best practices book. It's really hard to find out what is the proper way. And in my company, we found this testing architecture after six months. And if you want to know more about it, just let's have a look. So in conclusion, I don't have time for question, but time for conclusion. You can ask me questions in the hall if you want. But basically, please don't use mocks unless you have a very good reason because it's hard to simulate a complex problem. Because mocks, it will be very hard for your other people who will replace you in your company. It will be a lot of work for them to refactor the code base. On the other hand, interface testing is a lot of work. Dependency injections, you can use them anyway. It will really improve your code API. And at the end, just ask to your colleagues, to your boss, to your product owner, what you should test. And then you will find out, OK, maybe I need 100% coverage, so I will test proper integration tests with interface testing and all this crazy stuff. Or if I need less, I just write more and maybe it will be fine. So ask around yourself and don't have too much headaches. So that's all, don't have time for questions, but if you think your boss doesn't give you enough money, just call us, we will give you more. Thank you.