FastAPI and Celery: Building Reliable Web Applications with TDD

Speaker 1 [00:04]

So I'm Avanindra. I work at HelloFresh as a software engineer. And today I'm going to talk about building reliable web applications using test-driven development, using FastAPI and Celery. In my day-to-day job, I mostly use Flask, FastAPI, Celery, a little bit of React to build internal tools. tools. And today, our goal is to yeah, my goal is to introduce these concepts with the help of test-driven development. Two years ago, when I joined the company, I didn't know much about test-driven development. And when I learned about it, I was quite the concept was a little bit counterintuitive for me. And since then, I have fell in love with it, and I hope I can convey the same feeling to you. For some reason, my next button is not working. A little bit off. Yeah, hopefully. Yeah, all right. So let me begin with the acknowledgments to the creators of Celery, the community around it, and to the Python community in general. Since I'm not sure, given that it's the last take, that everyone, will you remain awake until the end of the talk, let me present the key takeaways right in the beginning. The goal really is that the test-driven development is one of the best tools or methods to design your system as it makes the feedback cycle faster. For designing anything or improving anything in general, what you need the most is feedback. And that's what test-driven development brings to the table, faster feedback. And the second takeaway is kind of going to be that dependency injection is going to make your code more testable. And most, I mean, although we will discuss specific case of FastTips PI and Celery, how to test them, how to write them, but these are the kind of two broad principles which apply to any software testing in general. Coming to the outline of the talk, we'll go a small introduction of what TDD is, what test-driven development is, how to set up our development environment using Docker, how to build an API with FastAPI, and then we will offload some of the job to Celery, long-running tasks, some conclusion, and further resources. I will try to do some live demo. see how that goes. So, this is what astonished me, when I learned first about it, that you have to start by writing a failing test. Well, if you write a test, of course it's going to fail because there is no code to support it. So, that is the first step of TDD. And then you try to write just enough code to make it pass. You don't implement other features which you have in mind. You might have thought of so many features which you want in your app, but no, you're not allowed to write any of it. Only just that much which makes your test case pass. And then the most important thing is to refactor both your test as well as your code. And just keep doing it over and over again until your software is kind of complete, until all the tickets which your product manager has given you is complete. So that's the summary of TDD, but why should we write the test first? This was really astonishing to me, and it is a better documentation? Not really. I mean, if you write your test afterwards, still serves as documentation. It doesn't change anything. Does it reduce the cost of fixing defect? Also not, because if you write afterwards, still the cost is lesser when you visit like five months later. The effect is the same. Really, the code quality, does it improve? Maybe. There is a maybe here because of the feedback cycle. But really, I would to stress again that the faster feedback is the key here, at least in my experience. Also, I don't want to say that TDD is the silver bullet here. There are limitations, like there is a little bit of overhead involved. Learning curve is also there, because no matter how much I tried in the beginning, I couldn't write tests first. It was just a matter of habit to write the code and, you know, build, do something useful, you know, with your time. And really it works very well if you use other extreme programming techniques like peer programming, code reviews, et cetera. I mean, I can't talk much about the theory because, well, I don't know. But also because there are better books and resources available for it. The first book by Kent Beck, excellent book. Also the second one, these two are classics, but they both have examples in Java if you don't like them. The third one is exclusively for Python. I strongly encourage all three books. Yeah, so what is the goal? I mean, I don't want to present just the slides. I really want to get going with some code here. What What are we trying to do today? We are trying to implement an order service. What does the order service do? It takes some input, some payload from a front-end or another service, takes and tries to do some processing which our stakeholders have asked, right? We'll come to what we do in just a moment. Let's briefly see what front-end payload will be and what information product service gives. So this is a microservice landscape, there is some front-end running, which will send a payload to order service, and it will try to fetch information about the product from another service, product service. And we will try to see how that goes, right? So this front-end service can give you the name of the customer, the email, some shipping info, payment info, and what are the items the customer has ordered, right, along with any discount code on the quantity. And the product service is kind of, yeah, it has the ID which the customer has sent you, the product ID there matches with the ID, the name of the product, the currency, the price, all the valid discounts, description, whether it's in stock or not. Things like those. A typical product service, right? But what do we want the order service to do? Well, I don't have a lot of time, so I can't build everything. But we want the order service to calculate the total price, something pretty simple. We don't want the front end to send it because, well, users will exploit it, right? They can just send arbitrary number as total price, possibly zero. So we want the calculation to happen in the order service. And just return a confirmation ID that, well, your order has been placed. And send an email to the customer saying that, well, the order is placed, and if the payment goes through, then the item might be shipped. So this is the overall feature request which we have got. Now we need to break it down into pieces of how we can achieve it, right? And then we will try to demo it one by one. First, the order service needs to talk to product service, right? For that, we will build a class which can get an item by using its ID. Since we have to build an API, we need to deploy a fast API application just with a health checkpoint, endpoint which checks its health, and an order endpoint to process the order. We'll do some refactor afterwards, deploy the Celery worker, and offload the emailing task to Celery worker. Usually you wouldn't offload emailing tasks to Celery worker, probably background task from FastAPI is more suited to it, but in this example, I mean, we'll just offload it to Celery. So let me start with the demo, and I have to move my VS Code into the other screen. This is going to be a bit challenging because I can't see my own screen. But let's see. Let's see. Let me zoom. Is it visible? Can everybody read it? But I think it's too big. So maybe we decrease the width. So as I said, we start I mean, I have set up the virtual environment and everything because of course we won't have enough time. And let's start with just one test. The test is pretty simple, there is product, we are planning to build a product service, and product service is returning everything as a dictionary, and we really don't want to work as dictionary, so what the product service is going to do is just convert that dictionary into an object, that's all the assert statement is saying here, that I want Whatever the output is coming from product service, I want to convert it into a product object, right? And this object would look something like this. This is from Pydantic. It has all the appropriate props, all the appropriate properties. And the discount is essentially a list of discounts. And we want to assert in the end that the ID is the same. We could do more, and we also want to be sure that the discounts are also in the right format. Of course, if we run this test now, we will get the... This is getting a little bit challenging. Let's hope my... Of course, it will fail. Here what is important, it fails in a specific way. It says that we cannot import product service because it's not implemented yet. So yeah, okay, even before we go there, maybe we should first take a look at how it looks like how our product service looks like. If I click on it, there is an API running. How this has been set up, if I take a look at my Docker file here, I have set up a small wire mock which mimics a product service. So, I'm not going to in detail with how this wire mock works. Suffice to say that it will return for some given valid IDs, it will return the description of product. And all I want to do is convert this JSON into a Python object which I can play with, yeah. So what I can do is I can just write an init method and a getById to enable that, right? I can make a request to get the object, and I can just convert it into an object. And that would that should make the test pass. Yeah. Usually, it's not enough to write just a passing test. Probably we also want to test that an invalid object throws up an error, right? And in this case, I want to raise a value error if some invalid IDs passed in. Currently, if we run the test, of course, it will fail. Yeah. I get this error. I can try to read it and understand what it does. I mean, ideally, I would put a break point and would try to understand what's going on. In this case, I just know that, yeah, the service is turning 404, and it's unable to do response.json. So, yeah. What I can do here in this particular case is whenever I so even before we actually implement Maybe I go ahead and go with what I said about refactoring. So we have made from red to green. Let's refactor it. I don't want to make a request in this function. Otherwise, it's doing too many things. I probably want to have another hidden function which makes the request. And let me disable this. This is the problem with demoing. So I have refactored it in a way that instead of making requests directly in this function, now I'm using get function. And if I run it, the test still continues to pass. And as soon as I... Now I try to build a new feature saying that it should throw the value error when an invalid ID is provided. it, right? And in this case, whenever 404 is reached, I can try to do this, right? And now the test passes. Again, I have not shown the whole path because ideally what would have happened is this product service should not have come as an argument. I should have built it, right? But now I see there is some code duplication going on between our two tests. And whenever there is a code duplication, it most likely points to that we need to refactor it out. And for this, PyTest provides us, I mean, we could just define it up here, to be honest, this product service variable but the preferred way in pytest is to write what is called fixtures and the way fixture works is that function you can define a function either up here and mark it as fixture like so with a decorator or you can put it in a special file called conf test In this case, this function, productService, will be evaluated before it's called, right? And that's what you were seeing earlier, that I can, you know, that I can run this fixture sort of here, and I can put the same fixture here, get rid of this completely. And all right, and I hope that even after my refactor, the code continues to work. So that's what we have done here, red, green, and refactor cycle. So we have refactored a little bit of the code, but during the refactor cycle, it's super important that all our existing tests should continue to pass. So that was a simple demonstration of building our product class, and if we go to the next plan that was to deploy a fast API application with a health endpoint. How do we do it in a test-driven manner, sort of? So before writing any other test, this is what we want to achieve, right? Again, there are some fixtures being used here, but since I have introduced fixtures, probably I can already say that you can imagine that somewhere I have this fixture, hopefully Yeah, testClient, which comes from FastAPI, which you can use. And the idea is that, yeah, that this testClient, it's almost like request object. You can do .get in it, and I want to assert that status code is 200, and this is equal to, this is the message, right? This is what we want to run. The problem is I don't want to run FastAPI directly on my terminal, I want to run it in a dockerized environment. So what I want to be able to do is to say docker-compose exec and name of the service, let's assume we will call it backend, and pytest and really this path. So what it essentially means, I'm telling Docker to run inside the backend container, pytest and this. Of course it will say that the service backend doesn't exist. And with this error message, I will go to Docker and I will sort of enable it, right? And then I will go ahead and implement the health endpoint. I'm not going to walk through the entire process here, but because writing health, yeah, now I'm getting 404. And I can imagine that in my API, really not even there, Just in my main, I have to define it, create a fast API instance, and just return it, and then it should pass. So that brings us to our second point. So again, we have gone in a test-driven manner. We have first written what we expected. We expect a health endpoint with simple payload, and that's what we are getting, right? Now the third step is to build an order endpoint to process the orders, right? Now if we see this checklist is what we want to do, right? We want to calculate the total price, we want to get a confirmation ID, and we want to send an email. And we want to achieve all three. So that is what our test should say, right? That given an order data, like a payload which we saw in the presentation earlier, okay, this is not the, yeah, let's say that first one. an order data. I want an email service, so this is probably let's ignore it for a moment. Let's say that we have this order data, and I want to go to an order endpoint, provide this order data, and the response code should be 201. I probably don't want to do anything else at this stage. I just want to implement a bare minimum of things. Right? And if I go to my views now, I have introduced, so I would go ahead and in a very painful manner, I will use the product service, let's forget the email service at the moment, we will use the product service and calculate the total cost by looping through all the products and applying all the discounts, sort of. But even before implementing, actually, we have not shown the red stage yet. We should have first tried to run this test, and it would have, of course, failed. And if you see, it says essentially that the order service is not implemented well. We went ahead of ourselves. I didn't... Yeah. So if you see, in this one, what we are doing is we are getting the order data, and we are calculating the cost, and we are just returning the order response. In FastAPI, we provide type hints for our payload as well as our response. And that ensures that our data is validated. So we must define how our input should look like and how our response should look like. And this is all based on type hints. So this is exactly the payload, which we saw earlier, right? The ID, the customer name, and list of all the items. And in the response, the only thing I want to add is the total cost. So it gets the order data, and it tries to return the total cost. yeah and once we have done all the calculation so yeah what what we want to achieve here is to return the total cost and let's see if we get some more errors it says API router is not defined that should come from fast API and now it says order service is not defined yeah that might be happening because in the service, I might have imported the order service, which is not yet implemented. And this one I already removed it, but now it says no tests. So let's see. What I want is this test over here. I definitely don't want to run this test just yet if I okay so it says here okay probably I have overwritten it in the PyTest.ini, not sure, sorry, sorry, so yeah I just copied the wrong path, I guess. And now it fails in the right way. Yeah. So it says that 404, and this is just because I have not imported the router in the right place. The problem is I can't see my own screen, and I have to see the other screen, so I can't see everything super clearly, unfortunately. And, yeah. So if I go to main.py, It should exist somewhere, yeah. So this router needs to be added. And imported. And then it passes, right? But we had decreased the scope of the test quite a lot. We were not checking if the email is being sent. So the thing is, we don't want to use a real email service to send an email during the test, of course. So we would like to mock it. And how we can mock it, so the fast API comes with this depends, which marks it as a dependency, which can be overridden during our test. And what I can say is that I create a mock email service, and I override the real email service with this mock one, and then I just assert that the sent email has been called once. If you want, you can also assert that it has been called with the right parameters, if you want, and so on. And of course, if you run the test, that should fail because, and it says, if you see here, it says that it, oh, it says it does not have the ID. So I mean, this is of course wrong, we want the confirmation ID, so we would say the confirmation ID should exist, rather. And now it says the mock to be called once, but it was called zero times. And then if you enable this small line here, when you actually send the email, now that should pass. And similarly, the total cost calculation. But this would not be the preferred way of doing it because we really don't want to put business logic in our views, in our router. So ideally we would want to create an order service and then send that and use that to do the total cost calculation, just like we are sending email via an email service. So the real way of doing it, the real test might be to use the same order data and use order service to do it, but due to lack of time, let's not refactor it just yet, because I think I have run completely out of time, as pointed out by the chairman. So I will stop it, but I will share the slides and the repo with you. I hope I have not disappointed you all with all the technical glitches and inconveniences. So sorry for letting you down. But I will share the GitHub repo and the slide in the channel with you all. Thank you very much. So we still have a couple of questions. When would you use Celery for long running tests instead of fast API background services or Huawei? When would you use Celery for long running tasks instead of fast API backend services? Background services works just fine for small tasks, but if it's really long running, I don't want to do it in a small thread. I would rather offload it to an independent virtual machine in a Celery. That would be better, at least in my opinion. Next question, how can I write the test for a method if it does not exist yet? Yeah, so I mean you would write the test assuming certain API in mind. And the advantage you will get is that you will assume this is how my function should look like. So if the function is not good to use, you know, if it's not very easy to use, you won't be able to write your test easily. And that would improve the API of the test you are going to write. Yeah. Let's do one more question. When are you in the middle of a project? How would you recommend to switch to TDD? That's a tricky one. I mean first you should write enough test coverage for the existing code base Little bit at least and then from the new functions you might try to use TDD But it might be hard on on a legacy code base to be honest. I have tried it and it's really hard Okay, thank you very much. Thank you.