Writing reliable software while depending on hazardous APIs

Speaker 1 [00:08]

Welcome, thank you for being here. I'm really happy to see the room completely crowded, and as we are in Germany, everybody on time, it's wonderful. Yeah, so today I will talk a bit about our experience writing, trying to write reliable software while all the dependencies, external dependencies we are using not always behaving exactly like we want. So my plan for today is to introduce the topic quite quickly, tell you a few stories that happened to us or a friend of us about real outages, seeing why I think naively that that should be a simple topic, but it's not, And then we'll present what we are using for the 12th, 18th, last month to try to solve this problem differently. I will try to go quite quickly in the first topics to keep a bit of time for demonstrations in the last part and then a bit of time for the questions. So just a quick poll. Who ever had to deal with a bug that was not caused by your system but an external one? And just by curiosity, is there someone who never had a bug caused by an external system? Okay. So nobody. Okay. You proved my point, but thank you. I was not sure. So talking with friends, especially one who is head of infrastructure at a bigger service company in France, we analyzed a lot of post-mortems. So post-mortems, those who don't know, are little documents that say, hey, we had a problem, we resolved it like this, and it was caused by that. So he analyzed the root causes of hundreds of post-mortems he had at his company. And he came to the conclusion that the two most common outage causes are, first, by far, human errors, and there's probably things we can do, but that's not the topic that interests us now. And the second one is external services failure. And failures mean a lot of different things. But mostly, all APIs are not equal. We all dream of very stable, GAFA-like APIs that always respond like you want in very good timing, like it's a racing car. But in fact, I present myself just after that, but in fact, when we're using internal systems to companies, to client companies, APIs were most often thought after the product, added on top of it by people who are maybe business experts, but not really neither web, neither API experts. So we can have some issues. So hello, I've already been presented, but I will do it again quite quickly. I use Python since 20 years now. I work both in a developer context and in a sysadmin context, whether 20 years ago it It was Linux machines built by shell scripts. In the middle, it was maybe Chef and Siebel. And now it's mostly Kubernetes and Helm. But yeah, it's still the same thing, having machines that work. And I also do development, so I've seen both sides of this. And I created a small company called MakerSquad. And our job is to build software. We say from zero to market. Before the idea, we help people, we help clients to understand what they need or what they don't need. And we help them conceive, design, and build this, and operate it. And as we are mostly working in business to business, B2B market, we work with a lot of enterprise APIs. For example, ERPs, Enterprise Resource Management Systems. And again, all the APIs are not equal. So what did happen to us? Well, first example that came to my mind was an adage we had on the said ERP, so some software that is managing the planning of all the operators. And in this instance, we had the truck drivers who are in charge of deliveries and maintenance. They had daily plannings. And we were consuming the same API as the mobile application. This API went down one day, and the symptom was mostly timeouts. So it was not really down, but it never responded. For us, it's down. But it even wasn't down. It's down, but after 30 seconds. So you have to wait to know that it's down. And the direct consequence of this API going down was that there was thousands of truck drivers that for two days couldn't work. At this time, we reacted because we had monitoring. So we've seen that there was problems with the, it's not status codes, but with the requests we were making. The direct consequence it had on our system is that we were using Celery to schedule background tasks. We have what's called an horizontal pod autoscaler on Kubernetes, which reacts to some metrics to adjust the number of workers, of salary workers, that are actually working. So as no worker could really work anymore, really work fast anymore, the amount of items started growing. So the pod autoscaler decided that it needed a lot more pods. So it spawned workers, and workers, and workers. And each of these workers, of course, were doing queries, API requests, to the underlying API, which was not good for them to recover. So we had manual action. We switched the scaling of the salary workers to zero, accepting that all software will be degradated for a little while, and we let them fix that. But for two days, 1,000 people could not work. What happens quite often also is what I call the ping-pong game with two contractors or two different parties that are responsible for some part of a system. And each one is saying, it's your fault. No, it's your fault. We had a payment processor arguing with us about some JSSID cookie that we were setting. JSSID, after a quick Google search, is a name of a cookie very specific to the Java world and spring, I think. So I was a bit doubtful about that, because we have only Python on our servers. But yeah, if I don't know for sure, I could not really say, no, it's not us. It was not us. But the chat can go back and forth quite a long time. We have a friend company also have an app launch where they were using an underlying NFT portfolio management API. I don't like buy, sell, and know what some people have. And on the launch, it couldn't hold the load because probably they had marketing, et cetera. So a lot of traffic. And they had to organize what they call war rooms with engineers on both sides to find what the issue was. And in fact, after three days of multiple engineers working on each party, the conclusion, not really conclusion, but the root cause was revealed by error by the boss of the API provider who entered the room and said, did we fix this index issue we have on the database? and he didn't know that the video call was on between the two rooms, so the other one knew it also. But if he didn't make this error, they could continue wasting engineering resources just for something that was already solved on one side, but for a stupid chat game. Also, we use a lot of tests. You need functional integration. But our experience shows that tests will never, never mock real-world data. For example, for the project with the truck drivers, we have ten countries, seven languages, a few different time zones. The one API we use are using naive datetimes. We are using UTC-based datetimes. But accounting in Laos doesn't work the same way in each country, so we could try to mock everything, but there is always a real-world case that only comes in production. So we have to find a way to see that. And of course, we have funny things like humans hacking the systems by, for example, putting do not bill in the title of a file, which when you start to expose this data to clients, of course, comes to the eye of the client. And I've seen on our side and on the side of, I had a lot of interviews with people to understand how they manage this kind of issues. Mostly the answer, of course, there is monitoring that shows the amount of regrets that are there. And mostly the answer is we put duct tape where it's important. reboot if HTTP 500, then send a Slack message. And there is this unwritten algorithm of if Slack message, then Michael should reboot the server. I'm caricaturing a bit here, but it's done in really large companies. I don't know in yours. Maybe you have a better solution, and I'm really happy to discuss this after the talk. But there is a lot of this duct tape code. What's important to me is that the end user should continue using the service. Why it doesn't really work, why it shows a little error message, which is most probably unintelligible, it's never his problem. But if I go on the client side, on the API consumer side, Consuming an API sounds quite easy, right? I imported HTTPx as HTTP just because any client would work like this. But I just need that, and I have my response. And what can go wrong? Well, actually a lot. So let's try to see. OK, there can be a cable that is unplugged. So we will catch the request errors. There can be, I put it above request error because timeout exception is a subclass of request error, but there can be also timeouts. And it's even worse than unplugged cable because, as I said before, you have to wait for the timeout to know that an error happened. There is the regular 500, which in theory should never happen, But in practice, I think we've all seen 500s. I didn't take the 400s. But there is also Error News 400s that should be 500s, and vise versa. But there is also the cascade failures stuff, like I was saying sooner with the salary workers that were scaling like crazy. Maybe you want to implement a circuit breaker, something that if some error rate will raise above some threshold, you'll define it, should not send the request anymore and just return that. How you handle that and how you handle that in your, for example, server workers is another question. But we have also interface changes. Everything works as expected, except that the data is not formatted like you want it. So I added some Pydantic validation. Maybe this Pydantic validation needs to have some value that more than just it's a string. Maybe a zero length name is not really good. And you have slow system. So maybe you will add cache system. Here I use Heashell, which is a very good Python library for cache, to add it to HTTPx, which will substitute the transport for HTTPX, so it's using the cache if available. But then you have people that will have some lack of respect for RFCs and will just expose, let's say, immutable, not immutable, but valid for quite some time, data without any side effects over a post method. So now I have to override the cache controller to say, yeah, first cache on this thing because I post, but I know that this post is not really a post. It's like a get. And, of course, even if I could have cut that with the Pydantic validation, there is always some time where the data should return one price and all of a sudden it returns two different prices and you don't know which one to use. This is a bit isolated, but it happened to us. So it looks stupid like this, but it happened to us in real life. For each problem we had, mostly we needed only one thing. We need to, after the problem appeared, we need to know what did actually happen. We did the duct tape, we did bad duct tape code all over the place, and at some point, we started to think again, because duct taping is cool. putting it on all your projects with duplicated codes within one project and amongst different projects is probably not very viable so we try a new approach instead of writing that again and again we built an open source software for that which acts as a proxy and will handle a lot of the so sorry it's not a remote proxy it's a proxy like a sidecar it's a proxy that you put just next to your application, and the idea is to offload all complicated and repetitive tasks around calling HTTP requests in a service that just does that and nothing else. As a schema, it looks like this. You have different applications. You have the in the middle that intercepts requests, stores them, adds I will list a bit of features and most importantly show it after that but it acts as an intermediary between your applications and all the external systems and by external systems I mean also the internal application which are not the application you're responsible for because the ERP, the CRM, any three letter acronym works here will probably have problems even Even if it's hosted just in the room next door. So for now, what we do basically mostly is a big audit log of everything that goes through that we index, including the content. It's not just monitoring. It contains all the traffic content. We implemented some caching with different backends, retries logic. We have different algorithms for circuit breakers. We implement a bit of load balancing of fallbacks. If the circuit breaker is open on service A, then fallback to service B. We added customization using Python code, so you can add change headers, maybe even change method for this post sooner. We could switch this post to a get if we want. Maybe it's not a good idea, but we can do it. Importantly, it just picks HTTP. So that means that on the project we use it, we just switch to endpoints, and it works without any code modification on our side. 18 minutes, OK. So I go very quick here because I want to take some time for a little demo. But it's open source. It's pip installable. It's also packaged as a Docker container. It's also packaged as a Helm chart for people using Kubernetes. It's extendable using Python. It exposes metrics for regular monitoring systems, like Prometheus, for example. And we've used it for 18 months for the oldest client and quite shorter for other clients. And it helps us, but yeah. And it's probably not ready for what I'd call high throughput yet. We have to define what that means. But if you have a few hundred requests per second, It's probably not a good idea, anyway, to have an audit log that stores everything. Even if we optimize things, meaning that we use Git-like storage to not replicate duplicated content, but anyway. So as a fallback, I have screenshots, but I prefer the live version if it's available. OK. Oh, that's not the good screen. OK. Where is it? Okay, so now I need to look at the same screen as you. So that's a hand, not handwritten, but GraphVis written schema that just describes the setup I launched on my laptop just so it's understandable. So we have HAARP running. We have two, sorry, we have four instances of HTTP running as four different services. In HAARP, we defined two services which will load balance. I will show you in the dashboard next, it will be easier, between those HTTP bins. And we have only one PostgreSQL storage here. We could balance with Redis for the cache, but it's a simple setup. And we have some monitoring with Grafana and Prometheus that we won't really use today. But we have also Locust, which is a Python project you may know, which is a load testing tool, so I will use Locust to generate random requests to the proxy. So which one? The dashboard first. So from 5,000 mail high, we have an overview that just graphs whatever goes through the proxy. We have an audit log view that shows everything that went through. You could see in the overview that this demo, I reset everything a few hours ago, so there is no real data before 10 o'clock this morning. But I will send you data in that. So if I use Locust, I will send, sorry, it's not really, let's say simulate 20 users. So I activate Locust so it will not target only, not target the host, but target in a one-webbing manner the API 1 and API 2 with a lot of different possibilities. So, here we see that things are changing a bit, and if I go to transaction, I will see the live traffic. Well, not really live, but whenever I refresh, I will have new thing because it's hammering the APIs. So, I see which endpoint it's targeting. I can see, for example, that this one is not finished yet, but it's an endpoint designed in a to respond after quite an amount of time. I can see it's too small. I'm sorry. I'm sorry if it's too small, but I need some space on the screen. Some requests. What I can show? Well, here I can see the content. Interface can be perfected. And the setup that I have here is, so we have an ingress, maybe not a good term, but we are exposing an API 1 on 8080, an API 2 on 8081, and this one load balance to API 1 internal, which all of those are instances of HTTP bin. So this one is load balancing to 1, 2, 3, 4 with only 1 and 2 active right now because It's in the default pool and not the fallback pool. The APA2 does exactly the opposite. It belongs to three, four, one, and two, with three and four in the default pool and one and two in the fallback pool. And for now, our health checks just says everything is up. So I need to go quite quick, but I will, let's say, scaled down the API 1 to 0. So now I have nothing to answer for the API 1. Oh, sorry. There is something on my screen that was supposed to. Well, I scaled down API 1 to 0. And you see that the probes detected that it's actually down a bit differently for the two APIs because I used a different algorithm for circuit breakers for API 1 and API 2. That's configuration. But API 2 reacted quite quicker, but the naive implementation means that after two errors, it shut down the system. API 1 used a leaking bucket algorithm, so it waits for threshold to be crossed. So it's a bit different. Here you can see also that the API 1 plugged with the little blue dots that indicate that. 3 and 4 as a fallback, because I defined that I always need to, if I can, I always need two services in my pool, and of course, because I don't have really much time left, I will try to scale up if I find my cursor back. And if you wait a bit, we should have both services restored. So this one, again, it's naive. It just said, oh, yeah, it's up. This one will need a bit more time. I'll show you after the end of the session. So that's screenshot. I need to move a bit fast. You have a documentation. Sorry, I will give you the URL after and a website presenting on that. We are trying to build an enterprise edition on that, adding privacy, security features. We have a lot of IDs. Some are already used by some clients. And we have an admin panel, because all what you've seen is made by, basically, a YAML configuration file. And we have an admin panel that generates this configuration. We're also building, actually, an Envoy-based alternative. So there is the ability to use either the Python-based proxy with more flexibility, more customizability, or the Envoy proxy, for example, for more throughput. And we have a lot of ideas of what we could do with that. One idea that I really like is that, for example, point of sale elements, we could use that to have really unreliable network in shops and restaurants. As I think I'm at zero, it's right here. So it's a very young project, but I really want, if you have stories about your outages, about your issues, about how you think it could or it could not resolve our problems, I really need to start talking about that, communicate with you. It's actually the first time ever we speak publicly about that, apart from a lightning talk in a really small room. So really interested in what you think about that, and I think we still have three minutes for questions. Thank you very much.

Speaker 2 [26:42]

Thank you, thank you Roman for the interesting talk and we have some questions on Slido. You can also ask if you're not there. So, if using a proxy how do you handle sensitivity data like token and personal data?

Speaker 1 [27:01]

So, there is different approaches. You can refuse that to be cached. We have extensions for some customers that will anonymize things. It's not the first goal of HAARP. We prefer to have the pristine, original information available. If we have sensitive information, I mean, it's project dependent, so it's up to you to customize and to anonymize things if you need so. But ARP is running in your infrastructure next to your project. It's not something that works in the cloud or anywhere. So the sensitive and personal data never leaves your infrastructures.

Speaker 2 [27:53]

Okay, great. So you suggest to use your API instead of a bunch of other APIs.

Speaker 1 [28:03]

Not really, because I'm suggesting to continue to use a bunch of other APIs, but to centralize them and instrument them. The goal is not to replace the APIs, because it's the exact same APIs you're still consuming. It's used in a wide range of use cases. For example, all the API gateways products are doing exactly the same thing but on the other side, on the producer side and not on the consumer side. So it's not really new APIs, it's just a tooling that's plugged in your network cable.

Speaker 2 [28:46]

Okay, one more question. Since you shared using Pydantic, how does HAARP compare to LogFire?

Speaker 1 [28:54]

I have no idea. I have absolutely no idea. I don't know log fire, so I've only seen captures. It would be a lie if I answer this question.

Speaker 2 [29:05]

Okay, then quick one quick one more can help catch streaming responses

Speaker 1 [29:11]

For now, it can, it works, but really bad. That's the quick response. But that's an issue that we're working on.

Speaker 2 [29:27]

Great. Thank you, Romain. Thank you very much.

Speaker 1 [29:29]

Thank you very much.