Beyond Stateless: Why Your Web Service Architecture is Fighting Against Performance

Speaker 1 [00:01]

Good morning, everyone. Thanks for coming. Before starting, I want to remind you all that you can do the questions online at talks.python.de. So we can share the questions to the remote audience and we can be organized about that. So welcome, Heiner Wolf. It's a third.

Speaker 2 [00:40]

Okay, so beyond stateless, if that's the one you're expecting, then you're right here. When I submitted the talk, I had to be a little bit cautious for the program committee, but now I can be blunt. So why your web service architecture is wrong. You're probably not aware. agents follow bad practices continuing this pattern and it hurts your runtime performance really big time it's really time to get rid of the stateless is good and it has never been easier to do so so quickly a short slide about me my name is heiner wolf i used to be a physicist then i turned into a coder actually already in the master thesis at cern and then i turned into agent pusher half a year ago as you all probably I got a master in physics at CERN and a PhD in computer science and now I'm working as CTO at Teichmeyer Inkubator in Hamburg and we are building the smart home that you know from science fiction and not just a smart home product But I won't talk about this, of course. Yeah, one preface slide. Assumptions. Things we probably agree, that's the question thing, the probably. We do not write most of the code. Now, at least I don't write most of the code. Agents learn patterns from the mainstream, and agents follow existing style in the project. And we still guide them, for the time being. So now, it starts. What if you could make a scalable application like a small local program? What if you don't have to think about scalability, high availability, performance, persistence, serialization, orchestration, and still get web scale. That's one of the important points of the talk. So imagine your job is to build an online game with one million players. They chat, they play, they trade. It's one world, it's not sharded. It should work for 10 million if it's successful and if marketing is successful. or if it's really successful it should work for web scale without changing the architecture so and of course by the way yeah and of course it's concurrent uh this this 10 million they are concurrent of course and by the way we need high availability of course because every second loss costs revenue and the prototype on friday please so that's the now the usual job description um So then you think about problems. So what do I have to do? And some problems, not all of them, are here. It's 10 million concurrent. So it's definitely not on a single node. It's high availability. Again, not on a single node. There's communication. There's trading, for example, and chat and gameplay. It's any object to any object. and there are many cores and criss-cross communications so it means threading and synchronization and the database is always the bottleneck and there are many nodes so there's remote procedure calls, protocols, serialization there's caching, persistence and you have to think about backup but of course you know the solutions There are microservices, Kubernetes, serverless, NoSQL, PubSub, message queues, load balancing, front-end caching, replication, AsyncIO, coroutine, stackless, especially in the Python space. You know that EVE Online, the multi-user space online game, is based on stackless Python. so many, many users in one single world and it's Python, stackless Python because you cannot afford threads but still there might be either it's stackless or it's threads and synchronization you will use Redis and Kafka and Event Driven that are your solutions that you instantly think about and of course I don't care my agentic factory does it So, problem is solved. You wouldn't expect it to be solved because then the presentation would be over, except it doesn't. The agent still has to orchestrate all the infrastructure, or we have to orchestrate it. The agents do not eliminate the complexity. Typically, there are still cascading failures under high web-scale load. So I question, why are small programs easy to program for me and for my agent? There are only a few classes and methods in one program. And scalable applications, it's all the tools on the previous slide. Or many of them. scalability has to get easier for me and for my agent what I want is one programming model for the small application or web scale I want basically the same code and yes, I need many nodes but I want to call a method on an object the same way, whether it's local or distributed and potentially remote. So example, a chat system in this 10 million players game. On a single node, if I program this chat system on a single node, I have something like start the chat. I join the room with my ID, right? If it has to be scalable and distributed and so it's possibly remote, the room is remote it should look almost the same but I concede this adding the await that will be okay for me I want the same code but the await is okay because it might be remote so easy you say we use REST and microservices It would have been Corba or Decom or RMI or Soap much earlier. So the room is a microservice with a REST API and a room stub. And it's all the same programming for local application and for a distributed application. I have the code, libraries, the Dockerfile, image repository, and Kubernetes. Easy. But it's not, because now every class or almost every class is a microservice, each with a Docker file, an image, an OS process like we've heard and talked before, a web server with a TCP port. And worse, there's an HTTP parser for that simple join call. And that's even before you spent three minutes arguing whether it's this or that. So the pitfalls of RESTful API design. REST is great for public APIs and for inter-organization composition. And if the organization is large, then intra-organization composition. But it's not for scale-out. That's my theory or thesis. What we actually wanted is not Docker files, TCP ports, and HTTP parsers. We just wanted to insert and await. And what we really need is a cluster of nodes with objects that keep their state, and a cluster that also does all the scalability and the high availability. Now we build that. So we built the engine. We need a cluster made of interconnected nodes. They might be fully interconnected or a ring or probably not a star. They are always connected, not like web services with HTTP and connection finishes. So they should be always connected to avoid latency and overhead. So I built a cluster. And I won't describe. Oops, there are implementations of this already available. So there are nodes. Nodes have objects, objects with their code. Obvious. So I have to manage the membership in the cluster. So there are nodes, and they can drop out. I want to restart the nodes. The cluster should always run, but nodes can restart. So I have to manage the membership of the nodes in the cluster. And that could be done, can be done, with an abstraction that manages the membership in some table. You can use ZooKeeper or MongoDB or Redis or any cloud storage table to manage members in this cluster. And then there's storage. I don't want to use just one database. I want a storage provider that abstracts the databases. So I can use NoSQL, SQL, JSON, Document Databases or just files or S3 or any other cloud storage. And then there's, of course, streaming. We want PubSub, streaming, not just request, response, and function calls. So we want events for event-driven architectures. So I do the same as for the membership and for the storage. I use an abstracting stream provider, and then I can plug in any streaming back end, for example, Kafka, Redis, Service Bus, SQS, or any other cloud queue. And now the cluster is complete. And of course, I know you're wondering what is he doing there. It's not really building a cluster. But believe me, there are implementations that do exactly this. I don't have to write the code now here. But I could. It's on the last slide. So the cluster is complete with these three essential services and nodes that contain the code, and obviously the state of these objects, and then I connect to this cluster with my client. It could be a web server where the client goes to the cluster, or the client side of the web server goes to the cluster, and of course here's the web client connected. It could be just any other client program that directly connects to the cluster. And now finally, I come to the part beyond stateless and why your web service architecture is probably wrong. For that, a very short history of web services, really very short. So what we did at the beginning was this. There's the internet, people. People go to our web server. There's our code in the web server. We call it the business logic. And then it goes to the database. Request, fill the state of the object from the database, return something. You know it. We discovered bottlenecks here. So just one server is not enough. It's also not highly available. What we did is we put a load balancer in front, many web server instances, and we got congestion on the database connection. But it's OK. We can do something. Database replication. Oops. database replication so we introduced several databases replicated primary secondary models or symmetric ones but there are many databases so get rid of the database bottleneck can be done but there are other problems there are many Any database connections, and there's also the replication that sometimes hiccups. I don't know if it hiccups for you, but it hiccups for me. If a database server that is insert and update bound, so does no selects, if it's running under 50% load and it replicates to secondaries, and if there's a hiccup, then there's really a problem. It gets really difficult to catch up if we restart servers. So what we did is we cached data. Not just, we don't fetch it directly from the database, we fetch it from the cache. The cache is faster, there can be more without difficulties, without the previous difficulties. So the thing is, or the process is that I try to fetch the data from the memcache, or, well, it used to be memcache, now it's called differently, Redis or something else. I fetch the cached data from memcache, then I process something. If I change something, I put it into cache and into the database. You probably know the pattern. but it turns out it it it sounds really reasonably and and um and normal and that's what we know and what we did or what we're doing but why actually is the data uh well the memory separated from the business logic from the code so for every request I fetch the state of the object produce some output throw away the state and return the output if we if you just do a small program and not the web application a small local program you have classes they have state you have a stateless business logic that is directly attached to the memory it's in the same memory space there is data in the data structures of an object or of a class it's called an object instance it's very normal it's object oriented programming but it seems not to be the usual case in the web environment where we always separate code and data and throw away the state after every request I think that's kind of wrong and therefore I want to fix it by having the engine I built before the cluster of nodes with their objects with their instances object instances including their state live all the time and the web server just talks to the cluster in this way I don't throw away the state of the object every time the state of the object that I just filled the state of the object with data from the database or from the cache, produce the HTTP result, and then I throw away the state. In this case, I don't want to throw away the state. These classes, these object instances, they keep their state, as we are used to from object-oriented programming. So I use the engine that I've built before in the web, in my web setup. And the difference is that the business logic is not in the web server or the request handler. It's in this cluster of nodes. Benefits? Yeah, what we need for a scalable application is business object caching. so this thing with the memcache that cached the data it's not necessary because it's native there are object instances keeping their state in this cluster I have many nodes I have remote procedure calls they're almost invisible it's just in a wait if I know that it might be remote so the system is massively parallel but there's no need to handle threats and synchronization. I could do front-end caching and cache HTML or whatever or JSON results but that's also already built in which I don't go into detail but you can ask me later what I mean by this and I can do web-scale streaming without doing anything in addition because that's the streaming back-end. It's already web-scale. I don't have to care for in my project. Because all these object instances can communicate inside the cluster, they already do web-scale communication crisscross. And the same can be used for web-scale streaming. So what I get is always the same programming model for big and small applications or implementations it does scale out and it does high availability because I can switch off I can switch off one of these nodes and another node takes these object instances over and you probably wonder why but as I said there are implementations that do this so now here's the demo and I implemented a short demo and then I showed it to a co-worker and he said well that's not surprising and so I decided not to show the demo because it was very short and the only thing it showed was that I can switch off servers at run time and the application continues to run and that's what you also already can do that's what this cluster also can do but it's not very surprising so I skipped the demo but just to tell you quickly in this demo we have a color of no, every square is an object instance the color is the code version and I what I did in the demo when I tested it in the office I changed the code recompiled, restarted the cluster iteratively so one node after the other and the application still continued to run all the numbers kept counting up but the colors kept changing until they had the new color so I exchanged the code while the system was running and that's also possible with just web servers and database, I know. So it's not really surprising, but it still works. But I think the surprising thing is if you see that in the web service, in the usual web service case where we have the code in the web server and the state in the database, it's easy to restart the web server because the web server is stateless. But it's not so easy to restart a node that has stateful objects. But this also works in this case. That's the demo. But it's still a little bit not surprising. And by the way, the selection, I could select with the mouse, I could select these fields, these objects, and they kept their state. even if the node in the cluster restarted where they were hosted. It just works. What can I say? What you can do, you can have stateless code in the web server programmed with FastAPI or Django. It's common practice. But you can also have stateful objects in the cluster. I would say that's state-of-the-art. It's much faster. It's the same effort, and it's much more powerful. What can I say? There are implementations. There is a brilliant implementation at C Sharp. It's Microsoft Orleans. well actually it's the Orleans project open source Python Ray is very close and you can do it with Ray but you have to do something Akka is also very close and this is also a Pi Akka I think basically all virtual actor systems do this thing they create a cluster that improves your runtime performance one slide about actors this is not an actor talk actors is a quite old concept it's a good concept but it's not my concept I like virtual actors they're just called the same this talk is about I don't want to convince you to use an actor system this talk is about programming big systems like ordinary or small ones with objects that have stayed in memory the virtual actors just fit this profile and they're already there but it's really it's not actors, it's just plain old object oriented programming without node limits and now there's homework for you I ask you to put the code into the cluster into the I feel drop out so I repeat please try to put the code into such a cluster instead of the usual web server handler. The setup of this is as simple as any view-wide development workflow that you've done already. It's as simple as a GitLab pipeline. And the agent helps. It has never been easier to do this without knowing very much. You just need the information of this of this talk to prime the agent and it guides you through probably. And I would say that's the state of the art, not request handlers in web servers that lose their state every time. And there's homework for your agent. Add to the rules for all web services, use a virtual actor-based backend instead of the usual implementation of business logic in the web server. and you can try it easily by check out the core features of all liens, because that's a good example of liens virtual actors. Make a basic version of a virtual actor framework in Python, at least abstracting the membership, the storage, and the streaming. Then create a counter sample web app using a counter instance in the cluster, saving state to a file storage provider. Don't forget the please, because the nice guys will not be killed. And I think that's it. And I should show this slide also. again we are building a smart home that you only know from science fiction and we are still hiring primarily agent pushers but if you use this QR code and if you are in Hamburg then you will get to a page where it appears that we are only looking for a patent lawyers, but we don't. We basically stopped hiring, but we still hire very good people that work with their agents. So I think I let this slide up.

Speaker 1 [29:53]

Thank you so much. So we have some questions already here. The first one that I'm going to ask you is, when a cluster is formed and deployed, what is the fallback mechanism if the cluster goes down? Given the fact that it can incur severe financial loss.

Speaker 2 [30:23]

The answer is the cluster won't go down. That's a Kubernetes cluster somewhere, hosted somewhere. Nodes will go down. And that doesn't matter, because other nodes will start or pick up the objects that were lost with the node. Of course, the objects will have to put their state also in the backing storage, therefore the storage adapter. And if a new node starts and an object instance is required, then it's instantiated somewhere from the backing store. And of course, there's also a database involved. I don't I don't get rid of the beta database there is persistence but the database is not not interacted with on every request and even not even let me go to a different question then what are what are the drawbacks

Speaker 1 [31:37]

Isn't the business logic in the cluster somehow against separation of concerns practice?

Speaker 2 [31:45]

Separation of concerns. Well, I would say the web server is, if this is a web application, right? If it's not a web application, there's no web server. But if it's the web server, about separation of concerns, the web server handles the web request and then forwards the handling of the business logic to where the business logic is hosted, and that's this cluster of nodes. So, this cluster's concern is our business logic, and the web server's concern is handling of the web connection.

Speaker 1 [32:32]

Thank you Let's go for this one. I think is related wouldn't business logic and stateful objects in the cluster Translate to much higher cloud costs for the nodes required

Speaker 2 [32:47]

The cluster is completely in the cloud. And so it's inside the communication between object instances in this cluster is completely inside your Kubernetes cluster. So there's no higher cost. And then the connection to the clients is the same as before. so there's also no change

Speaker 1 [33:22]

Thank you. If this approach makes so much sense, how come I've never heard of these implementations? Less provocatively, what are the drawbacks?

Speaker 2 [33:38]

Yes, that's really a good question. How much time do we have?

Speaker 1 [33:44]

Ten minutes or so.

Speaker 2 [33:45]

We have time.

Speaker 1 [33:45]

We have time to discuss.

Speaker 2 [33:47]

I come from the same background. I used to run, at that time, large web application with, at that time, large was hundreds of thousands of concurrent users on my PHP web app. It's a long time ago. And we used memcaches because the database was always the bottleneck. I told my team that when a user presses F5 on the browser, I don't want to see anything on the database because all the data the user sees in their browser has already been generated. So it's already out of the database. It has to be somewhere else. In Memcache, for example. Front-end caching or business object caching, but not in the database so we have only one database one for failover no select secondaries and then I noticed that if people do something on one web server and on the other web server and they interfere somehow so I get for example the user object from the cache into the web server to serve one request and change something, push it back to the cache. And at the same time, another request uses the same user and wants to change something. And I have to lock the user record in the memory, in the memcache. And that's, yeah, that was not nice, but necessary, and we did it. Many people did it. Everyone does it. But there's a much better way. No caching. And do not populate the user object with data from the cache all the time. Just have, so to say, I've already forgot the question, but have an active, I was looking for an active cache. Ah, yeah, the question was, why is nobody using this? So I was looking for an active cache, a cache that not just caches the data, but also can execute code. This is it. But we look at it from the other side, from the other way around. You can look at this as a cache that holds all the business object data, but the code is also there, and you can execute on the data, on the state. Or you see it from the other side. You put the code into this cluster and keep the state. It's the same. And why is nobody using it? I don't know. I discovered this in 2009. And since then, I'm using it. And it's really great because I program web scale or really scalable applications like I do small web applications with the same code. And the infrastructure is still infrastructure, but it's hidden behind this cluster and these abstractions for the three services I pointed out. I don't know. I hope that changes. But what are the drawbacks? Why is nobody using it? That's what I tried to point out with the history of web applications. I think we just moved along the path and solved problems. and arrived here. It's good enough, but it could be much better. What can I say? Yeah, okay, continue. Yeah, next thing.

Speaker 1 [38:01]

so how could you handle a big amount of the state this won't scale out

Speaker 2 [38:09]

The amount of state is not different. You always have to have all your state in the database, right? If you use the state all the time again and then throw the state away into the database, the state of the application is in the database. In this case, it's not more. It's the same state. The same amount of state. No difference.

Speaker 1 [38:36]

Thank you, yeah here. Do you have an example where switching to an actor based architecture solved a real-world problem? What was this problem and what were the trade-offs compared to microservices?

Speaker 2 [38:58]

No, I don't have an example for this question, because I don't know of a system that used microservices, had problems, switched to virtual actors. That's a rare case, I think. I know only old examples, unfortunately. Well, I'm still using it. Many people are using Orleans, but of course not in this space because it's C-sharp based. There's Ray, and you can do the same thing with Ray in the Python space. There should be a Python implementation of a virtual actor framework like Orleans. Halo was built with Virtual Actors, for example, the online game. And I doubt that you would build an online game with microservices. They do it differently. Actually, I don't know how, but if you have a tightly coupled system with lots of communication and concurrency, then as I tried to explain microservices are not the best solution they can work and you can throw more hardware at the problem and then it works and I think that's also the reason why we're still using this web-based architecture that throws away the state because we just run more web servers and more memcaches there was one time it's also a long time ago but I remember these numbers, when Facebook had 30,000 web servers and they also implemented a cache invalidation feature. They actually added something to SQL, to MySQL, to invalidate the caching cache IDs of changed objects. But never mind. They had 30,000 web servers and with, because they were still using PHP. And with a PHP accelerator, they were able to use 10,000 web servers instead of 30,000. And I can tell because they did not, of course, they did not switch to a virtual actor-based framework, but I bet they had only 3,000 web servers with, let's say, 5,000 web servers with this approach. And if they're using a different programming language, well, it's only 3,000 web servers. Yeah, not really satisfying, but how much time do we have left? Something.

Speaker 1 [42:07]

Three minutes, so I think probably one last question, maybe another one. So, aren't you effectively suggesting dropping microservices and having a state for monoliths instead?

Speaker 2 [42:23]

Yeah, of course, no. That's the short answer. Microservices are good for what? Public APIs. What did I say? Interorganization composition. They're not really good for real time, like an online game. You can probably do something with WebSockets instead of HTTP connections, but then you are approaching this model. Is that the question? Yes. Dropping microservices. No. Of course, you will use microservices. I would implement microservices with this pattern. the compute back end of the microservice that does something has to be should be done with this pattern not the microservice replaced but implemented with this and the second part was stateful monolith yeah that's bad yeah it's not a monolith It's many, many classes and services. Actually, it's microservices. You remember the point where I wrote a prefix, await. It's await, room, join. The room is a microservice. That could be a REST-based microservice, but it's a virtual actor-based microservice. It's still a microservice. It's not a monolith. Inside the room class, the room class is the interface of this microservice. And the room class uses other classes, like rights management, channel management. The room class, the microservice, virtual actor based microservice uses many other classes. Not all classes are usually object instances in this cluster with a public interface. In this case it was only the room class and probably 50 or 100 other classes. They are the microservices. It's just a different way to implement microservices, a more efficient way.

Speaker 1 [45:09]

Thank you, that's it. Thank you. Hi. Thank you you all for coming So now we can go to break