Simplicity Scales: Rewriting to a Django Monolith and Monorepo

Speaker 1 [00:40]

Welcome everyone to this session. Our speaker for today is Bruno and he's going to introduce himself in a bit. Please give him a really warm welcome. Thank you very much. Just a few housekeeping bits for this session. Please put all phones on silent, don't change rooms during the session and if you have any questions please submit them via the talks too and we'll take them after the session. Thank you so much. Over to you, Bruno.

Speaker 2 [01:09]

Perfect. Thank you. Yeah, welcome everybody. Happy you made it up here on the last day. I want to tell you a story today. I personally prefer the talks where somebody speaks of their experience, what worked, what didn't work. And I want to tell you today why for us the term simplicity scales worked, what we did over the last year and kind of the for us exciting journey We had going back to something a bit simpler, which we then made scale Perfect So I want to tell you a story how we went from a sad developer experience to a very happy one Yeah, but also objectively how we made it possible to decrease infrastructure cost or basically doubling the percentage of work that we do on forward feature basic PRs. And how we're going to do this is I think I want to explain to you first the situation we've been in to give you the context because throughout the entire presentation, I'm going to introduce a few choices where there is no silver bullet, right? It always depends on your specific context. I think that was a theme I've seen in very different talks over the three days. and my goal today is to give you our context so that when you are in a similar situation you can maybe pick and match a few ideas and see that it works for you. I'm going to walk through the actual technical journey, do a bit of high-level spots here and there and then analyze what has worked, what has not worked and give you a little summary and take-home message. Shortly for myself, my name is Bruno. As I said before, I'm one of the team leads at Itinerary. I have a classical education in computer science, and it's not my first PyCon talk, so if you're interested, you'll find the other ones on my website. Where I work is a place called Itinerary. We're a Swiss-US company startup, and we help scientists with their optimization problem through various techniques, physical air, robotics, and we're one of the pioneers from the self-driving lab space. And what is really important for our story today is we are a multidisciplinary team. We have profiles with an ML background, a chemistry background, a classical computer software engineering background like me, who all actively contribute towards our code base. And we're based, as I said before, in Switzerland in the US. So where do we come from? The story here is, I think, something everybody can kind of relate to when you've worked on a bigger code base. You work on a system where the responsibilities are maybe not clear. Interfaces are not very clearly defined. It's also not fully clear which system is supposed to be maintained by whom, what is every system supposed to do. You work in an environment where something that should be trivial takes a very long time to debug. You find bugs and you're like, why does it take so long to fix this? Or for example, you want to release a new version and that's not just a click of a button but actually a very long journey to get there. This was an environment where we were a while ago, and if you see like here a little kind of architecture diagram that we had, and if you see this and you think like, well, I don't get anything here, that's to illustrate my point, right? It took a while for engineers to get where we came from, and if I summarize kind of our problematic state, we had a legacy code base that organically grew over time with shifting I think this is very normal in a startup environment. You start off, you try different things, you have maybe a new client, you want to kind of satisfy their needs, so they stay, and you go back and forth, left, right. And I think this is really, as I said, very normal. We had the case that we had used a microservice pattern, probably not in the most ideal way, which led mostly to increased cost without the scalability gains. And finally, it was not fully clear who owned what part of the system, We had, like, a growing team structure started from, like, a very small engineering team, and it was not so clear who did what, who was supposed to do what. And I think the one thing I want to make really clear in this context is many of the decisions that might seem not ideal now probably had a good reason in the past, and in the context of the past might make sense. And I think this is something very important. If you also go into the process of proposing to refactor something in what sense whatsoever, keep in mind that your context now is different than the one it was in the past, and that they probably had very good reasons. You work with smart people, and you just assume their best intent. So that's where we started from. My goal now is to walk you through the thought process, right? To kind of, what was our context? Why did we decide certain things? I won't have the time to go into the technical details. If you're interested, let me know afterwards. But basically, the very first thing I would suggest and what we did, you ask yourself is, is this something we should fix or rewrite completely? You will have that question. And the underlying question is, do you want to do something incrementally or start from a scratch, from a blank page? And the idea, the reason why you want to ask that is because both of these approaches have their pros and their cons. probably an incremental fix would allow you to validate earlier if something has worked or hasn't worked, while if you start from scratch you have less constraints, you can basically do whatever you want. In our case we decided for starting from scratch for three basic reasons. We, as you've seen before, had a fairly complicated system and we deemed that kind of untying all the requirements and replacing bit by bit would have taken so much time that we would rather spend that time on something new and then along the way we decided that we would like to change some of our technologies used and one of the ideas from the management was that we would also take this chance and then update the product philosophy going from a process centric design to a data centric design of our platform and all of these introduce a certain amount of risk. We changed the product, but we also changed the architecture and some technology choices. And so to keep these risks down or kind of contain them, we started thinking about a technical playbook for this. The very first thing is every time we encountered a new decision point, problem, we really asked ourselves, can we simplify this here? Or can we rephrase that problem to instead use, we did before in that architecture, we had a lot of very custom solutions. which is great because it solves the specific problem you have but it adds a maintenance burden and so in most cases that was actually not necessary. And so we've asked ourselves is this something that we may be able to rephrase slightly so that we can use an existing open source solution for this. We tried to go really clear project management in the sense that we've spent quite some time up front deciding what we want to do and then aggregated this into various milestones. And then, I think this is very important in our case, after every milestone, we also published a version. We hosted a version first for only internal testing, but then fairly early on also for external testing with selected clients to get feedback. Are we on the right path? Is this what people want? The third thing is, from the start, one of the new requirements to everything was that we wanted to prioritize the developer experience. Reasoning here is that I think, I truly believe the upfront investment into making everybody's life easier will not only help to make them happier, but also faster. It will help them work maybe less error-prone. And I think this upfront investment will pay off much, much more later. And then finally, that's the only thing we hadn't had from the start. we decided also to instead of just going for something with feature parity in the new version to drop that idea and introduce some new features along the way. Why would we do that? One thing we saw is that initial clients were quickly asking why would we want to do the switch and to motivate these clients one of the things we also said hey there will be these new features they will only be maintained in that new version you will have to kind of do the switch if you want to use them and then from a requirement standpoint and this is really important because it motivates all the questions afterwards we are not working in an environment where we are expecting millions of requests per day we are really not at that at all while at the same time our clients really expect a very very reliable solution they pay a lot of money for it they want to have a reliable solution so from a technical side The availability aspect is very important. Then, through the nature of our product, we have to integrate various asynchronous ML systems. The existing ones, there is a huge, like a very capable research team that works on new ideas. So this will be a constant requirement. And then from the product requirements, we have, it's very normal to have like long interaction cycles here. So clients might not come back for a week to the platform because what they optimize just takes time to actually do. The reason I put this here is that they won't be there and say, hey, I need the results back, what I'm supposed to do in a second, right? If they don't come back for a week, it can actually take a little bit of time. We are in the area of B2B enterprise SaaS, so we really want to support integrating in the existing environments people might have. So that's something. and then we're still discovering where this product is supposed to go, so we also want to support short release cycles so that we can get stuff out, tested, and then maybe iterate away from it if it was not good. Overall, this kind of reminded me of the Django headline, which is basically, for those that can't read it, the web framework for perfectionists with deadlines. The two main technical things, as the title already spoilered, we're going to talk about today is us going back to a monolith on Django base and us moving towards a monorepo from a multirepo. Sorry. My bad. I jumped too far. Perfect. The reasons why we decided for Django and staying within the Python ecosystem is, as you've seen before, We don't have the requirements of multi-million requests per day, and most of the team already knew Python, right? So this is also something where if we have to get something done quickly, this works, and then Django is a very reliable choice. It's an opinionated choice, but it's a reliable choice. Many people have used it for various size of projects, so you can assume that it works well. Why do we move to a monolith? You've seen the microservice architecture introduce all of those problems without actually us needing these requirements, And I mean the whole point of this talk is think about your context. What do you actually need? Simplify it so that you can then scale on later Idea here being we go back to the monolith just for simplification reasons faster development And then choosing stable technologies along the way is really this idea of rephrasing the problem To go and see what we can use what exists. That's Django on Django rest framework. That's salary with a Redis back-end and so on. And then finally, we wanted to prioritize the developer experience. Well, for us, that meant mostly supporting that the entire application actually runs on a developer's machine so that they can actually test end-to-end what they're doing. That was not possible before. And then, so just to give a quick idea, on the Django side, we have been inspired mostly by the hacks of Django Style Guide. I don't know, for those that don't know it, it's publicly available on GitHub. It's a very nice, like, opinionated approach to how you should write Django or structure your Django applications very quickly. We have a single-page application that talks to our Django application via REST and WebSocket through an authentication and authorization layer. We then have various Django applications that have an API layer, which is basically taking the request, serializing it and then also serializing the output and then what the hacksoft guide suggests you is that you split your business logic into what they call a selector and a service the selector is for read operations the service is for the actual business logic those interact with the database and then in our case we extracted out the logic for integrating other stuff as I said that could be async ML libraries or third-party applications whatever that means. So this is kind of the architecture that we ended up with. Big advantage here is because it's so clearly structured, once you've learned this, you will find yourself very easily in various junk applications because basically at the end of the day they all look the same. So it becomes very easy for a developer who has not worked on part A of the application but on part B to also go in A, fix something because it kind of looks like part B as well. And then here, the bit where we wanted to make the developers happy, the requirements were that we had everything run on a single machine, we have a single entry point to start and stop everything, so we use just as a command runner, and we have really a just setup and just start, or I think we call it just run, basically to start the entire system, and then we dockerize only the utility parts of the application. That means the Postgres database, Celery Runner, those are within Docker. And then certain subsystems we allow you to mock. So for example, the long-running ML systems, those sometimes are mocked, but most of the systems we really try to keep as close as possible to what we would run in production. And here the idea is really to focus on the time between the code change and the actual visible change so you can see what you're doing. And I've asked my colleagues before, and I think this is kind of one of the biggest changes over the last year that made everybody happy, that it's actually running smoothly. In some cases, we have the situation that you cannot do something locally. That might be, for example, because mocking that system would introduce an extensive overhead or a need for a different architecture, or that system you want to use or mock is part of a product that you have to buy or the local machine is computationally not ready to run that system, in this case LLMs. What we did in these cases was, I'll give you two examples for the authentication. We didn't want to mock that part because it's just so core to the product itself. So here every developer gets a sandbox environment configured within our authentication provider so that they can test locally something already that is as close as possible. to the reality of the production environment, and in the other case, we use LLMs in various parts of our product. These we can, because they are either not publicly available or too computationally heavy, cannot be run on the developer's machine. In this case, we host them in AWS, so only the location goes to Bedrock, or in some cases where we have more complex things, we made sure that the subsystem that we wanted to test is very easily deployable just for the developer into another sandbox environment. In this case, it's like the focus here was on saying, hey, let's only do this where it's absolutely necessary, right? We want to still have everything locally as much as possible. This obviously introduces the requirement for the Internet when you develop, but I think in an age where everybody wants to use an agent anyways. The internet, you will need an internet connection one way or the other. To the second bit. So we've talked about rewriting to the Django monolith. Now the monorepo idea, I think the base point here is very simple. Everybody knows what a monorepo is. We had the situation here drastically simplified that various teams had various repositories. There was maybe some shared tooling, some different tooling, and some infrastructure components. And the idea is, okay, how do we take that, move that all into the same repository and have a few benefits, some of them namely being that we share the infrastructure and share the tooling and make it very easy for people to work on various things at the same time. This introduces one of the advantages that a developer, let's say in this case from an ML engineering team, can work on their ML feature but also quickly maybe change some of the things of the front-end, which was required. It's a little change. They can quickly do it. They don't have to learn however that front-end repository works. They can just quickly do it or update an infra component, whatever that may be. Another reason, and this one, because that's mostly what I'm doing, is let's say you want to update the tooling, improve it. I don't know if you've heard about like a new typing system at this conference. You want to bring that to your developers. Well, you just have to do it once and then everybody gets that for free. How we implemented the whole thing, very simple. We have a multi-language monorepo, so we have the React front end in there as well. And we just divided it by language and then we had an infrastructure bit. And the generated part is basically something that is not committed to Git. The idea here is that things where you generate an artifact that the different parts depend on, that's what go there. We used UV for the setup on the Python side. I won't go too much into detail here, but basically we said that we want to use one UV workspace per deployable unit to simplify that within a deployable unit you have shared dependencies, but between those deployable units you do not. In this case meant, for example, that the entire backend had one workspace where the versions are shared. And then finally, I talked about just before, we use really just as the one way to run commands throughout the entire monorepo. We have it at different levels, and the idea is basically that in this case you have the in the root of the Python part you have for example all the tooling definitions your rough setup your in our case it was pyrite setup and then that is shared via all the different Python projects to make sure that everybody's on the same version on the same rules and becomes easy to jump between those one advantage for us and I guess that's just to kind of illustrate what becomes now very easy is in our case, we type the communication between the frontend and the backend by generating an open API schema, and then we generate from that open API schema we generate the TypeScript types that the frontend will then use. That is very helpful for the development process, makes error finding very easy. And the good bit with the monorepo is that you can actually do these invocations of when that happens very easily, and you don't have to do any Git submodulation whatsoever. So that's kind of what we did. Now, where we ended up with, I wanted to, I was kind of curious what had worked, what maybe had didn't work, and so I went into a bit of an analysis mode for this presentation as well, and one thing that I noticed fairly early on is that one of the things we didn't even actually came, like we didn't want it, but happened anyways, was I mean, obviously, the onboarding became easier, but also it became easier to prototype with agents and, like, LLMs. And why is this important? I think for the individual developers, not so much, but now we have a product team that can very easily also help us, like, quickly kind of visualize an idea. Usually it doesn't go... Like, there is an iterative process afterwards, but it becomes very easy for them also to contribute, to brainstorm, and to build little prototypes. And then finally, I mean, the release process, we went from something like a few hours for one or two people to do something to a button click. So that was very nice. And then I wanted to compare a bit some metrics around how we did. So I looked at GitHub and the statistics I could get from there. And so what I'm going to compare is the last 11 months of active work on both of these. In the case of what we call V1, it's multiple repositories. In the case of V2, it's a single repository. What we see is there was a similar amount of work, so it was roughly the same amount of PRs and the same amount of average contributors over this. But one thing we have to keep in mind is that through new tools that I think at this conference many people talked about, there is obviously some difference in between those time spans that I have not spent the time to get out of the statistics here. And I'm just going to highlight a few special things here. In blue you see our old version, in green you see the new one, and one of the things I'm very happy about is that we actually see that the average amount of reviews and reviewers on each PR has grown. So I think that's what I contribute to the fact that we're all in the same repository and it becomes a lot easier to review other people's PRs. And then the other thing, I think that's the most important one, that's where the number at the beginning comes from, if you look at the different PRs, so we use conventional commits to tag our PRs, and if you analyze this, you see actually that we have improved the focus on the feature development because we see that here we nearly doubled the amount of percentage of feature PRs compared to the bug fix ones from before. And so if I summarize what I kind of went through with you today is we've went from our context and the problems through two technical decisions and then analyze a bit kind of what the development behavior is now with these decisions in our case and what I want you to take home with is that it's really really really important that you adapt the strategy to your specific needs right this is as I said no silver bullet here that you simplify whenever possible I think I've seen a few talks yesterday as well where it's very important to kind of adapt the complexity of your solution to your actual context. I am a believer that well-known frameworks usually are the better choice instead of the fancy new stuff and that you re-analyse your situation from time to time and see if it still works. This was from yesterday's keynote and I just liked it a lot because I feel like it summarises the idea a bit well. Don't always jump on the newest stuff that you see, but maybe just stick with something that works for your context. And then finally, if anybody's interested, the little plug, we're looking for some people right now in our team that have worked exactly on this, so if that sounds interesting to you, come by, say hello, and let me know. Perfect. Thank you very much.

Speaker 1 [25:35]

Thank you so much, Bruno. That was a wonderful session. Just a quick reminder that if you want to ask any questions, please use the talk tool so we can read them out. We've got a couple of questions here for you. The first one is, I didn't catch which Django style guide you mentioned. Could you please repeat?

Speaker 2 [25:51]

Yes, sorry. So it's called Hacksoft. H-A-C-K and then soft. That's what it's called.

Speaker 1 [25:52]

Yes. Thank you, and the next question is how many dependencies do you keep in the parent project versus Workspace members and how do you handle version conflicts of those dependencies? So

Speaker 2 [26:09]

So, what we do is basically that in the parent of the language, we really just keep the tooling dependencies, so rough, for example, or pyrite, and then within the deployable units, we share them all, like, for example, Django, we have Django as a dependency, we have Django REST framework as a dependency, and then we make sure that those match. I mean, every developer that adds something or updates something has to make sure that those match.

Speaker 1 [26:38]

Okay. Having a monolith architecture with many different projects is prone to a lower grade of flexibility and scalability. How would you overcome these issues in a growing monorepo, especially if projects are getting more tightly correlated with each other?

Speaker 2 [26:53]

So, in our case, I think the question here is for me, from my viewpoint and my experience, is the scale of where we are working with. And, I mean, we're a team of, well, I hope I'm getting this right, but like 18, 15 developers maybe at this time, right? So, we're not like a 500 people org or something. And at this scale, I think our code base is also not too huge. I mean, I think it's 100,000 lines or 200,000 lines or something like this of Python. I feel like the simplicity advantages are outweighing the problems that might come later that you get from having like this grow too much.

Speaker 1 [27:34]

The next question is, do you measure durametrics?

Speaker 2 [27:37]

We yeah, I was looking into this when I was thinking about what we bring today I think some of the Dora metrics like how often we release or how long does it take a feature like a Feature to go into the production. We don't measure or there are not comparable between the two So that's why I really stuck in terms of analysis to the ones that are comparable between the two But yeah, it's planned to maybe go do more in this direction

Speaker 1 [28:03]

Can you elaborate on testing? Did it also improve the graph showed thin lines?

Speaker 2 [28:08]

Yeah, so the testing generally we didn't so that's one of the things we're doing right now to look at this topic What we did in the beginning really is just move From the existing stuff that was like moved over mostly the ML systems. We just moved them over and right now We're looking into this topic, but due to fact that it's just simpler I mean if something's simple it's also simpler to test right so that's that's I think where the main advantage come from in the beginning

Speaker 1 [28:34]

Great, thank you. Can you give an estimate about the size of the new Monorepo EG number of lines?

Speaker 2 [28:40]

Yeah, so I think if I'm not wrong like we're like 200,000 lines or something so it's not a humongous project

Speaker 1 [28:48]

And the final question is, do you also start to extract repos once they reach a certain maturity or independence into a dedicated repo?

Speaker 2 [28:57]

No, so right now really I think we went from like 28 repositories to two. The second one is just the like playground for the ML people and so the rest is really in one repository.

Speaker 1 [29:10]

Great. That's all the questions we have. Thank you so much.

Speaker 2 [29:12]

Thank you very much.

Speaker 1 [29:13]

Thank you.

Speaker 2 [29:14]

Thanks for coming.