Python Monorepos: The Polylith Developer Experience

Speaker 1 [00:05]

Hey, hello, welcome to this session about Python monorepos and the PolyLith developer experience. Do you hear me? Yeah, great. My name is David, I live and work in Stockholm, Sweden, and I got here yesterday from Stockholm, so I joined the conference yesterday and saw a lot of good talks. and what I've been doing the last couple of years is of course Python development mainly but I've also worked with a lot of functional programming like Clojure and also Node.js and way back I was a lot in the Microsoft space like .NET and C Sharp and ASP.NET and stuff like that But the last couple of years, a lot of Python, of course, but also functional, Clojure, Lisp-style things. So I want to begin by, at least, Python monorepos. What is it and what is it good for? At least sharing my view on what kind of problems a monorepo can solve, and even maybe what it is, at least from my perspective. So let's begin with, let's say that you have your thing that you're developing. Maybe it's like a fast API thing with endpoints and you have like features and things like that. Naturally, you have that code in a repo, in a Git repo maybe on GitHub, and it runs deployed somewhere on the cloud probably. So this is like a single project single service repo so this is not a mono repo doesn't really make sense in this case but from my experience or the thing the things that i have been working on the last couple of years are like developing like platforms or larger apps and the teams that i have joined have chosen to use like a microservice way of solving problems microservice architecture And that usually means that you have, let's say that you have this big e-commerce platform. Then you have different features in that platform. And with a microservice approach, you would kind of group the different things that platform do into different contexts. And within each context, you might have like several services, independent services running. So it could be like a bunch of API endpoints or event-driven event handlers or whatever. But you have like a separate, the things this platform or large app does in separate parts, like in different bounded contexts, if you like, like domain-driven development. So you probably will have like a fair amount of services, a fair amount of code in different repos, in different Git repos, GitHub maybe. And the teams, at least from my experience that I have joined, we have kind of agreed on that maybe it's a good idea if every service has sort of the same kind of structure or built up in the same kind of way. You can organize code or maybe you have the same kind of setup and scripting and things like that. You kind of strive to have your many different services look as similar as possible to make it easier to work with if you're going to switch between these projects. But some things might, as things evolve, you might switch out some tools maybe you switch out like some tool for rough or you know this linting of dev tools and maybe even want to try out a different structure you see the last one has to do this kind of source layout which is popular among some teams so these services and get rip was maybe start to diverge a bit maybe you have like slightly different things and slightly different tool sets or inversions of those so that's that could be like a thing that to think about but I really think that this micro service approach to to this kind of larger things like platforms is a really good idea but it comes with with a with a trade-off and one of them is that you will have you all your code in in a lot of repos, depending of course on what you develop, but probably a lot of different repos. And another trade-off is that you probably will have code that is very similar or even maybe even identical in these different services. Maybe you have like copy-pasted some useful code in this service to another service because that service also needs that one. So a solution to that problem is to extract common code into a library, to a set of libraries. But you know, working with libraries can also introduce a different set of trade-offs, or maybe even problems, because then you have another repository to think about, and you need to think about versioning and strategies for releasing new versions and making the other services follow that pattern. I was about to say something more, yeah, let's skip that. So making a library can be too much of an effort for some things. Maybe you have this kind of small, almost like code snippets that are, you know, creating a logger or some dictionary parsing features or things like that, that it's not worth making a big library and have a separate repo for, so it's just easier to just copy-paste it, but you start to have like duplicated code across your platform. That's at least a risk. So one way of kind of solving those kind of trade-offs is to actually take all those services and libraries and have them in one and the same repository, Git repository. And that would be at least my definition of a monorepo, so you have all your code in one place and you deploy, you create different artifacts from that monorepo, you deploy different services or apps or whatever you are developing. But managing a monorepo can be really, really difficult, it can be a nightmare actually. I was part of one, I joined an organisation and part of one team in that organisation And they had this horrible monorepo that was like nobody wanted to work with it, and it was really difficult to understand what was going on, because they had custom scripts and things doing copying and symlinking and all of that stuff that someone, it made sense in the beginning, but all of those had left the company. And the people that were there couldn't really do anything about that. So what we did, we actually pulled things out of that monorepo into separate repos. And then we ended up in those trade-offs with the duplicated code and things like that. But here comes a solution. It doesn't have to be difficult. It can be really simple to use a monorepo, have these good things that you want from a monorepo, and solve these microservice several repos trade-offs. And this is where this thing PolyLith comes in. So what this idea suggests is that you have a certain structure in your repository. and it's it's it's almost like it's too simple so basically if you would structure your monorepo having these three basic folders basis components and projects so basis so if we think about all these services what what's probably common in all of these services is that they have some sort of an entry point point somewhere. So if it's a fast API service, you probably have some Python file with the end points, defining the end points. Or if it's a CLI app, maybe you have some main function somewhere. So that code, like, they are probably named app.py or something, or main.py. Those things would be put in the basis folder. So that's where you put the entry points. And And the rest of the code you would put in the folder called components. And what this kind of structure also encourages is that you write your code and you logically organize your code into namespace packages. And preferable in a fairly flat structure. So that you have like the thing that your code does makes sense with a folder name. So Kafka, well, it does something, maybe it's a Kafka producer and a consumer. So that's the components, the actual code that does something. And then on the third folder, projects, this is where you would put all the PyProject.toml files that you have in all your services. So there's no code put in the projects. All code lives in the components and bases. So the project will be like the definition of your actual service, like your fast API or CLI. So it's a PyProject HOML, maybe a Docker file, or like, you know, the infrastructure for your service. Now I forgot the next slide. Yes. So the goal of this kind of structure is to have it, to keep it simple. So have a really simple structure. You have code over here, you have your configuration over there. And then, just by expanding the components folder, you will see what's in this monorepo. If you have structured your namespaces in a reasonable way, you can see what's in there. Oh, there's something with Kafka, and I need it in the thing that I'm going to develop. I'm going to check that out. And the same thing with the projects. If you expand the projects folder, you will directly see all the services that are built from this monorepo. So just by having this structure, you will have a really nice overview. But the most important thing, from my perspective, is that it makes it easy to share code and reuse code. Because all of these services, they point to something in the components and the basis folders. You have one version of the Kafka producer. You have one version of the code that lives there. Everyone reference to it only. And if you make changes in one place, everyone will get that change. So it's really simple to reuse code. That's like the most important thing. So in this thing, this kind of setup is what is called the PolyLith workspace. So we have the basis, we have the components, and we have the projects folder. But at the root level, what we also have is a separate PyProject TOML made for development. So this is the one that you will use when you develop things. So that means that this TOML file contains all the bases, all the components, all your Python code and all your third party libraries that you use. That means that during development you have one development environment. So the kind of developer experience is like if you would have a single project. So you have access to all the code. And that also means that you have one set of linting rules and you have one set of typing rules for all your source code in your entire platform. So taking a step back from a sort of high-level view of what this is. So PolyLift is an architecture, but it also comes with a lot of tooling support. So these are basically the basic building blocks of PolyLift. So we have something called the bricks, and if you remember the bases and components folder, the common name for those are bricks. And PolyLith really likes Lego, so it's like Lego bricks. So if you have built Legos sometime, maybe you have used this kind of Lego base plate, thin layer where you can put like Lego bricks on top of and build something so so that's why that one of the folder is named basis because that's the thin layer that's like the entry points to you to your services and the components are the the Lego bricks that you build the things from so you can combine different components to create a feature to create features and then we also have something called libraries and that's basically the third-party things that you installed. The ones that you find on PyPI, like fast API requests and all of those good things that we use in our projects. So the combination of a selection of bricks and a selection of libraries becomes a project. And a project is the actual thing that you want to deploy somewhere, Like your service A and your service B. So in PolyLith you have one or several projects picking the kind of bricks that they need actually to make sense of that service. So that's kind of the architecture from a high level. So this idea comes from Clojure. And I worked in one team where we developed a fairly large app using Clojure and ClojureScript. And I was fortunate enough to work with a person who actually was the inventor of the PolyLift architecture. So he was in that team. And we used PolyLift to develop this, so I kind of learned how this thing works. And when that project kind of finished and I moved on and came back to Python again, so I joined an organization where we have this kind of this microservice set up with a lot of services. And if you remember those trade-offs that I mentioned before, that I could recognize those. So I started daydreams. What if we had like a polylith in Python? So I started Googling and did some research and couldn't find anything like that. And then it struck me, maybe I should try to develop something for Python. So I have developed this thing called Python tools for the polylith architecture. So I started off developing this for poetry. So this tool is built on top of package management tools like poetry. So that was the first thing I did. But now this tool also has full support for, like, Hatch, PDM, and Rye also. So it's a CLI tool that is, like, on top of one of these package managers. And of course, it's open source and free and things like that. Now I forgot the next slide again. Yes? Oh. So, I just want to show you what it is in practice. So what this tool is about is visualising your code or visualising your monorepo. So because once you have, if you have a tiny monorepo with just little code, just like my example, maybe it's not a problem, but once your platform grows and you have more and more services, you will probably want something to visualise things. So what this command does, the info command, so you write poly, that's the tool, and a command after, which is like info in this case. So this will show you all the bricks, all the components and bases, your Python code, and it will show all your projects, which are the services and apps and CLI things that you want to build and deploy somewhere. But what is probably more interesting is that you can see which brick is used in which project. So you can see that the Kafka brick is used in these projects, but the other brick is used over there. So you get an overview of what is used in your code and what you have in your repo. You can also inspect what kind of third-party libraries you use in your monorepos. You can see what libraries are used for an individual brick. A brick is basically a namespace package, a feature that you have fairly grouped into something logically. And you can also see what libraries are used in the projects and the versions. So the Kafka brick uses the Confluent Kafka library, which makes sense. You can also see which version is used in the different projects. So we can also zoom in to the individual bricks and to see the relation between how your code depends on each other. So to the left, we have some matrix. This is a super simplistic code example, so I have very few bricks. So it's just a tiny little matrix, but your code base would probably be a little bit bigger if you have many services. So you can see how they connect, who is using who in this matrix, but you can also zoom in for an individual brick. So to the right, you can see the message brick is used by the message API, and the blue color tells you that it's a base, an entry point, and the message brick itself uses these four different other bricks. You can follow the dependencies like this. I use this for when I go into refactor and think about should this brick be split into several or should I take these two and join them because it makes no sense to have this kind of small separation. So that's how I use it. And then we have a command called diff. So let's say say that I'm working on one of these things, I'm working on the Kafka part and changing something there, then I would probably want to know which of these services or projects, as they are called, are affected by my changes, because if I'm going to merge this to my main branch, I don't want to deploy everything all the time. I just want to deploy the ones commands that are affected by the changes. So the diff command will analyze the source code and identify which of the projects would probably need to be redeployed. So in this case it's two of these projects that actually uses this Kafka thing. How much time am I on time. Okay, two minutes left, okay. Then I can just super quickly go to my shell, because I've just showed you some pretty slides. So this is how you would run it, because I use poetry in this example repo. So I have, I say I run poetry poly info to get that info command. And I can do the same things to get all the libraries. So it's quite fast, even though this example repo is kind of small. But it should be pretty fast in in larger repos too. We have the PolyDepth, so I have this matrix of all the bricks used and how they relate. So just a quick look on how you would use it in your daily work. I think I also have my editor, so you will see that you have your bases, you have your components and you have your projects. So all the code shares the same top name space also. That's like an important thing. Then you have your entry points and your implementations and how they are used in the projects. And I think that was all I had time to share today. The code is on on GitHub. If you have feedback, we can talk about it today, or if you want to share something on my GitHub, we can also have a look at the documentation. I put a lot of effort in trying to make documentation as readable as possible. So, yeah, thank you for attending.

Speaker 2 [24:15]

Thanks so much, Dave. Please, do we have questions? Okay. Thank you so much. It was a very nice talk. I really enjoyed it. My question is going to be, so this is supposed to be like one Git repo, right? Yes. Because I worked a few years ago in companies which had multiple different repositories which were then merged in some Git submodel strategy or something like that. Okay, so in this situation, I was wondering, so you mentioned that each project can have its own versions of dependencies they can defer, and that's my case, actually, because some projects have some harder constraints, some don't have them. I wanted to ask how should it how how build of this project and linting of this project should look on my CI I was thinking of okay it's one repo so I'll probably just build I don't know I have five projects I'll build five docker images and then I'll run five different lintess in inside these images and they are gonna be pretty much independent should it work like that or I'm missing something

Speaker 1 [25:41]

think? It sounds like the same kind of view that I have, but I would say that ideally you would have the same version of a third-party library across all the entire monorepo, because that's when you can start sharing code. Because if you have one component or one base that uses something different, then you can't really share it with the other projects if they use... So, the solution to that is to try to isolate the dependent code as much as you can, like build wrappers around, like if you do something with AWS S3 or something, maybe you have like a wrapper that kind of has that, and if you want to change the version of the library, it's a change in one place only, so ideally you would have the same version of a third for the library in this with this way

Speaker 2 [26:44]

Thank you, that was really interesting. Could you talk about the advantages and disadvantages of polylith?

Speaker 1 [26:49]

branches of PolyLith versus M.

Speaker 2 [26:49]

versus pants and whether they are

Speaker 1 [26:52]

mutually exclusive or um yeah could you use them

Speaker 2 [26:54]

Yeah, could you use them together all?

Speaker 1 [26:56]

I haven't used Pants build myself, but I think they solve a little bit different problems. I think so, but they can probably have some overlaps. When I was starting to think about, when I thought about how I should develop this thing in Python, the first thing I looked at was actually Pants build, because it makes sense how things were, but then I kind of anyway leaned towards poetry, because it seemed like the problems that I didn't want to solve with my tool was solved in poetry, like, you know, poetry ad and dependencies like that, so I didn't want to care about that in my tool, so to speak, but, so I'm not sure about how exactly the differences and similarities between pants build but i think it can do a lot of things probably as what politics can do probably

Speaker 2 [28:04]

How do you cope with breaking changes in the components? But usually when you have such components like Kafka, library, in the library you conversion it. And when you have many projects, you can update them one by one. What I see in the polyleptic setup is that you have to update everything at once when you make a breaking change. Or is there a better strategy? I don't see it.

Speaker 1 [28:30]

I think that would be like the happy pathway, that if you can do that, then just do it, just fix it right away. But if it's difficult, if you have a lot of different things to think about, you could solve it in several ways. You can make another component, like Kafka V2 or something, and let that particular project use that one instead. and you can also some code maybe it's not fit to live in the polylyph then you can actually extract that into a library and use it as a third party library so it doesn't have to be like all in all your code must live over here and everyone must share the same thing but that's the ideal but if things don't really fit in that model you can adapt but the happy path would be just to fix it and everyone will get the new version. I'm sorry, Justin.

Speaker 2 [29:30]

Sorry, there are still more questions, but...

Speaker 1 [29:32]

the time is up.

Speaker 2 [29:34]

so possibly we can ask him.

Speaker 1 [29:35]

Yeah, I'll be here.