Monorepos with Python

Speaker 1 [00:02]

Hello, everyone. So yeah, so I'm basically going to be talking about monorepos in Python today. So just before we start, a quick introduction about me. So my name is Ali. You don't need to pronounce the full name. You can just call me Ali. I'm the CTO at Corridor Platforms. And what we do at Corridor is we are basically creating an IDE to help people write governed and compliant analytics. If you're interested in that, we can talk about it more later on. But let me just go to the actual topic, which is monorepos with Python. The first thing that I always hear whenever I say monorepos, at least to a lot of people, is, hey, but isn't monorepos the bad thing that you're supposed to use microservices and Dockers and Kubernetes and all of these kind of stuff? So let me just get started a little with what are monoliths, what are microservices, and then talk a little more on monorepos. So monorepo is basically something that's large and rigid. So it's a very generic word. It just means rigidity. And typically, when we say monolith architecture, we're talking about deployments. Like, oh, do we want to take our entire code base and deploy it on a single server? Or do we want to run it on a Kubernetes farm or whatever you may have and deploy it in different ways and forms? But deployment is good. And just changing your architecture may not mean that you have better code or something like that. Like, if you had bad code in a monolith, you're just going to have many pockets of smaller code when you move to a microservice architecture. So what does that mean? Other than just the deployment aspect, you also need to think about how are you going to develop your actual application. Over here, you'll see you basically have two different ways of developing applications. You have a very integrated way, which is, I have a folder with all my code. Or you may want to keep it a little more modular, like functions, classes. We've seen a lot of talks about how to break up codes and so on and so forth. And so that's on deployment. And the last one is also on packaging. So you could say, I just have a 5 GB Docker file or Docker container, which has all my dependencies installed. Or you could have multiple smaller Dockers where you're actually installing your packages or wheels independently. So every time somebody wants to make one line of code change, they don't need to download the entire internet, which is a typical problem with NPM packages. So Monolith can be in different aspects, whether it's deployment, packaging, or even development. And we're going to be more focusing on the development aspect. Whenever we talk about monorepos, it has nothing to do as much with packaging or deployment. It's more to do with how are you going to develop and organize your code in a clearer way. Now, let's go to advantages. The main advantage is that when you have a monolith application, you may say monolith is bad, but when you start developing something, you don't know how to break it up, you don't know how to make it modular, especially because requirements change. Let's say we started Corridor about five years ago, for the first one year, 80% of the code just kept changing every month. That's kind of how it is. But monoliths have the advantage of just saying, everyone understands everything that's happening within the application. Just because everything is in one place, you don't need to know anything too complicated. You just understand what's happening overall. So in general, developers can contribute to any part of the code easier. You don't have to chase around things. You can reuse and refactor codes quite easily. And also, deploying and everything is much easier with monoliths. It's just the old-school, simple way of doing things. So I would actually encourage everyone, if you're working on a new project, don't think about microservices, at least at the start. Start with just writing your code, and then you'll see how the structure evolves. Well, let's also talk about the issues of monolith. So the main difference is, as soon as a part of your application needs to be slightly different, that's when you have problems. What do I mean by slightly different? So let's say there's a particular API or a particular machine learning model that you have which just needs to frequently change, right? Just because of market conditions every month you need to update it or whatever it is. Or there is something which is much slower than the other aspects. So all of these are things which imply that your code base is normally uniform, but one part just needs to change quite frequently. And sometimes it's even like tooling. So we saw a lot of talks about Rust and Python and so on and so forth. So many times it's just that something is really slow. I need to rewrite it in Rust. Do I create a separate repo for it? Do I just use it in the same repo? All of these kind of issues. And can you guess what the solution for all these issues are? Well, it's in the title. It's basically monorepos. So monorepos are basically here to help with some of these issues. But again, I just want to be like the disclaimer over here is you go into something and you think that it's the best thing that you've seen in the world, but typically that's never the case. So monorepos are great, but then they also have their own issues. and you need to know when to use monorepos in the right way. So let's just maybe talk about what are the standard maintenance tasks that any maintainer needs to manage. So all of these are familiar, like install, format, linting, testing, packaging, et cetera. What you'll see is that a bunch of these tasks, if there is one team which is working on the entire code base, let's say your monolith code base, and now you want to start branching out, you want to say, oh, I want to write something and rust as an example. The image at nature is, all right, I'm going to create a new repository for it, and then I'm going to work on my Rust. And you may say, oh, this thing needs a Jupyter extension or something like that, so I need to start writing some JavaScript code now. So let me create a new repository for it, and I'm going to start writing JavaScript in a separate repository, and so on and so forth. But let's say if you're doing formatting. If you have multiple repositories of Python, and each of them, let's assume you're using black or something like that to format your code, you typically want to pin your version of Black, so that people are not formatting differently in different parts of your company or teams. Now, just imagine going to 20, 30, 40 different repositories and going and updating the version of Black every single time there's a new change. It gets very painful. So you may think in the starting that it's easy to manage two, three, four repositories. But over time, it becomes very difficult and can just become a lot more time consuming. And that's when you kind of have two solutions. We either say, let me go, and that's what we had to decide at Corridor a while back, is do we want to create now bots which will go and synchronize all our repositories, which is a solution too, or do you want to just reevaluate the correct structure that you want to use based on your current understanding of your organization? So a bunch of tasks which are difficult when one team is working on multiple repos. The second category of things over here is when you have multiple repositories which are very tightly coupled, like let's say a package one which really heavily depends on package two. It could just be, let's say, a bunch of APIs which are meant for machine learning deployment, and so you have a bunch of machine learning models which need to be deployed in those APIs. Now, every time that you're going to deploy the API, it's impossible to test or deploy the API, like an integration test, without the actual machine learning models. So therefore, they're pretty tightly integrated. One cannot really exist without the other in this case, at least from a deployment perspective. So therefore, packaging, deployment, And all of these activities are difficult when multiple projects are tightly coupled and you have different repositories for each of the projects. So what does that mean? So we're going to talk about the standard questions, like when, why, how to use monorepos. So when should monorepos be used based on everything we just saw right now? If there is the same team working on the majority of the codebase, monorepos are great. Because the team needs to know all the different kinds of dependencies or code changes they may need to do, not only in their project, but also other projects. If your projects are very tightly coupled, like one cannot work without the other situations, over there, again, monorepos are great, right? So there are normally two ways that projects work. They basically integrate tightly, or there's like a handshake, you know, like how APIs work, as an example. So you have a handshake of how two projects can work together. So if they're very tightly coupled, then monorepos are, again, good for that. And the last one is more of a realistic situation. You could say that, oh, my entire team is, my entire company of 1,000 people work very cohesively, and they're all working on different projects, and all of them are fairly related to each other. But realistically, if you're going to be using tools in the open source like Git, GitHub, et cetera, you cannot really be using, I don't know, 10 GB of code, put your entire company's worth of code inside one repository. and there are just real-life considerations that also come into the picture, right? Unless you want to rewrite Git, which is a project in itself. So typically, when you're using a monorepo, these are the three things to consider. The team, the projects, and also the tools that you have available. And the next question is audience, which is who is going to be impacted by the monorepo? Like, if you made a decision of using monorepo, who's going to be impacted by it? So there are three folks, right? Like, first is the end user. There should be zero impact on the end user. Because Monorepos is just a way of structuring your code base. So if you're saying that if I move one line on the top and my entire application breaks, you have bigger issues to deal with, I don't think Monorepos is the answer for that. So end users should just never be impacted. That's number one. Number two is developers. So as developers, we're used to a particular way of working, like how to install a package, how to clone multiple things, and so on and so forth. And this kind of becomes a little more complex, especially with Monorepos in the current way the way the tools are set up in GitHub and so on and so forth, access is a major concern that a lot of people have. If I put all my company's code into a single repo, should everyone have access to all the code at all the points in time? So things like that just need to be considered about how developers will be impacted when you're moving to monorepos. And the last one is CI environments, which is actually the most common reason why I see people moving to a monorepo, saying that, you know, we deployed something and our entire build broke because another component was not deployed at the same time, or the latest version was not used in tightly coupled projects, all of these kind of things. So you just need to realize who do you want to impact? End users should never be impacted, but CIN developers who should be impacted. And the last one, the most important one, is how should the code be structured with the monoreppers? So the first options, there are a few options here, so we'll just go through it. The first option is just have all the code in subdirectories. So let's say if I have a folder for ML, a different folder for reports, et cetera, you can just have all your code for ML and ML, all the code for reports and reports. In a previous environment, this could be a separate repository. This could be a separate repository kind of thing. But let's say all your code is just in a folder, which is then being pushed to Git. The second one is have separate packages for each part of your code. So you'll see that the PyProject.toml, the setup.pypyproject.oml, which has been added over there, just saying that, oh, ML is just an independent entity or an independent Python package, which can then be deployed separately, packaged separately, et cetera. Same for reports and so on and so forth. And when you have separate packages for each part, that's when you normally think about, should I keep the packages in the same repository, or should I keep it in a single repository? So this is basically the decisions that you have to make about how the code should be structured. And two of these lead to the fact that you're going to be creating a monorepo. Because if you have a single directory, it has to be a monorepo. You have to commit it together. Or if you're keeping all the packages in one repo, it again becomes a monorepo. All right, so let's maybe just take some examples to understand this whole structure and all of these a little better. So the most basic thing, right, like the steps that you need to move towards a monorepo or start using a monorepo is, number one, move all your code to a single repo. Or if it's already in a single repo, structure it out in the form that you want for long term and here is where you can decide the two options that we spoke about integrated monorepo which is everything in one folder or multiple packages in the monorepo and the second one is who needs to be impacted we spoke about ci and developers so that's a call that needs to be made and a lot of people have just come back and told me that you know i i love monorepos it'll help my ci environment and so on so forth but my developers don't want to move right so there is there are options of just managing your monorepo structure so that your developers have minimal impact or no impact at all. And the last one, most important one over here, is the tooling that's required to implement this. So there are different tools, and we'll go into some examples of these tools. But you just need to understand how these tools work and the benefits that they provide. All right, so the first point that we talked about, which is moving all the code to a single repo, there's a decision to be made over there about packaged or integrated. So let's talk about that a little more. Over here, the general thought process with integrated and packaged is, first of all, dependencies. In an integrated environment, because you just have a folder and then different files using each other, import statements are the way that one file can be used in another file. So your most basic unit is a file. So in an integrated setup, you normally say, I want to test this file, or I want to modify this file. You don't really care as much about packages and how it's going to get deployed. You just care about the files that you need to modify. In a package-based, on the other hand, it's more like the dependencies are managed by requirements for txt or your setup.py, just saying which packages does this particular package depend on, and so on and so forth. So the single fundamental unit in integrated is a file, whereas in a package-based, it's a package. And tooling. So in an integrated environment, typically, because you don't differentiate between, oh, this folder is for project one, this is for project two, and so on and so forth. You normally have a single set of tools that you use for all of them. Let's just take an example of Poetry or PIP or something like that. You're basically going to be using the same installer for the entire folder, whether you're managing requirements and so on and so forth. But in package-based, because you've divided them into separate packages, you can choose your tools per package. So if you have an existing setup where different teams are using different repositories, and each of them have their own package, like one uses black, one uses rough, et cetera, it's much easier to move towards package-based, because every package can just reside independently and still work in the same way inside your monorepo. So migrating is basically not as easy with integrated, because it's just a whole different way of thinking if you have an existing set of polyrepo packages. But in package-based, it's a lot easier, because it's just the same setup. And the last one, probably the most important for folks who are more interested in the CI part, is caching. So remember, we said that the single unit is at a file level. And most of the tools that we'll talk about, the biggest thing that happens when you move from poly repos to mono repos is, let's say you have test cases. You have different packages, five packages. Each of them have, let's say, 1,000 test cases, which are written out. Every time you're submitting a pull request, you don't want all 5,000 test cases to run. If I just made one line code change in package one, I just want maybe those 1,000, maybe even less than 1,000 test cases to run for that one package. In a polyrepo setup, that was happening automatically, because I would push to one repository, and then the 1,000 test cases for that repo would, let's say, run. So when you go to a monorepo structure, the biggest question is, how do you figure out what are the affected changes, like which files or which packages are affected, and only run the test cases, the formatting, whatever other CI processes you have, just for that portion? And that's the cool thing about integrated repos. It's like even if I go and modify one file, it can figure out that don't even run 1,000 test cases. Only these five test cases need to be run for that file. So all the dependencies, because they're at a file level, it's basically like saying every file is a package of its own. So you can just test that one unit, which is that one file. So integrated setups are a lot more efficient in general. But package-based setups are a lot more intuitive, because that's how we normally think about things. like what gets deployed, there's a wheel for every package, et cetera. So that's kind of the difference between caching and integrated and package-based. All right, so there are three types of monorepos that we're going to be talking about over here. The first one is just purely CI-based, which is saying, developers don't need to be impacted, just the CI environment matters. I just want to speed up my CI, which is Travis, GitHub Actions, whatever you're using over there. So over here, first of all, you can structure your code. In CI-based environments, you can actually use either one, like package or integrated. It really doesn't matter. So you can use whichever approach you want to do. And then you need to create some configuration in your CI tool. Like, let's say if it's GitHub Actions or CircleCI, et cetera, on what are the affected changes. Remember, we spoke about the biggest issue in Mono Repos is the affected changes. And typically, most CI tools have a way of dictating this. Let me just take the example of GitHub Actions. Let's say if I have the reports integrated tests, I can give a configuration of the paths parameter to GitHub Actions to say, whenever these folders or these files are getting modified, only then run my integrated test cases. So that's how my CI environment knows which test cases to run, depending on which part of the monorepo is being modified. So that's pretty simple. This is the most basic version, where developers are not getting impacted in any way. You could have multiple packages, Or you could even take, let's say, your entire monolith. Let's assume Django packages as an example. So you can just take your entire Django application. It has different modules. So that's kind of like an integrated setup. And you can just test different parts of your Django application or so whenever you're pushing into a CI environment. So that's pretty much it, because you don't need to do much. It's just a CI that gets optimized. The next one is package-based monorepo. So in the package-based monorepo, you'll see first you need to structure your code. And it needs to be package-based, hence the name. So when you're structuring it as a package-based, remember that it's like you could have multiple folders. And there are different folders with specific packages, like pyproject or setup.py, however you want to package it. It can be anything. I'm giving the example of Python over here, but it could also be an NPM package. It could be a Maven package, whatever you want. It doesn't really matter. So different packages which have different languages or different tools. And then you need to decide on which tool you want to use. So a quick note over here is that I found that the Python tooling for package-based monorepos is not that strong. Like, Poetry has a bunch of stuff as they're trying to do with Poetry Workspace, but that's still not released, or they're still creating plugins for it. But one of the best tools that I've seen for package-based monorepos, and this tool is language agnostic. So it's NX, for those of you who are familiar with it. It's basically very commonly used in the JavaScript ecosystem for Angular and React applications, Next, and so on. So Nx has some pretty generic ways of how it manages monorepos, which can also be used for Python. And that's the example that I'm going to be talking about, which is first decide on a tool, initialize it using whatever command is there in that tool, let's say nx init. And then you need to configure the tool. And that's the hardest part, I guess. So let's just talk about this a little more. First of all, let me go into a project that I have open over here. So here's an example Nx library. Let me just close a few things. So here's an example Nx library. You have an Nx.json, which is the core file which just tells Nx how to get installed and which version to use and stuff like that. And in that, you have some things about cacheable operations and stuff which we can come back to. And we'll also talk about default base. But along with the Nx.json file, you'll see that I have, in this case, two packages or two libraries. And inside reports, I have something called a project.json. So project.json is how NX finds what are all the projects in your monorepo, whether it's libraries, applications. They have some different categories, but every project.json is considered as a single unit or a single package in NX. And here you can define different targets. So NX has a very general thing, like just run a bash command, which is the run commands approach. So you have different targets for how to do an install, how to, let's say, run your formatters, how to run the formatters, your test cases with PyTest, and so on and so forth. And you can define each of these tasks for this package. The cool thing about this is that over here, I've written run commands for Python. But my other project, let's say, Data Lake or Reports or something else, or ML or something else, could be using a completely different language. And Nx doesn't care about that. So over here, we could be having an NPM command, or a Java command, Scala, Rust, whatever you want, like cargo, et cetera. Different kinds of commands can be here. And Nx is pretty agnostic to the language that it uses in this case. And you'll notice that along with these targets or tasks, you also say, what does it depend on, and what are the inputs? So you can chain your tasks, saying that, oh, before you run test cases, you need to build your application. so only then you can test it. And Nx will then realize that if you're trying to run the test command, it will automatically run the build and the install commands before that. So it really handles your task pipelines in a pretty clean way. And the second one is inputs, where it says, what are all the files that this command uses? So let's say if I'm doing pytest, I care about the source files and the test files. If I ever change any of my source files, I want PyTest to run again. If I ever change any of my testing files, or even the configuration files and so on and so forth, I may want PyTest to run again. So using the inputs, NX basically knows the whole affected, the thing that we were talking about in the CI environment, where you had to manually provide each of your files. Basically, it's kind of similar, but over here, you have just a little more elegant way of defining which files matter for this task. What this means is, in your CI environment, And we'll go into an example of this. If anyone goes and modifies a .py file, let's say source files, right? Like, source files are defined on the top over here. Anything in this folder is considered a source file. So if any developer is going and modifying a file in this folder, in your CI environment, you should be running the test command. And NX figures that out for you. NX also has a lot of caching operations, which I haven't gone into, like the cacheable operations over here. So they even have this whole cloud solution where even as a developer, if I'm running something, if none of the inputs have changed, then it'll just cache the results for me. And it's pretty elegant and cool. I'd encourage you to read up more about it after the talk. So that's how Nx works. You have an Nx.json, you have a project.json. In the project.json, you define your targets. And then you talk about all these dependencies and so on and so forth in project.json. A quick note here is that this could get cumbersome over time. Let's say if I have PyTest, I need to define every single folder, which can be a pain, so on and so forth. There are plugins in NX available, especially, again, in the JavaScript world, to go and scrape your code base and figure out the structure of your code and figure out these configurations for you. In Python, also, there used to be a plugin, but it's not maintained right now. So I haven't talked about it, nor am I using it, because it's been like three years since they released or something. So at least in my team, we've created a few packages, which hopefully will open source in the future. So that's basically how NX works. You can have better executors which understand your code base. But for Python, it does not exist right now. Now, once you've configured your tool, and NX knows what it's doing, or knows what your Monorepo is structured, now you need to start documenting and start using this tool. So here's where the benefits come through. You have three most common things that you would want to do with Monorepos. And there are many more, but I've just taken three. So you want to run all your test cases on all your packages. So you can just say nx run on all packages, all my test cases. And again, remember that the tests can be in any language. It can run Java test cases, Rust test cases, C test cases, Python test cases, all with this one command. Then you want to run something on a single package. So you could say run on the reports, run only test cases for the reports package. So you can do that. And this is the most interesting one, at least for me, which I was describing in terms of the whole caching operations, which is Nx tries to figure out if something is whatever is affected only run the test cases for whatever is affected and now you may ask hey what is affected even mean so you can define that in the Nx.json you'll see that the affected is basically the default base is the main branch so affected over here would be take the main branch see all the files that have been modified since your main branch your latest commit in your main branch and only run test cases for whatever has been modified. If I create a feature branch and I'm trying to do various code development over there, I can just run the test cases for whatever I've modified, and NX can figure that out pretty neatly. So that's basically what NX does. So that's how a package-based setup typically looks like. Let's go into the integrated monorepo. So the integrated monorepo is the one which is a little more non-intuitive, so I'm going to spend maybe a little more time on that. In an integrated setup, you want to structure your code as an integrated structure. So let's talk about this. And over here, you'll see that I have a folder called Corridor, and I just have different folders for all my packages, and just writing the py files over there. So you'll notice there's no setup.py, no pyproject.toml, and all of these kind of things. Just think of it like literally one large code base. So if you're going from the monolith world where you have one large code base, integrated may actually be easier as compared to moving with package-based. So it kind of depends on where you're going to move from. But in the data lake, I have all these files, and in any one of them, I'm actually importing things from, let's say, the data lake component and so on and so forth. Now where do all these dependencies and packages kind of come from? So remember that we said in an integrated setup, every file is considered a module or a package, which means that this histogram.py is considered as a basic unit and a package. You can even deploy histogram.py just as a separate Docker container or whatever you may want. It's just a completely independent unit. You may want to test it, you may want to deploy it, you may want to package it in completely different ways. And the thing that a lot of people ask me is, what does that mean? So does that mean that for every file I need to write a requirements for TXT and so on and so forth? Well, not exactly. Using your import statements, you can normally figure out what your dependencies and so on and so forth are. And there are a lot of tools which help you, especially with integrated setups, to manage this code base in a clearer way. So let's just go through that. So in terms of tooling, you have a lot more options for integrated setups, because that's That's normally the place when people say monorepos that typically mean integrated monorepos, which again, not everyone needs to go there. I think it's only required for some very specific cases. But integrated monorepos is the norm. And Pants, Bazel, Buck are all the common tools that exist over here. I'm going to be talking about Pants a little more, just because Pants is written in Python, much easier to use with Python. It's like Python native monorepos. And Bazel, Buck, and so on are more for other languages, but can also be used in Python and language agnostic in general. So first of all, you install Pants, can be as simple as just a shell script that you run, and again, configuring the tool. So here's where things become a little non-intuitive. When you run a Pants tailor command, in Pants, everything works with this configuration file called the build file. Build is a simple TOML file, nothing else. So let's go ahead and take a quick example of it. So you'll see something out there, but basically a typical build file is just a file where you can define your configurations for pants, and pants configurations are like functions in Python. They've tried to keep it very similar to a Python syntax, where you can define what exactly is in your, well, the build file basically defines the structure of your monorepo as well as it can. I'm going to take a different example just because that one is very minimal. So let's say I'm in the data lake setup, right? I can define all the test cases inside this folder using a Python test keyword. The Python test basically takes anything which ends with underscore test.py and defines that as all the test cases for the data lake folder. You can even have resources. So let's say if I have a CSV file, you can define that as a resource, like data files and so on and so forth. And you can have even source code, which is any dot py file. And you'll notice over here that my py files are reading my CSV file. So therefore, there's a dependency saying my py file requires the CSV file to run correctly. So you'll again notice, similar to what we were doing in Projector JSON, but just in In a different way, we are defining the different components, or every file is being mapped to a type, whether it's a test case, a resource, a source file, et cetera. And these types help Pants to understand your repository a lot better. Once it understands it, it can try figuring out things like what is affected. For example, let's again assume that I want to run test cases, right? So let's say I go and modify my CSV file and go and change some data inside the CSV file. When I push this out into a CI environment and so on and so forth and I tell Pants, hey, just tell me whatever has been affected and run all the test cases, it will figure out that this CSV file is being used in these set of Python sources, these set of Python sources are being tested by these set of Python tests, so therefore all the tests in this folder need to be rerun because the CSV file has been modified. So very similar to what the project or JSON in NX was doing, but at a file level as compared to a project level, right? So the entire point of all these configurations, just summarizing, is for the tool to understand the contents of your monorepo. And along with all this stuff about how to understand the contents of the monorepo, you also have configurations for how to actually distribute, package, and so on and so forth. So there's like a Python distribution setup, which says, I want to create a wheel, a wheel file for Python. And for this, you can use some setup.py which already exists, so you can embed a setup.tools.setup.py. And also you can say what are all the files that need to be present inside this wheel file. So you can essentially, like, you don't use things like the PyProject and so on and so forth. Again, you could use PyProject and tell Pants to read from your PyProject. So that is also doable. But Pants needs to know how to create an executable or a wheel file in this case from your repository. And the cool thing here is, because I just have the entire folder structure, I could have thousands of folders, at any point in time, I can just say, hey, I want a build file over there, and I want to create this as a separate Docker file. So let's assume that you were deploying your entire project as a monolith on this file. Everything put into a single Docker file and then just getting deployed. You can suddenly just say, oh, in this particular portion over here, I want to add a build file, I want to create a separate wheel from it, and then I can start doing different things in the wheel. And there's no code changes that you do at all, because in an integrated setup, everything is at a file level, so you really don't need to worry about changing your code, moving all your code from one place into another place, none of that stuff. Your code remains the same, and then Pants can understand what needs to be packaged and deployed, and there are many other functions, you can even deploy to Lambda and all these kind of things directly from Pants. So that's basically about configuring things, and the build file is where you'll see most of the configurations that are present. And coming to the final usage and documentation, so that's where, again, in PANS, if you want to run on all your projects, and note over here that we don't talk about projects, we just talk about files. So if you want to run all your files test cases, you can just say PANS test colon colon, it will run all test cases in your entire, so all 5,000 test cases in the previous example will just run. You can run it on a single file. Again, note, we never talk about, I want to run test cases for a wheel, or I want to run test cases for a particular package. It's always at a file level, because everything over here is all about files. And you can also say changes, like the equivalent to the affected command that we had in Nx. So you can say, I want pans to run all the test cases of everything that has changed since the main branch's last commit. So same thought process as Nx. It takes the last commit, figures out all your changed files, and runs the test cases. So just repeating one point, which is the most important point with integrated, right? Integrated is a lot more efficient, because you don't need to run 1,000 test cases every time you're pushing something in one package. You may only run five test cases or ten test cases. But it also may not be the most intuitive if you're coming from a package-based polyrepo repo stage, where you have multiple repos and multiple packages. So it just depends on which direction you want to move your code base into. So yeah, I think that's about it. Just to summarize everything we spoke about, you have monolith architecture and you have microservices. I think monorepos is a good segue which helps you structure out your code amongst both of them. And again, don't think of monoliths as a bad thing. A lot of people just think of it as a taboo. It's kind of like the go-to to function. Nobody should use go to, just use for loops. So Monoliths is useful in the right places, which is normally in the start of a project when you still don't know how your code is going to evolve. Once you start seeing some structures, you can move towards monorepos. And when you're in a monorepo setup, you can still move towards microservices quite easily, so therefore it's a good segue into structuring your code in a better way. And monorepos are great when you have a single team working on all the projects that you're working on, and all the projects are linked to each other in some form or the other. So in that setup, monorepos are great. If you have multiple teams who don't care about the other team's code, you're probably better off just keeping separate repos so that they don't break each other's code as compared to trying to help each other. And if only CI is a concern, then don't bother with changing your developers. Training your developers to start running new commands, like getting all my developers to change from pip install to nx install was just a big effort in itself, because no one remembers and no one does it how much of a documentation you write. So if you don't need to move your developer, change your developer experience, just don't change it and just focus on your CI. And the last one is package-based monorepos are easy to understand, and integrated monorepos are just a lot more efficient, mainly because they have a unit of a file. Right? That's about it. I think the references and so on and so forth you can take a look at from the slides later Thank you, Olli, for your great talk. And thanks to all of you for your questions. We will go to them now. Could you briefly elaborate on the role of Git submodules in the ecosystem of monorepos? So good question on Git submodules. So I kind of think as Git, well, let me just go one step backward. I think that when you're trying to structure your code, you can have all your code in one repo or all the code in multiple repos. When you're using multiple repos, you could be using different kinds of approaches of integrating your repos in some form or the other. A very common example is just using a Git bot, right? People have just written their own GitHub bots, which will go and synchronize all the CI tests or the dependencies across all the things. Like ContaForge is a very good example, which I really like, for those of you who use ContaForge. They basically have a bot which will go and update all the packages in ContaForge if anything needs to be changed consistently. Submodules are just another way of synchronizing multiple repositories. I think they have their own benefits. The biggest issue which I have with Git submodules is because it's at a commit level and not at a branch level, Every time somebody makes a change in, let's say, the main branch, somebody needs to go and remember that they need to update the commit in your parent repo. And that just gets, like, frustrating at some points. But again, remember, there's no one-size-fits-all. These are all just different approaches, and what works for you just works for you, and that's cool. These are different ways of managing your code base. If you have hundreds of commits from other developers every day on a monorepo, how do you keep on repo and think when you work on it? Well, I think the most standard practice over there is just pull every day or pull every half a day and just rebase. I'm not sure about, it kind of depends on what the structure of your organization is maybe and if somebody asks the question from here, maybe you want to elaborate so I can get a more pointed answer. But at least I typically recommend feature branches to be very short. You should not be having more than 10 to 15 commits, and every commit should not have more than 20 lines of code change. So very small atomic changes that can be quickly merged is what I would generally recommend with Git. I know different people have different preferences, so it just depends on what kind of setup you're using. Next question, in the package-based monorepo, how would you add ML as a dependency of reports, roughly speaking, poetry at PyPML versus poetry at local ML? Oh, okay. So the question, let me just rephrase that question, is if I have package-based repo, how does one package know that they need to depend on another package, right? like in a package one, package two. Let's say ML is being used in the reports. So whenever we install reports, I guess, the ML package also needs to be installed. So remember that in a package-based monorepo, every package can use their own tooling. So if you're using poetry, you just define it however poetry does it. If you're using requirements files, you can do a minus e and then the path. So whatever tool you're using to manage that particular package can just be the same. There's no difference. The only nuance over there, or the benefit over there, is that you know exactly what folder structure it's at. So let's say if you had two different packages and two different repos, you're expecting the developer to clone two things and then install this package in this guy's virtual environment. And that can be different commands for different people, because different people clone things with different names and stuff like that. But in a monorepo, it's a lot more structured, so you can actually hard code the path in your requirements file. And there's a very good plugin which got created in poetry very recently, which says, if you're using a relative path, then whenever you're packaging it, change it to an actual package in PyPy. So stuff like that can be used, but it all depends on the packaging manager and the tools that you're using in that package. Why is dependency management easier in monorepos? An example, and different packages have different dependencies. Well, I think I kind of answered that in the previous question, but it's just that dependency management is a lot easier, because let's say if I'm using a tool, and I'm expecting or let's say if I'm using Jupyter, right, a good example for everyone. When you're using Jupyter, there are multiple dependencies that are present, and you need to make sure they're all synchronized, like your Jupyter core, your IPI widgets, your Jupyter lab, Jupyter notebook server, all of these, you need to make sure all the dependencies are matching, if they're all being maintained by the same team, and they're very tightly coupled that, let's say, package version 5.x needs 5.x from another package, then with a monorepo, you can just make sure you're always using 5.x. If you go back in time, let's say a year earlier, all your packages are getting consistently downgraded to 4.x. So these kind of things just become a lot easier with the monorepo, because if all your packages are tightly coupled, they're just moving in the same pace, I guess, in terms of your release cycles. Even when you're deploying, you can just deploy everything, like 4.3, all of them just get deployed as 4.3. You don't have one at 4.3, one at 4.2, one at 4.10, and then developers need to figure out which versions are actually compatible, then you need to have a support matrix of what are compatible. It just becomes a lot more headache, right? So that's why monorepos are easier to manage for developers. There are so many more questions, but I think we have just time for one more. I think if the people have questions, they can come to you. Yeah, I'll be around the entire day, so you can just catch me at any time. Okay, last question. What's your stance on using Git submodules? As I said, I personally don't like Git submodules just because I need to keep synchronizing my other repositories. If Git submodule was at a tag level or at a branch level, I think I would have loved it. But sadly, it's at a commit level where you need to give the exact commit ID that you need to update. So before I started even researching into monorepos, we were using submodules. And when I was maintaining, I'm normally very clear about what needs to be updated, etc. It's easy for me to go and remember I need to update this, and this process, and this process. But again, how much ever you document stuff, the next person, like when the release manager changed for our company, the entire thing was messed up. So because of stuff like that, I think Git submodule has oddities. And therefore, unless you're really comfortable with it, I generally try avoiding using it. Again, very rare scenarios where it's useful, but I prefer either having a bot, which will then go and synchronize all my submodules to make sure the branch commits are the same, or just moving to a monorepo setup. Okay, thank you for sharing your ideas with us. To all of you, have a great day.