Kickstart Coding at Scale: How Project Template Automation Unlocks Developer Productivity

Speaker 1 [00:00]

I can set a destination directory where the product should go to. And if I do that, it looks like this. So here I use Pixie to create a temporary environment just for executing Copier. And then it asks me the different questions that I have defined in the configuration file. So here it asks about the product slug. I can enter a description. I can enter my username, my name, my email address. And then it generates a GitHub URL. I can select the minimal Python version. I don't want to use dev containers. And then in the end, there was a question about whether I want to use a GitHub Actions workflow to automatically update my log file. And that's all I need to do. And then it creates all these files, and I can start working on my project. Yes. And this is how the, like, we have already seen the cookiecutter.json file. This is the copy.yaml file, which works quite similarly. You can define different questions. We have seen these questions. And you can answer them and then insert the answers to these questions in your template. And this, for example, how the pyproject.toml file looks like. And the .ginger extension here indicates that it's a templated file. And here, for example, I can also insert the minimal Python version that we selected. And now the difference to cookie cutter is that we actually save the answers that we give to these questions to our repository. And this allows the intelligent update mechanism to keep our project in sync with the development lifecycle. Because here, it saves the git tag that we have used to generate a project. So in this case, it's version 0.17.1. It also saves the source path of our template. It also saves the minimal Python version and the project slug. And in reality, our internal template has maybe 15 further questions that allow you to really customize what you want and what you don't want in your project. And now we have discussed how to define such a template. I have also mentioned that we have maybe 20 different questions in our internal template. Now the question might be, OK, how do you test this? Because there is so much variability. And yeah, ginger can get quite messy. And it might not be obvious how a certain variant of the generated product looks like. And we have designed this. It's a GitHub Actions workflow that allows us to test our template. And this works by, for example, we open a pull request. We want to introduce a new feature to our template. Then we have this GitHub Actions workflow. And we just generate the projects for different inputs to the copier questions. And we just push them to internal branches of our repository with a random hash here in the end. So here maybe I've selected minimal Python version 3.13. And we have this abled left hook. And yeah, maybe we also generate this variant. and we generate a lot of them, and we push them all to our repository. And now the intelligent thing is that because our template contains GitHub actions workflows that themselves define the CI, we actually run the CI in our generated projects, see if it goes through. And if all different variants go through, we also pass the CI of our feature branch. And this is how we can have CI on our project template, which just inherits from the CI that we already define in the template. Now I mentioned the update mechanism a few times, just to give you some understanding of it. So let's say we have our current project, and it was started with this old version of the template, and we now have a new version of the template that maybe fixes a problem in our Conda build process. So what do we do? Copier regenerates a new project from the old template, and it can use the copier answers file to do that, because there all the answers are defined that we use to bootstrap our current project. And what we can also do, we can also generate a clean project from the new template, so how the template would look like if we just insert all the answers there. And then all we have to do, and this is a huge simplification, is a three-way Git merge, because essentially we have developed clean project old to the state of current project, and in the meantime, clean project old has evolved to clean project new. And this is a solved problem, and then you get your updated project. Of course, there can be merge conflicts, but it shouldn't be too many if you don't drift away too much from the template. And from our experience, things, for example, like the PyTest workflow or the Conda build process or like the CI that builds the Docker image, they rarely have to be touched. So this should not create too many too complicated merge conflicts. Okay. And we have automated this at large scale. So we at QuantGo have our fellow GitHub bots, QuantRanger, and every month it comes around and makes a PR that updates the template version to the latest version of the template, and it also gives a nice change log and explains what has changed. Here in this case, we introduced the agent's MD file and cloud MD file, and also supported Python 3.14. And in the best case, our developers that work on the project, they just have to accept this pull request and then they are on the latest state of things and don't have to think much about all the basic things that set up their project. And this really helps us to scale software development. Now a different concept I also want to introduce, though really related, is how we do migrations. So one example of a migration might be, or like one migration that we actually did, is we have a lot of GitHub actions workflows, and it's really easy to configure them in an insecure way, for example, by not pinning the dependencies, or by using caching in an insecure way, or by allowing template injection. So if you are not careful, then it might be possible that you actually execute the user-provided input, and there's a linter for that called sysmore, and it can lint those GitHub actions files. And what we wanted to do is we wanted to add this linter as a pre-commit hook to all of our repositories. But the problem now is we can't just add this in our copier template, because then these version update PRs would have all read CI, and then if some developers don't invest the time to resolve these issues, then this project becomes stale and we can't update the version anymore. So what we do instead is we do a migration, and we do that by introducing a question in our template, so in the copy of YAML file we just ask do you want to use this more for GitHub actions linting, and we actually set the default to false because of a technicality so that if you upgrade to a new template version, you don't get automatically this set to yes, so you have to opt in explicitly, and then we wait for the template version to be rolled out, so all templates have sysmo set to false, but the template would support it. And what we then do is then we start another bot process, and it makes automated PRs that just changes the question value from no to yes, and this is the migration that then allows developers to deal with this separate from any template update PRs. And then in this migration PRs, they would have to have a look at these errors and fix them in the project, then they can accept the pull request, and then we are good to go because then we have the security linting enabled. And then, of course, if all repositories are migrated, we can remove the option, just set the default to yes, and then the migration is done. This is how the PR looked like, so all it does is execute the copy update command and just change the value of the question. And then to keep track of our migration progress and also progress of our template rollout, we have built a dashboard, currently it's built in Streamlet, let's see if we change that, and the dashboard allows us to track the progress in our template development. So the most basic metric is we can track the template version. And here, we at Quantico have around 190 repositories managed by our copier template, and that's almost all Python repositories. And so basically we don't have any repositories that aren't managed by copier. And as you see, like around 40% were yesterday upgraded to the latest version, then another around 30% are a bit behind, and then it spreads out a bit, it might be projects that are inactive but weren't archived yet, but you have a great overview about this, and what we then have to do if we want to roll out a new version, like once in a while we look into these old projects and see whether we can archive them and then start with a project, but it can't happen that there's a project that still uses outdated technology or outdated ways of doing things that we aren't aware of, because we have this dashboard and we can track progress and we see what has been adopted in which projects. This allows us to have a great overview of things. We can also track the progress of these migrations, like yesterday we have rolled out Sysmore in around 70% of repositories, 6% have explicitly opted out for some reason, probably that's just some new projects that haven't set the value yet, and those not applicable repositories, they just haven't upgraded to the template version that supports Sysmore. And we have introduced this Sysmore option around six weeks ago, and I would say it's quite fast to migrate 70% of 190 projects within six weeks. So I think this really shows that we really are more productive with using this templating approach. I also promised in the abstract of this session a few lessons learned. So one lesson that we learned is that these update PRs really need to be as straightforward as possible Or we risk that they don't get merged. Because if it's just too much work every start of a month, then they just become stale. So we really want to make the process as straightforward as possible for our project developers. And one key part of this is that we avoid merge conflicts where possible. And we eliminated two instances where these merge conflicts could happen. First instance, it happens that in our copier template, we have these GitHub actions workflows defined, and we use third-party actions, and they are pinned to a certain version. And apart from that, our project repositories, they also have these GitHub actions workflow, of course, and they have Dependabot in place that just updates the dependency automatically. And because both the template and the project repositories touches these lines, that define the dependencies, we get a lot of merge conflicts here. So our solution is or was to just introduce a custom script that handles this case, and this eliminates 90% of all merge conflicts. And another small part is MergyRav. It's a Git merge driver. I really recommend it. You can also install it locally, and it resolves some merge conflicts based on the AST of the language that you are using. So, for example, we have a PixieToml file that defines our dependencies, And if the project adds a dependency and the template also adds a dependency, normally this is a merge conflict because Git does not know which order to put the dependencies. But, of course, we are not interested in the order because it doesn't matter. And MergeGraph can detect that and then just resolve the conflict by choosing an arbitrary order. Yeah, and then also we are quite happy with our migration process that I've shown you earlier. Copier also has an inbuilt functionality of handling these migrations. Basically, you can define, okay, when migrating from template version XYZ to this other version, then also execute this custom script. But we at least had a bit of trouble with testing this properly. And we found that the option of rolling out template updates and migrations independently by just introducing this additional question that we eventually change to the other value is the most stable approach. So we also found this is also a lesson that we have learned. Yeah, one final word. Maybe you now think, okay, it seems quite complicated still. I have to manage the template and all this automation. Maybe let's just pull a lot of the common code in repositories. So, for example, CI pipelines, let's just move them to one central place and define it one central place. In our company, if you want to build a Docker image, you do it like this, and then you enforce that all projects have to call your custom action, yeah, and maybe you also put a lot of other functionality in it, you also can execute pre-commit, you can build Python packages, and you can do a lot of things. But the problem is that this becomes really magic, and actually seeing this in the wild, all you get is that you imprison your developers, because then your developers can either use your magic pipeline or they can't use it, and if they don't use it, they have a really hard life, and if they use it, they can't really change much about it, so I think having a template and a paved road that your developers can be on, and it will be really easy to be in that route, but still allowing them to deviate is really like a key insight that you should have. And that's also how I want to end my talk. Yes, thanks for your attention.

Speaker 2 [15:58]

Thank you so much, Yannick, for a very nice talk. Let's open the floor for questions. For those who don't know, you can sign up and ask questions on talks.pycon.de. Now, we already have two questions. I'm going to start with the one. Do you only provide tooling with the template or also things like basic folder structures, et cetera?

Speaker 1 [16:19]

Yes, we also provide basic folder structure. But to be honest, in like a basic Python project, there is not a lot of folder structure that we could provide to our project, I think. So all we do is we have like a .github folder that contains all the workflows. And we have a folder that is called like your project slug, and you can put your Python code in it. And we also have a test folder. But I think this is about it.

Speaker 2 [16:45]

There is one more question by Yannick Khaas. You said that you create one branch per test case. These are probably more combinations of parameters you could actually test. How do you decide which cases to test?

Speaker 1 [16:59]

Yeah, so it's a quite difficult question, but I would say the answer is quite simple. I think this is just guided by intuition. So we just try to combine different parameters in those selection of test cases that we think are the most interesting. And in most cases, it's not really hard to come up with parameters to make sense. For example, there is no reason why the template should pass with minimal Python version 3.12, but not with 3.13 with all other parameters identical. So, yeah, these are some rules that we follow. But, yeah, I would say mostly it's intuition and just choosing ones that look sensible.

Speaker 2 [17:46]

Amazing. So if you have another question, there is time for you to go up and put your question there. Otherwise, let's thank Yannick a lot for a great talk with an applause. A quick question. Is the next speaker here Samuel Oslowich? if yes can we that are you the next speaker yes you can come to the front maybe thank you so much