PEP 581 and PEP 588: Migrating CPython's Issue Tracker Keynote

Speaker 1 [00:06]

I came from Vancouver, Canada, came here just for PyCon DE. I'm really honored to be invited here. I work as an engineer at Zapier. We're a fully remote company. If you are interested, we are hiring. Now, in my spare time, I volunteer. I help with the PyCascades conference. We are a traveling conference. conference we've traveled from Vancouver to Seattle and the next one is in Portland. I also volunteer for PyLadies. I help organize Vancouver PyLadies and I also volunteer at PyLadies at the global level. And in my limited free time, I still manage to contribute to open source projects. So one side effect of contributing too much to open source project is that they want to give commit right. So that's how I became a core developer of a popular project like CPython. Now a little fun fact about me, I am the emoji expert for CPython project. If you're wondering what that is, I don't know. My expertise about emoji is very rarely needed, so instead let me tell you about the core Python workflow. What is core workflow for CPython? The core workflow is a special workgroup within core Python development. We discuss improvements to the tools and infrastructure that supports CPython. Within this working group, we discuss topics like experience to contribute to CPython. So if you have questions about how to get started contributing, how to create pull request to CPython, how to write a commit right, how to write commit message, have questions about the dev guide, have questions about any of the bots we use in CPython, all of those are within the realm of the core workflow. And to give you an idea, these are all the different pieces that makes up the core workflow. First, we have the issue tracker. That is something called BPO, bugs.python.org. That's where you'll go to find issues or bugs to see Python. If you're interested in contributing, looking for things to work on, you will go to that BPO. Now the code base is something else. It's not in BPL. There's another website to go to. It's on GitHub. So if you need, once you find an issue to work on, you will need to make a pull request. That's on GitHub. And then we have various ways to communicate. We have different mailing lists like Python Dev, Python Ideas, Core mentorship mailing list and recently we've started experimenting to communicate other ways like chatting through Zulip or discourse and if all of these are not enough we need to meet in person so co-developers and contributors still go to events like conferences like this one we have sprints we have the language summit. So it's very complicated, the workflow. Now, in the past two years, there have been a lot of improvements to the workflow. It began in February 2017. That's when we finally moved our codebase from Mercurial to GitHub. That is very recent compared to the age of Python, like 30 years, right? GitHub is very recent. And pretty soon after we moved to GitHub, we realized something. The previous assumption that everybody, every developer was born with GitHub account, that everybody knows, understands Git branching, that was wrong. We realized that. In fact, after the migration, a few core developers who could not keep up with the new workflow have stopped contributing. But we reacted quickly. Soon enough, we realized that it's not good enough. We come up with better documentations, better toolings. We have new bots. And now our workflow is better than ever. But realize that it took time from the time we moved to GitHub until we have all these two links in place. So it means between this period, maintaining and contributing to Python was not ideal. It was a little bit of a chore. It was a little bit painful. So now, the workflow is so much better than ever. We have lots of bots. When on GitHub, I think it can be further improved. I think it would be great instead of using the current bug tracker, the BPO. How about we move to GitHub? We start using issues on GitHub. So I wrote a tab about this. That is what tab 581 is about. It describes the reasonings of why we'll be better off on GitHub. There are a lot of nice features. like emoji. It has very extensive APIs and we've had great success using those APIs. We made lots of bots and automations, things we cannot live without now. And part of the motivation is that from time to time I hear from people they find the current issue tracker is so intimidating that they don't want to use it. And this is not the only person who said this. I've heard this from lots of people. This is a more recent example. People are reluctant to file issues on our bug tracker, so instead they just message me on GitHub, saying, my data please take a look, can this go into CPython? And I'm sorry. We can't do this. If you have bug, if you have feature request, you have to actually go to the bug tracker and use it. I know it's painful. And thing is, we have 7,000 open issues already. It's not like we really really need more. It's not like we need to make filing issues. We need to lower that barrier of filing issues. It's true. But I care about your contribution. I care about your experience contributing. And that's why I'm looking into this. I feel this is an area that we can still help. It can be improved. Now, why do we need to move to another issue tracker? Like, if we've identified there are issues with the existing tracker, why don't we just work on improving it instead of moving to something else, right? This is an option. There is another PEP that proposes, let's stay in the current issue tracker, let's improve that. That is PEP 595. I'm not the author of that, I'm not the one driving that. But I agree, I think it would be great if we can continue using open source project like Roundup, which is what BPO is, instead of using proprietary system like GitHub. I think it sounds great in theory. In reality, I don't think people actually really, really want to develop an open source issue tracker. We've been using the same issue tracker for like 10 years. It's always been open source. Everyone can contribute, but we don't. the bug tracker has mainly stagnated. Lots of you come to me asking how to start contributing to open source, but what you really want is work on one of those 7,000 open tickets in CPython. You don't want to work on this, right? I mean, I'm sorry if I sound pessimistic, but I'll be happy to be proven wrong if somebody comes in if lots of you come in and say hey let's do this I think it'd be great but meanwhile I think the issue tracker is so important so crucial maybe we need to leave this to the professionals so so what's the holdup like if we want to use GitHub issues what are we waiting for isn't it just a matter of like taking this box it doesn't show good here but there is a box in github that says enable issues it's not that simple migrating issue tracker is such a big project that we need to write yet another tab. So that's tab 588. This is the complete plan on how we will do the migration. There are still a few things we need to address before we can do the final migration. Previously with tab 512, we moved to GitHub and then we build all these different tools this time around we want to have all those tools and documentation in place before we do the final switch one of the biggest risk is that we need to acknowledge this is github is proprietary solution it is possible it will suddenly disappear without notice right everybody's worried about this. I think it's a valid concern, but I think it's not a unique problem to CPython. 4 million GitHub users like you face the same problem, so it's not a unique problem to us. It should not be a blocker into the migration but what we're trying to do to mitigate this is that right now we've been doing we've been capturing daily backups of the complete github data of CPython so we will not lose data that's that's the best we can do right now but eventually we will need to figure out a failsafe plan like now that we have this backup, what do we do with it? What if tomorrow GitHub disappears? What do we need to do with this backup data? Maybe we need to export it to a different issue tracker. So that's something we need to think about. So maybe we'll move to GitLab, who knows. I've been talking about BPO as the issue tracker, But it's actually not just an issue tracker. It is also our CLA host, the contributor license agreement. So it holds the database of records who have signed the CLA. This is something we need if you're contributing to CPython. So if we're not going to use BPO, we need something else. We've been looking into an alternative solution in the last year, but a couple months ago after much research and discussions, the solution we were thinking about is not going to be good enough, so we might need to build something new, something custom for ourselves. And if this is something you're interested in helping contributing, I have written the complete requirements of what the CLA host will look like what it needs to be able to do i wrote like nine different points of what the cla host should be it's in that issue um core workflow issue 39 360 so if that's something you're interested in contributing follow the issue and then we're going to need to have a test repository a place where we can play with like we want to experiment with a new workflow like how do we use issues on github the idea is to copy over about hundreds of existing issues from the existing bug tracker to github and play with it and discover problems maybe we need to do improve the documentation, we need to build appropriate tools, so we need to do this. I'll also need help someone to actually write that script to import from existing bug tracker to GitHub. So, if this is something you're interested in, let me know. Once we have that repo set up, maybe we're going to start writing, updating the dev guide. This is also something I'm going to need help from people because I'm way too familiar with GitHub. I know what needs to happen, how to do things. I don't know what people want to know, like what other things that needs to be documented. So, again, if you're interested in writing documentation helping with this if you're new especially new to github i think this is um a great way to help to contribute we're going to need help triaging issues now that if people were intimidated in contributing in filing issues in existing bpo github will make it so much easier sorry, for people to file issues, right? With that, we will need help triaging, categorizing issues. And the triage role is actually something new on GitHub. It is in beta, and CPython has been granted the beta access to it. All the roles and responsibilities of the triage team is fully documented on Dev Guide, you can apply to be a member of the Python triage team. All it takes is one core developer to vouch for you. So it's easier, less barrier than becoming a core developer. So again, if this is something you're interested in doing, check out the documentation and apply. We still need more bots, more tools. We're not done building bots. About two months ago, I've requested feedback from core developers, asking them what are the features they still want from GitHub, what are the blockers they could foresee if we were to make the move to GitHub issues. all the comments have been collected in this issue the core workflow issue 359 um looking there are about 20 different comments there like feature requests so if these are some of these are probably we need to get help from github it needs to be changed from github site i will be passing some of these comments to github and ask them what they can help us with, but there are also some things that I think we can build from our side, like by using the APIs. So if you're interested in building more bots, GitHub bots, GitHub tools, check out this issue for inspiration on how you can help us. So one of the recommendations from the Python Steering Council is that for the success of this migration project, it should be handled professionally. So maybe we're going to need to hire a project manager, similar to how the migration to warehouse was handled, like how the Python end of life is being handled. We have a professional project manager for that. So maybe we need the same thing. We might even need a few core developers, maybe, full-time. I'm going to leave that to the PSF to handle how this can be professionally managed. But what you can do is donate. If you follow this link, your donations will be used directly to support Python development. now if you have other questions about this you have other ideas you can tell me about it or post in our discourse, it is under the core workflow category, I will be watching that category, so I hope you learned about CPython's workflow, I hope this gives you ideas on how to contribute to Python other than writing the code itself. If you'd like to get in touch, I'm around for the rest of the conference. Follow me on Twitter. My name is Mariana. And if you do appreciate my work, sponsor me on GitHub. So thank you so much for listening. Grateful to be here.

Speaker 2 [19:43]

Well, thanks, Marietta. So I think the lightning talks are in.

Speaker 1 [19:48]

20 minutes, no?

Speaker 2 [19:53]

so yeah you can come back in 20 minutes I don't think there will be anything in between thank you very much again thanks for coming from Vancouver and you're round

Speaker 1 [20:04]

Thank you.