5 Years, 10 Sprints, A scikit-learn Open Source Journey Keynote

Speaker 1 [00:04]

I am honored to be here today to present at the 2022 PyCon Germany and PyData Berlin conference. I have never been to Berlin and I hope one day to make it there. I'm going to do a brief introduction about myself and then I'm going to go into the sprints, the in-person sprints, the evolution of the in-person sprints, and over to the online sprints and the lessons learned in open source. My name is Reshma Shaikh. I was born in India and moved to the U.S. when I was six years old. I attended Rutgers State University of New Jersey for both my undergrad and master's in statistics. I worked as a biostatistician in the pharmaceutical industry for over 10 years and I have an MBA from NYU Stern School of Business. I am particularly excited to present here because back in November of 2019, I submitted a CFP to PyData NYC to speak about the sprints I had organized. Up until that time, there had been five, and my CFP was not accepted. So I'm really, really happy to be here. New York's a tough crowd. You can find me on Twitter, GitHub, and LinkedIn at RacheMass. I've been doing community work in the PyData space for over seven years. Some of you may know me from the documentation that I wrote up for Fast AI. I was a diversity chair for DrupalCon 2020, which you may remember was supposed to be in Berlin, but pivoted to online. Maybe you may have read some of my blogs, such as why women are flourishing in the art community, but lagging in Python. I've been an organizer for the New York City chapter of PyLadies. In 2019, I received the Non-Focus Community Service Award, and they're the scikit-learn open source sprint, which is what today's keynote is on. Some background of my community work, I was an active organizer and contributor to the women's group focused on machine learning and data science, and about two and a half years ago, I founded DataMela. I'm also on the Scikit-learn contributor experience team. A little bit about DataMela, it's a global community for underrepresented persons in data science. What we do is we organize events on data science and open source. We organize open source spins. some people know them as hackathons and we create pure rate resources for the community on data science and inclusive practices. All levels are welcome beginners and experts and Data Umbrella is a fiscally sponsored project of Open Collective which is a registered 501c3 non-profit based in California. There are various ways to connect with Data Umbrella our meetup group has upcoming events listed we have a webinar library on YouTube and you can follow us on Twitter at and dollar one word before I dive in I want to acknowledge all the folks over the years who have been involved in the sprint there have been many many people who have contributed to the sprint I'd like to thank all the organizers and volunteers a special thanks to the scikit-learn team for patiently reviewing pull requests by so many beginner contributors and I'd like to thank the community for joining and contributing at the sprint with so much enthusiasm them. This keynote is one person's perspective on the sprints that is mine. How did the sprints get started? So I met Andreas who is a core developer on the scikit-learn team at a PI data NYC back in 2015 and I had invited him to do a workshop on scikit-learn to my group which he so kindly did. And so several months later, I received an email from Andreas. Hi, Reshma. I'm applying for an NSF grant. As part of that, I'd like to do an annual second learning beginner's print at NYC, particularly targeted at women. We did a sprint in Boston that was very successful in showing people how to contribute to open source software. I'd like to do this again and on a regular basis, probably starting with second learn, but also trying to involve other Python projects. I'm writing to you to ask if you and your group would be interested in getting involved to get the funding it would be great if i could get a statement of collaboration from you saying that if i get the funding your group would participate in a sprint and outreach thanks andy i thought to myself what is the sprint exactly um and what i did was i wrote back and said hey andy that sounds amazing and something that would be great for us to be involved in we would love to learn more details about it so what i learned is that a scikit-learn sprint is a hands-on hackathon or workshop where we work on issues in the scikit-learn github repository we work with a pair programming partner and we learn to contribute to open source and some of the prerequisites are having a github account having some knowledge of python and scikit-learn and it's helpful to have some experience with git as well the objectives of this sprint were to increase diversity in open source and also one of the goals was to increase diversity with long-term contributors and the types of contributions that participants would make in the sprint would be easy issues documentation reporting bugs or opening up issues i also did an interview with andreas right before the first sprint and uh he had also indicated that that there's one woman in the top 100 contributors to the Scikit-learn network. So the first sprint happened, it was in March of 2017 in New York City. And then the second sprint we organized, it was in September of 2018 in New York City as well. This is my very first pull request to Scikit-learn, it was a code contribution and even though was labeled, quote, easy, it was pretty hard for me. You can see at the conversation left that there are about 35 items in the thread, and it took me two months to complete that. I should have started with a documentation contribution, but I spent the first couple of hours of Sprint running around, and I missed Andreas's Sprint intro presentation. So, we observed some challenges with the sprint. The first was that it was listed as a general meetup event, and there were a couple of people who attended the sprint without a laptop, so they weren't able to contribute. Some of the attendees had little or no knowledge of Python or Scikit-learn, and it wasn't the best use of our time. Some attendees don't read the meetup event description and the prerequisites, and so they left early. and we had a long wait list and so it was a shame that some other people couldn't have joined. And about a month after the Sprint, I received another email from Andreas Mueller. Hey Reshma, is there a way we can follow up on the PRs from the Sprint? See what has happened and what is stalled? Do you have a list or should we just go by date? Cheers, Andy! So I followed up with the Sprint folks. I sent numerous emails via Meetup, LinkedIn, and not a single response. So what happens when contributors are at a Sprint and they don't follow up on their work? It's not great for the maintainers because people started work that is unfinished. And to be fair, we hadn't communicated that at the Sprint, so we were learning along the way too. I reached out to folks in the community that I knew that had participated in either the 2017 or 2018 Sprint and asked if they would assist in finishing up some of the PRs that were open and these three lovely women helped with the PRs, Melissa, Tashae, and Kat. As well, I also worked on a few of the open PRs from the Sprint which was great practice for me and the majority of those were documentation. So some observations from the sprints is that interest level is high in the community. There are a lot of people who want to contribute. And also attrition is high. And so it's sort of challenging to sort of estimate how many people are going to attend. And also, you know, with people not following up on their work, and they also sort of missed out on the experience of finishing up the pull request and the satisfaction of having their pull request, which is amazing. So after those two sprints, I wrote up sprint impact reports, and it's about how many people attended, how many peers they worked on, and some lessons learned. And it was good to be able to share that with the community, and I published that in January of 2019. So results from these sprints are not easily measurable, and occasionally some become apparent. Somebody named Emily Thompson read my sprint report and organized a sprint with her friends to contribute to Scikit-learn. That's really great. And then in February 2019, Tanya Allard, I had shared my report with the NUMFOCUS DISC committee, which is the Diversity in Scientific Computing, and so she sent me an email and said, thanks for sharing this with us. I'm organizing a mentored sprint for gender minorities in the Kaifeng community as part of Kaifeng 2019. If you have some time, I would love to to pick your brains around the lessons learned and general suggestions you might have. And so it's been good to be able to share what I've learned with the community thus far. And I learned that documentation is important and impactful and it's good to see that people are finding their way to it. Now let's look at the evolution of the in-person sprints in 2018. We implemented the application form and what that did was it required participants read and agree to do some prep work. We also communicated about following up with work, that it was part of the process of contributing, that it would take a while for the PRs to be reviewed. And so encouraging people to finish up on their full request and letting people know that it does take some time. We encourage documentation for first-time contributors. And so with this adjustment I also increase communication a lot more emails a lot of some announcements on Twitter encouraging New Yorkers to get enough sleep and save their partying for Saturday night after the sprint and to arrive on time so they don't miss the introduction and watch their emails for updates there was a third sprint that was organized and which was the first one for 2019 which was in Nairobi, Kenya after discovering there's a very active tech and data science community there. So how do you find a Scikit-Learn maintainer to travel to run the sprint in Nairobi? Well, I discovered that there is a Scikit-Learn mailing list and I asked on there and I connected with Adrian Jalali who is also somebody who lives in Berlin and you may know him because he is one of the organizers of this conference. So the sprint happened. There is Miriam there on the right who was the on-ground organizer there and they all had a great time and a dream traveled all the way from berlin to nairobi so the next sprint which is number four was in new york so that was the third sprint in person in new york and um you know we had 45 sprint attendants which was a really large crowd even with the luxury of having three core developers in the city of new York. It was a pretty busy and intense day, and we had very, very serious contributors. In fact, I may have overbooked and had too many contributors. We were pretty exhausted. Well, the maintainers were pretty exhausted. And so we celebrated the 10 Days to Finish the Sprint and the organizers. And so the sprint number five, which was the third sprint in San Francisco, happened in November. It was loads of fun. San Francisco people are a bit more relaxed than the New Yorkers. They take long breaks and enjoy chatting with each other during lunch. So the results of these adjustments for the in-person sprints in 2019 are as follows. The majority of sprint pull requests and work has been completed, which is remarkable and wonderful. New contributors see satisfaction and experience of having their pull requests merged in. The maintainers are a lot happier to see their contributions come through in lieu of increasing the number of open PRs in the repo. There's one challenge with these adjustments, which is that it puts a greater burden on the organizer to do more screening and communicating for the sprints. Meanwhile, in Berlin, Berlin is a perfect place for an in-person sprint because one of the core developers lives there, again, and Noah Tamir and I connected and I shared my sprint experiences and resources with her such as the application and such and you know i want to congratulate the berlin team for organizing the sprint now we have the pivot to the online sprints so we had one two three four five that i organized there was the berlin sprint and for each of these sprints there's about 30 to 40 participants So in 2020 there was an in-person sprint planned for June in New York City and then the pandemic hit. And upon realizing that the pandemic would last throughout the year, you know, my goal was to have at least one in-person sprint a year in New York and with that I explored the possibility having an online sprint. What prompted the move to the online sprints? Well, there was the pandemic. There had been a lot of inquiries from people who found out about the sprints from areas outside of New York and outside of the United States that said, you know, how could they participate in the sprint? I had given, you know, I was really considering providing opportunities to folks outside of metropolitan regions, particularly the Global South, and also with my move to Data umbrella the mission it lined up with the mission of data and the mission of data umbrella is that thinking globally is important and imperative as I mentioned before I was part of a women's group before and I pivoted over there I think intersectional diversity is important which considers dimensions of diversity in addition to gender and that welcoming including allies is crucial in moving the needle on diversity outcomes, particularly in open source. So in June of 2020 was the first online spin, which we'll refer to as the pilot spin, and then in 2021 we had three online spins, two of them were focused on Africa and the Middle East and one in Latin America. You can see from this map on the left where the core developers of the library are located. the majority are in France, which is where the library was created. There are some in the United States on the east and west coast, one in China, one in Australia. And with these sprints, the online sprints that with a focus on Latin America, Africa, and the global online sprint, you can see that participants joined from a variety of countries. This is just a listing of some of the countries where people joined from. Next, I'm going to discuss some of the lessons learned. So what we learned as maintainers and organizers of the Sprint is that continued documentation, which is the text that's available, can be intimidating or inaccessible. There are other mediums such as videos and written transcripts of those videos, which as supplements can be more accessible and in fact far more accessible. We discovered that we needed translations for the Latin Lyrica Sprint. These sprints provide an opportunity to the core development team to receive feedback on documentation and improve the process, and that regular bi-weekly office hours are helpful after the sprint. We also learned that there are some open source barriers to entry. For instance, Git, contributors may know Python machine learning, they may not know Git so well. Familiarity with RSTD structured text, which is the documentation format that is used for a lot of Python libraries for documentation. There are issues that they could be better labeled or accessible to contributors on where they can begin a work on. So contributing documentation is available and yet a lot of people still don't know exactly where to begin contributing. So Andreas created about a half hour video for Crash Course in Contributing to Open Source Projects. And one of the reasons we did that is because the sprint is typically eight hours. An in-person sprint is eight hours, but it's really tough to have people online for eight hours. So this allows people to review the video prior to the sprint and break up the timing at their convenience. and that video has 3.7 thousand views and you know what this indicates that is that these resources are helpful for people who don't attend sprints because not everybody can. I also created a short video step-by-step instructions on a pull request to scikit-learn and that video has 1.4 thousand views. I reached out to Melissa Weber because I realized with my contributing experience that it was important to understand Sphinx a bit better for documentation contributions and so Melissa did a fabulous webinar for the data umbrella team community and her video right now has 5.6 thousand views it's been immensely popular I also reached out to Thomas then who is a maintainer of scikit-learn and there you know, some questions that have come up throughout the sprints and just general knowledge that would be helpful to the community and so he's done a fabulous presentation on three components of the very people request and that video has about 700 views and it's a really great video and I encourage you to check it out. And here I'm sharing the experience of a couple of our sprint participants, Ana Valiz from Venezuela and Fortune from Nigeria. So in the next section, I'm going to go over behind the scenes what it takes to run the online open source sprints. So pre- and post-sprint office hours are helpful. The sprint is four hours, so a week before we do pre-sprint office hours for one hour, and two weeks after the sprint, we do one-hour office hours to answer any questions on open PRs. There's a sprint prep checklist where people can go through and watch the videos and take notes and look at the transcripts and translations if they need that. We do pair programming and pair programming has been received with a lot of favorable comments and enthusiasm, more so than we ever, we get a lot of positive feedback, more so than we ever did in the in-person sprints. Repeat sprints are helpful so people can continue to return until they get on a cadence of contributing and having curated issues on GitHub is also helpful. There's also the engagement of long-term contributors. When I see people contributing after the sprint, and this is really an imperfect way to fund people, and I invite them to join the next sprint, whether they are located in that region of interest or not. And so we have at least four people, probably more, who have been continuing to contribute from their first sprint. So the objectives of the sprints, did we meet them? Did we increase diversity in scikit-learn? I don't know exactly how to measure that. Did we increase diversity with long-term contributors? We seem to have made a small dent, but wow, it took a while to get there and it was with a tremendous amount of effort. So I've organized almost 10 sprints. I attend a lot of sprints with Pi Data Global, Pi Cascades, JupyterCon. The MBA side of me jokes that it's for competitive intelligence, but really it's to research and learn from other projects and make adjustments. So I want to share some observations from my experience in contributing to open source, using open source projects, and observing the activity in the Scikit-Learn repository, as well as the conversation on social media, particularly Twitter. I think these suggestions will be beneficial to the whole community, whether they are contributors to a library, whether they're consumers of open source, whether they're maintainers. The first thing that I would say is that these practices will help everybody, which is that reading the contributing guide, actually, this is for the contributors, read the contributing guide. There's a lot of good information in there before jumping in. And before jumping into an issue, really read it through thoroughly from the top to the bottom and see what the discussion has been best for. The other thing that I learned along the way is that Nadia Albal has written this amazing, amazing piece of work called Roads and Bridges, which is the unseen labor behind our digital infrastructure. It is 140 pages, but it's been the best investment of my time in being more informed about open source, and I highly recommend it. And I'd like to share Jeff Choaklit's philosophy on contributing and being part of the open source community, which is lead the campsite better than the founder. There are other ways that people can contribute to open source, not just by submitting a pull request, which is donating to the open source project. You can also donate via GitHub sponsors, via Numfocus. You can also donate via Benevity, which is a company sponsored matching program. You can sponsor the project with one of your favorite maintainers. You can ask your employer to donate to projects used by the company. You can also donate to communities such as Data Envelope Highlights. I think it's also important to recognize that contributing to open source is more than code. There's documentation, there's reviewing pull requests and opening up issues, there's helping out with website updates for people who have CSS and HTML skills, there is doing social media for libraries. We see that new teams are emerging in open source projects such as communications, triaging, websites, setting up a blog, organizing events such as scripts. Some of the observations that I've observed and I think that are good to share is a lot of the projects have a mailing list, so do join the mailing list and first assume the observed communication style and then ask instead of jumping directly to a list server and not really knowing how it functions. You know, it's important to use inclusive language. Don't use a, hello, sir, assuming that the person who is on this issue is a man. When using terms like, dear sir slash madam, it excludes non-binary persons. And also, hello, guys, or hey, guys, is not so inclusive. You know, when people, if they run into issues with the library, contribute by opening up issues using constructive language move the issue forward. Complaining on Twitter with language like this library is stupid is not as constructive. And as mentioned before, do read through the GitHub issue first before jumping in. It takes a while to receive a response, so do think politely after a reasonable amount of time for people who have opened up a PR. Don't expect an immediate response. Do be an informed and educated open source consumer and contributor to be a direct contributor. And you know what open source projects are slowly discovering is that they're also recognizing that non-code contributions are essential to moving their projects forward and that providing non-traditional on-ramps to contributing such as videos and transcripts and welcoming other skills such as communications documentation from websites. There are some cascading impacts of the 2020 online students, which is that Juan Martin Loyola of Argentina was invited to the contributor experience team for Scikit-Learn. He was a really avid contributor and very, very helpful to the community for the participants of the students. Mary Westman, who's located in Berlin, has taken what she's learned from her community to the PyLadies Berlin Hack Nights, as well as continue to contribute to Scikit-Learn. And we have a group of returning contributors that come to the office hours and contribute to the library. I'd like to thank all the corporate sponsors who have supported the Sprints over the years by providing venues, space for food or books. I'd like to really thank particularly Code for Science and Society and the Moore Foundation for providing fiscal support of the three online sprints in 2021. And I would like to thank CDI for supporting Data Umbrella with the two-year grants. And I really want to thank the PyCon Germany and PyData Berlin for organizing this conference and the invitation to speak. You can find me at Reshma S on Twitter, GitHub, and LinkedIn. Again, if you want to connect with Dave Impella, we're on Meetup and YouTube.

Speaker 2 [29:15]

Hello. Well, thank you for the talk. We have a few questions already on Slido. First one is, do you have any upcoming online streams and how to participate?

Speaker 1 [29:41]

Yes, there is an upcoming online sprint for the NumPy library in June, and the best way to find out about that sprint is to join our meetup group, Data Umbrella, or follow us on Twitter, or sign up for our newsletter.

Speaker 2 [30:00]

We have another one. How can core developers make it easier for people to contribute?

Speaker 1 [30:08]

So a few things, having a variety of different ways that people can, documentation, for example, videos, some simpler documentation for contributing, having office hours where people could ask questions in real time.

Speaker 2 [30:30]

And another one, do you organize, well I guess you did answer that already, do you organize sprints for other libraries and can they reach out to you to organize sprints?

Speaker 1 [30:42]

So this is something that I have been looking into and considering. It does take a tremendous effort to do a sprint. And so I've been thinking of what are ways to scale it? What are ways for other projects to learn about it? And it's something that I'm working on and I haven't quite found a way that is scalable. And with the resources that I have to be able to provide that for the interest level that is out there yet.

Speaker 2 [31:14]

Do people who sit here, how can they help you if you don't have the bandwidth?

Speaker 1 [31:22]

How can they help? That's a good question. There's a lot of, I would say there's a lot of help needed in terms of organizing the sprints. A lot of administrative work in terms of applications, websites, dealing with spreadsheets, getting materials together, things like that. Pretty much, it's organizing a small mini conference.

Speaker 2 [31:47]

Fair point. So how much has scikit-learn improved through this, friends?

Speaker 1 [31:54]

So, you know, improvement in a way is difficult to measure. I, you know, I asked myself the question, I had a nice long discussion with Andy, Andy Muller, and so, and with other people too, and it's difficult to measure, but I guess one question that I asked is what, what would it look like if in the past five, six years that none of these sprints ever happened, right? I believe scikit-learn has over 2,000 contributors to the library I would say two three four hundred of those came in through the sprints maybe two three hundred and so you know what would it look like without the sprints I believe it's I believe it's made a difference even in terms of the video views that people have been able to access who weren't even attending attendees of the spread

Speaker 2 [32:46]

Yeah, fair. Are sprints suited for less experienced programmers?

Speaker 1 [32:56]

I would say, yes, they can be. I see that Marin Westerman in Berlin is organizing hack nights. I think shorter events, regular events, can probably build up experience that's needed. I know that, for instance, the Pandas Library, Pi Data, and Pi Ladies London have about monthly meetups where they spend two or three hours online as contributing to Pandas. So that's like another variation of how to contribute to a library without doing an official sprint.

Speaker 2 [33:30]

Nice. Can a company organize an internal open source sprint? Would there be support for that from already experienced maintainers?

Speaker 1 [33:42]

So that, you know, that is a really good question. I have seen companies reach out and say that we would like to organize a sprint. And it's really, you know, a commendable initiative. I think that one dimension that is not considered in that is that maintainers need to be involved in the sprint as well. for instance the meetings just to organize it of curating the issues following up and so that you know open source projects are type really like understaffed under-resourced underfunded and so while companies who want to have sprints is you know it's a good initiative i think what companies can consider is to add on a sort of um to consider that effort at the top that goes and finding a way to support that as well.

Speaker 2 [34:36]

So I guess I have a follow-up question to that. Do you think you could maybe put a price on it if the company was willing to pay the maintainers or data umbrella? Do you think that would be possible for you or the maintainers to kind of accept that and then work with the company on that?

Speaker 1 [34:55]

Yes, definitely. And I think that price would, that would be a better discussion to have with the maintainers because they would be giving a significant amount of their time for that. But yes, the price could be put on it.

Speaker 2 [35:13]

Do you think the work around organizing sprints is worthy when compared with the same time dedicated in hard developing it?

Speaker 1 [35:27]

Could you repeat that, please?

Speaker 2 [35:28]

that please yeah I also I'm trying may if I want to interpret the question

Speaker 1 [35:28]

Yeah.

Speaker 2 [35:38]

maybe it is do you think the work that people put around organizing sprints worth it

Speaker 1 [35:46]

You know that is a question that I have asked myself, a number of times, I would say that knowing if only I knew then what I know now, which is for sprints to be bit more impactful to commiserate to commiserate with the effort is if a, let's say if a library If a library has some structures in place, for instance, they have regular community meetings, they have teams outside of contributing codes such as documentation teams, website teams, social media teams. I think that if infrastructure is in place for continued contribution, then I would say that sprints, yes, I would say that they do bring value to it. These are all things that I wish I had known five, six years ago. Thank you.

Speaker 2 [36:36]

I guess a part of that is also like by doing this friends The whole team kind of learns how to do sprints easier and the materials are there I guess the first one you and everybody was a lot more involved in preparing the Documentation and after a while it became much easier because you already had all those documentations. Is that

Speaker 1 [36:58]

Yeah, exactly it. You know, and some of those documentations, there's actually a vert, there's a part two of the video that Andy did as well. And so whenever, whenever we run a sprint, there's, there's a feedback form that we send out. And so people give feedback. And there's also feedback via blogs and Twitter and just very casual feedback that they give us organizers. And I make the list of all of that. And then I incorporate it into the next sprint. And as well, you know, for instance, part two of Andy's video, answers a lot of those questions for instance how do you work on two prs as at the same time um and so um and so yeah we we do learn a lot along the way i mean i would say that scikit-learn has learned a lot over the past five years and so i can learn is not an easy library to contribute to and the fact that so many people have been able to contribute with you know certain issues that the maintainers have created particularly thomas fan in terms of you know validating numpy documentation so people can get started on an easy issue and then moving on to more intermediate and and hard issues.

Speaker 2 [38:07]

nice any more questions from the ground if not we thank you very much for the amazing talk and thank you