Binder - lowering the bar to sharing interactive software

Speaker 1 [00:03]

Thanks a lot. And I've not done anything yet, so maybe you should save your applause for later. And also a round of applause to you guys for making it to the end of the day and still being able to attend talks and still taking in new information. I will talk about a project called Binder that is a huge amount of fun to work on and a huge amount of fun to talk about because the people I work with on it are excellent. the people who use it are super nice and they love using it so in principle this is a super nice talk to give because everybody loves the project and the only thing that can happen is that I screw it up in my talk so what does Binder do? it makes it very easy for you to run other people's code without having to install anything on your laptop you just have to have a browser and that's very useful if you're in the business of getting your ideas into other people's head because they'll have a much easier time understanding these ideas if they can run the code that goes with these ideas. And I will show you a demo of this and tell you a little bit about how it works. But to start, we have to go back about 5,000 years. Because 5,000 years ago is roughly when we think the first libraries started appearing. and that tells you that already 5,000 years ago people realized that writing down information on clay tablets in this case was a good idea it was a more efficient way of getting information from one person to another person than having to tell stories mouth to mouth and the cool thing about these tablets is they were a huge amount of work to create because you had to wait for the clay to be just soft enough and then you had to write on the front page of it. But once it was dried, then you would put it in a shelf. And because it's so thick, then you couldn't read anymore what's written on the front. So people would write on the back of it what was on the front of it. And really, that sounds like a book to me. It's as thick as a book and it has writing on the front and the name on the back. So since 5,000 years we've been doing this. Then, a while ago, we invented paper. Now books had lots of pages in them and you can explain more ideas per book and libraries looked something like this. And the interesting thing about it, the earliest big libraries like this, actually you had to prove that you were a gentleman and scholar to get access to it because I think people realized that having access to information in written down form was actually quite a powerful thing and you might not want everybody to be able to do this. So today, libraries still exist. You might not have been to one for a very long time, but they still exist. And for some reason, architects love designing libraries. So they are extremely modern buildings usually these days. And they still contain a lot of books. However, the last time I've been to a library was when I was a PhD student and then I would not go to the library as a building but I would go virtually to the library because most of the papers and documents I wanted to read were in PDF form now so you didn't have to go anymore but in principle the idea is still the same not only a clay tablet, a book made of pages of paper, a PDF, it's all the same idea it's got static content in it it's easier to transport around now and duplicate but the basic idea is the same so these days there's a lot of people who will tell you that paper is obsolete and they don't mean the paperless office which has been a great success as we all know what they mean is the research journal article paper and this animation is taken from a great article in The Atlantic which I think you should read is interesting in all sorts of dimensions commercial versus open source software the future of journal articles etc so what I would like to tell you is that if you make a claim like this you should bring some good examples of how you propose to replace paper because in some sense paper has been the market leader in information transmission for 5,000 years now, including PDFs as paper. And hopefully, at the end of the talk, you'll be convinced that Binder is one way that maybe we can make progress on static content. So if you're a modern citizen in a modern society, you might be interested in the latest going-ons. For example, in machine learning or in AI, maybe you're a data scientist and that's why you're interested in it. or you're a researcher into machine learning. And all of these are reasons that you would like to read a research article on this topic. So this particular article is published by some guys at MIT, and they investigate or propose a model that can take a picture and a question written in human-readable text and figure out what the answer to the question is given that image. So that's an interesting thing to be doing. and if you want to understand how they really do this you would start with reading the paper and if you're a normal person like me you probably read the paper and you don't understand anything about how this actually works if you're a researcher who's deep into this field you probably read this paper and you have more questions than you had at the beginning about details of how exactly they do what they claim to do and potentially you're thinking yeah you cherry-picked all your examples etc etc so the next thing you want to do to try and get a better understanding of the idea behind the paper is to try and run the software that they use to produce the results in the paper. And for the researcher, that will give them a chance to answer all their detailed questions. As a novice, maybe you'll get a better idea of how it actually works if you can run the code and play with it. So luckily, these guys put their software on GitHub. and now the question is how long do you think it will take you to get this to run on your laptop and I offer people a range of options between never and weeks, days minutes, seconds and nobody ever wants to propose that it will only take minutes, everybody says well this is going to take at least a few days if you're brave enough to propose that it's only going to take seconds for a not cherry-picked example, I would like to buy you a drink afterwards. So, the question is, can we do better than taking days to get this working? Because if it takes days, you're putting this on your to-do list of things to do when I have time, which we all know is code for never. So, of course we can do better than this, because this is what Project Binder is about. For repositories that work with Binder, you will see a badge that looks something like this usually slightly less big that you can click and if you do this you get a new tab with a spinner and this is going to run any amount of a few seconds to a few minutes depending on what exactly is in the repository in this case it took just a few seconds because I had clicked the link before so all the building and cloning of the repository had happened already. And what you get afterwards is a notebook which is what the authors linked you to in the badge. And this notebook tells you stuff about how to reproduce figures in the paper. And without reading any of it, just to get started, we can execute this by just pressing shift-return here. and eventually we get to a bit which says this will reproduce figure 1 and you remember this was some picture of some objects and then two blobs in the output image so if I run this hopefully we will produce the same figure as we just saw in the PDF and indeed it does so this is nice because now, after just a few seconds or maybe a few minutes, we're already convinced that at least their code can reproduce the figures that they have in their paper. Now, this is not the most exciting result. That's not why we're here, but it's the minimum hurdle we need to clear. So the next thing that we could do is spend a little bit more time reading the code, which I've done for you, and figure out that we can actually edit this So the question that they were asking about the image is how many large metal cylinders are there in the image and the answer is two and I just edited a little bit of it that turns the question into large rubber cylinders So now we can rerun this and this is the first next step you would take to actually trying to do something useful with this project Does it still work when I edit it a little bit or did they pick an example and that's the only example it works for and in this case the answer is 1 and if you go and read the paper and understand the color coding of the objects in the picture etc, you can check that the answer is indeed 1 so this is what MyBinder or Binder does for you it gives you a way to run other people's code and get going with trying to do interesting things with other people's work and building on top of it. So that was Binder. And in principle, we could finish now. But let me explain a little bit more about how it works. So an alternative title for today's talk could have been Running Other People's Code. And depending on who you are, you either complete that sentence with is fun or is terrible or whatever, this XKCD cartoon shows you a flow diagram of trying to figure out what version of Python you've got installed on your laptop, on your MacBook. And you're trying to figure out one little detail about somebody else's install. And I think it's not completely unrealistically caricatured how many different things could happen. So you can imagine if you're trying to install something or deal with something which has lots of different libraries, this picture would be terrible. So there's different approaches to dealing with this problem that different people have different software installed. One of them I call the IT department approach. This works very well if we all work for the same company and we have a competent IT department. They will give us computers that have stuff installed and they will lock it down so that we cannot change it. And that has the advantage that now sharing between us is very easy because we all have the same tools. The disadvantage is that every time you want to install a modern version of something, you have to go and negotiate with the IT department and they have to install it on everybody's computer. So there's a disadvantage to that. The opposite approach, which you might know from R&D or if you come from a research background, is the wild west approach. Everybody just installs whatever they want on their computer and probably in a few months time they could not even reproduce what they had installed in the particular configuration that they have. The advantage is that you can use whatever tools you want to be using. As a result, very often people now do what I call the kitchen sink approach so they send you everything and the kitchen sink. So they will send you a 20, 30, 40 gigabyte Docker container or virtual machine or something like this, which contains all the stuff you need to run their little piece of software. But that's a little bit like trying to understand, you know, how exactly the helipad is attached to this oil platform. And when you ask, you know, the manufacturer that question, what they do is they send you a boat, which is big enough to put a complete drilling rig on top of it and say, here you go. You go like, well, I'm just interested in how the helipad attaches. Do I really need to be able to receive a whole oil platform? And I think the answer is no. So what we should do is something like the IKEA manual approach. Now, if you go to IKEA and you buy a new living room, they don't go like, oh, yeah, we're going to put a complete living room ready assembled on the back of a truck. and, by the way, please remove one of your walls so we can slide it in. No. They send you small pieces of paper and flat-packed furniture and you are able to follow the instructions to construct the furniture inside your living room. And while there's lots of jokes about being able to follow IKEA manuals, it's quite amazing what kind of sophisticated furniture you can build following them. following them. So the idea then in technical terms is Docker files. So instead of shipping you a Docker image, I just send you a Docker file, because that's a text file that's very lightweight and nice. So the problem with that is constructing a good Docker file is an art in itself. Now if you want to install a tool called Less on a Ubuntu image, I think you should put at least these six lines in your Docker file. And I cannot remember what they are. I have to look them up every time. And really, this just means that constructing Docker files is like writing on clay tablets. In principle, all of us could do it. In practice, probably we would never write anything down because it's just so much work. So what can we do? Well, let's look at how that paper did it that we just run a minute ago. And this is the complete screenshot of the GitHub repository and if you're sitting close enough that you can read what's in the repository you can check for yourself there's nothing suspicious there if you're sitting further towards the back you can just believe me and there's nothing out of the ordinary about this repository but also like in a magic show where they ask can we levitate this elephant and you know that they're going to do it. But you also know no way that you can levitate an elephant. You know there is something special about this repository. And I will tell you what it is. So BinderHub is a piece of software that ProjectBinder builds. Repo2Docker is another piece of software we build. And BinderHub uses Repo2Docker. And Repo2Docker does exactly what the name suggests. It turns a repository into a Docker image. And it does that by doing what a human would do. So it git clones the repository and this is the magic trick part. It analyzes it and it looks for certain files that it knows about and recognizes, in this case an environment.yaml. And then it goes, well, if there's an environment.yaml probably what the human wanted me to do is to install it. So that's what it goes and does. and then it runs a Jupyter Notebook server. And that's how you get what we just saw earlier. Now that's essentially what happened in the background while the spinner was running. The cool thing about Repo2Docker is that it understands you. So we made a lot of effort to recognize files of specifying dependencies that are already in widespread use in the community. So requirements.txt, environment.yaml, REQUIRE in all caps. I don't know why Julia people like to shout, but you know, that's what they use. There are a few things where we had to invent our own. So for R, for a while we couldn't find something that our community likes to use, so we have install.r. In the meantime we also support an all caps description, which is like the setup.py of RLand. And in the end if you really love writing docker files you can also do that so there's lots of other formats that we support that are not listed here and they keep getting bigger so there's currently a pull request to add support for the nix package manager which is some tool for making really reproducible builds so soon we will support that as well and the idea is that if you look at a repository you should not be able to tell necessarily that there's anything special about it that makes it work with repo to docker. Ideally, everything we need is already there because you already use it. So back to sharing. So now we could share this with somebody else and we'd have to tell them, oh, please install docker on your laptop and then pip install repo to docker and then you can run repo to docker and the URL of the repo and they will reply back and saying, docker, what is that? I use Windows, by the way. And you're like, oh my God. so can we make it even easier? maybe they can just click a link and of course so this is what is behind the badge if you click it you end up on a website served by BinderHub and you see a little spinner and all that BinderHub is doing is telling Repo2Docker to please build an image on the fly and what BinderHub combines is 3 bits of open source software Repo2Docker which takes repositories, turns them into Docker images, JupyterHub which is a multi-user Jupyter Notebook server and Kubernetes, because we have lots and lots of Docker images and Docker containers that we want to run and orchestrate auto-scale the cluster, etc so we give that job to Kubernetes and if you want to talk about in detail any of these topics you can find me afterwards and we can talk all evening, all of tomorrow about any of these things if you want to. Another cool thing is, you might say, well, Jupyter notebooks are nice, but I really like RStudio. And the answer is, you can use RStudio as well if you want to. So you don't have to use a Jupyter notebook as the front end. We can launch RStudio for you. You can even go one step more exotic. so Mathix if you happen to know what it is is an open source implementation of Mathematica notebooks and Wolf Volprecht who is also here but I think not in the audience spent a few days one weekend making a binderized version of this so you can now run the open source version of Mathematica on my binder which I find very meta given the arguments between Wolfram and Jupiter about who invented notebooks, etc. You can go even one step further. So this is the documentation for spaCy, a natural language processing library. And in their documentation, they have these little black boxes that contain code that show you how to do what the documentation just talked about. And the cool thing about it is you can run them. You can edit them and run them, and they are executed on Binder. So you can deploy your own Binder hub. If you went to the DAS tutorial earlier, you were using one deployed by the Pangeo people. There's also one at a Leibniz Institute here in Germany. They were actually the first people we knew deployed a public Binder hub. And so ProjectBinder runs mybinder.org, which is our public demo or infrastructure of Binder Hub. And we're very thankful to the Gordon and Betty Moore Foundation who bankrolled this exercise because we rent a lot of servers to do this and they pay the bill for that. So to wrap up, I'll talk about the people who make and use Binder. I would recommend to all you guys to come and join our community and work on all these tools that we're building because there's room for lots and lots of different skills from designing logos to writing software to DevOps. Whatever you're interested in, we probably need help with it. And now the question is, 868, what is that? And it is the number of unique repositories launched last week on mybinder.org. And I find that amazing because it's not just five popular repos. It's lots of different repos which are launched by lots of different people. another way of looking at this is this is not a map of the world this is a map of the world where every country that had at least one user in the last 80 days is coloured blue and the more blue it is the more users there are from this country there's a few countries missing like North Korea, Cuba, the Vatican and a few countries in Africa but I think we're working on turning this map blue and I find this fantastic because it means it's not just people in Germany or in the US or in Switzerland who are using this but people all around the world who are giving free compute at scale to is actually a very big deal and then the final bit of showing off which all this telling you how amazing Binder is as a project is somewhat weird for a quiet geek who builds stuff is in August I gave a talk at JupyterCon and I somewhat confidently predicted we will cross 2 million binders launched by the end of the year so it turns out probably we will cross that line much before the end of the year so we're already at 1.7 million binders launched this year alone and around 80,000 binders are launched every week and this is just amazing that this many people are using this Some of these users have their logo shown here So I looked a little bit in Germany University of Heidelberg uses it, LMU uses it There's a few companies, as far as I can tell, who use it for stuff It's a very broad audience Our biggest user is jupyter.org slash try So if you want to try out anything related to Jupyter you can visit that page and click. And so going beyond paper used to be a question mark, now an exclamation mark, because I think this is actually a way to go beyond the static piece of paper. One very cool example is this textbook, which is from UC Berkeley, and it's teaching data science to everybody who enters UC Berkeley. So not just computer scientists, mathematicians, statisticians, programmers, whatever, but everybody. And it has a normal textbook explanation of the Monty Hall problem, or the Ziegenproblem. But you can click the interact button, and you will briefly see a spinner, and then you get a notebook that basically, actually the website was generated from the notebook. So all the examples that are in the textbook, you can actually run for yourself. and hopefully by being able to do that you actually understand why if offered to swap you should always swap which door you're selecting so this is the final slide by now it's an open project, you should come and join we are very nice people we're from all around the world, we build open source software we provide public infrastructure and the really final slide something to try while I'm answering your questions is visit this short link because that will let you launch your first binder right now thank you very much

Speaker 2 [25:36]

We have five minutes for questions.

Speaker 1 [25:37]

for questions.

Speaker 3 [25:39]

If I want to run Binder on-premise, Kubernetes, and GitLab, how much work would that be? So do I have to write my own adapters somehow, or is there something built in? Because the docu mentions explicitly GKE and GitHub.

Speaker 1 [25:59]

yeah so we support um a lot basically we're trying to work towards being able to support anything which git clone understands so right now you can put a github uh gitlab.com or a git repository url in here what i would do if i was you and you had an on-prem git lab i would write a small adapter so that it would be convenient to talk to your git lab host and then the question is does your Kubernetes cluster support all the things that we need because saying you need a Kubernetes cluster is actually not sufficient to and off the top of my head I think the answer is yes because with JupyterHub on Kubernetes the most tricky thing is a dynamic provisioning of PVs and we don't do that for BinderHub so I think you should be able to manage The Gitsys cluster, for example, is deployed on bare metal. So they managed to do that. But you can come and ask us questions in the Gitter channel if you get stuck.

Speaker 4 [27:14]

Well, thanks very much for your talk. Very interesting and an amazing project. I would like to ask, what are the resource limitations for a notebook? Because if you run data science applications, they will usually have to use anything, large GPUs, anything. So what is your typical resource?

Speaker 1 [27:32]

So the answer comes in two parts. If you deploy your own binder hub, then the limit is whatever your credit card can stomach. On mybinder.org, we give you two gigabytes of RAM and one CPU, just because that already costs enough money to do that at scale. If you use the Pangeo deployment, I don't know what the limit to the number of cores is that they will let you use. I think in the tutorial today they used 20 cores per person and they had 20 people in the audience. So, yeah, the short answer is one core, two gigabytes on mybinder.org if you deploy it yourself depends on who's paying.

Speaker 2 [28:24]

Yeah, thanks, Tim, for the great talk. It was very interesting. For those of you who don't know, a little trivia, I met Tim at lunch. He's a very nice guy. But now to my question. I think the project, if I understood it correct, is great, but I really, I as a sort of, when I want to reproduce analysis, for me it's perfectly sufficient if people pre-specify their hypothesis and basically share their code on GitHub as soon as possible and do formal version control on it. How would you say does your approach then with Binder even add benefit in this situation? Or is it just me now being a little old school and wanting to basically do it from GitHub?

Speaker 1 [29:22]

if you're happy that people just provide you the source code maybe some manual instructions on how to set it up and get it running then there's not much we can do for you. In my experience though you can spend essentially an infinite amount of time trying to get other people's code to work and usually there's a mixture of you not being able to follow the instructions the instructions not being precise enough hidden dependencies etc and that then leads to an infinite email chain of trying to figure out why it doesn't run and I find that massively frustrating I find it much better if I can check at least if you can set it up somewhere that a machine can execute it and reproduce it then I'm happy to now spend some time because I know at least in principle it's possible to reproduce this whereas if it's just your instructions my assumption is going to be it's impossible to reproduce and I'm going to read a different paper instead.

Speaker 2 [30:25]

Thank you, Dean.