The future of the Jupyter Notebook interface

Speaker 1 [00:03]

Okay, thanks everyone for coming. So, we'll be talking about the future of the Jupyter Notebook, but before that, a quick word about me. So, I'm a technical director at QuantStack and also a core Jupyter developer, working on JupyterLab, Jupyter Notebook, JupyterLite, Voila, and a bunch of others. At Quantstack, we work on two things mainly, so it's one big part on Jupyter, so we contribute to many Jupyter open source projects, and also working on package management with Mamba and Macromamba, and also CondaForge. So we are pretty active there. So first, before talking about the future, let's talk a bit about the history. So Jupyter Notebooks are a great tool for what we call interactive computing. So in interactive computing, the workflow is pretty much you have a compose step, and then you submit, and then run, and then you get outputs, and you can view your output, and then if you want, you can repeat this whole process. That's pretty much the flow that you have with such tools. It's been possible to do something like this without a notebook for a long time, using IPython that you probably know about. In IPython, the terminal is like the UI, so it's like a supercharged Python console. But there are a few problems, is that you have no narrative, no memory, and there's no way to reproduce what you have in your terminal, or no way to communicate about your results. But in the end, it all boils down to this thing we call the REPL, the Read Eval Print Loop. And this is what made the Jupyter Notebook successful and also useful. It's very easy to do all of this, do all of this interactive computing workflow, but with a narrative, a way to keep the narrative, the memory, reproducibility, and communication. So IPython, back in the days, I don't know who here has used IPython notebooks. OK, it's not that many, maybe just a quarter of the room. Looked like that, so it looked like a classic notebook, but it was still called IPython. And then back in 2015, I think, there was a Jupyter announcement to kind of dissociate Python from the notebook itself. And it's been super successful. Millions of users, millions of notebooks on GitHub already, maybe more now. And it's heavily used in data science, education, finance, and a bunch of other domains. And when you work with a notebook, under the hood, this is what happens. So you still have a server somewhere. And then you have a notebook file. And then you have an execution environment where your kernel runs. And you interact with all of this via a web browser. So it's a web tool. But the notebook file follows a standard file format that all of the Jupyter frontends implement. And the kernels are also following a standard called the Jupyter protocol to be able to talk to the Jupyter server. So there you have a quick overview of what the Jupyter protocol means. So you can have multiple kernels, but you can also have multiple frontends. So if you don't like the notebook, you can use something else. You can use even application like Spider. But there are also other frontends, like Google Colab, for example, or other ones that are proprietary that still use the protocol underneath for the kernel communication. But then they provide a different frontend. But in our case here in this talk, we're going to talk about this thing that you might recognize. So this is called the Classic Notebook, and it was released in 2015 as a successor of the IPathon Notebook. And back in 2015, so here we're just going to have a quick review of what it means to build a web app. So you need to have some JavaScript somewhere. And at the time, it was built with jQuery, RequireJS, and Bootstrap. Maybe some of you know that. But then came JupyterLab. So JupyterLab was a restart of this default notebook UI. So the classic notebook stayed there, so it's still around. But this new project was born. So with JupyterLab, you can do more things. You can work with terminals, code consoles. You can have multiple tabs open at once in the same browser tab, so it's very, very powerful. It's also very extensible, so you can add extensions if you want, that someone else made, or your own. And it's also built with modern web technologies. So it's not leaving jQuery behind, but embracing more modern approach to web development. So yeah. In 2018, there was a blog post on the Jupyter blog to pretty much announce that Jupyter was ready to use. So this is what it looked like. It was not even version 1 at the time. It was 0.32 or something. But it was ready to use, so you could use it instead of a classic notebook if you wanted to for your work. So it was there. And in this blog post, it's still available online if you want to read it, there was a paragraph saying that JupyterLab 1 will eventually replace a classic notebook. Throughout this transition, the same document format would be supported by both the Classic Notebook and JupyterLab, which is still the case. But there was still this kind of direction of dropping support for Classic Notebook at some point in the future. When you use JupyterLab, this is what it looks like. So you can do a lot of things there. And it's not even showing everything you can do there, because you can have things opening on the bottom on the right as well, pop-ups. It's really like an IDE for the web, pretty much. A bit simpler to use than an IDE, but very similar. And some people have been kind of afraid of using this. So we'll see a bit after what it means. So it's kind of a different look compared to this simple notebook that you have on the page. The problem with the classic notebook, so the one that we saw built with old web technologies is that once Jupyter app development started, it was very few maintainers and also very little time invested in the classic notebook maintenance. So that's also why it hasn't really changed that much over the five years or more. But the problem is that some people were still coming to the Jupyter notebook repo. They were asking for features or bug reports. there were reporting bugs pretty much interacting with that and it was just like no one replied because there was really no one really working on that so it was kind of this weird situation that we had a tool heavily used but really no redevelopment happening there or any maintenance happening there so after a while a couple of Jupyter developers they started a weekly limiting to start processing those new issues and try to at least answer the people saying that they should consider something else in the classic notebook at some point because it's not maintained and so on. So you can have more context on this if you check the links there. But it was not enough because it was still taking a lot of time and it was not clear what to do in the end. Should the repo be archived? should the classic notebook just die? At some point, Zach Sater, also a core Jupyter developer, opened this issue on the notebook repo, and it got a lot of activity. See more than 120 comments there. It was about, what should we do with this classic notebook? And the reality of the decision right now is that some people, at least a lot of people, still use the classic notebook because they like the simplicity of it. In that case, the simplicity is a feature, especially in education. If you want to teach something, you want your users to only focus on the notebook and the content. That's it, not the terminals on the right or these little distractions that JupyterLab might add to your presentation. And the idea is to have something that is really document-centric, and remove the noise around. And also, some feedback on JupyterLab was that the tabs in tabs, so you can have tabs in JupyterLab that you can reorder, it was considered a bit confusing for some people, because for them, they already have tabs in the web browser, so they didn't really understand why you need to have tabs again in an app. So it's really a matter of preferences, and also workflows. So now I'm curious. I'm going to do a quick poll of who uses the Jupyter Notebook, the classic, on a day-to-day basis mostly. OK, so maybe half, and the others, JupyterLab, or maybe something else. So let's say JupyterLab. OK, so maybe 50-50. Just kind of expected. I would say it's not even a surprise anymore. So the problems are that the classic notebook is not maintained. It was also very difficult to extend. So power users of the notebook, they were like, oh, I want to have this new thing. I want to have something on the left, a table of contents, for example. But it was not really made for extensions and things like this. that's just transitioning from the iPython notebook originally. And JupyterLab has a very clean way of extending the interface and with very easy plugin system that you can develop for. All web technologies, also jQuery. Now, if you say jQuery to a web developer, they will ask you, why are you using this? But it's still used a lot. So legacy software sticks for a very long time. And yeah, very big user base. So that's a problem, but it's also an advantage. In our case, it's a problem, because how do you move a big user base without losing them? So the initial plan was to progressively sunset the notebook, and that means providing ways to customize your apps so it's a bit easier to use for the people who don't want to use everything, or have a simpler version, or make it easy for people to have their notebook workflows work in JupyterLab. But after taking into consideration all of this, the conclusion was what came out of all of this was maybe you shouldn't sunset it, because that would mean archiving the repo, and then the last version on PyPI would be the last version of notebook, and that's it. So instead, there's been some discussion. All of this happened at the end of 2021, I think. And this led to something we call the Jupyter Enhancement Proposal. It's number 79, if you want to look it up. And the title of this proposal is Build Jupyter Notebook v7 of Jupyter Lab Components. And that means the notebook interface, this document-centric interface, is still important to have as a Jupyter offering people still want that so we're not going to take it to archive it or just sunset it instead we're keeping it but we are making it easier to maintain and also develop against that means building it with JupyterLab components so JupyterLab is an extensible set of extensions and what happened a bit before all of this actually I started this experiment called Retrolab. And the idea was exactly to do something like this, rebuild a classic notebook. So it just looks the same, but with the JupyterLab components. Means you can reuse themes. You can reuse a lot of extensions from JupyterLab. And the main idea was to keep really the classic notebook look and feel. And as part of this CHEP, so the conclusion was, OK, we already have something that works and that was built end of 2020. So we're just going to start from there and then continue notebook development based on Retro Lab. So if you can see it here, it's a bit pixelized. Behind is the classic notebook. In the front is notebook 7, the new version of notebook. And the idea is that it should look almost the same. Of course, it won't be exactly a one-to-one copy because there will be some styling that will be different. and also a few things that are maybe put differently. But the idea is that a user can adapt pretty easily from the old version to the new one. And this has a lot of benefits. So first, keeping this document-centric notebook interface is important. But that also means, since it's based on JupyterLab components, that a lot of things can be reused from JupyterLab directly. There are debugger. So if you want to teach debugging to a classroom, you can open it on the left side in the notebook. You couldn't do it before because there was no debugger in classic notebook. Accessibility is super important, especially also again in education. In TripDeliab, there has been a lot of work in accessibility to make it better and improve it. Theming and dark mode. So there is a built-in dark mode. Now it's there by default, the table of contents. That also means you can also use it in other distributions of Jupyter, like JupyterLite is able to use the new notebook based on JupyterLab. So we can have a static website with our JupyterLite. And maybe one of the bigger features is the real-time collaboration. So I can show you this quickly. So now you can collaborate with Notebook on the left, JupyterLab on the right, with your users. So it has been a lot of work on real-time collaboration in JupyterLab. And this just works automatically with Notebook 7, the new version of Notebook, because it's using the same stack underneath. Users don't really care about the technical details, but this is one of the features that has been very asked for over the years. It's really cool that it can be reused like this. Let me do a quick, OK, I'm going to grab the URL from here. I thought I had it already. You can also try it on Binder, so if you follow the presentations and so on. So FileTree looks like this, not exactly the same a classic notebook but you know you shouldn't be too much lost you can find your running tabs here so i'm going to open this notebook here was most important and you get a classic notebook view right here so let's do what we all do just shift enter until the end and you get the plots the widgets as well so widgets are working jupiter lab and also notebook they also work here so everything you expect should be there but let's say you want to use the dark theme now we can it's by default night it's a third party theme you can also install them you don't have to go through this kind of tricky configuration you had to do in a classic notebook so it's there so you can for example enable the debugger and then open the debugger panel here and then you can start putting bread points here so if i just interact again We see that breaks. So it's kind of nice way of Using a debugger. So you have only the debugger here and then the notebooks In JupyterLab you have all the components here. It's really just focused on that. So yeah, that's That's one thing you can do and we made it easy so that you can jump to JupyterLab easily if you want So there is this drop-down thing. You can just switch between the two versions if you if you want to and also Okay, maybe I will go back to that later about migration plans, and let's continue here. So what about extensions? That's one of the most asked questions, like RISE, for example, for making presentations from a notebook or NPGrader. So there's been work to port them so that they work in JupyterLab, and that means they will work in Jupyter Notebook 7 as well. I think there are still some work to do on the rise, but should be available soon. And like we said before, you can reuse many extensions that already exist. And if you are not sure, you can check this link here. There is a matching table between a classic notebook extension and extensions. There's also that we're after. With this rollout of Notebook 7, so now we are Notebook 6, this thing I showed you is going to be Notebook 7. There has been a lot of work put into making this transition as smooth as possible, because based on all of the discussions, we know there is a huge user base that can be caught off guard by this. So the plan is to also distribute another package called NB Classic, which is serving you the classic notebook UI. So you can access it from here. And if you still want to use this for now, for the time being, because you're not ready, it's fine. You can. You can still access it from here. And you see at the very top, and that's been the case for a couple of weeks already, if you installed a new notebook, a Notebook 6 version, latest, and also NB Classic, you will see a little banner there that warns you that something is happening. And this is the banner. You can also have logs here in the terminal. If you started from the terminal, that would tell you that a change is happening soon. Current state is that notebook 7 is in beta. So you can try it. It's behind the dash dash pre. So it's the same as before pip install notebook, but you need to put the pre-release flag first. Once the final release will be installed, there won't be the pre, and the pip install notebook will install notebook 7. It's also available on Conda. So we are really looking for feedback. And if you have issues, if you are worried, please let us know because everybody is super open for hearing about all of the different use cases and make it as smooth as possible. Development is happening in the notebook repo, the same as before. The classic notebook UI is now in the Admin Classic repo. And the Notebook 6.5 is still supported for maybe one or more years. But it's only really the security patches that goes in. There won't be any new features. And there is a branch in Notebook repo for that. So what's next? There will be a migration guide. There is already a migration guide, actually, on the docs to cover a lot of different aspects, like extensions and also how to do your migration. There will be release candidates after. And since Notebook 7 is based on JupyterLab 4, JupyterLab 4 will have to be released first. It's also in beta, but it's expected soon before JupyterCon this year, which is early May. And the final release of Notebook 7 will be soon after. It's not clear yet when, but it will be after. So yeah, reach out if you have any concerns. So that will be a huge help. please get feedback or try it and see if it works for your workflow. If it works, then it's perfect. A lot of people will see that as actually a super cool improvement because they get a lot of new features by just upgrading to the new version. And go to the forum if you have more general questions or you don't know where to ask, just go to this course, it's fine. I love this, it's like the work of more people than you can imagine. So it's been in the works for many years actually. And also, the GPU community has been very helpful to give feedback and also raise concerns and all of this. So we'd like to thank them for this quick shout out. If you want to know more about Jupyter and everything orbiting around it, you can come to Paris in two, three weeks, early May. And it's jupyter.com. There will be also people working on other Jupyter tools, so you can get answers to all your questions there. Very interesting keynotes. Also, you can check the website. It's going to be great. And yeah, the slides are on GitHub. So I'm going to leave this slide. And that's the last one. Feel free to check it out and ask me a question if you would like to. Thank you for attending. Yeah.

Speaker 2 [22:59]

Two questions one is does collaboration mode also work in jupyter lab

Speaker 1 [23:04]

Yes, because it comes from from JupyterLab actually, so Notebook 7 is just reusing the same extension for enabling real-time collaboration.

Speaker 2 [23:16]

Okay and then another one. What is the difference of using the option to go to the simple interface inside JupyterLab and use RetroLab? Is it the same thing but embedded or not?

Speaker 1 [23:29]

Yeah, so first of all retro lab was the ancestor of the new notebook now retro lab is Not there's no development happening in retro lab anymore It's all happening in a notebook 7 and the simple mode it was I would say it was an attempt of doing This kind of document-centric view but in Jupyter lab But with simple mode you still open your files on the left You know, it's not in a separate tab like it is with the notebook application. So For many users, especially the ones using the classic notebook. It was just not enough or not Not helping that much. So It's still useful for some some use cases the simple mode, but it's it's still different. Yeah

Speaker 2 [24:16]

Okay, then another one. Does Notebook 7 support Copilot?

Speaker 1 [24:22]

So, there's been some work, actually, it's in under the JupyterLab organization on GitHub. There's a project called JupyterAI, and that is going to give you access to large language models or something similar from JupyterLab. But again, Notebook 7, since it's just a kind of remix of JupyterLab, it's just the same almost building blocks. It means that you will be able to use it. So it's not released yet, but the folks working on it, I think they are probably going to put a release soon, and I would expect that to also work in Notebook 7. But there is no co-pilot yet, but hopefully soon.

Speaker 2 [25:07]

Okay, then there's another one. What will be new in JupyterLab 4?

Speaker 1 [25:12]

A lot of things. Accessibility has improved a lot. Maybe it doesn't speak to everybody here, but actually it's a big thing. Real-time collaboration has been improved quite a bit, so we have better support for multiple users. like it's still available it's already available in 3.6 right now but it's been some issues so they have been many have been fixed during jupyter 4. what else do we have there is a new extension manager that you can install extension from pip before it was a bit clunky the one that is built into jupyter because it was downloading things from npm and then rebuilding it now it's a bit streamlined and i think i need to check the docs for for more but yeah it should be easy to find

Speaker 2 [26:04]

Okay, that would be it. So thank you again for your talk. See you around.