A Tour of JupyterLab Extensions

Speaker 1 [00:03]

Hello everyone, my name is Jeremy and today we're going to do a little tour of the JupyterLab extensions. So there is a repo on GitHub, so if you want to follow along, there is actually a binder link. You can run the same presentation as I have at the same time, or you can also check it out later if you want. Okay, so before we start, I'm currently working at QuantStack. At QuantStack, we specialize in scientific computing, so we work a lot in the Jupyter ecosystem. that means building tools for Jupyter and building on top of Jupyter. We work for example on the Jupyter app extensions, iPad widgets, and also on C++ tools such as Extensor and also the C++ kernels. So if you have more questions or if you want to know more about what we do, feel free to talk to me at the end and check the website here. If you want to find me or talk later, you can also do that on GitHub and Twitter. All right, so when we talk about extending Jupyter, there is actually a lot of different ways to do so. Maybe not 99, but there is this blog post from Yuvi, who is a co-contributor of Jupyter, working on JupyterHub. And he took the time to go through all the different kind of extension points we have at the moment for extending Jupyter. It's quite funny because the first one is the JupyterHub extension. This is exactly what we're going to cover today. But it's not only about this, it's also about notebook extensions, server extensions, Jupyter kernels, IPathon magic, and so on. So, as I said, we are not going to cover all of this today, only the JupyterLab extensions, which is already quite a lot. So let's get to it. So about JupyterLab, quickly, it's the next-gen UI for Project Jupyter. So you can think of it as the kind of the succession of the classic notebook. It turned the version 1.0 early this summer. So that is quite important because that means it's both ready for users, which was already the case like one and a half years ago. But more importantly, it's also ready for developers. So it means that if you want to make an extension for GPU lab, the API is now stable until, of course, we reach 2.0, where we're going to break everything. But yeah, that's the main point. And that's why I wanted to show you now like a tour of these extensions that can be built as of today. One of the main selling points of GPLab is that it's extensible by design. That means that basically everything in the GPLab is an extension. And if you make an extension for GPLab, it has the same rights. It kind of lives on the same level as all of the others. And when you install GPLab from PIP or Conda, you are going actually to get 60 of them, like 60 extensions. but you don't necessarily see all of them when you look at the UI because I'm going to show you an example of core extensions afterwards. So all of these extensions are built in JavaScript, TypeScript. They're using NPM to publish them. So this is really built, kind of the modern way of making JavaScript web apps as of 2019, which is also a huge improvement compared to the classic notebook. All right, so a quick word on core extensions. So we are actually currently looking at JupyterLab. And, for example, if you look at this file browser, this is an extension in terms of JupyterLab. It just ships automatically when you install it. The same goes for the console. So here I open the console. Can type code, execute code. This is also an extension that is built in JupyterLab. And of course there is also the notebook extension, that's the one that it is with the notebooks, and that shows me the notebook in the UI right here. But today I want to focus more on the third-party extensions, that means all of the extensions that we as developers can build on top of GPLab. To give you a little idea of what we can do, I put a little video here. So I'm going to play it and show what it means at the same time. So this is running a simulation and on the left side you have a 3D scene of a robot and on the right side you have a notebook that actually controls the simulation. So this is pretty nice because here the only thing we did was to extend the GPULAB interface and use these extension points to basically be able to do this. So here we have a 3D model on the left side, but technically it could also be like a real physical robot. And the good thing about this is that we can actually reuse the whole UI that already exists in Jupyter, such as the notebooks and consoles and so on. All right. So that was just a little intro to give you an idea of what we're going to talk about. So we're going to start with 10 of them. That's going to be the first batch of extensions we're going to look at. Then we're going to look at 10 more, but a little bit faster because we don't have that much time. And at the end, I will show you a little bit what is coming up. So extensions that have been released, that's the ones that we're going to look at now. And at the end, we're going to look at the ones that are going to be released in the future. okay so one important thing when we work with notebooks and especially in teams is being able to version code the notebook is not different than code it's just code in styles and and then we get a json file that we need to put somewhere and share with all people so there is an extension in jpeg lab called the jpeg lab git extension and what it does it's a little bit cropped but yeah, so there is an icon on the left side, you can click on and you get this Git UI, so this presentation is actually a Git repo, so you see all of my commits here, so the history can go here and I see that my notebook has changed, I can stage it, write a message and press commit can also pull latest changes using this button as well as pushing new changes using this one. So basically a simple Git workflow, but that is usually more than enough for basic work with notebooks. But the good thing is that you can actually use Git also in a terminal. And this extension is just hooking itself on the existing Git command that you have installed on your machine. So you are free to basically choose what you want. If it's just You know, quick and dirty commits in a UI, that's perfectly fine. And also a bit more complex rebase and so on using the terminal. This is definitely possible. All right, so that was about Git. But before we actually commit notebooks, we often want to know what changed in the notebook, and we want to be able to diff them. And fortunately, there is an extension for that called NBDyme. So NBDyme is actually a tool that you can run via the command line. integrates really well with git so you can configure git so it actually shows nbdime diffs and not give diffs and it's very useful because if you take a classic notebook in the json format you're gonna see a diff of two json files which is not very handy but nbdime is able to recognize the cells and do a cell by cell diff which is very very useful and here this is even better because we can actually use it in jupyter lab so for example i have this cell i'm going to change this value execute it i'm going to save the notebook and i can click on this and we see that basically a diff of the notebook we even see the video i executed before because now it's added to the output so it's also shown here so it could be used to really make sure that we are not committing outputs for example if the the policy we have in our git repository is to never commit outputs then we can really quickly see that and also yeah have a better diff than just JSON files okay so still about git diffing and now let's say we want to also ensure that everybody is using the same code format that's great because there is an extension for this so it's called the code formatter this extension is meant to be used as a kind of agnostic code formatter extensions, so basically supporting multiple languages, not only Python, but also multiple formatters. Here I'm going to show you how it works with Black, but basically you could do the same with AutoPep8 and other formatters. So this is a very badly formatted cell, and I would like to format it using Black. So what I can do is I can open the command palette and search for format. Here, since I have installed black, it shows up in the list. I will have installed another formatter. There will be another entry. So I can just hit enter, and now the cell has been formatted. So this is really useful if you want to limit the number of diffs you have with other colleagues or something like this. For example, cells that have not been modified, they will stay with the same code style in the repo. Okay. next one is the table of content extension so it's going to be pretty quick because it does one thing but it does it well it has this little icon here on the left side and you can navigate through the notebook so it's basically the kind of passing all of the titles of the markdown cells and creates kind of entry points for you and kind of anchors so you can navigate through the notebook very easy so very useful if you have a very large notebook I want to basically jump around in a matter of seconds okay quick open is meant to bring you the functionalities of this control shift P shortcut you get in other IDs in VS code sublime PyCharm and so on so you want to fuzzy search a file that's what you can do with quick open so you get this little icon on the left and let's say I want to open the readme file I can search for For Remy, press Enter and I get my file open. Okay, next one is the sidecar. So I think we need to change the ratio because I don't think you were seeing the left side on the screen. Never mind. Yeah, the sidecar. So the idea with this extension is that in JupyterLab, you have on the left side different entry points for extensions here but i don't know if any of you knew that you can actually have things on the on the right side of gplab so you can actually do that here by doing a right click and switch side by side that will bring this uh little icon on the other side and basically move the extension on the other side and the sidecar extension does the same but programmatically so you can use it to display widgets on the right side and here Here I took the example of... I'm going to take IPyCanvas as an example of a widget to put on the right side. So IPyCanvas is being developed by Martha, and you can also check it out, so you will see in a minute what it's doing. So I'm creating a canvas, so we can think of a canvas as an HTML canvas you get if you work with JavaScript, and you can draw things on. And here I'm going to draw a face. It's a poker face, not very happy, but it's there on the canvas. So I'm going to scroll, and it's already kind of already gone, right? I don't even see where it is. So that's the whole point of using the sidecar extension, is that instead I can put it on the right side, and I can start scrolling with the notebook, and it's going to stay there. And even better, now we can actually make the face happy and start modifying the widget. So it's also useful if you work with maps, for example. You want to have a map on the side using Alpine Reflect and make some modifications. So now we're going to keep that face with us the rest of the presentation. No, I'm kidding. It's not true. Let's close this. Okay. All right. So speaking about drawing things, there's an extension called Draw.io. for gplab so maybe you've already used this website called draw.io you can draw diagrams and then export them you can save them to xml also like png and svg and that's the great thing about this gplab ecosystem is that you can actually reuse existing libraries very easily so that's the that's what happened with this extension so that this is basically a wrapper around draw.io and it means that if you go back to the launcher you can create a diagram here And you get the same thing, you can start creating your diagram and then save it to a PNG and potentially embed it in your notebook. So you don't have to leave the GPULAB anymore, you can just do everything in the same interface. Alright, so one thing I've heard quite often is people want to be able to use the space that you have here at the top of GPULAB, which is usually empty. but if you use this top bar extension this won't be the case anymore because now we can start making use of it and this is particularly useful if you want to be able to put some indicators, one of them that is quite used is to show the memory usage or system metrics, so that's what the top bar does, it exposes this space for other extensions to place indicators or other widgets in there and for example this one is like a little switch to toggle the theme very easily instead of going to settings and so on can be useful to some people and here we have the ability to put some text so yeah just put the name of the presentation so we know where what we're doing at the moment and another thing you can do is you can actually rearrange the the different extensions here so you can drag and drop them if you want okay so for those of you working with sql and who want to basically run sql queries very easily in gplab there is a gplab sql extension the same as before we can go back to the launcher and click on sql and here we basically enter a url to the to the database we want to use. So I'm starting to type SQLite that is because in my examples repo here I have an example database so test.db, enter and now I can explore this database very easily in gplab. I can look at what is in the employees table and then I can go back invoices and I can so select for example select all from invoices where total I think is to that greater than five control enter and I'll have all of my invoices that have a total greater and of course if you know SQL you can do a bit more advanced things like order by total or desk for example and basically that's what you get so that's useful when you have SQL queries embedded in your Python code let's say you use SQL alchemy for example you still want to be able to run those SQL queries kind of like this like a one-shot query just to make sure that work okay so the last one for the first batch is called the cell tags maybe it's also like some kind of hidden feature of the notebooks but notebooks can actually have metadata about the cells and some of this metadata is all the tags so if you go here to the this section here the notebook tools you see that you have possibility to add tags so let's go back to this and let's say this is a special special cell and now we have a tag called special I can click on another cell but then I can select this tag here and select all cells with this tag and now I'm back to the one I actually marked before so this is useful if you want to for example navigate quickly in a notebook like this but it's also useful in a for example if you create a pipeline on a ci system and you want to for example format sets automatically based on their tags or do more advanced stuff that you can do programmatically all right so as i said now we looked a bit more in details in for in 10 of the extensions and 10 more are coming but we're not going to spend as much time just to give you an idea of what is available so once again it's like the whole purpose here today is to to just give you an idea of uh the extensions we have and the fact that gplab is not you know lagging behind the classic notebooks that extensions are being forwarded over to gplab and makes it uh as usable as the classic notebook okay so first one is the go to definition extension and as the name suggests So, for example, if you have used other IDEs, you know that you can usually do a right-click, go to Definition, or click on F12 or GD using Vim, and it brings you to where something was defined. So in the case of functions, if you do that, you will be able to jump back to where the function was defined. So again, this is like bringing IDE-like features to Jupyter apps, so you don't have to switch over between two different IDEs. if you are looking for these kind of features. Something very similar is the GPULAB LSP extension, and LSP stands for Language Server Protocol. This is already being used in other editors to give you more introspection utilities for your IDEs. And the good news is that we can actually integrate that with GPULAB. So it's still kind of a work in progress, but the results are pretty good already, and it's already possible to try it on Binder if you want to just go to the repo okay so maybe some of you have heard about Voila which is a tool that turns your notebooks into a web application and dashboards and yeah so there is also a HPA lab extension for Voila. I will show you a little example so let's say we have this notebook here this is the basic notebook you get also in the documentation of Voila that can interact with it but I'm still seeing the code and I want to see what it would look like if I start sharing with Voila. So there is this little icon here render with Voila. I can click on it and on the right side I see the preview of the notebook rendered with Voila. This is also usable here and you can actually start iterate on your notebook on the left side and using the render and save checkbox here it will automatically reload your preview as you type okay so speaking of yeah live previews one is more like for educational purposes is the Python bytecode extension so if you're interested in knowing how your source code is being you know translated into by a bytecode you can use this extension so it's the same kind of thing you type Python code and and then you get the bytecode on the right side that is being automatically shown. Okay, so Matplotlib is probably one of the most used plotting libraries and we've been using this Matplotlib inline for years and it's been working really well but gives you static images. So now there is this API MPL API widget that you can use in JupyterLab and it comes with this JupyterLab Matplotlib extension and what it does, it basically adds some kind of interactivity around the plots you make with MyPlotLib. So this is pretty nice because you don't have to rewrite your plots, you can use the same API as you've been using for a long time in the MyPlotLib API, but still get some basic interactivity. Okay, so often we also want to know what kind of variables we have in our kernel at a given moment, and this is what the VariableExpector extension is doing. So here you can see that as we type, we create some variables, and we have this right panel that shows what variables are being alive in the kernel at the moment. It's a bit similar to what MATLAB is doing with these variables you have on the side. And it also has support for showing numpy arrays as a matrix is pretty useful. so it's still about introspections and like kind of looking under the hood of what is happening there's this kernel spy extension that you can use and it's especially useful if you're making an extension for gplab that is talking to a kernel or making a custom ipad widget so what it does is basically intercepting all of the messages being sent to a kernel and being received from a kernel so you can basically see what is happening with your extension as you build it. Okay, so I think there are three left for this second batch. So it's going to be a bit more random. And this one is about integrating Heroku in JupyterLab. And this is especially useful if you want to, for example, in one click, deploy an application to Heroku using a notebook. So again, this is taking Voila as an example, but it could be also in any other Python application. And it adds a little kind of UI sugar on top of Heroku command line, so you can do everything in JupyterLab. Okay, so let's say you work at a company that kind of enforces a very strict way of how a notebook should be written. And this is where you can use this templates library, the templates extension. So you can create templates, save them, and then you can create new notebooks using templates that you choose here. So let's say you have a logo of a company at the very top, then this is the extension you can use for that. Okay, and just to show you that it's not only about features and extensions, but also about theme. So themes, again, they have the same kind of level as your other extensions in JupyterLab, so you can create new theme using the extensions. And this one is the Material Darker. Once it's installed, it's going to show up here, I can click on it, and then switch to the new theme. So where do we find all of this? Yeah, I agree, it's a very interesting question. There are actually different ways. Compared to other IDs, there is no marketplace yet for GPULAB. We don't know yet if it's going to be the case or not. but at the moment we can go on github and search for topics so the jupyterab extension topic that requires people writing extensions to actually put this topic but this is often the case because it's documented in the cookie cutter for extensions so you can go there it is 140 at the moment there is also the extension manager which is now part of jupyterab it's not enabled by default but you can enable it here and click on enable extension manager it's going to add this little icon here on the left side and then you can look at the ones that you have installed and i could also search in the in the box for new extensions and finally there is also a awesome jupyter github repo so this is not only about jupyter but also other jupyter resources there's a section about jupyter extensions so if you're looking for other Jupyter-related tools, and the difference with the rest is that it's curated, so it kind of needs an approval, so it means that usually we get some high-quality content there. Alright, so what's next? As I said, now we're talking about extensions that are in the making, but not released yet, And one of them is a debugger UI for JupyterLab. And this is what we've been working on at Constac with the folks from 2Sigma as well. And it basically brings you a debugger interface to JupyterLab. So everything is being developed on GitHub. So if you are interested, you can also check it out. There's a rich text mode extension being built by the interns at Cal Poly in the US. So it's like a WYSIWYG editor for Markdown cells. same kind of thing as you get on using Word. A package installer to install pip and corner packages without using this pipmagic or pipcommands in a notebook. Commenting extension, similar to the Google Docs style where you can put comments and then you can share in the notebooks with the comments with your colleagues. And many more, of course. So if you are interested in this, feel free to check out all of these resources. with that I say thank you and if you have any questions just come to me at the end hello so we have a time for one or two questions left does anybody have any questions for Jeremy yeah you were first Thanks for watching.

Speaker 2 [27:17]

Hey, thank you so much for a great talk and great extensions as well. And I was just wanting to ask, like, if you're working with PIP, you've got, you know, a requirements.txt, you can have in a project or an environment.yaml with Conda. Are you thinking about doing something similar for like Jupyter extensions or is there a good reason why you think that wouldn't be a way to go?

Speaker 1 [27:45]

Yeah, so the question is about how do you manage all of these extensions in a similar way as with requirements.txt or environment.yaml. And at the moment, there are two ways of doing that. So some extensions are being kind of bundled in the Python packages. So if they, for example, some extensions here that are demode require a component on the server side as well, which is the case for the Git extensions. and here what you can do is you can actually install automatically the gplab extension by putting it in the right folder and then rebuilding the gplab so that's one way and the other way is to install it by hand using the gplab jupyter lab extension install command like here and and it's going to install the extension in your environment yeah so that's the two different ways at the moment but i agree that it would be great to have some some kind of spec to be able to manage extensions, and also there are versions of it in an easier way. Thank you very much, Jeremy, for the wonderful talk, and I think you can enjoy the coffee break now. See you later.