Panel: Turn any notebook into a deployable dashboard

Speaker 1 [00:04]

All right. Hi, everyone. My name is Philip. I work for Anaconda as a consultant, but on the side, work on various open source projects, of which panel is one. It's a project I started about a year ago. Learning from some of the experience I've had, we're consulting for various clients to solve issues I've seen over and over again in various data science workflows. So who of you has a Jupyter dashboard with a plot in it? okay that's good or a table an image an equation whatever well most of you i thought so um and maybe you might want to explore these things or share them with people is that a common need i would think so so panel is for you then so panel as i said is a pretty new library it's built on bokeh which is generally thought of as a plotting library but there's actually a lot more behind bokeh, and panel basically lets your notebooks double as apps or dashboards. So you can use just about any plotting library, image type, or other objects and embed it in a panel. So the comparison, I mean, we just had that excellent talk about Dash. The comparison to Dash obviously becomes pretty obvious, so I might as well address it right away. What we try to do is we try to kind of address meet people closer to where they are already. So they already have a notebook, they have some analysis in their notebook, and they want to kind of create a little dashboard or app that they want to share with people, and then revise and kind of adjust things in the notebook and then revise, repeat, and kind of build a dashboard that way. So while we've geared it towards usage in Jupyter Notebooks, it actually doesn't require Jupyter Notebooks. You can just write a Python script and deploy that just like you could with a Bokeh app. Before I go through the typical workflow that you have with panels, I'm going to quickly take you through some demos of the kinds of things you can achieve with panel. On the panel website, we have a nice little gallery of some more sophisticated examples. We're actively building this out. I announced panel about six months ago, and we're actively working on improving the examples. There's examples of simple apps and then the usages of the various APIs that panel provides, to build dynamic UIs and so on. So that's quite nice. And one thing you need to know about panel is basically it consists of three different types of objects. The first are panes. Basically the panes provide a view onto some object. You've got some plots, some bokeh plot, plotly plot, bigger plot, whatever. You put it in a panel, in a pane, and that visualizes it inside panel. Then there's layouts. Currently there's columns, rows, tabs, widget boxes, and so on. And then there's a widgets which give you the interactivity you might want so there's a whole range and we're actively expanding these okay then we have a website of various examples on examples.pybiz.org so these are more involved examples that kind of take some subject matter and go through them and then i'm briefly going to take you through some examples so this is a panel app we're running on Bokeh server where I have a few demos lined up. So the first demo I'm going to show you is this Gatwinder demo. So here we've embedded four different plots from different libraries. Here the first one is generated using HVplot as a Bokeh plot. Interactive, we've embedded an Altair Vega plot, a Matplotlib plot, and a Plotly plot. And we've linked it to this widget. So if I drag this, all the four different plots update. And so whatever block calling library you use, one of the common ones in Python, the panel will just support your current needs, meet you where you are, basically. Another nice demo we've built is this cross-filtering demo. This is basically all the world's glaciers. This plot is generated using Datashader. So it's several hundred thousand data points, but DataShader will scale to millions and basically allows you to cross-filter on these. So the nice thing this demonstrates is you can embed your plots in here. They can be linked, so you can have interactions on the plots themselves, either using Bokeh or Plotly. You can have updating text, and you can have buttons and so on. So the other nice thing is because this is all built on top of Bokeh, which has a really nice layout engine, You can see that when I drag this, these plots are responsive and everything updates as I drag it around. So the next demo I'm going to show you is a portfolio optimizer. Basically, the idea here is that you have stock data sets, you have various stocks, you can upload a little file. But the thing that this is meant to demonstrate is, again, you can have plots, you can have buttons, you can upload things, but also you can have tabs. For example, if you can't fit something on a page, it'll let you lay things out in this way. Okay, and then finally I'm going to show you a more involved demo, which is this dashboard we built, or someone else built, for the LFST. This is the largest telescope in the world. It's currently being built. So they want to analyze quite a lot of data. They have billions of stars, obviously, that they're scanning for. And so this is a bit more involved app. It uses something called templates, which I'll show you later. But basically, this allows you to do more complex things. So here we're cross-filtering on a couple million points. And so when I drag this, you'll see the other plots update in response. you can do quite complex things. So this is not the standard thing, laying things out with rows and columns, but rather we're actually embedding things in a template, and I'll show you how that works in a bit. Okay, so going from there, I'm going to take you through the standard workflow that you have at Panel. So we're going to start with a dataset, as you so often do. So here, this dataset, I don't know how well you can see this. I might increase the size a little bit um so this data set is basically just a common data set of room occupancy it's basically measurements in a room as people go in and out of this room measurements like temperature humidity light co2 it's just an example it's just trying to show you what the workflow might be um so what we've already done is we've already got a notebook we've already done some analysis right so we've got a plotting function we've got some little analysis so what this is doesn't really matter but I'll quickly take you through it. So we've taken the rolling mean of this over time, then computed a residual, and then we're trying to find outliers by basically taking points that are away from this rolling mean by a certain number of standard deviations. And so we've already got this. We've got this function. Oh, well, live demos. I need to actually run this. So we got a manifolded plot. Easy enough. Everyone's familiar. but now you need to share this with your boss or your boss's boss who doesn't know that code he doesn't want to go and manually edit this rolling window or pick a different variable or whatever so you might want to explore this in a nicer way so as a starting point you can simply say p and non-interact give it this function and it creates some widgets for you if you're familiar with ipy widgets this is basically the same thing it kind of inspects the function it's pretty magic and it gives you fine it's a good start it's interact is being as magic as it is kind of doesn't know that a window can't be negative really or that sigma's you shouldn't use a negative sigma on this data set doesn't really make sense so what we're going to do is actually give it some additional information we're going to tell it basically the window size shouldn't be should be between 1 and 60 we're going to tell it the various variables that we have in this data set the range of the sigma values, and then we're going to run it again. Cool. A bit more useful. We can now pick the different variables. These are now in a sensible range, and we've got a GUI, right? We could share this with our boss, which is fine. So we're going to do that. We're going to see what that looks like. So if you run the .show command on this object, what you get is a separate server, a Bokeh server, we start it up, and you can see what this app looks like. So within just two lines of code, we've now generated this little app, which is interactive and kind of does something that we want. But it's not very pretty, right? And our boss doesn't, he doesn't really understand what a sigma value is. So we're going to add a little bit of text. But before that, we're going to look at, inspect what this actual object is, right? We've run this interact thing, and we've gotten something back, and it visualized itself, and it all seems rather magic. But what this actually is, if you print it, is a column, which contains another column. And in this column are the widgets. And then there's a bro in it as well, which contains the map.lib plot that we were showing. And then you can also see here that we've got these nice little indexes, which tell us how to index into this object. So we can unpack what we've gotten from this interact thing, and then build something that's actually kind of nicer. So that's what we're going to do here. We're going to add some text, define some markdown text, unpack our little interact objects, get the first column, the row containing the plots, and then put that next to the text with two of the widgets because, again, our response didn't understand what a sigma is. So we're going to just ignore that and just use two of the variables. So now we've got something that's a bit nicer, a bit prettier. We can actually show this to someone. So that's a good start. And we can now, again, use show to show this little app and share it with someone. What we can also do is, as I said, we've dropped this little, this extra widget. But actually, because things are linked, you can have things linked across cells. So maybe do something else here. So things stay linked across cells. And also note that because everything is linked as well, you can also just change the variable from Python. So programmatically, we can still alter the widgets and things update. Okay. So, so far, we've done everything with interact. As I said, it's kind of a magic thing. It's kind of, it does some crazy stuff inside. It does this introspection and things. And maybe you want to be more explicit, right? You want to declare some widgets that you've declared yourself. So here, what we're going to do is declare a variable widget, which is basically a radio button group. We'll see what that looks like in a minute and in slider And then this is fairly similar to what you saw with DASH in the last talk Basically, you create a reactive function where you say actually this function depends on these two widgets So whenever these widgets change, I want this function to be re-executed and my plot to update So that's what you've done here. We've got this reactive function and then we can just put this reactive function in one in our layout and panel will know to inspect the decorated thing and find out what it should listen to and And so, if we do that, we again get, I need to remember to run these things, we again get our little app, and this time we've got a nice different widget and so on. Okay, so throughout this entire workflow, we've used matplotlib, which is nice and fine, but maybe we want to use something else, maybe we just want to swap out for a different library, So what we're going to use here is HVplot. HVplot is basically a drop-in replacement for the pandas.plot function, but it also works on X-arrays, DAF data frames, intake, GeoPandas. So it's a really neat thing that we've been building as well, which you can check out. So what we've done is simply we're going to swap the plotting function we had before out for one that uses HVplot to plot this mean and average. And now we get an app that's now interactive. We've got a little interactive OK plot here. we can zoom and pan and do all the nice things like inspect the individual data points and so on. So whatever plotting library you use, you can just drop it in, and it'll just display it. Okay, so this was just an example of an example workflow that you might have. You probably already have a bunch of notebooks with some outputs, some plots, some tables in them. You can just drop those in there. But once you've actually, so we've seen the .show part, Like, how do I basically stand up the server really quick to look at what it would look like as a standalone app? But now I want to run it and deploy it somewhere, right? So what you can do is you can simply, instead of adding .show at the end of your little object, you can say .servable. And what that means is then that particular object will get picked up when you execute this thing. So when I run panel.serve on this notebook now, again, I'll get the fully deployed app running in a fairly scalable way. So, what this shows is that Panel basically can take you through your entire data analysis workflow. You can start by just using your standard tools, then drop them into a Panel, add some widgets, some interactivity to your thing, and finally you can deploy it to be shared within your organization, or in some cases even outside of it. Okay, so I'm going to show you some other features that kind of make Panel quite nice and stand apart from some other tools. So one thing you can do, what we've seen so far is everything we needed calls back into Python. Every time you change a widget, it has to go back to Python, just like you saw in the dash talk. Ask it what is the updates that I need to apply and so on, which is fine if you want to deploy a dashboard. But if you've got a simple little app, that doesn't seem that important. You don't really want to stand up a whole web server just to do that. And it's just a simple app and it's got like 10 different states or whatever. So what we're going to do is say, actually, I want to embed the output in here. So what this is doing now is we can actually see what it's doing. So you can see here, it's actually going through the different states of these different widgets, recording what the state at every stage is, and then embedding that output into either the notebook, or you can have a standalone HTML file where all this output is saved into, or even have it export to JSON so that every time you change a widget, it fetches that update from the JSON file. to update it. So you don't need to actually have a full server. You can simply say embed and then you don't need the web server with it. So I'm going to have to let this finish. So if you want to instead you can also say save this. So currently this would get all get embedded into this notebook itself. What you can also do is say actually I want to save this into a separate file and then I want to embed this as well. So this should be just about finished now. We're going to let it go through the final little states. Okay, so it's now, yes. So now, obviously, if you have a million different states, that's not a sensible thing to do. In this case, what it's done is basically sampled the parameter space in a fairly discrete way. You can also tell it to sample it a bit more, and so on. So embedding is a quite nice thing for a simple app that you just want to share as an HTML file. So another nice feature is templating. So you saw an example of that earlier, but I'll briefly take you through how that actually works. So, so far, we've built all our, in our example workflow, we've built everything using rows and columns. But actually, if you want to have a nice little template around it, make it look a bit more pretty, you might want to drop it into an actual HTML template. So in this case, what we've done is if you use Materialize, which is basically an implementation of the Material UI style, and then the way this works is you basically, it's a gender template, so you extend it a little bit. You extend the basic template. Then you can either say, basically, just iterate over all the plots or panels that I've added to this template, Or you can explicitly give it names. So you can say here, at this place, insert the Vega plot. At this place, insert the Plotly plot, or whatever you have. So in this case, we've done that. We've declared a number of plots, a Vega plot, a Plotly plot, a Holovies plot. We can then say, so this currently doesn't render in the notebook yet, but we can launch it as a separate server. So here we now have a nicer looking UI on top of things. We have our Vega plot in there. We've got our Plotly plot in there, a nice little 3D plot, and then we've got a nice little HoloViews app built on Bokeh, which allows you to basically interactively zoom and pan and kind of explore really quite large data sets and filter and change the color map and so on. So let me go back. Okay, so these are some of the features that Panel will support. Going forward, obviously Panel is still a pretty young project. We're still actively building out our documentation. We want to have detailed guides on how to actually deploy this thing on different cloud services. So if you want to help with that, join us at the sprints on the weekend. We're also going to build some more examples and expand the gallery. And another nice thing that we want to do is add more components. And part of that work is we've been engaging with some of the Jupyter folks to support Jupyter widgets bidirectionally. So the Jupyter widgets ecosystem is fairly large. They've built a lot of nice components. And it would be nice if we could reuse those. So the idea is that not only are you going to be able to put your panel app inside an API widget and then deploy it using some of their solutions for dashboard deployment, but you're also going to be able to put Jupyter widgets inside panel to kind of reuse that ecosystem. So that's something you want to do. And finally, I want to compare a little bit. So we just had a talk on Dash. There's also the Jupyter folks have been building something called Vola, which also makes it easy to deploy your notebooks as a dashboard. But before I do that comparison, I just want to say how really great it is to see that the Python ecosystem finally has these solutions. Ours has had Shiny for years and years and years. And finally, we're seeing these solutions emerge in the Python ecosystem. So while I'm going to be doing some comparisons, I definitely do not want to denigrate the other libraries. Dash is super nice to build probably more complex apps than Pano will let you do. Voila makes it super easy to just say, Pano, Voila, serve my notebook. You don't need to do any extra notation and so on. But there are some differences, right? So compared to Dash, you saw Dash is kind of closely tied to the kind of structure you see in a HTML DOM tree. So it's pretty closely tied to HTML concepts. I think Panel is a bit more Pythonic and less more concise in some ways. And also Panel lets you use a variety of plotting libraries just out of the box. And you can also embed Panel things inline in a notebook. So if you don't actually want to deploy this thing, you can do some stuff just in a notebook. So it's kind of more geared towards this full life cycle of a data analysis workflow where you start in the notebook and then you kind of explore a bit and then finally you might deploy this thing. And then I also wanted to compare it to Volat. Volat is also a really nice library. They've been doing a lot of work to make it work well. Bola is tied to the Jupyter ecosystem Which means that basically it just runs the Jupyter server It starts a kernel for each user or each session that's started So unlike that, Panel will start a separate Bokeh server Which means it doesn't necessarily have that overhead of starting a kernel for each user Which can be somewhat significant And it also means Panel will let you share data across different users right? You've got some computation, you've got some data that's expensive to load, uses a lot of memory. You can share that and then reuse it. And then finally, the final difference is Bola usually just serves all the output in your notebook. So if you want to kind of narrow down what you want to serve, you write a template. So both have trade-offs, but in panel you basically designate, this is my output, this is what I want to actually serve in the end. Okay, so thanks everyone You can panel an HV plot Which I've shown today Are just some of the libraries that we work on We also work on libraries like Datashader Which did the nice resampling that you saw earlier HoloViews, GeoViews, Param, ColorSet So check those out Go visit us at holoviz.org It provides a nice tutorial Taking you through these tools You can find me on Twitter and GitHub Under philipjfr and this is my email. Thank you. Thanks, everyone. Doesn't even have questions. Any questions? Yeah, thanks for the great talk. If you're using the save option, saving the HTML file and using embed equals true, what's like a practical limit for how large of a data set you could embed that is a pretty good question so I haven't scaled it up to a thousand different states it's sensible right it's not too crazy especially if you can then spill that out to some JSON files so you can just dump those somewhere on a web server or something but it certainly won't scale to like if you want to have five widgets which each have a hundred different states I mean that's the combinatorial explosion, and it's crazy. So it's really geared to smaller, you just want a one-off thing that you want to send to someone. Hi. You mentioned some new features that you want to implement. Could you give some examples of what you see happening in the future? Sure. So one of the things we really want to push on is the API widgets integration, but then different components. So some of the components I want is I want an editable tables, some other things I can imagine. We definitely want to push the templates, templating a bit further. So I'm currently working on, for example, being able to show the templates inside a notebook so you can kind of see what it's going to look like and then deploy it. But beyond that, it's really driven by users. So we've had various users contribute new components, things like a VTK renderer, any new plotting libraries that you want to embed. So it's really driven by user's needs. Thank you again for this great talk. This is definitely a tool I want to try out. One question would be, if I want to use this in a semi-production environment, just show it to some customer or something on the web. Is there a simple way of implementing some basic authentication with this? Yes. So this is something that's, well, right now, no. But in about two weeks, Bokeh, so since it's built on top of Bokeh, we get all the nice features that the Bokeh team implements for free, and they've just implemented an auth provider. So you can simply basically write a little Tornado auth provider, and then you get it for free. So we're going to have examples on that very, very soon, in the next couple of weeks. All right. Thank you very much. Thank you, Philip. It was an amazing presentation. So let's give an applause to our speaker once more and have a nice lunch, guys.