Jupyter Notebooks for Print Media

Speaker 1 [00:05]

Thank you everybody for coming to what is kind of a niche subject in my opinion. Maybe not the most like sexy topic, but it is Jupyter Notebooks for print media. And when I mean print media, I mean like literally like printed documents, like stuff that you would print out on a printer. So I don't have a like great live demo this time. I have some examples of using this library and be print in action. but I'll start with just some background and motivation and there's a decent amount of background this library is basically glue a lot of the libraries I write are basically glue, but standing on the shoulders of giants and other important libraries, many of which I'm sure you've heard of already, some of which hopefully you haven't, so it'll be something new but kind of gluing them together to accomplish a task that I need to accomplish, but that I think is probably something that many people have to deal with I'll show a little bit about how the library works. It is not done yet, like the library itself. Maybe open source libraries are never done, that might be a philosophical question. But it's not that far along, we'll say. It's something I would call alpha or beta stage. But it's progressing quickly, and I think it has a good spot that it kind of fills in in the ecosystem. Part of this talk is also going to be a rant, but you'll know that part when we get there. So I have a question here, which is just that how many people in the audience, I assume most people are developers, but how many of you work with non-developers? All right, so most people in the room are honest, and the rest don't want to admit it. I think most of us work with non-developers, even if it's something like an engineering manager, but I have to work with non-developers quite a lot. And whenever it comes time to present things that I work on, when I go out, when I go on Hacker News, or when I go on GitHub and look at the trending things, it's always apps and dashboards and widgets and things like that, which is great, and I'll mention a bunch of the libraries that enable apps and dashboards and widgets, and also my talk earlier was about apps and dashboards and widgets, so speaking from both sides of my mouth today, but that's all right. And I think it's very useful when you're iterative, the stuff that you're working on day-to-day can be delivered in this form. And I put it in quotes because this is usually what gets asked of me, is that someone just wants a PDF. Which kind of hurts a little bit when you're like, oh, but I made all this cool interactive stuff. It's like, no, I just want a PDF, and then I'm going to mail it to me, and then I'm going to print it out, and I'm going to read it on the plane or something like that. So there's also some kind of aspect of, well, okay, fine, if I have to produce something like a PDF, is there an overhead for that today? I think most of us probably would use something like LaTeX for that. But if I'm living in kind of a Jupyter ecosystem for my research that I'm performing, do I have to like context switch in order to produce these documents, even if you're just going to Microsoft Word or PowerPoint or something like that? It's not like a native experience, at least not for me. I like Jupyter, I'm comfortable in Jupyter. Doing things in PowerPoint is difficult for me. Another big thing which I face in a lot of the topics that I have to do, maybe not everybody does, but it's about parameterised reports. So if you're not producing one document, but a suite of documents, and I'll give some examples a little bit later on, obviously the overhead increases from there. So if I'm producing a Jupyter notebook, now I have to take that notebook and maybe generate PowerPoints or something like that. There are some tools for that, but it gets pretty annoying pretty quickly, and again, you have to context switch out of that place where you are conducting your research. My thesis around doing this all notebook-based was a lot of times, at least in the things that I do, a researcher produces a report that they deliver to a business user, but then they also want to interactively work and further inspect that report, so they oftentimes have to go back into the notebook later on and maybe toggle some things. So I didn't really want to have a separation between the place where research is done and the mechanism by which research is presented to other researchers or kind of non-developer, non-researcher business users. So I give an example from finance because I work, I should have introduced myself, I'm Tim, I teach at Columbia and I work for a hedge fund in New York. in finance oftentimes like financial analysts people at banks have to deliver kind of on-demand research so for some stuff like if a client calls you and they're like give me your research report on apple it's a big enough company that this is something that is like highly curated you know every every pixel is perfectly placed but in finance and in many other places you also get talks or calls like, give me a report on the New York City Municipal Water Finance Authority bond, which is like a bond that is produced to fund New York City water projects. And there are millions of municipal bonds. There are millions of municipalities, and all municipalities need to fund their endeavors. So you can't really build curated reports on a million different things unless you have like a massive budget or maybe lots of interns. So with these things you need the ability you need to be able to deliver on-demand reports and again finance a little a little bit slow here and there we like things in pdf form or an email form but other industries that's also true. A couple of other examples that I could think of on the fly news is one and I'm not going to take a like moral stance um but i used to work for a data vendor and we would work a lot with like news brokers i want to say news aggregators and a lot of them would produce like via natural language models like news on demand um but certainly in the age of llms it's hopefully a lot better quality maybe maybe a lot faker who knows um but being able to produce like news reports or something like that uh on demand um sports politics voting finance transit weather These are all things that tend to be maybe a little bit more in paper form or that you might want to produce reports on demand. And one that is particularly relevant in finance in particular, but probably in other industries, is like machine learning research, incremental progress reports. In finance, you have a lot of like ongoing model assessments. So every month you'll have to deliver the performance of your model versus expected performance to ensure there's no like substantial degradation in how it's working. And again, these are usually non-technical developers or non-technical people reading them. Either business users or lawyers or compliance and stuff like that. So they often want it delivered in PDF form. Alright, if I'm using Jupyter, so again, my thesis here was basically, if I'm using Jupyter for my research, maybe I can use it for my paper. And I also mean this in the context of academic research. I think a lot of these tools, Jupyter, IPython, came from academia, and a lot of the tools started being able to produce nice academic reports, but we've kind of strayed a bit, and I think that that's also a viable place. If you've ever written a paper in CS, ML, things like that, people tend to be living in notebooks, and then again, it comes time to do the paper, switch over to LaTeX, and kind of have to context switch. This is the rant part. So, Jupyter obviously needs no introduction, but this is a bunch of tools that kind of exist in the space. Nbconvert is really the big one. I have them open here, Nbconvert, if you just Google it, they'll find their documentation. This is sort of the Swiss Army knife of notebook conversion, taking a Jupyter notebook, converting it to HTML, PDF, whatever, but kind of just as is. So it provides, I mean, I rely heavily on this for my library, but it provides the backbone of being able to convert things, and it relies on other libraries like Pandoc underneath to help with this. Voila is a fantastic library for taking a notebook and turning it into a web app. Panel is another library for doing the same, basically, and more, but taking a notebook and turning it into a standalone web application. Jupyter Book, some folks have mentioned, there was a talk earlier in the week on this which I thought was great, and this is a great project. It is probably the closest comparison to the thing that I built. It lets you take notebooks again and construct, or sets of notebooks, construct documentation such as this page. It often looks very similar to this. The closest thing actually to NBPrint, which is what I wrote, is this project iPyPublish, which is still maintained here and there, but the goal for this was sort of the goal for mine, which is write your notebook, write your code, publish it as some kind of PDF. It relies on nbconvert, just like mine does, and it relies on LaTeX a lot internally, but this is sort of the goal. People want to be able to produce these things. Now, the rant part is that even though a lot of these tools were kind of built with LaTeX in mind, built with research papers in mind, I'd say like the vast majority of the development work has focused on apps, there are apps everywhere, apps, you know, every single conference has a kind of app tool as a sponsor, basically I want to just write Python, I never want to learn JavaScript, I want to take my notebook, I want to take, you know, standalone Python functions and I want a web app, right? And that's cool, the rent part, you know, obviously a lot of venture capital money has flowed into the notebook space, and every single venture-backed notebook startup has a notebook platform, and on that notebook platform, one click, you can publish an app. But really, not that much has been done in the print space, not much really focuses on PDFs. Again, PDFs are not that sexy, but critically important. And again, JupyterBook is probably the closest, Myst is fantastic, Myst Markdown, that whole ecosystem. back to that a little bit later on. But really, you know, none of these things totally did what I needed them to do, which is to produce print-oriented, you know, media, print-oriented reports, allow me to configure the type of things that you want to configure when you're writing, when you're producing print-oriented media, and have a couple of those extra bells and whistles like parameterization, so I can produce, you know, do some like model fitting, I can produce and reports, stuff like that. So I've been thinking about this library for a lot longer than it has been actively worked on. Like years at this point, I think. But a lot of stuff has happened in the meantime to make it a lot easier to do and make my life a lot easier. So there's not really that much code in the library. Like I said, it's mostly just glue. But I think the results are pretty cool and we will show some of them in a minute. As always, it's open source. so you can go to nbprint I should have put it in order github.com slash nbprint slash nbprint and it lives here it's basically pure Python there's some JavaScript components as well and it is just a bridge between nbconvert as we said and the w3c paged media specification so this is kind of browser web specification for things like CSS rules and basically how browsers should handle directives that indicate that print is the target of the web page. I didn't really know this was a thing a couple of years ago. It's come like really far and the browser support is quite extensive for it. But the browser has baked into it all kinds of stuff, how to slice and dice pages so that when you do, for example, like command P, it knows how to lay things out. It knows what goes where. It's able to kind of put things in the right place, split pages correctly. And this was kind of the missing link, I think, for a long time, because it means you don't have to rely on LaTeX for all your very nice typesetting and stuff like that. You can let the browser do most of the work. The key kind of junction point is it's still a little annoying to do, but thankfully, there's another open source library called PageJS, which does all the remaining heavy lifting between nbconvert and kind of what the browser can do. This is the description from the Page.js website. Page.js is a free and open-source JavaScript library that paginates content in the browser to create PDF output from any HTML content. This means you can design works for print, e.g. books, using HTML and CSS. When I read that, I was like, what? No, that's not a thing, right? People aren't making books in Google Chrome, right? That can't be possible. But they do, and it's amazing. So people actually use this library. They don't use my library. It's not that popular. But they use Page.js to produce real physical media, like books. There's a bunch of examples on the Page.js website. You can see that they, I mean, this sounds sort of dumb, but they look like books, right? They have all of the features of books. They have all the layout that you want, table of contents, pagination. You can do basically anything that you can do with a book you can do with Page.js. So this was amazing to me, and I'm still shocked at some of these things, right? Like, they just, they look good. They look exactly like what books should look like. So I was like, okay, I've got Page.js, I've got mbconvert, I'm going to plug them together. There's a tiny bit of extra that needs to go on top, which is that Paper Mill is another library, lots of glue, from Interact, sort of out of the Netflix ecosystem, which lets you do parameterization. So you can parameterize a notebook, you can say, I actually want to run five different copies of this notebook with five different input parameters. So you can imagine you're doing some model assessment. You need to toggle parameters, generate a report for each of those sets of parameters. You feed it into Paper Mill. It will kind of execute notebooks and let you execute with those parameters. This was a disclosure I gave at the beginning. Still a work in progress under active development, but I want to show you guys some stuff now. This is kind of high-level overview. I let you configure from YAML, just because, again, if you're going to be parameterizing reports, you're going to be generating lots of reports. It's sometimes more convenient to kind of just use YAML, do everything statically, as opposed to starting from a notebook, but you can start from a notebook. NBprint will help you go from YAML to notebook. From notebook to HTML, we use NBconvert. I have a custom template, which NBconvert has lots of hooks for extending. So a custom template to produce an HTML document that is linked, you know, the notebook outputs with the page JS. So there are directives that you can put in your notebook to say, put this in the top left corner, or put this in the bottom right corner, or include a table of contents, stuff like that. There's also custom hooks for like arbitrary CSS and, you know, JavaScript. And then once it's in HTML form, you can look at it. It's very nice. You can hit control p, you can kind of print it, or nbconvert also supports something they call webpdf, which is basically headless Chrome, render it in Chrome, and produce a PDF from it. So you have an end-to-end solution for parameterising reports, generating them, producing PDF documents. Cool. That's everything I just said. Let's look at some examples. Okay. So, again, it's a work in progress. You will find bugs. A couple of hooks that I think are interesting, So this code is way too small for anyone in the audience to see. But a lot of the things are done as Pydantic models, because Pydantic is popular, but also it's convenient when you want to kind of statically configure things to have static typing. All of the different things that you can deal with, like if you want to write your own section of your own page, you have these hooks for arbitrary CSS and JavaScript inputs, so this is pretty heavily relied on. You can define via static YAML configuration. This is an example. It's going to be a little bit tricky to follow, but I can specify things like please produce NBconvert outputs, just output to HTML, don't do web PDF, just go to HTML so we can look at it, put it into this folder, parameterized inputs. You can provide context in your notebook because this YAML file is going to programmatically generate a notebook. You want to be able to pass variables around in the notebook so that after you've created the report, you can go back to the notebook and toggle things. And then you can, you know, set up your page. In this case, I don't know what I did. I have a example page. I've got the bottom left has a little logo. The bottom right has, I think this defaults to being a like page number. And then I've got example report, just text in the bottom. So you can configure basically the various regions of your document. Page.js supports, you know, top bar, like there's three regions, top left, top middle, top right, left top, left middle, left bottom, or left center, or something like that. You basically have 12 regions around your page. And then you can put content in there. That's probably what you wanted to do. So here are some Pydantic models. They're all callable, which I'll show in a second. You can configure them hierarchically. If you recognize this syntax, It uses Hydra to go from configuration to notebook, or configuration to Pydantic model, and then Pydantic model to notebook. You can see I'm basically laying out the cells of a notebook. So this would be the first cell. It will instantiate one of these things, potentially with some parameters, and then call it so that it can render into the cell output. So I did some table of contents, some markdown content. Obviously, all these things you can configure in the notebook as well. And you probably wouldn't put, so I think I put code in here at some point. You probably wouldn't do this. You would write your own custom module. But I just did it as an example. So you have all these different things. Here's another good one. Just kind of same deal. What else? Here's an example of what one of these callable models might look like. So if you're writing your own thing, you might produce something that looks like this. You've got your kind of pydantic model. You accept inputs. These can be configured via YAML. If you're doing the notebook generation part, there's going to be a bunch of metadata injected into the cell, but then it will just call this. Okay, I've shown a bunch of boring code. Nobody's that interested. Fine. Let's see something real. So I took one of these things and I ran nbprint on it to generate a notebook, and it produced something that looks like this. This is a bit ugly, admittedly, because this is a very complicated layout that I did. The goal here was mostly that if these things had slightly better names, once Once I've produced my report, I can come back here and I can kind of toggle parameters, so like new output, or whatever, new input equals five, because a lot of times these are for interactive research, but if I run this thing top to bottom, you'll see something that maybe kind of looks something like a report. This is a financial report, so we've got like, you know, company name, some tables, it looks okay, some text content, some author names, whatever. So I won't spoil a surprise at the end, I think it's a pretty good result. But I'll show you some other examples before we get there, but this is sort of what it looks like in the notebook, you can toggle things. If we start from that simple kind of research example, you might end up with something that looks like this. So each one of these is a notebook cell. This was a notebook cell, this was a notebook cell, this was a notebook cell. This is a Jupyter notebook, as weird as that might seem. You can see my very interesting research that I'm conducting, which is just lorem ipsum generation. If I had more time, I actually had a paper I was going to reproduce. You see you've got things like multi-column layouts, inline images, all the stuff that you would expect. And again, critically, if I haven't messed it up, nope, I have messed it up. If I Command-P this, what you see on the page is what you get when you print it out. And that was really the key thing. We want to target print-oriented media, stuff that is going to be printed out. This one's kind of boring. Let's look at a more interesting one. So this one has some slightly more interesting stuff going on. We've got a header page. We've got a little logo, table of contents. You've got kind of jump to sections. That all works fine. All your kind of nice bells and whistles are there. I can put stuff in the bottom. I can do page counters. All the stuff that works in Jupyter also works here. I didn't do an interactive chart, but things like Bokeh and iPyWidgets also work here, which is kind of interesting. If you don't really want to go to print in the end, you can still use it for just the kind of rendering step. And again, let me, because I didn't spoil this one, this is just the PDF version of it, right? So again, your table of contents all works in the PDF as well, so you've got that linkage. And what you see on the page ends up being what you get. So no more having to think about, like, I've got widgets, so I can't go directly to PDF, you can go to the web have it look however you want it to look with all the interactive libraries that you want you want to use matplotlib, great, that'll work out of the box you want to use bokeh or plotly, render to html you still get a PDF in the end and this was the one I wanted to do this is a real report from Morgan Stanley I'm not affiliated with Morgan Stanley but I googled financial report and this was one that came up so you see these a lot these happen a lot in research This massive thing that investment banks do is producing this stuff all the time. But I wanted to use this as kind of like a benchmark of complexity as far as layout is concerned. And maybe unsurprisingly, this is what I came up with. So again, this is a Jupyter notebook, which is kind of crazy. It looks very similar. I didn't do one-to-one. You've got all your complex layout. Active Jupyter users might note that this has the CSS rules and stuff from a notebook. So we get the highlighting of Panda's data frame here, all the layout stuff, every element you see on this page is a cell of a notebook, I didn't do the whole thing, I just did the first page. And again, moving this thing to PDF, what you get in the end is what you saw on the page. So that was basically it. Next steps are really just like, I want to integrate with Mist, because Mist is great. I want to do landscape mode, so that you can do presentations. I actually might use Reveal, which is what I use for this presentation, and then just providing a library of standard components, examples, and then reports that can consume other reports. So, I need help with this library, I'm still working on it, it's under active development, I need users, I am a user, but please check it out if this is something that's interesting to you, or just for fun, and let me know what you think. Thank you.

Speaker 2 [23:56]

Thank you so much. We have quite a few questions. So one of the questions is about your opinion on... I hope I'm pronouncing this right.

Speaker 1 [24:08]

Oh, is that it's one of the notebook? So I had a list of them. I Didn't put it in the presentation, but I counted and there were like probably five or six of these different tools I mean, I think they're all good. I think it's good to do stuff in the space But and I use like I use all of those different tools to deliver Some portion of like research results that have to be delivered But there is still this gap of just like there's a spot where you know it needs to be easy to use you need to be able to go back and forth produce a PDF report but then reopen the notebook and I don't think there's really anything comprehensive that does it if there is let me know because I'll delete all this code tomorrow and just use that

Speaker 2 [24:53]

The next question is about what you would recommend to comfortably create slides from a notebook.

Speaker 1 [25:01]

There's been a project, I think it's called Rise, which is basically... NVConvert supports Rise as an output. And it produces... Rise is essentially like Reveal.js based, so this is done with Reveal. It's just like a framework for doing presentations. I forget what the argument is, but if you pass in an argument here, you can put it into print mode, and you'll just get slides. So this is really the best thing to do for that. It doesn't let you configure all the things that you want to configure. So again, that's the same deal like you could just produce pdfs with nb convert natively um but it just like kind of flat gives your content you can't do certain layout things that you may or may not want to do you can't do multi-columns um but it's also there's little things like line numbers and stuff like that if you do it in pure nb convert you end up having to just write your own templates and they're not i mean they're not hard to do um we can look at the one that i have but they're not like the easiest things to do either um and if you are starting to maintain a library of these nbconvert templates, it can get quite complicated quite quick. So you have all these hooks. It's Jinja, so you have all these different blocks that you can provide. Really, at a certain point, you're going to want to plug in a JavaScript library and do things from JavaScript, which is basically what I did.

Speaker 2 [26:17]

Thank you. So the next question is how would your library work for math-heavy report, so for example using a lot of LATIC and multiline equations?

Speaker 1 [26:29]

Yeah, so this is another part where I didn't really do any work. It was kind of all done for me. So when you do stuff, when you nbconvert to HTML, you can have Mathjax integration, and the default nbconvert template uses it. So when you... Did I not put any in any of these? There should be some. Yeah, there's a tiny bit. I did just enough. I'm not the biggest LaTeX fan, if that wasn't obvious. But you get Mathjax rendering on the inside. So basically before the page JS pagination, but after the initial rendering, it'll do a mathjax run and that'll do your conversion. So there's still wonky layout stuff that can happen. I'm fighting with it all the time. But for equations and things of that nature, I should have put it in the research report really. It's not that bad.

Speaker 2 [27:17]

So the next question is, is there also a way to create links between notebooks like scientific references?

Speaker 1 [27:26]

references? Yeah, so within a notebook is kind of the next thing, and I really don't want to do

Speaker 2 [27:26]

Yes.

Speaker 1 [27:31]

that, which is why I'm looking into Mist. Mist does all this stuff, and it does it really well. You can do it today via these contexts. If you're producing it from a YAML, context gets passed around, so you can do it that way. It's a little clunky, but I think Mist is really the right thing for that. It's on the, it was one of the kind of next items. The other big thing is a lot of times, again, if you're doing a model investigation, or like, I don't know, you're investigating parameters to your model, you want to just run, do a bunch of runs, or in finance, we do like backtests and stuff like that. So if you want to do a bunch of those things, and then produce a report that consumes those, if you're producing 1000 different backtest runs, you don't really want to go and manually look at those, but you still want the artifacts because you're going to consume them, pick the best 10, and kind of pull those artifacts aside. So that's the kind of report consuming report stuff, which I also think is important when you have notebooks referencing other notebooks, or if you're building a corpus of these things, you want them to have stable links to each other.

Speaker 2 [28:28]

So maybe one last question. Can you generate PDFs with CMYK colors in a specific color profile, like the ISO-coated V2300?

Speaker 1 [28:38]

Probably. I mean, I don't know specifically. I think if you're thinking about some of the stuff, some of the Page.js examples, you're somewhat bound by the browser. So it's sort of whatever the browser can do. Even Page.js is just responsible for doing, this is a preview of it, but it just does some of the hooks to help with pagination. What you actually get in the end is just what the browser provides you. So Chrome is not the most featureful one um but it will you know it gets the job done um if you don't have support for those things in the browser or you're not able to configure those things via like a javascript framework or css then you probably won't be able to do them

Speaker 2 [29:22]

So, unfortunately, we've run out of time.

Speaker 1 [29:25]

Yeah, I'll hang out.

Speaker 2 [29:27]

So, thank you very much. Thank you.