High Performance Data Visualization for the Web

Speaker 1 [00:05]

Thank you, thank you everybody, thank you for coming to my talk. I tried to pick a pretty bold title, and I realized in my time slot that we all picked very bold titles, so hopefully I can live up to high performance data visualization. My name is Tim, I teach in the computer science department at Columbia, and I'm a quantitative developer at Cubist Systematic Strategies, which is a quantitative and systematic hedge fund. It's part of 0.72, as I wrote to you. Used to stuff always going wrong, students coming in late, all that fun stuff. So I have a bit of a presentation, but I want to kind of jump as quickly as possible to the live programming part. We've got background music now, too. Oh, perfect. So I'll try and run through this relatively quickly. I tend to talk pretty quickly, too, so if I'm talking too quickly, just wave and yell at me to slow down. I just wanted to go through some background and motivation. I think people are probably familiar with most of that already. Talk a little bit about perspective, which is really the key element of this talk. And then, of course, no good talk is complete without stuff going wrong in a live demo, So I'm looking forward to that Background everybody knows about web frameworks in Python very easy to use. There's a lot of them I listed a very small number of them here. I should have mentioned that my talk will be on my github But this links to one of these, you know Popular how many stars do you have web frameworks? repos on github and And, of course, data visualization components are also everywhere. There's all kinds of different ones. You don't suffer for lack of choice. And I don't want to disparage any of them. Some stuff is still a little clunky to do. In particular, virtualization is an area that, for whatever reason, nobody seems to care that much about, but it's very annoying when it crops up. This is my Airflow instance at home where I run random stuff. And if I click on that 728 past DAGs, I got very lucky, they almost all passed, but usually it's like 12,000 failing tasks. This thing is fully paginated, so even something like deleting is difficult to do from the UI, you have to scroll through things, I'm sure everyone is familiar with paginated grids. So in certain contexts that's good, in other contexts it's annoying, or not being able to see all of your data or not being able to work across all of your data can be limiting. So many frameworks either can't or won't support unbounded data sizes, so for virtualization we mean basically being able to scroll infinitely in a grid. Dumb grids are also kind of everywhere. So key features that you're probably going to want are things like filtering and sorting on columns, spark charts, those are the little inline charts in the middle of the grid, and maybe some of them support pivoting, there's not too many of them. This is a screenshot of AG Grid, which is one of the most popular JavaScript grid frameworks. You can see this is just an example I took from their website, where column pivots and row pivots are a little wonky. It's not really clear how to configure these things. We have some aggregations going on. Also grids themselves aren't always adequate, so in this one above, usually what we want to do afterwards is something like a heat map or a tree map, throw something on top of it to further gain insights from the data that we're looking at. And there was a great talk yesterday about tracing, and I should have grabbed more screenshots from it, but what usually ends up happening is, because grids are insufficient, you end up just throwing as many other kinds of visualization as you can. So everyone has seen dashboards where it's just like, how many different charts can I put on the same page to make it completely unreadable? And that talk, again, I'm going to just steal, I'm going to plagiarize. This is mostly for us to put in our corporate headquarters and stuff like that, but no one actually gets a ton of insight from these types of things, and certainly if you've ever gone to start configuring a Grafana chart, it's a little bit difficult to dig in and slice and dice further. And the last thing is, all of these things, you can probably work around, you can probably sort something out, and then you realize that your data is streaming, and that you actually need like live updating stuff and you need all your aggregations and your pivoting and your you know your chart and your grid to all be updating in real time um so you can do kind of polling but then you also have to manage your polling and your pagination so i'm sure people have hit things where like you know you have inconsistencies between your client and your server um and you know to their credit obviously uh people don't pay a lot for the open source software you know so there's no money back but uh web frameworks and visualization frameworks both have not made this super easy. It tends to be kind of assume static and then try and handle streaming later on. So here's like an example from the FastAPI docs of a streaming WebSocket server. It's, you know, a bit long. And if this is kind of the baseline example of just getting something working, you can imagine that if you're doing things like, now I've got end users who want end different aggregations, right? I've got some static state as of this point in time, a new user connects, I need to flush state and then provide incremental updates from that point onwards. All of these things become super difficult, and there's not a ton of tooling in the open source ecosystem to help with this type of thing. So with all that said, I work on a project called Perspective. We originally wrote it at JPMorgan, which is a bank. It is basically a streaming pivoting aggregation engine with some, you know, visualization components on top of it. This is like the read me, you know, elevator pitch. I have all these links in the slides, again, I'll make sure they're on my GitHub. But basically, it is a C++ based engine, like a data engine, you can think of it almost like an embedded database, but not really. Not designed with the same constraints in mind. It is bound to Python in the back end. On the front end, the same C++ engine is compiled to WebAssembly. And then on top of that, there are UI plugins which are written in a mixture of Rust and TypeScript and vanilla JavaScript. So we have this symmetric C++ engine, which is kind of cool. So it means you have the same engine running on both the client and the server, which again is only possible thanks to Wasm. There's a fair amount of gil-free processing, though I'm sure I have a lot of bugs in there still. You can do kind of pure client architectures, meaning you can embed the entire engine just in JavaScript. You can pull all your data and do all of your operations purely from the client. And you can also do kind of client server architectures. I'm going to demo the latter today because this isn't a JavaScript conference, so we've got to have Python in the mix. And between the client and the server, we have virtualization via diffs, which are all sent over Apache Arrow. On the front end, it's web components, so I'm going to show some HTML and JavaScript code. Hopefully people are okay with that. It's very minimal. There are plug-ins for different views, so if you want grids, charts, stuff like that. And the key thing, I think the most important feature is the fact that it exposes all of its configuration to the end user, so you can basically hit the configure button and start playing around with it and change the view to suit whatever questions you're trying to ask of your data. Okay so that's basically it for the presentation part. So again, this is the GitHub page, it's hosted under Finos, which is the financial fintech open source organization, it's part of the Linux Foundation. There are a lot of examples linked here, so I'm going to pop through some of them. They might deform a little bit when I make them larger, so hopefully it doesn't look too bad. But this is kind of like the canonical example. This is a superstore kind of simulated data set from tableau So you've got your kind of grid here. You have the same data manifested as a tree map You have pivots here row pivots. You've got some like regional and statewide pivots. You've got some Column pivots in terms of I don't know what these are categories and you know various other things And of course you can kind of pop this thing open we can inspect what these pivots are, group by region, group by state. I can undo those ones, split by category, split by subcategory, kind of undoing the example to get back to the original kind of flat grid. So I can bring all my columns into view. Right, and again, you have your big flat data set. You can now ask these questions that you want to ask interactively. So you can group by, I'll just do something quickly. I can group by category, and I'm seeing that I'm now aggregating the numerical columns by some default, in this case sum, I'm aggregating the categorical columns by some other default, which is count. If I want to just do something simple, I'll just do sales and profit, sum of postal codes. I don't think that's super meaningful. Profit. And then, of course, this is a grid form. I can immediately pop it over into, let's say, heat map form. I can pop it over into a tree map form. I can look at this data. I can duplicate it. Now I've got a different form of the same data where I can, you know, maybe I want to look at it in both forms, a tree map and a data grid. I can see these things side by side. There's a little bit of a layout engine here to look at these different things. This is, again, kind of a fake data set. Here is a real data set. This is the New York City city bike data. So one of the plugins available is a map. I don't know what the tool that we use for the map is, but kind of map with the latitude and longitude, we can kind of, it's a scatter plot on top of the map, so I think it's colored by available bikes or something like that. But again, same deal. I can, in this case, I'm just sorting by last reported. I think the data's pretty old. You can kind of get a view for what it looks like. And when you couple these components together, because they all kind of work in the same way, you can build some interesting charts. This is probably a bit too small to see, but this is Olympics data. So it's medal winners for Summer and Winter Olympics from 1896 to 2016. So in this case, I've constructed, and again, apologies for a bit of the zooming, I've constructed, it's one data set that I've now kind of bifurcated into multiple visualization components. I've got a heat map here, I've got a scatter plot here, And I have aggregated grids on the left-hand side. Again, all of these things, I can come in and see what I'm doing. But because they all work in the same way, I can do things like cross-filtering. And this, I didn't have to write any code for. We'll look at the absence of code that I wrote. It all just kind of comes for free. So I can do something like, let's just cross-filter by archery. It's not going to be a particularly interesting chart, because these are both grouped by sport, I believe. But if we do something like by country, we can see cross-filtered breakdowns. Now, some of these aren't going to have data present. We can kind of see the cross-filters apply. And again, because of the component, it's all fairly straightforward to see what's happening. It's just applying filters across all of the available viewers. Okay, that's like the quick, like, here's a bunch of somewhat pretty-looking charts. I want to go through, like, a real demo. because I think it's useful to see code. I'm making a little streaming data application. I'll see if I can get through the whole thing now and be still somewhat intelligible. I'm using a couple of frameworks to make this a little bit easier. So perspectives is one of the key ones. I'm going to start out in the notebook. But I'm using a data generation tool that I wrote called Superstore. It's basically a version of just kind of categorical data generator, the time series generator that got deleted from the most recent version of Pandas and a couple of other things. I'm also using a library called CSP, which is a Python stream processing library. There'll be a talk at PyData London about this, but it is a kind of high-performance... If I have a high-performance front-end, I need a high-performance back-end, too. So it's a Python stream processing library written in C++. Cool. So I made some sample data, which is just some machines. I got inspired by the talk yesterday. So I've got a bunch of machines that are doing, that, you know, they're going to have a region, they're going to have a certain number of cores, they're going to have some, like, category, an edge machine, a core machine, a worker machine. And the Superstore library is just generating fake data for me. I've got, you know, the ability to look at the machines. I can look at, you know, an individual machine. Let me pick a reasonable one. This one, cool. So when I run the usage data generator function, it'll give me CPU memory. This is all fake, but I needed something that looked good. And I don't have that many machines available to me. I can ask for the status of machine. It'll give me some last update time and some status. This one happens to be operating at capacity because the CPU usage is a bit high. And I can generate some jobs, whatever stuff that runs. I just gave the job a name and some units that it's going to consume. Using that CSP library, I'm just going to construct a streaming graph. Again, the details of this aren't particularly important. The key thing is that I have a couple of these perspective widgets, and I'm going to connect it to ticking data sources that are going to populate those widgets with data. So I will generate a static list of machines. I'll generate new usage information every second. I'll generate a new status report every five seconds. I'll generate new jobs randomly every five seconds as well. and I'll push all of this data into perspective widgets. Okay, cool. Hopefully everything will work. Using ipyewidgets just to get a bit of layout. It might look a little squashed on the screen, but I'll see what I can do. All right, so I've got my four little perspective widgets, and I can kick off the stream processing, and we'll see things are populating, things are ticking in real time. Not a ton of code. Most of this code was boilerplate for the data generation, but fairly easy to kind of create these things, and again, they're interactive, so once I'm here, I can do things like grouping by kind, so now I can see, you know, sum of CPU usage across all of my machines grouped by the type of machine. I can interactively kind of toggle these things. I might want to just do a bar chart. So here, I'll change the status widget, which is this one, to an X bar chart and do a couple of, you know, sorting and aggregations. So now I can see this thing. It'll update every five seconds. Cool. So we wanted to still have the availability to do something interactive, things like widgets and like Streamlit, these kind of pure Python, let you play around, let you slice and dice your data. But the key deliverable, the key target is to be able to build a real web application, high-performance web application with relatively minimal code. So I did that as well, fairly short. This is just taken verbatim from one of the perspective examples. And again, I'll make sure all these are available on my GitHub, so you can click from there. I basically just have a thread to handle Perspective's integration with the outer web server. We support a couple of them. I picked FastAPI because it's a very popular one. So we just have to hook in so that we're able to kind of process incoming data, process outcoming data, stuff like that, run it in a thread. I have the WebSocket handler where I basically hook up into FastAPI, or in this case, Starlet underneath, and we have this perspective starlet handler and I basically just call run on that and the rest of this stuff is just fast API I never know how to do cores properly so I threw this in at the bottom and making sure I can mount that index.html this is that same graph from before just instead of perspective widgets I have perspective tables we don't have the widget interactivity, it's not widget based just because that would be a little bit constraining but same deal, generate some random machines create some usage information create some status information updating one second five seconds five seconds and push it into perspective again this is just the code from above I'm calling I'm instantiating these are from the notebook I'm instantiating these tables and I'm hosting them so that they're available and then again this is just kind of run it with uv corn the index.html is also pretty straightforward hopefully this will work I know the wi-fi when everyone comes into the room is a little slow, so hopefully it renders still. But a little tiny bit of layout, and then my perspective viewers. Again, with web components, they're all kind of self-contained, so there's a custom web component called a perspective viewer. I gave it some IDs. I basically just wait for the web components to be ready, connect to my web server. For each of those things, I load the table via the WebSocket connection into the front end. And then again, I wanted this to be an X bar, so I did a little bit of configuration. All the configuration is driven by JSON, so pretty straightforward. It's more code than I would have liked, but I can probably make it shorter. But it's basically just this part. Instantiate a web worker locally, connect across your web socket, and then load up your tables. So if I run this, which hopefully I haven't broken anything yet, I keep tinkering with it, so it tends to break. I should have kind of same view and it'll update about every second or five seconds. Same deal, not a ton of code probably less than 50 lines total probably less than 20 lines of real substantive code. And of course this is a high performance talk so that's not really good enough so let's make it a little bit faster let's just make it like 10 times faster that's probably a good starting point and this is where I kind of skew from meaningful charts to you know stuff that maybe is not quite as intuitive to see but it'll get the point across that this thing is very fast and again this is doing this is pivoting in real time so these are pivoted by kind so you got active idle capacity these jobs are being generated that's probably also not good enough so it was a high performance talk let's make it a little faster and my computer is going to get hot but we'll see how it goes alright, so now we're moving in real time again, this is pivoting in real time and just because I wanted to this is a slightly more complicated view and they're running at the same time this one is a multi-aggregation grouped by kind, region, and something else we've got our little spark bars in there I mean, nobody can see what's going on but we've got our little spark bars for CPU. This one is also aggregated. These are two separate clients connected to the same server, all being fed with the same data. It's all kind of symmetric. I'm not gonna show this, but I can edit things, and that will also propagate from the client to the server across to all the other clients. Yeah. Pop back over here. So there are a lot of other examples on our GitHub. It's all open source, it's Apache 2 licensed. I didn't show there are computed column support via eXPRTK, we basically stand on the shoulders of giants to enable this. WebAssembly, Apache Arrow, eXPRTK for computed columns, and again, it wouldn't run as fast, so I didn't, actually it had some computed columns in there, but I didn't show them. You have a whole expression language that you can use inside the grid. There are some vanity examples here too, like ray casting, which you'd never do in a grid, but we do full ray casting via computed columns. There's a live webcam one, which probably won't work, but I'll try it anyway. Allow, allow, let's see if it does it. Oh yeah, that's me, all right. A bit weird. And really the key thing here is it takes a little bit to kind of figure out what you're doing, figure out how to configure things. If you know Pandas, you're in a great spot because all the operations you're doing basically the same, and we tried to align the language grouped by and split by to match with that. But once you do kind of figure out how it works and get the things, kind of understand the parts that you need to understand, it really puts the power back into your users' hands. They're able to, you don't have to provide them these crazy pictures like I had before where you have dashboards with tons and tons of different charts. You really only need your one grid with your flat data, and they're able to explore, dive in, construct the charts that they want to see on their own. So kind of taking the power back from the developers having to build everything for them into the end users, BI users, people who maybe are less comfortable jumping into code. Cool. That's all for the talk. It's all on GitHub. I have to make sure that it's linked from my profile. And all of the libraries I used here are open source.

Speaker 2 [22:02]

Thank you so much, Tim. Actually, we have a ton of questions. We'll try to get through them all. First question is, is it possible to put perspective together with Kafka and make real-time visualizations from there? Yeah, in fact, I would recommend

Speaker 1 [22:13]

Yeah, in fact, I would recommend that CSP library, which is a Python stream processing library, also has a Kafka adapter. So it makes it easier. I know there's things like Faust that basically let you do this, but CSP is kind of not reliant on Kafka, so you can use it with other message buses, but you could very trivially connect to Kafka, pull data into CSP, do all kinds of things, pump it into Perspective. It's straightforward. Actually, I think we have a talk at the Databricks conference, which is basically one of the upcoming conferences, which is like Kafka and Spark streaming and kind of these other external applications.

Speaker 2 [22:47]

Okay, so make sure to go to their conference then. Can it be integrated into Streamlit?

Speaker 1 [22:54]

There is there's a panel version. I think someone tried to do a streamlit version. It's just a web component So fundamentally, we're only limited by the fact that The maintainers are pretty busy And we have there might be a like community one that we have done in the past But if somebody is interested in doing that we're definitely willing to help out and it should be very straightforward It's a web component on the front end. There's a Python integration on the back end. It's it's not terribly complicated to do

Speaker 2 [23:23]

Okay, great. Do you have examples for streaming non-relational data, for example, graph data?

Speaker 1 [23:29]

No, probably not. I would look around at what's through here. We don't do pure graph data, it is relatively focused on time series data, just because coming from finance, and some categorical data. Even location data is a little wonky, but I don't think that's explicit, we don't want any of that data. It's mostly built around tabular data, it could be extended but we don't have any current plans to do so.

Speaker 2 [23:59]

Okay. Question, it says, I'm going to read this literally, for noobs, what is virtualization?

Speaker 1 [24:06]

Oh, yeah. So that's basically where instead of taking all your data from your server and sending it to your client, you send data to the client as it requests it.

Speaker 2 [24:06]

Oh, yeah.

Speaker 1 [24:13]

So imagine like scrolling in a grid. It's like infinite scroll on a social media application. Like it doesn't give you all of the stuff. As you start to scroll down, it gives you, you know, the information that is maybe one render away. And this is very useful because, you know, if you have big grids and things like that, you don't want to send all the data across necessarily. We do support that model as well. That's kind of client-server, pure client architecture. But yeah, basically just means as you scroll, diffs are being sent of the new data that needs to be in the kind of rendered frame.

Speaker 2 [24:44]

Okay. Can Perspective be used to do EDA on static data sets?

Speaker 1 [24:50]

Yeah, I think so. So one of the examples here is a file drop. So if you, the blocks shut down, so we kind of did our own blocks. But you can upload your own file here. I think it supports, it might support Parquet files, but I don't know. But CSV files, you know, the gold standard, you can upload one and start to kind of do slicing and dicing. So I'd say probably the majority of the use cases, just because static data tends to be so prevalent, do fall under the static data category. It's just, it was built with streaming data first. Doing static data after is pretty easy. And it also supports streaming. So it makes that bridge of like, you don't have to re-architect your application to go now have streaming data.

Speaker 2 [25:30]

Okay, great. Was reduced data analytics in Excel and removed ton of VBA one of the main initial motivations to develop this?

Speaker 1 [25:40]

to develop this? Yeah, so I think when large banks invested in Python and C++ and stuff like that, one of the goals was to move stuff. I mean, that's how Pandas came about. It wasn't a bank, it was a hedge fund. But the goal here, moving stuff out of Excel into something more performant. I think one of the things that was lost when that was done is that the end user was very fluent in Excel. The end user, I mean, they're getting better and better, but traders and stuff like that are not great at pandas and Python, but you wanna still allow the end user to investigate their own questions, as opposed to having the developer have to do everything for you. But that was one of the motivations.

Speaker 2 [26:22]

What is the maximum amount of data points for the map visualization in perspective?

Speaker 1 [26:26]

For the map, I don't know. So the framework can go pretty high. I don't have any exact numbers, but you start to get limited by basically bandwidth across. So if you're loading a gig of data from the server to the client, there are browser limitations around four gigs. But if you do client-server, you can obviously go way beyond that. It's just a matter of some of your aggregations. If you're doing an aggregation that has to look across all four gigs, 10 gigs of data, it's going to get a little slow. So, there are things in place to basically limit the total data size, and there are different kind of architectures. It's in the documentation of deciding whether you want to bring everything to the client. That'll obviously be slower to load, but then faster when the client can interact with it, or do a client server. So, there are some options. It's not a big data tool. Again, it's coming from finance, and we don't really have big data in markets, at least, but it should handle sufficiently sized data, I'll say.

Speaker 2 [27:21]

Okay, can Perspective be used in a standalone HTML? Yep.

Speaker 1 [27:26]

Yep. And actually you can embed it in an HTML. So you can, you know, if you bring your data with your HTML file, you can just have an HTML file that you send to somebody and then it's fully interactive. That would be like the pure client architecture.

Speaker 2 [27:37]

Okay, we're going to try to do one or two more questions. Is it possible to create UIs for end users, basically pre-configured charts that can't be edited?

Speaker 1 [27:45]

So all of the configuration of these things, I'll just pick on one magic the gathering. That's a good one It's it's probably gonna break because it's pretty heavy on the server and the Wi-Fi is not great So I picked a bad one But all of the configuration for these layouts is done via JSON This one also has a bunch of stuff on the server striking out with all of my examples So these types of things you can configure it's just a JSON plug-in state So the view configuration and the data configuration are separate or separately configurable, so you can have your data and then kind of apply a view configuration to it. A lot of the applications that we build with this too have a save load button so that you can configure a layout exactly how you want it, hit save, that's just a JSON, you can store it in browser cache, and then you can swap between layouts or you can host them on the server. So it's very easy to configure the kind of view.

Speaker 2 [28:37]

maybe one final question then we're out of time is it possible to slow down animations like getting 10 events per second but then just updating the plots every 30 seconds

Speaker 1 [28:47]

Yeah, so Perspective itself will just update as fast as you give it stuff.

Speaker 2 [28:47]

yes so

Speaker 1 [28:51]

But again, if you're in the Python space, that kind of is the bread and butter of a library like CSP, which is a stream processing library to do things like windowing and bucketing and gating. So I picked it for this demo just because it works very well for exactly those types of things you want to do.

Speaker 2 [29:08]

Okay. Well, we're out of time. Thank you very much. And if you have any questions...