Interactive Visualization of Traffic Data using Bokeh

Speaker 1 [00:02]

So welcome to my talk. Thank you for your interest. Again, I'm talking about interactive visualization of traffic data using Python. Just a few notes about me. So my education, I'm a physicist. I graduated in theoretical physics of superconductors. And now I'm working as a traffic data analyst and transport model at the PTV group. The PTV Group is a leading company that provides software and solutions for traffic planning and logistics. So we have about 800 employees. We have 2,500 cities that use our solutions. And we were, by the way, founded about 40 years ago as a spin-off of the University of Karlsruhe, which is now the KIT. Furthermore, we have our headquarters here in Karlsruhe. But we have company locations all over the continent, and our slogan, as we see here, I think they summarize pretty much what we do. We plan and optimize everything which moves and people and goods worldwide, integrated and in real time. So, about me. And kind of the part what we want to see here is how to create something like this. So, interactive maps where you can zoom and pan and see the data. And in this case, it's just the locations of the PTV group. So you see it's all over the world, except in Antarctica. But in a lot of continents, we can zoom in here and look here. So that's the head border here in Germany. See, it's in the eastern part, and we are probably about here. So I can show you how we can process traffic data and show them on such a web map interactively that you can also give to your friends, clients, and colleagues. Okay, about this talk, I'm going to present basically Python packages that help you with the data processing of traffic data and with the visualization. So it's not going to be about network modeling or anything related to that. So I would recommend there, if you go to the SkitHub page, There's a nice library called Network X that serves for that purpose It's also going to be very interactive. So I'm going to show you a lot of code. It's interactive So there's nothing that can go wrong actually, I think Good just short about traffic modeling as you can see here There are kind of two parts of traffic modeling. There's the demand part So basically from where to where do people go or want to go and there's the second part which is network So network tells you all the streets, all the links, and basically the possible routes that a person can take to go from one position to the other. And if you combine them together, you get an assigned traffic model, as you can see here for Karlsruhe, and the widths of these blue lines tell you how many people will drive there. So you see this main road, like the central road, that's the main road where the people are going to drive, for example. What we're going to focus here in this talk is the demand part. However, you can also, using the tools that I show you here, visualize and show the network part very easily. Okay, a short thing about GIS objects. So there are three basic GIS objects. First, a point, which is, for example, a location, as we have seen before in this map. Then a line string. That's a line, so, for example, a street. And there are polygons, and this would, for example, correspond to an area, for example, an area of a building or a block. And there are this multipoint, multiline string, and multipolygon objects, which are just a collection of these objects. There are a few Python libraries that are nice for working with this data, called Shapely, Fiona, Folium, PyProjection, and so on. I'm not going to talk about these. Instead, I'm going to use GeoPandas, which is kind of a high-level wrapper that uses a lot of these libraries to make data analysis of geographical data easy. Just as a note, there's a talk this afternoon, Processing Geodata Using Python, where they're going to show a lot of these libraries that I just mentioned here. Okay, main part of the talk, I'm going to look at the New York taxi dataset that probably a lot of you already know. So it's the dataset of 2017. We could also do any other dataset that we want, but let me first show you what we have here. So I hope you can all see that. I can make it a bit bigger. Maybe I think that's better. The new taxi dataset consists of two things. First, the taxi zones that contain the geometry information. So we have like zones in Manhattan Queens and so on that tell you this is Texas on one two and so on and you can See here. It has an object ID. So the zone number it has a name Newark Airport here Jamaica Bay and a polygon and this polygon. These are the coordinates, right? Sorry the X&Y coordinates of of, as a list basically telling this is the shape of the polygon. Okay, on the other side, you have the origin destination data. So that's the geometric part, that's the destination data that contains the information at which time is a passenger of a taxi picked up and when it is dropped off. Right, in which occasion, also in which occasion it's picked up and dropped off. So the information from where to where, at which time do the passengers go. So we need, I am using two tools to process this data. This one has a problem, it's geographical data, so we need to process it with a geometric data analysis tool and this one is big, so this is about 11 gigabytes and this is already maybe too large for pandas, so we're going to use the Dask library just to show you. Okay, first let me start with the taxi zones. so we are going to use the GeoPandas library that's basically Pandas plus a geometry column, so a column that contains information about the specific GIS object, and it has some nice additional methods, for example, intersections, differences, spatial joins, and also some basic geometric plotting, so it's really Pandas plus geometry. Okay, I have to go down here. Just to show you an example, it's really easy to read in, for example, shapefiles. So shapefiles is one of the base formats or the standard formats for sharing geometric objects or geographic objects. You can see here I'm just importing GeoPandas as GPD. Can everyone read that? Okay, nice. And then instead of PD we use GPD and then we can use the read file method to directly read in the shapefile, really easy. We can then with head just display the first three objects and as you can see it's a data frame which has an additional geometry column containing the polygon, so the geometry of that object of the zone. There's also some basic plotting capabilities, so it's basically this one line, so it's like in pandas, you can say this taxisomes data frame plot. We also specify here a column and a C map, and as you can see, what this does is just it plots this data, so you can really see here, we see Manhattan, I think this is Queens and Harlem and I don't know what these boroughs are but here I can directly tell them hey color the different boroughs with this color map so really easy one other nice thing that you can do directly here is you can convert your geo data to different coordinate systems right there's a lot of coordinate systems I don't want to talk about them here but the usually they're not in a Certain projection that you need so here you can just use the to CRS function you have to pass the e PSG code So that's a standardized code a number for this projection, and you can then Directly convert this polygon as you can see here here with a number that's up with minus 8 and here It's 9 so it's just converting that to a different projection directly what I did here is I'm Converting that to a web Mercator projection because that's the projection that is used for all the plotting libraries and the web So all these webmaps, they're based on the WebMaker.projection. Okay, what you can also do, and what I always would recommend, or mostly, is you can simplify the shape of zones. Shape files or such zones can be very detailed. So a lot of little, which you see, I don't know if you can see that, but here there are like really little features. You don't really need them normally for just plotting something. and with this simplify method that you can see here you can simplify these polygons directly so you can maybe you can see it here that they're simplified and you don't see the structure here this increases the performance of your plots right if you're plotting too many points too many polygons it's just hard for the browser and it's starting to freeze okay um and this next part i'm just showing it here but what this does it um takes this geometry column over here, this polygon, and basically gets rid of it and just writes the information about the x and y coordinates into Python lists. As you can see here, right, there's the x and there's the y coordinates. And it also splits and separates this individual, this multi-polygons like here, so this one zone is a multi-polygon, it's a lot of islands, and I'm going to split these into individual zones, right? Always the same location, because we're going to plot them all individually okay that was the geometric part now we're going to look at demand data so again as i told you it's there's a lot of people taking the cap in the in new york i think it's about 500 000 people a day so there's a lot of data it's a csv format that you can download on the web and what we're going to use here is dusk which is a python library that basically parallelizes your numpy pandas i could learn libraries so you can use all your cores you can use a lot of you can use all the ram and all the stuff it's more than that it's also something that helps you parallelize all your python code but i'm not going to talk about that A cool thing, it works on your local machine, so you can set up a local cluster to use that, or you can use remote clusters, so you can calculate everything in the cloud. It also uses lazy evaluation, so it only keeps the data in your memory if it's needed, and it throws it away when it recognizes, I don't need it anymore. Just to show you what we do now, so we have this table, and you can see we have to pick up and drop off locations, so from here to here, we have the date times and what we want to do, we want to have this big table of about 11 gigabytes of CSV, we want to group by this, by the location ID, the weekday and the hour. Because that's what we want to plot. We want to see in which zone and which weekdays on Monday, Tuesday and so on, and in which hour, how many people do, are picked up here and are dropped off here in this zone. So this is the table that we want to generate. It's basically a simple group by. So what you should do with Dask, you first should start a local cluster. So that's this part of the code. You have to import it and start your local cluster. And what it does here, it basically instantiates a lot of Python, instances parallel and looks at the communication between them to compute your results. And as a next step, we are going to process this taxi data, which is done with this part. So it's basically Pandas code. So instead of import Pandas SPD, you say import Dask DataFrame STD. And you can then basically have the same APIs as with Pandas, so you don't really have to care about most of the stuff that Dask does. It just does automatically everything for you most of the time. So here you can see we read the data in. we apply some daytime operations so we um parse the dates we filter something and in the end we're going to group by everything that's what we do so if we do all that you see in 400 millisecond it obviously processed 11 gigabytes of data or not no it didn't because this just defined the graph so it's lazily computed this just defined okay i'm going to do this and this and this and and this and this on my data and then just said okay I'm waiting till you say I should compute it so that's what happens here you have to basically say I'm going to compute now this pickup and drop offs table this group wise and if I do it yeah it starts and with this cluster you can see here there's a little there's a link to a dashboard which opens up in your browser and you can now see here that all my worker cores, I have eight cores on my machine, so you can see that all the cores are now busy, mostly reading in, applying daytime operations and stuff like that, so you can really see what happens, and you can also see that in the graph, as you can see here. Okay, so you can see here at which step you are. Okay. I'm going to stop that computation now and I'm just going to read the data in because it takes about two minutes and as a next step we're going to use Bokeh which is a really nice library for building interactive web applications, web interfaces just using Python. Just as a simple example, you can see here, this is all you need, basically a bit like Matplotlib. It looks a bit like Matplotlib from the API. You define a figure, you define your data, and then plot the line. You can add hover tools, all stuff like that. And if you execute that code, you see you get a parabola. It has a hover tool here, which tells you about data. And you can also zoom and drag around in this plot. Really nice. All interactive. Okay, now as a next step, we have our data that we read in and processed. And we now have to combine these two. So we have here the geometry information, and here we have the information about the demand. And what we do here is we now join these two information. So join information for every hour, every weekday. And if it's pickup or drop-off, we are adding this as columns to our zone information, our geometry columns. So you can see here, we have our location, our zone 1, with that X and Y coordinates. And we have, I think, 336 columns here, really a lot, you can see, that are named like this. So these are the number of passengers that are picked up on a Monday, so Cirrus Monday, at 0 a.m. to 1 a.m. in the zone 1. So an average 0.13, right? And this goes all the way to a Sunday at 11 p.m. So as a next step, we want to plot the data. Now we have the data frame with all the data, and we want to plot that. So that's actually not that hard. You have to do some imports. Here what I have to do, I have to define a column data source which basically translates our data frame that we just had to something that this Bokeh library or this JavaScript understands, right? It just converts this basically to a dictionary with all the columns containing within. And we then define a column mapper. And in the end we're defining our figure. This is again just a figure. We have a title, some plots, widths, heights. That's all not too important, but what is cool here is you can directly add a map in the background. So, just using a simple API, basically, either you provide your own tile server, or you use already built-in tile servers that are defined here. You can just be this figure.addTitle. You can add a background tile. And now plotting the data is as easy as setting p.patches. p.patches, what you do is you give it your source. So that's basically your data frame, right? The data frame, that's the data that you're going to plot. The x and y coordinates of the polygons, these are the columns named x and y, right? So you refer to the columns in your data now with this API, and you define your fill color. you say that's the passengers column and in the end what you do is you say show P and then it's plotting that you see we have a map with a background we can zoom in we can drag and you see as it's expected that in Manhattan there's the highest demand right there's a lot of demand here and these two these are actually the two Newark and John F Kennedy airports so that's where the most of the people are picked up, right? Okay, there's another nice thing, just easy with this hover tool, you can very easily add a hover tool, so now when you hover over the individual zones, you can see how many passengers are picked up, right? Very easy. Now to the kind of important part, I think, is you can add interactivity, very, very easy here. So there are a lot of widgets that are provided with Bokeh, for example, a slider widget, as you can see here, and you can lay them out very easily with this layout object, and here you see what I defined is a slider that has an hour from zero to 23. We have our weekdays, Monday to Sunday, and we say, do we want to see the pickups or the drop-offs? And the interaction can, in Bokeh either, you say, when I click that, call a Python function, But that can only be there, you can only do that if you have a Python server running in the background like I do here. If you want to give that to someone else as an aesthetic HTML, so that he doesn't just need his browser, then you have to use JavaScript callbacks. But they are also not hard to do. I'm really not a JavaScript guy, but this code, basically everyone should understand that. So you see, I say here, okay, when I change the slider, make this callback function that I defined here exactly the same with these two other widgets. And what you do, you pass basically all your Python objects here in this args as a dictionary, and this translates this Python object to JavaScript objects that you can interact with to browse them. And then what the script does, it's taking the hour from the slider value, it's taking the number of the weekday and it's checking if it's pick up or drop off, what these values are. And in the end, what you just have to change, this source, that's the data frame that we passed, we're changing the data of the passengers column, that's the column that defines the colors that are plotted on the individual zones, by this one column that corresponds to this selection. so hour 13, Wednesday, and drop-off. That would be the column drop-off DO to 13, for example. And then it's called source change emit to update the plot. So that's everything. Long story short, what you have here now, you have your plot, and what you see is I can hover over it still like before, but if I change the slider values, you see that the colors are updated. So you see the demand dynamically. And there's some nice features that you can see, for example, that the demand shifts a bit to the south in the evening because there are more restaurants there. You can see that on a Monday at 11 p.m., there's way less traffic than on a Saturday, as you would expect. Stuff like that. You can change your column mapper to logarithmic to see more structure there. And, yeah, that's basically everything that you can. There's a lot of more stuff that you can do here, but this just shows how to provide basic interactive capabilities and you can just pass that to someone and he can then analyze that and finds his own conclusions about the data right okay another cool thing is basically with just saying it's giving this comment output file right output file and you can specify a file name and at html file you can give a new title with this comment then this all this part here that's just again copying the plot that we did before and then we show show this plot what happens is now it's not showing that in a notebook it's now showing that here as an it's exported that to to a file on my that i defined and then it opened it here so this is just some setting html file that you can pass to everyone that you want you can just pass it to and you can see it on his iphone on his ipad or any modern browser don't Not an internet explorer, but everything else probably. Right, a real browser. And here you see it's still interactive because it's all JavaScript in the background, right? Also the function that is defined to add interactivity, it's JavaScript. So I think this is the really amazing part also about this library. Not only that it's interactive with Python, but you can also export the interactivity to JavaScript directly, to HTML. Okay. Last thing that I want to talk about is data shader. Data shader is an amazing library to visualizing huge amounts of data, which I mean millions and billions of data points. There's no plotting library that can, no normal plotting library that you know of where you can draw maybe 100 million points, no way. So the idea here is that it's, instead of making this, like, this is one million normal distributed points. You don't see anything, you just see a circle. It's making a grid, and then counting the points in this individual grid cells. And then colors it, depending on how many points there are. And then, as you can see here, you can see that it's a Gaussian distribution, or some distribution here. Instead of here, you don't see anything, here you can see something. And I don't want to go through that code, because I cannot give you the data, but there's a other example that I can show you later, I can give you on the homepage. What I have here is some data, and the data provider is INRIX, by the way, it's a data provider that provides you with GPS data. And what you see here is that for, we defined, I think, 16 tunnels or outpasses, and it says They give us all the data that you have for this for two weeks that went through this pass right and you can see here So here the gotha tunnel for example These are all the trips that go through the gotha tunnel and these are all the trip that go through the Prana That come here right so you see you got a tunnel more the southwest and the Prana is more used by people from Munich and The region right which makes sense These are 70 million data points processed in real time always updated So when I zoom in here, right, you can see it's re-aggregating and you can even see the individual GPS points that are recorded. Or you can zoom out and see all 70 million data points aggregated on a level interactively, right? Here you need Python in the background, by the way, but still it's an amazing library for, for example, viewing 100 million data points and having a feeling about how the data is structured, especially on a map. traffic data Okay, thank you for your attention There's a most of the material that I showed you here is on the is on this github page there's a notebook and all the data you can directly download and Yeah, I want to thank you for your attention and if you have any questions, please ask Thank you very much for this interesting talk right now it's time for questions

Speaker 2 [26:18]

Hi. Yeah, Boki and DataShader is very impressive. What I've seen, I mean, you worked exclusively, I think, in the 2-byter notebook, which is, if you give this out to end customers or users, let's say, maybe not the best way because the feedback is always that they might accidentally temper the code. um so there's also the opportunity the possibility to host this have a pokey server yeah and just stream this as a web page uh but the data shader option there or support there is not that mature do you have any uh experience with that

Speaker 1 [27:00]

No, I'm more from a data analyst, data scientist point of view, so I'm not really using clients, but I'm more into getting data and going to analyze it by myself or by some colleagues of mine.

Speaker 2 [27:01]

No.

Speaker 1 [27:13]

And for this, for example, this HTML export is amazing because I can just pass that and you can open a browser and look at all the data that I aggregated for them. But yes, the data shader part, I normally show them on my computer, yes. I don't know anything about the data shader support for Bokeh Server. I use SpokeServer once in a while, but I've never used DataShader with a backend. Yes, I'm not from this deployment side, I have to say, yes. But I know there's opportunity, and you can see some web pages that do that.

Speaker 3 [27:44]

Okay, thanks for the talk. That was very interesting. Can you recommend data sources where you could get shapes like for Europe or Germany, for example, if you want to visualize something by zip codes or like by areas? Because you always see these international examples, but it's hard to find data that you could use with German data.

Speaker 1 [28:08]

I cannot really tell you. I think the shapes of German Gemeinden, right, or Kreise, they should be available freely, I guess, by the Statistische Bundesamt or something like this. If you want, you can send me an email. You should be able to find me there or send me a message via GitHub, and then I can tell you more about it because normally my colleague is more about this stuff. He knows everything about the data sources and where to get the data from. I'm more using the data and he's more on the side of gathering together the data and defining where to get the data. But a lot of the data is actually commercial, so the last one is a commercial data source. That's why I cannot give it to you. There's OpenStreetMap. There's OpenStreetMap, of course. There's OpenStreetMap, yes. I cannot tell more about it. Yes, there's OpenStreetMap. You can also use that, right? You can also use OpenStreetMap as the background layer, for example. If you're just thinking about layers, if you want to see the plots, as I showed you, it's like this. If you want to really interact with the geometries, you need the data itself, not just the tile and the background. But OpenStreetMap, I know it's a valid data source. I don't know if it has information about the kreise and stuff like that. I don't know how recent they are. So normally commercial data is in some parts better, not always. But for example, on the street level, commercial data usually is better. I just know that. But if you look at where, for example, the next truck stores and so on, OpenSphere has an amazing quality. I don't know how many people are involved in that, but it's just an amazing project, I think, yes. yeah let's thank patrick again and enjoy your lunch