Visualizing Interactive Graph Networks in Python

Speaker 1 [00:02]

The audience, that's good. Welcome, everyone. Welcome to Berlin again. Thanks so much for being here today. I mean, there's so many awesome talks happening in parallel. So I'm really happy that you decided to be here with me. Or maybe you wanted to go to the Pandas workshop, but you ended up here by accident because the rooms were messed up. Well, now you're stuck. um it's really cool to be among the first people to speak at this conference it's also a huge responsibility to have a good start and um i'm gonna talk about graphs today so i wanted to start off with a joke but for me that did network well three people got the joke that's that's fine for me that's all i'm asking for um one second seems like i got there we go all right but what is this actually about today like in short it like i want to tell you that analyzing graph data is really really cool and it's getting more and more important but once you have larger networks making sense out of them is getting really really hard so what we will learn today is how to visualize interactive graph networks uh in python and we will do it on an example and in fact we will analyze the skill sets of data scientists and how these skills uh like relate to each other in a larger network and the tech stack it's python it's network x traitlets ipy widgets and plotly so and this is basically how the result will look like you have like a little bit of hover you have like the possibility to adjust the layout, it recalculates stuff, you can filter some nodes, you can put them more nodes, then you can obviously remove edges, and you can at the end search and find nodes in your graph. This is basically what we will talk about today. So who am I and why am I talking about this? My name is Jan. I'm a data scientist at Instafo. and Staffo is the startup in Heidelberg and we deal with automating the recruiting process and we're trying to make it as fair as possible relating to the first talk today. We're not using supervised learning so we don't have the problem of introducing bias to our data which makes this entire thing much more ethically correct. And I'm also an organizer of Pi Data Südwest which includes Mannheim, Karlsruhe and Heidelberg and obviously I live in this very pretty city, Heidelberg. So if you have questions about my employer, the company I work for, please approach me personally or send me an email to my work address. If you have questions about Pi data in the region that I'm from, if you want to speak, attend, sponsor, anything, I mean, many of you might have cool talks. Just let me know either in person or through my personal email. So before we dive deep, just a little refresher on graph networks. So we're all on the same page. a graph obviously consists of nodes and edges and edges connect usually a pair of nodes nodes can be all kind of things they can be people computers documents anything really and they can have what i call arbitrary metadata so for example in this case you can see the node it's a user on facebook he's 25 years old and he has 240 friends and through this edge he's connected to a different person and as i said edges describe how nodes are connected can be you are friends with you are connected with or a scientific paper is cited by another scientific paper and what is important to know is that edges can be undirected or directed weighted or unweighted directed and undirected when two people are friends on facebook that is undirected because it is mutual when you follow somebody on twitter that is directed obviously because it's one-sided and you can say that every edge has the same weight so it's like a binary and edges present or it's absent or edges can have like their own weight you're like much more friends with this person than you are with this other person for example so why is interactive graph analysis importance and before i start please raise your hand if you have seen spiegel mining reverse engineering of spiegel online before nice so uh for those of you who haven't i really really recommend watching this talk it's by david kriesel he's a really cool investigative data scientist and he scraped spiegel the journal for many many years and he did all kinds of interesting analysis with the content that he received. And I will show you the result of his research in a second. But what I'm trying to tell you is that with small graphs, you can produce like a static image and it's fine and you can look at it and it's pretty. But the bigger the graph, the more information you as an analyst also need to process and the more difficult it gets. And from my personal point of view, once you have more than 100 or some 100 nodes and edges in your graph, it becomes almost unfeasible to really comprehend what's what's going on there's just information overload it's it's overwhelming so um what i'm talking about here for example is again david's talk what he did is he rendered a graph with tags that spiegel associate to their articles so a tag might be like finance or it might be refugees or something and they they he looked at how these tags appeared together in articles and he rendered a network and the network looked like this so you can you can barely see it but you can see some colored pixels every colored pixel is like a node he had like 7 000 nodes at the end and several thousand edges um it's impossible to analyze this picture as it is static graph analysis visual graph analysis is uh it makes no sense here so you need interactivity and to be fair david had some interactivity in his plot he had the ability to zoom in and zoom out which was key but i believe you can do more than that you can do more than just zooming in and zooming out especially when you have so much data and such a big graph present so what was my personal goal i wanted to develop a proof of concept for a self-service interactive analytics tool for graph networks that runs in jupiter because i like jupiter that is easy to build and maintain for a data scientist who probably doesn't necessarily have a like an academic training in software engineering uh processes medium-sized graphs reasonably fast like some hundred nodes and edges shouldn't be a problem and looks all right doesn't need to be super pretty but it needs to have some nice aesthetics and regarding the interactivity that i imagined at the beginning when i was doodling this on like a piece of paper i really wanted to zoom in and out obviously i wanted to get additional information when hovering over certain elements i wanted to be able to filter for nodes and edges i wanted to search for notes and i wanted to update the graphs layout on the fly which we will talk about in a second so as david's cool data set with the spiegel text is not available as far as i'm concerned i had to come up with my own data set and this is also when my company comes back into place what we will deal with today is skills that every data scientist should have. So basically, it's a data set composed by us. It contains the 250 most common skills for data scientists, their relative count in percentages of all users having that skill, and the similarity between each pair of skills. So 250 squared, basically. So if we're thinking about it, we're dealing with a weighted graph, where the weight of the edge is the similarity, but it's undirected because similarities are always mutual. And to give you a certain understanding of how this data looks like, I wanted to give you the least scientific, most silly way of presenting data, the word cloud. And you can see some cool stuff in there. Obviously, big data is very big and very often used. machine learning, AI, data science, data mining, Tableau, Hadoop, big data analytics, deep learning. There's many things on this graph. It ranges from data engineering to classical data mining to deep learning. You name it, it's basically all in there. And all kinds of cool Python frameworks are in there as well, as we will see. So I had the data. I had a goal. What I needed next was what I call the master plan. So I wanted to build an interactive network visualization and I wanted to put it on top of four pillars. Visuals, so basically what the user will see, the information that I will convey to the user. Then network algorithms, things like for example calculating the positions of individual nodes in a graph. User input, the ability for a user to specify filters, search boxes, et cetera, by themselves, and the underlying model logic. So if I press this button, this should happen, and then this should update, and then something should appear in the front end. So these were my four pillars. And then I looked for the corresponding packages, frameworks, to make this work. And to me, it was a rather easy choice. for visuals i chose plotly for the network algorithms i chose network x for the user input i used ipy widgets and for the underlying module logic and model logic i used traitlets and i will talk about all of these four packages briefly so let's start with plotly plotly is probably the one that is the most questionable in this set because there's many many many choices when it comes to visualizing stuff in python even visualizing interactive stuff in python to name one that is actually supported by numfocus that would be bookie or bokeh that is also a very very cool tool why did i use plotly first of all for those of you who don't know plotly is an interactive open source and browser-based graphing library for python and it also exists for r and javascript The main reason why I used it is because it had a new major release this year, the version 4, which arrived in July. And now it comes with a quite nice grammar of graphic-likes API, similar to ggplot in R, that works really well with Pandas DataFrames. And I was curious. I just wanted to try it out. So I decided to go for Plotly and not for any of the other existing tools. so if you want to see how it looks like you basically import plotly you have a data frame with certain columns and all you can do is you say i want to plot a bar chart on this data frame on on the x-axis i want that column and on the y-axis i want the other column and you get this so in three lines of code you get a rather interactive web-based visualization so very convenient network x again there was not many things to choose from it's a python package for the creation manipulation and study of the structure dynamics and functions of complex networks and it also has a new release this year the version 2.3 and basically what i needed this for is to calculate the layout the positioning of my notes because in my case my notes they didn't come with a natural position like an x and y coordinate that you might have when you work with geo data in my case i had to calculate the position the coordinate of every node based on how they are connected to other nodes and this is that's called a node or a graph layouting algorithm so again basically it's very simple only a few lines of code you initialize a graph you add a couple of nodes and a couple of edges and then you just run an existing layouting algorithm on top of the graph and what it returns is a dictionary ID of the node and then the X and the Y coordinates. So that's a very convenient format to use further down the road. Traitlets was something that I introduced I explored during my research for this project. It's a framework that It lets Python classes have attributes with type checking, dynamically calculated default values, and on-change callbacks. It's also in version 4, and the latest release was just this month, and it's part of the IPython universe. So what this means is you can have a class, and this class has a class attribute number, or num, which is an integer. you can use decorators to tell this class to validate every change that I attempt to put on this attribute and here for example I'm validating that the new value for num needs to be greater than zero otherwise it's throwing an error and I'm also telling you that every time it's changing it should print like a log message basically and these two methods I don't have to add any logic they are executed auto magically whenever this attribute num is trying to change so here you can see an example i'm like establishing an example object i'm setting num to one and automatically it's printing a message num changed from zero to one i didn't have to trigger this this is triggered completely automatically and when i'm trying to set it to negative one i automatically get an error message num must be positive so this is nice this way of dynamically triggering methods updating other attributes based on how things change make your logic in the back end much more much simpler and then the last component of my master plan ipy widgets it's an interactive html widget for jupyter notebooks and the ipython kernel it's already in version 7 so there has been quite a lot of major releases for this package. And again, it also has a very, very nice easy API. In this case, I'm establishing an integer slider with values between zero and 10, steps of one and the default value of seven. And once I execute this in a Jupyter notebook, I immediately get a very nice HTML widget that I can drag and play with very, very, very easily. So these were all the four components that i needed and i started working on it i was like okay i have my data i have my master plan let's start working on it and at some point i looked at my code i was like oh my god this is really really awful code what what is going on like i had this one monolith class with like uh the the the front end defined and the back end logic and the network algorithms and everything was there and i was like this is a mess i cannot show this in front of people i was getting a little nervous i was like i need to i need to do some research and i had to confront myself with possibly my worst fear most data scientists worst nightmare software architecture design patterns and i was like how do you actually write software and uh one thing i came across in ipy widgets tutorial was um how to build these kind of dashboards using model view controller which was something i hadn't used since my bachelor and so the more complicated master plan looked a little bit like this i had different views i had a view for the figure which is the actual graph i had a view for how people can like define layouts i had a view for how people can filter notes and edges i had a view for how people can select and find individual nodes. On the other side, I had models. Models that took certain inputs, calculated certain things on the fly using traitlets. And then I basically had a controller that linked everything together. And I wanted to make it as nice as possible, but still I realized, I mean, you see all those red arrows that it's not only going like nicely horizontally left and right sometimes i had to like use like a shortcut where it goes right down into another model and then trigger something else again data scientists are usually not trained software engineers sometimes that's a problem but the result was a little bit clunky but working nicely separated modular implementation where for example you could test all the features using the model classes without having to touch or change the views or anything. So it's a nice separation. And now we go to the live demo. What could possibly go wrong with a live demo? So I just want to show you the code. Basically, in my case, it's very simple. I have... Can you guys see this well? Otherwise, I'll zoom in a little bit. I have a class which I call Network Analyzer. Before that, I import nodes and edges that I have prepared I can show them to you again. So the nodes look a little bit like this. You have an ID, advanced analytics, you have a size, again it's a relative count which means that the largest node has a size of 1 and all others have a value between 0 and 1. So I can again like for example say sort values sending equals false and you can see uh oh it's it's not a serious never mind there we go here you can see the the biggest notes let's maybe put there in the top 10 i mean you've all have a good understanding because of the word cloud of course but here's some more information so you can see big data machine learning art data analysis sas artificial intelligence it's all there and now maybe let's look at the edges quickly that's a great question uh let's let's uh move questions to the q a but i will definitely answer that one um edges basically has a very similar layout so i can say sort values but by weights and ascending equals false. So for example here mudplotlib and seaborne pytorch, tensorflow, theano, lasagna has a weight of 1. Just a heads up, we have a normalization script that does reasonably well normalizing skills, but some people are so crazy when it comes to stating their skills that it's really difficult for us to normalize everything. So we have some funny looking skills here. But the majority looks pretty alright. And for example, the similarity between PyTorch and Qlik Sense was the lowest in the entire data set. So that, I believe, makes perfect sense. But now we have to zoom out a little bit so you can see the entire graph. That's basically how it looks like. You have the network here with a couple of nodes and edges. You can hover over the nodes. You get the coordinates and the size. You have a nice color bar that also tells you a little bit more. but now for example i want to use more notes so i hover i go into the filter tab here and i say i want to have all notes and i hope i don't break my system now because it's not as efficient as i think it would be and it's calculating all the there we go you see it's fast as hell and that's how it looks like when i present all the notes that are bigger than one percent of the relative count but now i have too many edges so i say well yeah let's do it like this that looks better and i could technically do things like adjusting the layout i could use different kinds of layouting algorithms when i choose something like spectral like you see it adjusts very very quickly and but now let's go a little bit deeper into the I will focus on only a few edges. And one nice thing to see is that here, for example, you have Keras, Neural Networks, Panda, Scikit-learn. The entire Python frameworks are very, really close to each other. They're building a little island above here, TensorFlow. It's connected by everything through machine learning, which is the bigger node, which is spreading in all directions. If we go from the left, we see big data analytics, data visualization, big data analysis, data analytics, so the more pragmatic stuff in data science. Down here, we have Tableau, we have Qlik, we have ETL, data integration, et cetera, Power BI, SaaS, and data warehousing. And here on the right, we have AI and Kunstlichen Zelligens. The main insight that I wanted to share with you from this graph is that contingency ligands is possibly the furthest away from deep learning and neural networks, which is because everybody's using it, but it doesn't have strong patterns with the actual skills that it takes to work in artificial intelligence. It's actually similar to big data that is also not in the ETL data warehousing area. And yeah, the last thing I can show you is I can do things like this, and I can say, where is the AI node? And I get navigated there right away. So this is something you can build very quickly. It was a little bit of a hassle at the beginning, but now it's in a nice structure and easily maintainable. So let's wrap it up. What is the outlook of this? What I don't like yet is that when in Plotly, you zoom in into a scatter plot, the scatter doesn't increase relative to the zoom. So they stay the same, which is mind-blowing to me. Why would this be the default behavior? So you could avoid this using like circle layouts and stuff, but it would involve some more implementation. Right now, my logic for rendering the figure is very dumb. Every time I make a change, it discards all traces, and then it has to re-render everything. Obviously, it would be cleverer only to render the delta, so the new edges or remove the old edges, etc., which would make this entire visualization much faster and more responsive. the node information should be updated you might want to see there's some other stuff than the xy coordinates and edges should also display information right now they're just dumb lines that you cannot hover over or anything that would be cool if they could highlight the nodes that they are connecting and stuff like that I have a couple of ideas and while building this I was introduced to panel which is another framework for building dashboards there's also dash from plotly so there's also other tools to consider for building the entire frame, rather than traitlets and ipyewidgets. Maybe I will do an evaluation of that. Some good resources for you. There's really cool tutorials by Plotly on how to visualize network graphs and how to use widgets, but not together. This is why I'm using this, or I'm doing this. And there's a really nice three-hour workshop on Jupyter widgets by Sylvain Conley, one of the main contributors to Jupyter. This is where I found this part with a model view controller in notebook 903 that's it my time is over i think i made it just in time thank you so much for being here i hope this was somehow insightful for you i'm up for some questions and thank you very much yeah we'll just repeat it for the recording so why was python not mentioned there i was i was so surprised when i just couldn't find python among the 250 most similar skills to data science one thing is that the data set that we're using it is not from 2019 it's a little older um it's not from 2005 or something but it's like from 2016 python was already well established back then but that's why for example r is still extremely prevalent in the data set and also we noticed that python in our data set has very strong ties to a more classical software engineering so to things like Django it has strong ties to C++ it has strong ties to Java and it doesn't but the funny thing is at the end the Python frameworks are still displayed so Python seems also to be a very generic skill that people put on their profile all the time it's like when people put Java on their profile and you know they're lying because they haven't used Java in 15 years but they still put it there Python is similar to that so that's why the patterns that we can observe are not as strong as for other skills so the question is what's the bottleneck calculating the layout or visualizing the the traces so the the graphical components of the graph from what i could see on on this network and you have to consider it's 250 nodes but it's fully connected so every node has a connection to every other node and using things like a spring layout or a kamada kawaii layout on reasonable parameters goes very quickly unless you exaggerate with the number of iterations it goes very quickly which is very very nice the problem right now is Plotly, because it's a little slow with calculating the traces. So I think the biggest improvement in terms of speed would be only calculating the new traces rather than recalculating everything. But then again, this accounts only for this particular notebook graph. And I would have to see how this scales. For example, if you have 1000 nodes or 100,000 edges, probably then calculating the network becomes exponentially or quadratically more expensive. More common than a question, when you want to define every aspect of an organization, because globally it's very quick, but if you want to have full control, you would have to go a little low level. But then of course it's a purely JavaScript library, and then you have to use a different language. But still, in that case, you would then have full control, for example, as you just mentioned, the width of all the connections of the edges and already behavior. So the comment was that if you want to go full control, you should use D3, which is JavaScript only. And you're absolutely right. As a data scientist, I give it my best to avoid any kind of JavaScript and rather find hacky solutions that only involve Python. For example, for the width of the edges, that's something Plotly cannot do with the Plotly Express. But you can hack it a little bit by just calculating individual traces for individual edges. Because then again, you cannot assign an attribute to the edge width, but you can just loop through all edges and just set the edge width statically. It works quite well, but then again, it's slow. But for such thing, I would probably team up with a front-end expert that could help me with this. But yeah, definitely great comment. I have some kinds of questions, and we have more questions around the next few days, so that's not it. Yes, thank you very much. Thank you.