Practical graph neural networks in Python with TensorFlow and Spektral

Speaker 1 [00:04]

Because we are a little bit late, so I'll start Let's start with a quick agenda We'll start with an introduction Then we'll talk very briefly what you can get out of this workshop. So, you know in advance Then we'll prepare the environment and we'll have one proposal for you, but you can also do it in a different way Then we'll talk about GNS which stands for graph neural networks Then we'll briefly talk about types of GNS, and we'll only discuss a couple of them. Let's say a couple of classic types of those architectures. Then we'll talk about Spectral, the main library that we'll be using today. And we'll start with seeing where this library fits in a broader graph neural networks landscape in Python. And then we'll talk about basic components of Spectral, which will be layers, data sets, and data loaders. And then finally, we'll do some practice. And we'll have a couple of practical exercises that hopefully will be valuable for you and maybe reusable for you in your own projects. And at the end, we'll have 10 minutes of questions, Q&A, with you and with the online audience as well. You can prepare your QR codes reader because there will be some interactive things that you can access using QR codes. If you don't have one, there will be links as well on the presentation. OK, five sentences about myself. My name is Alexander Molak. I'm a machine learning engineer and researcher at Iron Scales. I'm also a machine learning researcher at TensorFlow. This is an independent research initiative where we are mainly focusing on traffic prediction and traffic optimization. My main expertise is natural language processing. I'm also interested in doing a lot of stuff in probabilistic modeling and causal inference, which is closely related to graph neural networks, because it also leverages graph structures heavily. I'm an author of Sunday AI papers, and usually I'm saying that it's a weekly LinkedIn-based microblog. But last week's were very, very busy for me, so now it's not weekly anymore. It was not weekly. It will be weekly again, or biweekly. I'm also interested in complex systems psychology and neuroscience, and this is part of my academic background. And I love traveling with my wife, running. I enjoy vegan food and languages, both natural and computer ones. OK, what can you get from this workshop? There are a couple of things. First of all, you will get some theory behind GNNs. But as I say, practical workshop will mainly focus on practice. We'll just do theory in a bird's eye view, let's say. Second, you should get some practical knowledge of how to implement GNNs using Spectral and TensorFlow slash Keras. We'll mainly focus on Keras interfaces because they are simpler. But everything that we will do today can be also done with more advanced lower-level TensorFlow training loops. So if for whatever reason you need this, you will be able to use those lower-level interfaces as well. And at the end, we'll also devote some time to preparing a custom data set in Spectral. So this is something that is pretty important in the whole graph neural networks workflow, and which is different than anything we know from classical, let's call it, machine learning. So we'll do a short exercise on this. And I will share some actual code that we are using at TensorFlow with you so we can see how you can build those data set objects. OK, so let's jump to the environment. This is the repository for today. On the top, you have a link. Here we have a QR code. You can use it as well. And my recommendation is that you use Colab. So if you open Colab and you type this link on the welcome screen, you should be able to open the notebook. You can also go to the repo. You can clone it, and you can build the environment locally. There's an environment file there. But I do not recommend this for the workshop, because there might be some complications depending on your system. And we'll also have a little bit of a speedup if we use GPU. So this is another thing worth mentioning. When you're in Colab, when the notebook is already open, you can go and change your acceleration type. So this is the direct link to the notebook. And now, if you want to see how to do it, so you have Colab. And then you click this thing here, GitHub, and then you should put this link here. And when you click enter, then you should have a notebook, PiData Berlin 2022, 04 PiData Berlin, Wi-Fi. And then you click this thing, and this should open the code for you in the new window. OK, so let's jump further. So now, we'll forget about the notebook. Of course, leave it open, please, because we'll get back to it. But now, let's talk about GNNs. So what are GNNs? Does anybody has an idea? It doesn't have to be a good definition. It's just about your ideas. We say graph neural networks. How are they different from regular neural networks? So Alexander said that graph neural networks operate on the data that is structured as a graph, and that they are trying to build a representation of this graph. And this is a very good answer. Thank you for this. So let's see. So first, they operate on graphs. Maybe you also have the same name and the same presentation in your computer. No. Not the green cell. Yeah. Another important thing is that they need to be permutation invariant. What does it mean? It means that the nodes in the graph, if you take a graph, you can present this graph in many different ways. So you can maybe say like A, B, and C. And you can rotate it. Or in 3D, you can rotate it this way, this way. And it's still the same graph. You can think about proteins in the human body. They might have different positions. And we might look at them from different perspectives. But still, we want these proteins to be the same protein independently of how we look at it. So you can think about nodes in a graph as elements of a set. So they are not ordered. The only thing that matters are edges between those nodes, so connections. So we don't have an idea of where a given node is, about the location of this node. But we have an idea of a connection of this node. And as Alexander said, we will try to build a representation of a graph that will abstract those connections, those local connections of those nodes, and in a sense, map them to some space where we will have a concept of position. And this is very related to what we just said. They need to provide a meaningful way to represent the data. Now, if you think about neural networks, I will give you something that I think is very closely related on the conceptual level. If you think about word embeddings, for instance. So what word embedding algorithms are doing, like Word2Vec, or maybe transformer embedding layers. They're taking words that are categorical entities, and they are mapping them into some space, some continuous space. And they map them in such a way that words that are similar in the meaning are close to each other in this space. So if we compute something like maybe cosine similarity, cosine distance, or angular distance, or maybe Euclidean distance between those points, if those two points representing similar words, like maybe dog and puppy, they should be relatively not far away from each other. While words that have meaning that is very different, like maybe a rocket scientist, should be somewhere further apart, or maybe, I don't know, a truck. So here, we are doing something similar. But we are not using words. All the words or sentences could be seen as a graph, which is like a chain graph. But we are trying to map nodes that are similar to points that are not far away from each other, and nodes that are dissimilar to make them further apart from each other. And when we are building this data representation, we can use three different types of information. So it's information about the node itself. So those nodes, at minimum, they need to have something that identifies them uniquely, maybe a one-hot encoded vector with one in one position and zeros everywhere else. Or they might have some features. So they might, at minimum, have one feature. And we want to use this information about node itself. We want to use information about nodes' neighbors in terms of their maybe identifiers, maybe of their feature vectors. And we want to use the information about the local structure. And local structure is defined by the neighborhood. So how a node is connected to other nodes, or with how many nodes it is connected. This is a picture from this paper, which is called Semi-supervised Classification with Graph Convolutional Networks by Thomas Kipf and Max Welling. And this is a paper from 2017, so it's a couple of years ago already. But this is one of a very, very classical graph neural networks architectures that they proposed. And what they are doing here, basically, is that they are taking some graphs. So those black things are edges between the nodes. Those circles are edges, obviously. And now we are taking all the information that we have in these nodes. And we put them through a couple of hidden layers, one or maybe a couple of hidden layers, and we build some other representations. Now, what we can do with those representations, we can, for instance, use them to classify the nodes. But we can also use them to classify the graph, the whole graph. We can also use them for any regression task and any other task that you can think of with classical machine learning. So you can think about graph neural networks as a type of a special encoder. So we need some special encoder to deal with graph structure. But then, when we are in this representation layer, we can use this representation layer in any way that we could use any other representation obtained in any other, using any other encoder, deep encoder. And I want to show you, I want to share with you one example, which comes also from the same paper from Kipton Welling. This is a data set that is called Karate Club Network. It's a very simple, as you can see, very simple, relatively small graph data set. And this is how you embed it. When you embed it, so all of each of these nodes, they have some features. And then you can embed them in this two-dimensional space. And this is what you get when you just embed them just like this. Oh, sorry. I think they even don't have features. This is just a random weighting. And then when we use a graph neural networks, in particular in this case GCN, a graph convolutional network, you can see that after 25 iterations, those points are starting to group together given their class. So their classes are marked by their colors. After 50, it's even more ordered. then 75, 100, and finally 300. And you can see that all the blue nodes are here, green are here, and those purple are here. Now, those edges that are here are not anymore in those representations. They are in the picture just to show you how those nodes were connected in the original graph. We have a question. We have two questions. OK, but you are first. Where do the colors come from in the first place? The colors come from the classes. So in this data set, you have like four classes, not three, but four. Sorry, I missed one. And these nodes are colored according to the classes. So each node was, does it answer your question? Yes. OK, great. And you have a question. Sorry, what's your name? Thomas. Thomas. Very nice to meet you. I'm Jacob. Jacob. But the random weights are all similar to one class? So how are they classified by the weights? No, so I don't want to confuse you now. So let's forget about this random weights. OK? They are classified because of something, right? In the original data set. Now, we apply a graph neural network to them. And based on the connections of those nodes, and maybe some features. I'm sorry, I don't remember if this data set has node features or not. Maybe it has. Then we build a representation for each node. And now, how we do it, we'll show this in a second. But basically, we are trying to leverage these three types of information that we mentioned on one of the previous slides. So information about node itself, if it has a feature vector, this information is preserved. Then information about its neighbors. And then the information about the local structure. So how this particular node and this particular node will be each of those nodes at some point. Does this answer your question? Yes? OK, great. Thank you. OK, so let's talk about graph representation. We'll talk about nodes, edges, and features. So we actually already talked about this. Nodes are just nodes. Edges are connections between those nodes. And features are vectors, or scalars can be, if they only have one feature, that are combined or connected to a given node. Now, from a computational point of view, how do we represent? So we have something that is called adjacency matrix. Adjacency matrix is basically a matrix that has a shape of number of nodes and number of nodes. And it has zeros everywhere where there are no connections and ones where there are connections. So for instance, if you imagine that we have a graph that has three features, this matrix will have three rows and three columns. And if you have ones on the diagonal and zeros everywhere else, this graph will be a little bit strange, because it will only have self edges, So self-loop, so-called. So edges from the node to itself. But if you have, for instance, like 0, 1, 0, and then all the 0's, this means that node number 1 is connected to node number 2, but everything else is disconnected. Does it make sense? OK, we have a question. What's your name?

Speaker 2 [18:10]

Just from my understanding, this is the place where

Speaker 1 [18:11]

It's that place where the...

Speaker 2 [18:12]

Permutation symmetry

Speaker 1 [18:17]

In a sense, because you could actually mix those nodes, right? So you can change their positions, and the matrix will be different. But still, what the neural network will be looking at are connections. So it doesn't matter if node 1 is node 1. I can also move it to be node 2 now. the network should learn the same representation basically we have another question so it's a very good question we can have directed or undirected graphs and if it's undirected we will have to have one like one here and then in the same mirror image in the column space and in the row space, right? Does it answer your question? Yes? Thank you. OK, and the second thing is a feature matrix. So we will usually have some features. And those features, this feature matrix will have a shape of number of nodes versus dimensionality of features. So if we have like five features, we have three nodes and five features, it will be three over five. OK? Does it make sense? Any questions? We have one question. I'm not sure if it will appear later on, but do we have features on the edges themselves of what the relations are? So what's your name? Costa. Costa. Costa. Okay, so Costa asked a very good question. Can we also have features on the edges? And the short answer is yes, and we have graph neural network architectures that might also work with edge features, but just for today, we won't use them, okay? So today we're only using node features because it's already a lot of stuff going on. Yes, we have another question. I'm not sure if I understood the question. So you said, what's your name? Yes. Yes. So we usually have like an edge feature matrix as well. And by the way how those networks are constructed, we take those features into account, like edge features into account. To do that, you do have to increase the dimension. That's what I'm asking. You have to, right? The dimension of? Of the matrix. You have to add... The feature matrix? He's asking if we need to kind of introduce artificial nodes to fit it into a current setup. No, so we usually have a separate matrix for this, for edge features. We might have a separate matrix for each edge features. Sorry, if you want to, we can discuss this after. But I don't want to devote too much time to this now, because we won't make use of this. OK, so how does it work? So let's take this graph. And for now, we'll just focus on this node, node A. So to build a representation of node A, we'll take some function of nodes B and C. Why B and C? Because these are the only nodes that are connected here. And for now, let's forget if it's directed or undirected, just for the case of simplicity. And I call this A prime because this will be like another layer. So let's say this is our input layer. And then we are constructing the representation for the first layer. And we can stack those layers on and on. Now, this looks simple. In fact, it's a little bit more complicated, not much. So usually what we do, we take some function of b and c. And this function could be split into two functions again. Usually we'll take something like multi-layer perceptron of b and of c, and then apply some aggregation function here. So maybe it can be a summing. Maybe it can be average. Maybe it can be max. And then when we aggregate those features, we pass them through some activation function, usually a nonlinear function, to make it more expressive. And we apply this to each node in the network. And here, it assumes that this is directional in this direction, right? Because otherwise, b should also be a function of a. And there we do it. And for each layer, we repeat this set of operations. So now, one important thing is that you can notice that if we have just one layer, A will be only looking at B and C, so-called one-hop neighborhood. If we would like A to be able to see nodes that are further part in 2-hop or n-hop or k-hop neighbourhood, we need n or k layers. Okay? So, depending on your problem, you might want a very deep network or maybe a very shallow network. I have a question. Can you just quickly tell me what is A prime different from A? Yes, of course. So we say A prime here, because this is a representation of node A in the next layer comparing to this. So this is like an input layer. And then we have the first hidden layer. And we call it A prime. Does it make sense? OK. Thank you. OK, so now let's talk about types of GNNs. And we'll talk a little bit more about how they work. So there are many graph neural networks architectures. I will tell you about three of them today, classic ones. These are called GCN. That stands for Graph Convolutional Network. And this is, in a sense, a generalization of what we call a convolutional layer in computer vision or sequential modeling. Then we have GraphSAGE, which is like a more, in a sense, more versatile architecture that can have different aggregation functions. And finally, the third one that we are talking about today is called GAT, graph attention network. So as you can imagine, this one leverages the attention mechanism when learning those representations. And we'll focus on the first one and the third one today. So we will talk briefly about GCN and GAT in terms of theory, how they work. And then we will implement both of them. And we have many, many more. So even if you go to the website of Spectral, which is the library that we'll be using today, they have implementations of many, many layers, starting from simple layers like GIN to pretty advanced stuff. OK, so how does GCN work? We'll introduce a little bit of mathematics now, but not too much. So now what I want to do, I want to take this notation that we had like AB and so on and make it a little bit more abstract. And we'll say that H is a representation of any node in the network, node V in this case, and this L here in the bracket means layer, stands for layer. So representation of node v in layer l is representation of each of its neighbors. So we take all the neighbors of v, we call each of those neighbors u. So in this case, for the node a, it would be b and c, assuming it's directed. In this case, actually, it doesn't matter. And then what we do, we divide this representation by the number of neighbors. So this is basically what we're doing here. We're averaging. We're averaging representation of all the neighbors of our node. And then we multiply this by some weight matrix. And this weight matrix has two important properties. First, it's learnable. So we learn this matrix during the training. And second, it's shared. And when we have this, when we take this average, because we have summation and division, we send it to some nonlinearity. That's all. That's graph convolutional network. Do you think there are any problems with this approach? This is a tree at the moment. Maybe what do we do when we have circular references? OK. I need to be faster in my way. So the comment is that we might have circular references. So we might learn the representation of the node itself, in a sense, in a circular way. Is that correct? Am I understanding correct? Yes. That's true. It can happen, but it's not necessarily a bad thing. One thing that is a problem here is the averaging. Because you can think of it like, because of averaging, there will be situations where we won't be able to differentiate between two different neighborhoods. So maybe you think, let's forget about D and E, or maybe different. Let's say, it's not a good example. Well, by averaging, we basically might build the same representation for two different neighborhoods, assuming that they have certain values in their features. So average of two elements of 8 is 8, and average of four elements of 4 is 4. And they would be represented identically here. And this might be a problem. It doesn't have to be a problem, depending on your problem. We had a question, and we had another one. Answered? OK. Yes. Sorry, maybe I missed that. What is the initial value of those representations? Is it scalar or vector? And also, does this matrix always with the break matrix, does it square and change the dimension? OK. So the question is, what is the initial representations of those, of this? And the second is about if this matrix can change the dimension. OK. Yes, so this doesn't have to be squared. So we can compress those representations, or we can expand them, anything you want. And this, in the first layer, this is the input feature vector of a given node. So I said that it can be scalar, technically, if you have just one feature. But even more technically, we even represent a scalar as a vector in computations because we want to have dimensions in place. OK? So this is. Yes? There is no learning rate .. I'm not very good at machine learning or anything, but I think in the machine learning, you usually have a learning rate in this formula. JAN-FELIX SCHWARTZMANN- A learning rate? Yes. So learning rate is, so everything, things like learning rate, and batch size, and everything you can think of also applies here. So now we are only looking at, let's say, functions that are mapping nodes to other nodes. So we are just focusing on this very, very small part. And then, of course, when you do this, you apply this one time in one layer, maybe another time in another layer. And then finally, you go to your classification or regression layer, and there's some error, and we want to backpropagate, we have learning rate, of course. And we have enough. Yes, yes. So all the methodology from neural networks applies here. This is only this part that is, as we said in the beginning, it's different than what we have in classical machine learning. Because we have this encoding, and we have this very special structure in the data. Thank you. It's a very good question. OK, GAT. very quickly we change something here what have we changed no averaging exactly what do we have instead huh yes but alpha we didn't have alpha there look yes yes we don't know what alpha that's a very good that's a very good remark but what is it what can it be edge weights okay we could think of it this way yes specific weighting factor depending on connection that's a very nice answer maybe to drop out some connections very good intuition number of connections okay what do we have here in the index what I V and you what are those nodes exactly neighbors somebody said good yes your first yes so so we We have a remark that we know that W is being optimized in alpha. We don't know. Of course, we don't know so far. Very good. Attention, yes. How important is node u to node v at the given time? Yes, and all the answers were very good. So in short, this is a vector of attention weights. And we had another from Alexander, a broader, extended answer. that this is a coefficient that tells us how important is node u for node v. So node u in the previous layer for node v in the next layer. And this is a tension mechanism. How it's implemented? Well, in the paper, it was implemented in one way. You can think of different tension mechanisms, and you can apply them here. But basically, this is what we've said. So instead of just averaging, we are taking a weighted average. But, and this answers to your question, we also learn those weightings. So this is another parameter in the network that we are learning during the training. OK? Yes? Are those constrained in a way that they have to come up to one or two? The question is, if this is constrained, if they have to sum up to 1. Yes, in a sense, because they are normalized. So there is a softmax operation over those weights. OK, so we had GCN that averages neighboring representations. And we have GAT that learns attention over those representations. So now we'll go to something more down to the earth. And we'll talk about spectral, the library that we'll be using today. And I need to speed up because we are over time. But hopefully, everything we talked about will help us be faster in the next part. OK, so GNN landscape in Python is pretty rich already. And I will just talk about three types of libraries. Maybe there are more. I don't know. I will share with you what I know. So first, we have back-end agnostic frameworks like DGL, Deep Graph Library. So you can use this with TensorFlow. You can use this with PyTorch. You can use this with MXNet. Next, we have PyTorch-based frameworks like PyTorch Geometric. And this is a very, very mature ecosystem that has also many sub-libraries like PyTorch Geometric Temporal and so on and so on. You have the research community around PyTorch Geometric is very rich. So many papers are implemented, and so on. And finally, we have TensorFlow-based frameworks. And we have TensorFlow GNN that we are not using today. And we have a very good reason for this, because this is like a very, very, very early stage library. And I think it's still in alpha. And there are many problems with it. And there's finally Spectral that we will be using today. So why Spectral? I told you a lot of good stuff about Python Geometric, and I think that's all true. But if you use TensorFlow, or you prefer TensorFlow, or your production system is in TensorFlow, you will probably pick Spectral. DGL might be your choice as well. But as with Keras before being integrated into TensorFlow, you are not able to go super deep sometimes. So if you want to do something more advanced, something more low level, Spectral probably will be your way to go because it integrates smoothly with both Keras and TensorFlow. And you can go as deep as you want with custom training loops and actually virtually anything. OK, so let's think about main components of Spectral. So I will talk about three of them, though you might argue that there are more. And first, the most important one is layers. So we have convolutional layers. And convolutional layers is a broad term that embraces everything we talked about. So GCNs, GATs, GraphSAGE, and many, many other layers. So this term convolution is used very broadly here. It captures all of the graph layers that they have. And excluding pooling layers that are just taking these representations and pooling them together into something that can be then passed to a classical dance or classical convolution or whatever other type of layer you want. Then we have base layers, which is just base for building your own classes, and you have models. So you have those layers implemented with other layers, combined with other layers, pretty often as it was done in original papers. So in the GCN paper by Kip van Welink, or GAT paper that introduced this GAT network by Velichkovich and colleagues. Next we have data loaders, and this is very important because we need a different approach to data than with classical neural networks or classical machine learning because we have those feature matrices, because we have those adjacency matrices and in some cases we also have other matrices that have indices which nodes belong to which graph because we might have a huge data set that contains many graphs which is called a disjoint mode. We will talk about this a little bit more later. And finally, you have data sets. So if you want to, maybe you just want to experiment with something, or maybe you don't want to start with your very large data set, you want to do a quick POC, you can use some of those data sets. And they have everything, all the classics are available, like Cora, which is a citation data sets, TU data sets, which is a bunch of very broad scope of graph data sets. So now it's practice time. We can go back to the notebook. Here's the old ink still. Great. Oh, is this like this? Maybe. Yes. So if you are in Colab, one important thing is that you need to uncomment this stuff and click Shift-Enter. I think it will be installing for a while. So I'll tell you what we have in imports here. So we import OS, which will be only important for our data sets at the end, then NumPy for some manipulations. We input spectral as SPKTRL, TensorFlow STF, TF Keras as Keras. And then from spectral data set, we will take citation data set, which is Cora, the data set I mentioned before, and TU data set, which is actually a bunch of data sets. Then we also import two data loaders. So we said that this is an important piece of code that is very, very helpful because of graph structure. And as you can see, we have two different data loaders, single data loader and disjoint data loader. So single data loader will import something that is like just one single graph. And disjoint data loader will import something that is a set of graphs, but that is encoded as one huge matrix, which is basically a set of matrices put on a diagonal of a huge matrix. And each sub-matrix on this diagonal will be a different matrix. Sometimes it's easier to encode a graph like this and more efficient to process. And then we actually don't import this. And we import data set and graph. And both things are something that we will use for data sets. Do we have anyone who is still installing the libraries? No. OK, so we have all the libraries installed. OK, so you can import the libraries. But before, please go to, in call up to your, I don't remember how it's called, to check if you have GPU turned on. It will be faster, yes. So please make sure in Colab that you have GPU. If there's anyone who doesn't know how to do it, just let me know. Okay, one, two people. Okay. Okay, so we have, I will show you then because it will be faster. Because we have more than two people. Colab. And open this. And open this. So you go to Runtime. And then you go Change Runtime Type. And then Hardware Accelerator. And you pick GPU. And then you click Save. Do we all have it? Yes? Okay. Yes, sorry. I'm sorry, yes, unfortunately. Okay, so let's wait one minute to turn on GPUs and reinstall if this is necessary. Yes, of course, you can make a question. What... Okay, so it's a very good question, thank you. So the question is, what is the most practical problem that you can solve with graph neural networks? So from business point of view, pretty often it's about social networks. well business point of view and scientific point of view it's about for instance proteins modeling drug interactions predicting toxicity of a given substance it's also about in neuroscience it's about connectivity in the brain so trying to understand something may be built a generative model maybe build a predictive model very good question so the question is is there a way to deal with time dependent networks like transport networks for example yes and there are special packages for this like Python geometric temporal where you can relatively easy you called graphs that are changing in in time, and make predictions in time, as well. OK, one more question. Is there also something like skip connections in these architectures? Very good question. What's your name? Nicolas. Nicolas. Nicolas asked if there is something like skip connections in these architectures. There can be. So you can think about it, as we said in the beginning, you can think about it as a type of a special encoder. And now what you do inside this encoder, And also in the whole architecture, sky is the limit. So there are architectures using skip connections. And this is something that tends to work very well in certain contexts. But today we won't do this. Actually, we'll have an exercise so you can try a skip connection as well. But you can definitely do this. And you can do this within the graph block. And you can also do this when you're in the dense block. So you have already encoded representation and so on. OK, do we have anyone still waiting for the installation? Nope, great. OK, so if you click this second cell or the third cell, you should see something like this, physical GPUs and spectral intensive observations. Do we all have it? OK, assuming that we all have it, thank you. OK, let's get the data. So we'll start with the CORA data set. You can use this link if you want to read more about it. We're using this citation class, which we imported before from spectral data sets. And we say that this is CORA. And we are normalizing our X features. So let's do some exploratory data analysis. We said that from a technical point of view, if you want to encode all of this, we have something we call adjacency matrix. And then we said we have something that we call a feature matrix. So let's see what is the dimensionality of our adjacency matrix. You can see that this is 2,700 something over 2,700 something. Why is it a square matrix? Because it is a node versus nodes. Exactly. This is nodes versus nodes, because we are looking at connections between nodes. So on the diagonal, what would we have? . Yes, but if we have ones on the diagonal, what does it mean? Self loops, exactly. Very good. And you? . Self-sighting, yeah. We should not have them, right? OK, so we have in those data sets, we have this adjacency matrix. We said feature matrix. We have it. We also have something called y, which is labels, OK? And in this particular data set, because this is just one graph, one huge graph, not that huge, one, well, reasonably large graph. We also have labels. So for each note, we have a label. How many labels do we have? Seven labels, exactly. Good. So now it's time for your first exercise. Display the label of note 77. What is the label for this note? So although this is just one graph, We are saying data set 0. We are indexing this. There's just one index. But this is because we want to be generalizable. So this is your data set. If you want to access labels, it's dot y, the attribute y of your data set. And then we need label number 75. Who has the answer? 77, sorry. You're right. OK, so what does it mean? So this is one, these labels, this is very important thing, these labels are one hot encoded, right? Which means that we will have one on the index that marks the number of our class. So we should do something like this. And this is the last thing. So it's like because we are in Python, so it's 0, 1, 2, 3, 4, 5, 6, 7. Something wrong? No, good. It's 6. Sorry, sorry? Yeah, so this is 0, 1, 2, 3, 4, 5, 6. Instead of saying it's by 1077, it's by 1076. Ah, OK. That's a good point. So I'm telling you that Python is zero index, but then I'm doing this. Exactly. So this is the other label. Thank you. Thank you. You're right. You're absolutely right. OK, so let's understand features. Features are encoded under this attribute. You can find them under this attribute x. What is the dimensionality of the features? What is the dimensionality of the features in Cora data set you have the answer on the Exactly 1433 this means that we have 1433 features for each note those features are one hot encoded words from the articles, okay, so we have Actually, they picked 1433 1433 words that they fought maybe are Are they have some predictive power and they said this word is in this article this word is this word is The other one is not that's all So now let's see. What is the distribution of this you can see that this is extremely sparse sparse So what we see on this histogram is we are counting how many? How many words per article do we have? you can see that the like the median of the mold is like 20 I don't know if it's medium yeah so it's like this this the larger mold is 20 which means that we have only 20 ones out of 1433 this is very very sparse extremely sparse okay So now there is something that we are using for this particular data set. It's just like it's not important in general. It's just an item in this data set. We have those masks. TR stands for training. VA stands for validation. And TE stands for test. And this is basically trained validation test split. So we will use this to mask certain nodes in our data set. and then use those nodes for validation and for test. But they will be invisible during the training. Okay, now it's time for your exercise number two. We need to check how many training examples, how many validation examples, and how many test examples we have. So you should replace those ellipses here with something like this to get the answer. Okay. Do we have an answer? Okay. Go on. I think you were the first. exactly so I'll do it as well here on the screen so we can see and test so this means that our training set is very very small a validation set is larger which is well in a sense is good and our test set is even larger so this means that if this network works well we would expect that it would it should have pretty good generalization right at least in this realm of something similar okay so now we're going back to loaders so loaders we take this data set so it will take those X's A's and Y's and I's and E's if it's there, and they will structure them in a way that will be compatible with the library interface, and it will make it possible for us to use all those training structures that we know from Keras and TensorFlow, so high-level APIs, making just the whole work much, much easier. So first, we'll use the functional API. Who have used Keras functional API before? Who has used the Keras model subclassing API before? One person, two people. So the idea about functional API is that you start with some inputs. Then you build some layers. And you pass these inputs to these layers. But not to the layer constructor. You just call those layers on those inputs, or whatever. Then you build some, maybe another layer. And then you also call this layer on this thing. Here we have two things, because this is a graph layer. So we need our data representation that comes from the dropout layer. But we also need, and this is critical, we need this NA, which stands for our adjacency matrix. We need to pass this adjacency matrix so this layer knows where are the connections between the nodes. And then we continue with this. I will show you a couple of more examples to familiarize you with this. Then we take another layer. This layer is not a graph specific, right? It's just a regular dropout layer. So we only pass the data representation, OK? We are not passing adjacency matrix. Then we have another layer, which is a GAT, which is a graph layer, which means that we need to pass what? Adjacency matrix, exactly. So we pass our data representation that comes from the dropout layer, the second dropout layer, but also we add the adjacency matrix. And finally, we encapsulate all of this in a Keras model. So we take the model class from Keras. This is Keras from TensorFlow, but it doesn't matter. And then we specify inputs. And we have two inputs, because we have a feature matrix, which is called nx, and adjacency matrix, which is called na. And then we specify outputs. We have just one output. This is our final GAT layer. And an important thing is that, look, The activation function of this layer is softmax, which means that we will use it for? Classification. Exactly. So we don't use any other. If that would be like a deep layer, we could use or or hyperbolic tangent or anything like this. Here we are using softmax. We could also use nothing and build a regression problem. But for this, we will typically use also a dense layer after this. OK, let's try to run it. We have one question there. No, we don't have a question. OK. OK, let's set up some parameters. So you were asking before about learning rate. Now it's time to set up the learning rate. So you can see that in this 1.4.2, we are setting the learning rate, number of epochs. We say 10,000, but we just say 10,000 to make it like, OK, If it needs to, it will run. I will finish much earlier. We also set patience to 30. Why can we use something? Why do we? Why might we need something like patience? What would that mean? Yes? If the warning process starts calling, there is a change in the load function, or the function we are trying to optimize. which is early stop, so there is the parameters are definitely not working in the favor of the model. Exactly. So for instance, we don't want to waste resources. If the model is not learning anymore, we will use something that is called early stopping. And patience is 30, which means 30 epochs. So if for 30 epochs there is no progress, we'll just stop the training. And as we will see in a second, we'll also get back to the weights, the best weights that we found on the way. Another question? You don't use batching because... Again? You don't use batching because... Batching. We are using batching. Not in this context. We'll talk about batching in the second exercise. I don't want to go into this, but that's a very good question. So, for now, let's compile the model. We are compiling the model and we are training the model. And as you can see, we are using this early stopping callback here. We are also reducing learning grade on plateau. I don't want to get into this. This is more like deep learning stuff. And let's run this training. It should be relatively fast. You can see that we are starting with validation accuracy of 0.3 so this is not very good also accuracy might not be the best metric here but for now we are going with this you can see that now around epoch number 155 we are at 96 already so this is good yes so the seven labels in this data set they mean the area that the article is coming from so maybe it's computer science or neuroscience or psychology or biology or something like this from all of them like so so the the this words are independent like they are they are this the set of words is the same for each area for all the areas sorry the set of the set of so the set of words that consists our features is independent of the labels okay so maybe we have a word computer and it will appear in I don't know maybe we have a word neural network and it will appear in mostly in artificial intelligence papers and maybe just sometimes it will appear in psychological papers but very rarely right but we don't have so these words they are coming from all of the areas and I don't I don't remember how they pick them the same words what do mean by this yes yes yes so so so yeah so you have a biological article so in this biological article you will have words like cell or life or something Well, this particular network has a slightly different purpose, so we want to classify the nodes. So, you're sending me an article, nodes. So each node is an article. One node is one article. So you want to classify the map based on the keywords? Based on the keywords but also on the connections, right? Because we are taking citations into account. So we could show that if we take just the keywords and we ignore connections and we train a regular neural network on this, then our results will be much, much worse. OK? And that's the idea of this encoder. OK, so my model has trained. I don't know about your models. Let's see how the loss goes. It goes OK. We could actually stop probably a little bit earlier. And let's evaluate the model. It's 0.998. So this is a very good result. Let's go back to the size of those training and test sets. And so based on 140 articles and the information about the citations, we're able to predict almost all of the 1,000 articles correctly. Yes. Again, again. Why don't we? don't we why do we cover all of the yes is is less yes I understand I don't know you can go to the Cora website and check this maybe they had some idea for this yes Arbitrary, like a graph structure, you mean? So when we were talking about those GCNs and GATs, and we were talking about those weight matrices, we said that they have two important properties. So one important property was that they are learnable, so we are learning them in the training. and the second one and i think this should answer your question is that they are shared which means that we could potentially push a sub graph or another graph and it should be able to generate predictions based on this training and we'll see an example of this in the second exercise so you can so you can learn those weights on one graph and then generalize them or use them on another graph and they should work as well Does it answer your question? Yes. So the question is that from the formula that we've seen, it seems that those networks are taking into account the immediate or one hop neighborhood into account, right? That's true. And the question is if we can extend it, we can extend it by adding more layers. So adding each layer is going one hop further. Of course, this is not entirely true because we can have those self loops Not direct self loops and so on and then it might be different. But yeah to count this for your training we are masking we are masking the we are masking this yes Yeah, but we are splitting before we start training. So this representation is being propagated during the training, right? So if it's cut off before the training, then you are altering your data in the end. You are altering your data, yes. Okay, that's really different then, okay. Yes, but this is how it is. So of course, there are many problems with this type of networks if you want to go very deep. So you might have things like over-squashing or over-smoothing, which means that we might go, if you think about it, if we do this like, if number of layers goes to infinity, the representation of nodes goes uniform, right? Because I'm taking all the information from my neighborhood, and again, and again, and again, and again, and each node is becoming essentially the same when number of layers goes to infinity. So there are limitations to this framework, for sure. So this is a very good intuition. OK, so I have another exercise for you, but we are a little bit out of time. So my proposal is that we go with another structure, and you can do the exercise at home. Or we can do the exercise now, but then we might not be able to do the second part. So who votes for going for the second part? Who does want to do the exercise instead? OK, so that's clear. So we're going for the second part. OK, so for Cora, we're trying to classify articles. And articles were represented as nodes in our graph. Now we'll try to do something else. So now we'll try to classify graphs, OK? So we'll try to take a bunch of graphs and their labels. And we'll try to learn weights that we could take and apply to another set of graphs to meaningfully predict their labels. And so we were talking about those data loaders and so on. And I told you that we can encode sometimes many data sets in one matrix. And this is precisely these things that we're talking about. So we have a large matrix that contains, that has many adjacency matrices arranged on the diagonal. And then we have another matrix that contains features for this. So we can do this for storage reasons, for computational reasons, and so on and so on. And this is what we will have here. So now we are reading a data set. And this data set is called proteins. So now, each single graph in this data set will represent one protein, OK? So now, the question for you is, how many nodes are in the eighth graph of proteins data set, and how many nodes are in the 172nd? So, this is how you do it. We'll go further. You can answer the second question yourself. And here we do something else when it comes to train test split. are just taking like a random split but then essentially we are doing something very similar but instead of using single loader we are using this joint loader and this joint loader is something that will work with a graph like this with a graph encoded like this so many graphs in one in one matrix and now we are we will use so in the previous example we used graph attention network now we will use a GCN, so the first architecture that we mentioned today. Now, one difference between those two situations is that now we have a couple of differences. But we do not only have x and a, so features and adjacency matrix, we also have this i. And this i is a set of indices that picks particular graphs from this huge matrix that contains all of the graphs. So it just takes like, OK, maybe one graph, one, and two, and three. And now we are also using something that is called model subclassing API. And we inherit from Keras models model. We have some init where we define our network structure. And what we have here is we have this GCN layer. And we have maybe more GCN layers if we want. We have some global average pooling. Then we have a dense layer, dropout, and dense again. That's all. And then an important function or important method in this model subclassing API is this call function or call method that calls the whole thing. So I will run this. Again, some setup, learning create. Only 10 epochs this time. We instantiate the model, run the optimizer, and train. What is the dimension of the input layer? The dimension of the input layer is the dimension of features. Not the nodes. No, because nodes are so in a sense, what is going on within this GCN layer that we've seen earlier is actually inside this GCN layer, you have a set of small neural networks between nodes. Yes. But if you think about, so this is a tricky question in a sense, right? So it takes connections, and it has this set of neural networks that are building those representations, then applying this nonlinearity that we talked about before. But then the dimension of this is channels. So it will take, and the input it will take for each node, it will take a feature vector dimension. Maybe it's like, I don't know, 2,000. And it will squash it to number of channels, maybe 36. And in another layer, maybe we want to expand, 64. And then maybe expand again, 1,128. Then we want to compress 16, and so on. So we can think of it like, in a sense, so it's like if you think about, if you abstract the graph structure, it just goes like a regular, I don't know, convolutional network or dense network. But behind the scenes, it also applies those little neural networks that compute the representation for each of the nodes. But the global feature dimension for each node is defined by channels here. Make sense? So then it's time that you pass an input, you put the feature of one node, then you run it, and then the next time you put the feature of one node. So inside, it's all, you can make it very efficient on matrix multiplication. So you just multiply adjacency matrix by the feature matrix by the reverse of one of those matrices, inverse of one of those matrices, and you will get all those computations. So it all happens simultaneously, in a sense, inside, right? But from, let's say, abstract point of view, the dimensionality is just driven by those channels. And this dimensionality will be identical in each layer, will be identical for each node. Does that answer your question? Yes? What is the intuition behind using What problem are these models in the graph? So with this, let's leave it for Q&A, OK? That's a very good question, but let's leave it for Q&A, because I want to complete this. OK, so as you can see, we actually started with a better result than we got at the end, right? And we don't have early stopping here, which means that we just finished with those weights, which are still with the weights giving us this result, which is still, well, reasonable. So there's an exercise for you. You can add more layers. So we talked about listening to this one hub versus k-hub neighborhoods. So you can add more layers for yourself and experiment with this. And finally, I wanted to tell you about building your own data sets. But actually, I think let's leave this five minutes for questions. You can read this here and also check the Spectral website. And you have this class that is actually the actual code that we used at TensorFlow for one of the experiments for our traffic optimization experiments. And you can look at this code to try to understand what is going on here. One thing that is important in those data sets is that they should inherit from the data set class, and they need to have this read method, where read just reads the graph from a given location, and then you can do anything you want in this read class. And another thing that is very important, you should output a graph. So graph is imported from Spectral, and it should have x, a, and y, and maybe also e if you have edge features. And this, if you structure your code like this, it will be ready to be passed to any TensorFlow or Keras training API that you would like to use. OK, great. So I go back to the presentation now. And you see my screen like this. It's just not good. Okay. So here is the survey. You can click this and answer a couple of questions that should not take you more than one minute. I'll be very grateful for this because your feedback will make my future presentations better. So you can help other people to oh, wow, sorry about this. is going on with this now it should be stable and I'm ready to take your questions and questions from people online because I suppose we also might get some something from there

Speaker 2 [78:48]

Yeah, so before jumping into questions, let's thank Alexander for this very nice tutorial. So there are not so many questions on the live chat. There was one about...

Speaker 3 [79:06]

Mm-hm.

Speaker 2 [79:07]

If the...

Speaker 3 [79:09]

note

Speaker 2 [79:10]

node V is contained in the subset of nodes in your formula.

Speaker 1 [79:14]

Even though they take you

Speaker 2 [79:15]

U in the number of nearest neighbors of V is V in that set as well.

Speaker 1 [79:21]

Ah, so this will be a self-loop, so-called. So actually, in fact, GCN, for instance, is, if I remember correctly, the original implementation takes the node itself into account. So it takes its own representation as well. But sometimes it's worth to experiment with this. And even if your architecture takes the self-representation into account, you might want to add self-loops if you don't have them in your data set. You might want to add self-loops in your adjacency matrix. And it might affect your results positively. This is my experience, at least, with some architectures.

Speaker 2 [79:55]

Let me just finish with these ones and then we can jump into the question from the audience. So another question is a what's a let's say, how do you compare like a GNNs as compared to forgetting the graph structure and just using NNNs or LSTNs or things like this

Speaker 1 [80:12]

or things like this. Forgetting the graph structure? Yeah. So like catastrophic forgetting like in Transformers or like...

Speaker 2 [80:19]

I think it's just simply like, let's say,

Speaker 1 [80:19]

I think it's just simply...

Speaker 2 [80:22]

what do you think if you have a network, like a graph neural network, and instead of using GNNs, you just use a classical neural network?

Speaker 1 [80:28]

classical neural networks. OK. Yeah. So multiple experiments showed that if there is a graph structure, and this graph structure is meaningful to the problem, then graph neural networks will give you better results. So I encourage you to, in the repository, you have also a folder with papers. I encourage you to go to those papers. These are papers describing GCN and GAT. And you can see also comparisons between graph neural networks and regular neural networks applied to those problems.

Speaker 2 [80:55]

All right. Those were the two questions we have, so we have questions from the audience.

Speaker 1 [80:55]

All right.

Speaker 2 [80:59]

Okay.

Speaker 1 [80:59]

Okay, so let's let's start with with the at the end. Sorry

Speaker 2 [81:08]

soft questions

Speaker 1 [81:11]

First is how does it scale if we

Speaker 2 [81:13]

if we start to speak about like the millions of nodes. And my second question would be, how do you deal with bipartite graph? I'm thinking let's say you have a series of customers and a series of vendors.

Speaker 1 [81:26]

Yes. So very good question. Thank you. Scaling. Scaling is a real challenge. So with millions, you're probably still good. But, you know, there are things like Pinterest or something where they have like millions, thousands of millions. So, nodes, thousands of nodes, so what people are trying to do in those circumstances is trying to, they are trying to take SAP graphs, learn on SAP graphs, or sample, and there are many sample techniques that are trying to make sure that you're not losing too much information. And the second part of the question? Bipartite graphs. With bipartite graphs, I don't have experience with this. But you can do it. So there are methods designed specifically for heterogeneous graphs, where you can have bipartite or tripartite graphs. And you can also have multiple types of edge features. So there are methods for this. They are actually, let's say, trying to take those methods like GCN and just apply them to more general settings. If you go to Spectral Documentation, you will find methods like this. Or Pythage Geometric, you will find. And there are links to the papers.

Speaker 2 [82:57]

Mm-hmm. Let's have the question here. First things, the presentation was really awesome. Thank you. I wanted to ask about the data set for graph, the classification of the whole graph. Like, is it common that you have just feed, you have a data set of just many graphs, or is it always, you usually just have one graph and then you just sub-sample it in the sense that you create smaller graphs out of it and feed it to them.

Speaker 1 [83:27]

So you're asking about this particular way of storing multiple graphs? Or in general, is it more common that we work on one large graph versus many smaller graphs?

Speaker 2 [83:38]

Yeah, because you want to classify the whole network, right? The whole graph.

Speaker 1 [83:43]

The whole graph, you mean?

Speaker 2 [83:44]

you mean yeah that was the second problem right yes we're classifying we generating one label out of the whole graph yes so would you like do you do you then have multiple example crafts so you have for example 10 example graphs and each one has a different label and feed it to the network or do you have like usually have just one network available and then you just sub sample it into smaller graphs and then try to add.

Speaker 1 [84:11]

So in case of problems like with proteins or maybe toxicity of certain substances, you usually have graphs for many substances or many proteins or many whatever entities you can imagine. And you will train using many different graphs. And then you will come up with a new graph, maybe a new protein or maybe a new drug, and you will try to predict the outcome of this drug or the interaction for this drug. In social networks, you would usually have one large graph. Or sometimes you can also sub-sample this graph. You can cut this graph into many graphs and so on. And you can try to work on these, let's say, local structures. So usually when I'm maybe on Facebook, what is the most important are the friends that are closer to me than those who are, you know, I don't have many friends in common with them. So depending on the distribution, you might make different decisions. I don't know if this answers your question or complicates it even more.

Speaker 2 [85:15]

even more, but... No, I guess there are different uses.

Speaker 1 [85:18]

use cases yes yes yes precisely there are different use cases so with the different types of graph neural networks do you have any what is intuition when you use the different kinds so just to try the different because in normal neural networks I mean there's certain neural networks that are using certain problems right what is the intuition error where should I read up maybe it's a huge answer. So, yeah. Yeah, so, well, I think it really depends. So it really depends on your problem, it really depends on the data, like how feature-rich it is, how deeply you want to look into the graph, because in some problems, actually, one-hopped neighbourhood is good enough, or two-hopped neighbourhood, right? And sometimes you have something, you might have a problem that's really simple, and then it doesn't make sense to apply a a very complex method. So I can share with you our experience from TensorFlow. So everything with graph neural networks started a couple of years ago when Pavel Gora, who is the leader of this initiative, he built a graph neural network for predicting waiting time in a subset of a city, waiting time on traffic lights. And he built a graph neural network that was topological, which means that nodes in the neural network were actually mirroring the topology of the crossing structure. And this was a pretty simple architecture, but it gives very good results. So I was working on the same problem with the same data set using graph attention in one network, I was not able to beat those results. I was able to come very, very close. And computationally speaking, this architecture is much more complex. But I was not able to beat this result. So it really depends on your problem. There are also architectures like GraphSage that might have, let's say, different faces, depending on how you configure them. So I don't have a definite answer for you. Sorry.

Speaker 2 [87:39]

I guess maybe we have time for one last question.

Speaker 1 [87:43]

Or otherwise, I don't know. I think there were at least two questions here, so I don't know Okay

Speaker 2 [87:49]

Maybe we can take the last one here.

Speaker 3 [87:51]

Yeah, also, thanks for the nice tutorial. Thank you. So I'm not sure, judging by the formula I saw, it's basically you just insert the features of the neighbors into the neural network, but do actually, yeah, things or ideas exist where the actual structure of the network. So say you have nodes without any features, and the only feature is that they are connected to some other nodes. So, for example, you want to learn, like, clustering coefficients or something like that. So really inherent network structures. So to use this in graph neural networks.

Speaker 1 [88:27]

Yeah, so you can have dummy features. You can use dummy features, for instance, just once, right? And then it will just compute number of nodes, of your neighboring nodes. Then you can normalize this, and so on and so on. So you can one-hot encode them, right? So all those methods will allow you to learn the graph structure without true features in some way. Those ways will be slightly different, and applicable in slightly different problem sets, let's say. But yes. Does that answer your question? Partially. MARCO PAGLIABOSCHI- Partially, OK.

Speaker 3 [89:09]

So I was wondering for example if you want to see in a social network if somebody is so to say an opinion leader So that would be judged by the number of contacts and the way for example in the graph Convolutional network you just average over the number of nodes in the network So actually not even the number of contacts a node has goes into the

Speaker 1 [89:26]

Not in the network, it's only the neighbors.

Speaker 3 [89:31]

Yes, yes, but but but if you feed the data of the nodes into it into your training network I didn't see and maybe I'm wrong in that so I didn't see how the information of the number of contacts for example enters into into your neural network the number

Speaker 1 [89:45]

The number of?

Speaker 3 [89:46]

Contacts, for example. So just the number of contacts, the number of edges that go into node.

Speaker 1 [89:49]

Going to node. Number of contacts?

Speaker 3 [89:51]

Connections?

Speaker 1 [89:51]

Connections?

Speaker 3 [89:51]

Yeah, number of edges, say.

Speaker 1 [89:53]

So number of edges.

Speaker 3 [89:53]

So number of edges.

Speaker 1 [89:54]

So you could add a dummy feature that says 1.

Speaker 3 [89:57]

Yes, but you would have to calculate it in advance.

Speaker 1 [89:57]

Yes.

Speaker 3 [90:00]

So it's nothing that you feed your graph neural network with your graph, and it figures it out automatically. So that's what I mean. So it doesn't.

Speaker 1 [90:07]

It could, but let's take it offline.

Speaker 2 [90:10]

Okay, so one thing that it's clear is that this topic is very interesting and we can go very deep But I think it's a first glimpse to the ones that we weren't exposed to this topic. It was a great introduction So, thank you very much

Speaker 1 [90:24]

Thank you so much.