Practical Session: Learning on Heterogeneous Graphs with PyG

Speaker 1 [00:02]

Yeah, hi and welcome. Nice to see so many people here. I'm really excited to speak here. We will present a presenter session, a practical session, on learning about heterogeneous graphs with pi dot geometric or pi g. And I myself am Matthias Pfei. I'm from Dortmund, Germany, and we'll be speaking here with Ramona, which is my colleague at Kumo AI. And I will go through GNNs and why you should be interested in them, and why are they the best choice for machine learning on graphs. And then I will also introduce PyG, like best practices, challenges, and so on. And then we'll hand over to Ramona, which will go through the practical session, which you can also follow on your own devices. And so just as a quick introduction, why we should care about GNNs in particular. And so if we look at the modern deep learning toolbox, then we see that this toolbox is heavily designed for simple sequences and grids. And so we traditionally use, like, convolutional neural networks to process images or transformers, and we also use recurrent neural networks and so on to process sequences. But not every problem can be framed as such, because, like, not every input we can think of can be represented as a grid or a simple sequence. And so the main motivation here, or the main question we ask is, how can we develop neural networks that are much more broadly applicable? And so if we look at the real world, then we see that nearly everything can be represented as a graph. And a graph usually consists of a set of nodes that are our entities in our world. And then the edges between these nodes denote how these two entities relate to each other. And so, for example, a social network is a prominent example of a graph where we have users as entities and then we have connectors connecting these users and so on. Other examples are, for example, traffic networks, communication networks, and so on. And we can also see that on all different kinds of scale, we can use graphs. So we can really use graphs for describing high-level concepts such as social networks, But we can also think about describing molecules as a graph and use neural networks to process them. And so the main problem we have here is that graph data is different from grids and text. And so in particular, they can have, like, arbitrary sizes, which means, like, the number of nodes, for example, may be different. There's some complex topological structure beneath them. Secondly, we usually have no fixed node ordering or reference points. So if you think about, like, a CNN, then it is clear that if we apply a CNN, then we can directly reference, like, the top right pixel to the current pixel. But there's no such definition in the graph. And so we usually speak about no fixed node ordering here. And then graphs are often dynamic, which is in contrast, again, to images or text, because I have rarely seen a graph that is not changing over time. And lastly, we are speaking about, in practical cases, we are mostly interested in heterogeneous graphs. And that means that we have different types of nodes and different types of edges. And to these different node types, we have different node and edge features attached to them, which makes them kind of a multimodal problem to tackle. So if we look at the traditional machine learning pipeline on graphs, we are questioning ourselves, like how can we capture this network structure as input feature? Or how can we combine this network structure that we processed and combine them with attributes such as nodes and edge features? And especially how can we use that to perform tasks at different levels of the graph, such as the task at the level of nodes, edges, or graphs. And then we see that if we put that into machine learning pipeline, then we actually use many, many different tools and frameworks to process them. So for example, we use Pandas or NumPy for feature engineering. We do some data splitting and so on with sklearn. And then we process that all into like a tabular data layout and then use XGBoost or neural networks, MLPs on top of them to apply them. And so there are two prominent approaches in doing that this way. And the one I mentioned earlier is known as some kind of preprocessing node features. And the goal here is that we kind of capture network structure as input features. And so one way to specify, like, how important a node is in a graph may be the node degree, which simply describes the number of neighbors attached to that node. We can think about more complex input features, such as node centrality measures, clustering coefficients, and so on. But the ultimate drawback here is that all these features are super arbitrary. So we do not really know in advance what features we are really interested in in our network structure. It is also expensive to compute, especially some kind of clustering, coefficients and so on, and ultimately we are losing signal because we just, the final machine learning model has just input to the features we preprocessed in advance, but not, has no access to the raw graph. The second traditional machine learning model, which was kind of popular like three or four years ago, is known as the shallow embedding model, and that references the lines of work of DeepWalk or Node2Vec, and those are all like random walk-based, which means they're built on the intuition that we embed each node in an embedding space based on whether Whenever two nodes are nearby in the graph, we want their embeddings to be nearby in the embedding space, and if we have two nodes that are far away in the graph, we want also their embeddings far away in the embedding space. And so we usually train that in like a word2vec or contrastive learning scenario where we sample random walks and then push all the embeddings of this random walk, of the nodes inside this random walk together, and then sample negatives and we push them away. And ultimately that works well for some kind of link prediction task, but it has drawbacks as well. In particular, it is super hard to scale because we basically learn a shallow embedding matrix for all nodes in our graphs, and if we think of really big graphs, that is like a giant embedding matrix we need to learn here, we are using like an unsupervised learning loss which does not necessarily align with the task we are interested in, and it is transactive by nature. That means whenever a new node comes in or we want to apply that model on a different graph, then we're basically stuck. And so graph neural networks got really prominent in that sense because it does not have all these limitations. And in the essence, graph neural networks are best described as some form of neural message passing scheme, in the sense that nodes send messages to their neighbors, they got aggregated, and these messages get learned and trained. And so I personally see them as some kind of generalization of any neural network architecture, And I find this kind of cool because what we have here is basically that we're using this input graph and use that as our computation graph inside our neural networks. And so that's actually a new paradigm of how we define neural networks in the sense that we no longer have static computation graphs in our neural networks, but we actually input the computation graph into the neural network. And so that naturally allows us to use graph neural networks as a super high-level concept to go, for example, from CNNs to GNNs in the sense that we are moving away from discrete neighborhood sizes and allow dynamic neighbors arbitrarily positioned in space. And we can also see them as some form of generalization of transformers, which is, in the concept of GNNs, just a neural network that operates on a fully connected graph in which every Every node is connected to every other. And then we're using attention to pass messages between these nodes. And so how does that work in the formula way? And so graph neural networks can be described as a way to update node representations by repeatedly transforming and aggregating information of direct neighbors. So we're having this function here called f, and that is parametrized so it can be learned. And it takes in the input features of the node we are interested in, and all the input features of this direct neighbor. And then it uses this information to compute a new feature vector, a new embedding for this node we are interested in. And so that scheme below here shows it quite nicely. We have the central node here, which is marked as a green node. The blue nodes send their messages to this green node. They get aggregated, and that information is then used to compute a new representation for the next layer. And so they have some ultimate, graph neural networks have some nice properties and strengths. For example, all these GNN layers can be trained because all of them are differentiable. So you could think of them as having some kind of subneural networks that describe how we aggregate information in each different layer. And so if we're looking at the graph here on the left, and we want to compute a representation for this target node, then this will amount to the computation graph shown on the right, where all these little gray boxes denote how we aggregate information at all different stages. Importantly, each node uses the same set of parameters, or these parameters are shared, which makes GNNs inductive by nature. So we have no problems using them and applying them on new nodes or graphs. And then GNN layers as in CNNs and other architectures can be naturally stacked, and that helps us to reach long-range information from nodes far away from the target node. GNNs, in the essence, just are responsible for giving us a node representation, but we can use this representation then for a given downstream task, and the most popular tasks are known as node classification, graph classification, and link prediction. So in the node classification, we want to predict the class of a given node. So we have a neural network that gets the graph as input, and a specific node V we are interested in making a prediction for. And so we can directly use the representation produced by the GNN, which is here H in the else layer, and use that representation and input that into an MLP to make the final prediction. And that is all trained end-to-end. Similar in the graph classification case, we are using GNNs to compute a node representation for all the nodes in the graph. We aggregate them together in the permutation invariant way, and then apply MLP on top to get a prediction for a single graph. Oh, no, sorry. And in the link prediction case, we now want to make a prediction for a source node and a destination node. So the neural network gets as input the graph, and then the link we want to make a prediction for, in that case, V and W. And, for example, we can then use an MLP as well, which is concatenate the embeddings of the GNN to make a prediction for a given link we are interested in. Again, importantly, and that is probably why we're here, implementing GNNs is ultimately challenging, especially because it does not really follow the traditional deep learning toolbox we all know. One of these challenges is sparsity and irregularity, which means that all our graphs we want to process are ultimately dynamic and may change in size and neighborhood sizes and so on, and there arise questions such as, like, how can we effectively parallelize this data and so on. There's an underlying constraint of heterogeneity in the underlying data, so we have input features that can be numerical, categorical, that can be even images and text, and that may also be different across different node types and edge types in our graph. And then, as mentioned earlier, graphs are usually dynamic, and it's hard to find scenarios in which graphs will not change over time, so you need to take that into account. And then there exists various different requests on scalability in the sense that some graphs may be more sparse than other graphs. Sometimes we even work on highly dense graphs or fully connected graphs and so on. And then there's this tradition or this classic contrast between operating on many, many small graphs, which might happen if you think about like a molecule data set where each molecule has like 10 or 20 nodes, but we're actually working on a data set that contains like millions of molecules, versus operating on a social network where we just have a single giant graph, but that single giant graph may consist of millions of billions of users. And then there is the applicability to a set of diverse tasks in the sense that our framework or how we implement GNNs should not restrict to which task we want to apply that GNN. So we have node-level, link-level, graph-level tasks as mentioned earlier, but that also should naturally support tasks such as clustering, pre-training, self-supervision, and so on. And so our solution to that, which is fully open-sourced, is called PyG, or previously known as PyTorch Geometric. And the goal here is that we have a PyTorch-based framework that bundles the state-of-the-art in graph representation learning by implementing various layers, architectures, and examples known from latest research. And so I'm especially proud of over 350 contributors now. And in particular, the whole project has kind of grown from like a single person project to a project that is now maintained by many, many people and many, many different companies, such as different universities, our own startup, Kumo AI, as well as NVIDIA and Intel. And so in the essence, PyG consists or contains four major components. One is that it contains graph-based neural network building blocks. So it contains message passing layers, GNN layers known from latest research, graph-based normalization layers, pooling layers, readout layers, and so on. And they are all defined on unified interfaces such that it can easily be extended and you You can easily implement your own on top of that. Then we are having a graph storage layer, and the graph storage layer helps us to define graph-based data sets and graph-based data loaders. So in particular, we have support for homogeneous and heterogeneous graphs. You can have direct access to over 200 different benchmark data sets, and we also support ten or dozens of different sampling techniques, which I will talk more about later. And then we have graph transformations and augmentations, which are kind of applied before we load in the data into our neural network architecture. And these augmentations are useful, such as like dropping nodes and edges during your training routines, which amounts to some kind of dropout, but you can also use that to transform your graph so that it fits into the needs of a GNN, such as defining meta paths on top and so on. And importantly, that scheme also, or this concept of transformation also allows us to not only support graphs, but also meshes and point clouds in the sense that we can use k-nearest neighbor search on the point cloud to construct a graph on top of that, and then use PyG to process that graph. And we are working hard on making PyG the central place to practically learn about GNNs. So we now provide video, call-ups, blog posts, and application-driven GraphML tutorials that let you practically learn about GNNs in the first place. And so I quickly want to show the general pipeline of how PyG looks in practice. And the main goal here is that we're trying to achieve is that PyG is like super modular. So in general, we want that each of these different steps can be easily swapped out. So you should be able to easily swap the GNN layer, see the performance. You should immediately see how the impacts of graph transformations and so on. So in the end, it always starts with data you want to operate on. And as I said earlier, you have access to predefined data sets, but you can also create your own data sets on top, and then in your Python script, you just simply instantiate that graph and apply the necessary transformations. And so the graph creation step in PyG looks as follows. So we always expect a graph in either a COO or a CSR format, which is like a super popular sparse matrix format. And that allows us to model complex heterogeneous graphs. So for example, we can instantiate a heterogeneous data object and then just write information to that object for different node types and edge types. So for example, we can write node features to a user and a product node type, and write that to the attribute x. And then we can also write edge type information using this tuple annotation here, where we want to specify information from a user to a given product. And here we can define the edge index, which is basically the graph indices that describes this connection, we can add edge information such as add features or the time of these edges and so on. Importantly, what I want to highlight here is that in academic graphs, these data sets are already curated and ready to be used for us. But if you want to apply your GNN on your own industry graph, then that basically means you have to think about how you move your input graph to the expected format of PyG. And so what you need to respect is, for example, what is the best graph design that I can pick from that is suitable for this underlying message passing scheme of GNNs in the sense that you want important information close by to target nodes. And then you have to think about how can I put continuous features into my neural network architecture that we're speaking of, feature selection, feature encoding, and so on at the level of graphs. After that, you're basically free to define any task on top of these data sets. And we usually use this Y notation here to specify, like, node labels and so on. And so, again, in academic graphs, these training labels and data splits are predefined and ready to be used. But in the industry setting, you have to, again, think about, like, how can I build my own training labels and importantly what are the data set splits I want to use. Especially it has been shown that random splits are like overly optimistic and better scenarios might be temporal splits and so on. Next, once you define your data set, you need to think about how you can obtain mini-badges from that. And PyG provides over ten different sampling techniques that work by sub-sampling subgraphs, and otherwise the graph would not fit into GPU memory. And so the general procedure here is that our graph is stored on CPU memory. We sample a subgraph, move that subgraph to the GPU, and apply a neural network on top. And afterwards, you are free to define your own custom GNN layer, and you have different flavors to choose from. You can use predefined GNN models that are used or that are built using state of the art practices. You can build your own GNN layer with customized building blocks, as we are doing here, in the sense that we are using like a Sage conf layer, which is a popular variant of a GNN layer, and then defining our computation graph on top. But you can also go one step further and just define your own GNN layers and use that in your custom GNN pipeline. And afterwards, the training and inference routines are like super straightforward and similar to vanilla PyTorch in the sense that you just iterate over the data, you move it to the GPU, you apply the forward path, you compute the loss on top of that, and then use back propagation gradients as descent to move the model parameters to the right direction. And so PyG is not limited by any of that, so you can have full customization from model architecture to training routines, but I think it's important to note that there are some concepts where you need deeper knowledge about GNNs, and in particular it's important to know that the best model architecture is usually both data and task dependent. So what is the best GNN for the given task? How can we capture long-range information? You might find scenarios where you need to deal with class imbalances, overfitting and so on, and that is hard to answer. It needs to be studied on a case-by-case basis. So that is the general overview of pipeline, of PyG, and I quickly want to go a bit deeper on heterogeneous GNNs before I hand it off to Ramona, and heterogeneous GNNs are known as graphs where we have different node types and edge types, and that is here a classic example of a heterogeneous graph where we have paper notes, and these paper notes are connected by which paper has cited which paper, and then we have author notes, which are described by which author has written which paper, and then these different authors may also be affiliated with some kind of institution here. And then you can think about tasks such as what is the subject area of a given paper. So the general message passing scheme of a heterogeneous GNN looks super similar to what I have presented before, but it has some small differences. In particular, we are now only iterating over the neighborhoods that are bound to a specific edge type. So we first iterate over the number of relations in our graph, then apply a distinct GNN on top of that for each edge type we have in our graph, and then aggregate all these different embeddings we get together to get a final representation of a node. And so the general computation from a homogeneous GNN to a heterogeneous GNN may then look as follows. So on the left, we have the classic computation flow of a homogeneous GNN, which applies like different GNN layers, applies a nonlinearity between them and so on, and then finally reads out these node embeddings. But in a heterogeneous GNN, it would look like follows. So we may have two node types, here denoted as H1 and H2, and then we create distinct GNN layers for each edge type in our graph. And here in this setting, we would have three different edge types. And then the computation is defined by which edge types aggregate information from which source node to the destination node. And so since this looks similar but is still different, we found a way in PyG to automatically convert homogeneous GNNs to heterogeneous ones using the PyTorch Torch.fx package. If you don't know about that, I heavily suggest you look into that, because it's pretty cool and lets you do really crazy things. And so in the essence, what happens here is that we create a homogeneous graph attention network here, which is known as GAT. And that just computes that is a two-layer GNN and that computes like 72 output features and then we call this two hetero function here which takes in the model the node types and the edge types in our heterogeneous graph and that gives you a heterogeneous GNN back that is built on similar principles as the graph attention network and what that does under the hood is that it duplicates the GNN layers for each edge type, so that we have distinct model parameters for each edge type. It then transformed the underlying computation graph such that messages are exchanged along different edge types, as shown earlier. And then we're using this concept of lazy initialization here, which is known as this minus one in the input channels, and that allows us to handle different input feature dimensional dimensionalities without specifying them beforehand. And with that, I'm happy to hand it over to Ramona.

Speaker 2 [29:15]

Do you have any questions before we go into the practical sessions do you think you need anything before? That This is the QR code for the colab notebook for all of those who want to Also play a little bit around with it then just scan that and open it on your computer Okay. I don't see any phones anymore, so I guess you all have copied it. Yes. It doesn't open? It doesn't work? Can you share it over there? Thank you. I think we are A5. I can already start talking about the task we are looking into while you are copying the link. So we're looking into the movie lens data set, which is a data set containing information about movies and users that have rated the movies. And we're going to predict the rating a user gives a specific movie. Is there still anyone who doesn't have the Colab notebook open? Okay. But you have the link, right? Okay. So first, the notebook works a bit faster if you use a GPU, so if you go to edit and And then go to notebook settings, then you will be able to choose whether to use a GPU or not. So you can change that, and then the notebook, the training will be faster. And first, for installing PyTorch Geometric, you first need to know which Torch version we have. And, like, the first cell gives you the option to check for the version. And then we see that the notebook already has preinstalled PyTorch 2.0. And with this information, it's pretty slow. With this information, we can specify the Pytogeometric version we want to download. So there are two libraries. We have the PyGLib, which is the foundational library. And then we have PyG, which we both want to install in the notebook. And then additionally we have, like, three other libraries which we need for the pre-processing and for the explainability, but they're not specific to PyG. I think the installation also takes a couple of seconds. So as I said, like, the goal of this practical session is to learn how to load a set of CSV construct a heterogeneous graph data object from it and then learn a model on top of it for in our case a link regression task so we want to know how a user would rate a movie that the user hasn't seen before here we saw kind of like how the graph should look like so we have like movies and then users and And we have information about which user rated a movie. So this is kind of the graph we want to construct. And like the data set is publicly available. It's like a toy data set created from the group lens research group. And we can just like specify a URL and then download the data set. It will then pop up here, and we can read in. We're especially interested in two different tables, which is the rating tables and the movie tables, and we can load them from the CSV file into a Pandas data frame. We will see that each movie is assigned to different kind of genres, and we also have the title information and the release year of each movie. Additionally to the movies data frame, we also have a ratings data frame, which gives us information about the links between users and movies. So we see which user rated which movie with how many stars. So this is like the base data set. And I would kind of like to ask you to add manual ratings. So there are like two different ways how we can add our own ratings to that data set. Because then in the end we also get personalized predictions. So if any of you have IMDB ratings, then it's very easy to export them and upload them. an instruction. It's, like, 30 seconds. Is there anyone who has IMDB ratings in an account? Okay. Yeah. If you like, then you can do it. For the others, there is, like, we have in total 10,000 movies. So the most rated movies are, for example, Forrest Gump or Jurassic Park. And you can specify how many movies you want to rate. So let's say we want to rate four movies. And then you can enter whatever movie you want to rate. Let's say Avatar. Then it gives you the whatever is in the movie data frame, which sounds similar as the title you entered. If it's one of the titles you want to rate, you can give the ID and your rating. And if you specify a couple of them, then you can in the end get personal recommendation and then also see which of your ratings influenced the predictions. I'm also going to rate a couple of them. But yeah, you can also ‑‑ I can also just upload my ‑‑ how does that work? Okay. I think that doesn't work at the moment. But yeah. Did anyone try to upload the IMDB ratings? Yeah? Almost. Almost. Okay. In the end, you need to add your ratings to the ratings that were given in the data set. I'm still seeing some people adding movies and writing. Okay. Anyone still wants to add more movies? More ratings? Okay. Then I continue with the preprocessing. So we have now the pure table of ratings, including our own ratings. And then we also have the table of movies. We cannot put this as it is into a graph data object because we have, for example, the title, which is text and cannot be processed by a neural network. So we need to preprocess it a bit. So we first have the genre column, which is just one hot encode. So we receive a 20-dimensional vector. And then in addition, we also use a pre-trained sentence transformer model for the titles. This might take a couple of seconds. And we'll download the model and then transform the titles. We then concatenate the genre and the title features to one movie feature vector or matrix. We don't have in this data set any information about users, so we don't know, like, their gender or age or anything else. So we will just use an identity matrix to for the neural network as input features. Then like those were like the features for the nodes. But then we also want to somehow store in the data objects the connection between those nodes. And we do this with the rating CSV, where we have the links between them, and we will construct something called an edge index, which contains the information of both which user linked what movie. And it will be in the form of two times the number of ratings. And what we do first is, as in the table we got, we don't have consecutive numbers. So we have, for example, movie 1, 10, and then 12. But there's no movie in between. And just to have it, like, more compact, we map both the user IDs and the rating IDs to a consecutive range. So we create, like, a data frame containing the original user ID and the mapped user ID, which is consecutive. Same for the movie. And after creating this, we use the map user ID and stack them together and receive a tensor which looks like this. And contains all the edge or link information. So user zero, rated movie zero, and so on. We now are ready to construct the data object. So as Matthias already showed in his slides, this data object would look like we have a a heterogeneous data object, and we would now pass the user features to the user attribute in the data object. Same with the movie features. And then we also add the edges to the object. Additionally, in this case, as, like, the target is the rating, we add edge labels. So what Matthias showed in the table was using an Y on a node, but in this case we're not predicting node type or node classification. We don't have a node classification task, and that's why we use an edge label. We also use one of the transformation functions, which means we transform the directed graph, So at the moment, we only have connections from users to movies, but we want also that the GNN is able to pass messages in the other direction so that we can also learn something from movies to users. So we undirect the graph, which then also undirects the edge label, but we don't want to have the edge labels in the other direction, so we delete them again. And with this, we create a heat-to-data object which looks like the following. So we have 9,000 movies, and we have 100,000 edges between them, and also labels on every edge which represents the rating. And then we also have the reverse edges, but no labels for it. We are ready for training a model, but then before we need to split the edges into a training validation and test set. So we specify that we have 10% of the edges part of the validation set, 10% of the test set, and we split it randomly. Here we see that we have 8,000 edges being part of the train data. So the first data you see is the train data and the second one is the VAL data. So you get back with this random link split function, you get back two new data objects. And what is also new here now is that we have the edge label index. And this one is required for telling the model on which edges we need to make predictions. So in the training data, we make prediction on all edges. But then, for example, in the test and validation data set, we have 10,000 edge label indices, but 8,000 edge labels, meaning we take the edge labels from the training set, use them as the basis, and then predict on 10,000 new edges that we add. Similar as Matthias described in the presentation, we can construct the model quite easily. We have here a specific way of constructing the model as we have link prediction or link regression task. So we first learn an encoder, which learns an embedding for the user and for the movie. And then after passing the graph through the encoder, we apply a decoder, which combines the embedding of a user and a movie and then gives a rating based on the two embeddings. The GNN encoder is, I think, basically the same as the one we showed on the slides. So it's a homogeneous graph with two graph layers, graph convolutional layers. And then when we initialize the model, we can just transform it to a heterogeneous model. And the decoder is then, as I said, it combines. So you give in the edge label indices, which tells you which links you want to get a prediction for. And then you combine the user embedding and the movie embedding and apply a normal linear layers on top of it to get a final rating prediction.

Speaker 1 [48:37]

TrueHitro exists for now over one and a half years. It's pretty stable, but there are some corner cases where it might fail if you use a super specific GNN architecture. But importantly, you don't have to use that. So you can easily convert a GNN layer to a Hitro GNN layer and then build your pipeline in a more fine-grained way, I would say. But I felt this two-heter conversion makes it super straightforward to build most of the classic GNNs and just a useful helper function.

Speaker 2 [49:30]

So, here we see how the model then looked. So, we have, as Matthias also said, we have a different convolutional layer for each edge type. And then in the end, we have the linear layers, which predicts the rating based on the combined embeddings of the user and the movie. And the training of the GNN is like very similar to any other PyTorch model, we first put both the model and the data on the desired device, and we initialize an optimizer where we specify some learning rate, and then we have like a training loop, which just applies the data to the model. And then so here we see the one training loop. So we pass to the model one X dict, which is like just all the features for every note type as a dictionary, and then we have the same for the edge indices. So this way you can access what we specified earlier in a dictionary. So here we specified it as an attribute of the data user, and here we get it as a dictionary. And then additionally we also pass the labels we want to get ratings for. Then we get predictions out of the model, and we combine that or compare it to our actual ratings and compute a loss from it and then update the model parameters based on that loss. It should go pretty quickly. So we see that we have like a training error of 0.8 and a validation error of 0.89 or 0.85 for training, which kind of means that, like, the prediction of the rating is 0.9 off from the actual rating. This is kind of due to that the data set is very small, and we could be also haven't, like, fine-tuned the model a lot. So that's a very basic model. And we can just apply the same to a test, to the test data, and kind of see here visualized for which user, which movie, what we predicted as a rating, and what the actual target was. It's just because otherwise the data frame doesn't ‑‑ it's just for visualization. So now coming back to our own ratings we gave, as we have now trained the model, we can just Like, this cell implements that you get a random movie, which, for example, here is for me the reader, and then you can in the next cell get how many stars you would give to this movie even if you haven't seen it. So it would just generate a random movie that you haven't seen, and in the next cell, we will see that the model thinks that I would give 4.3 stars to this movie. I should probably watch that. Final last step. Since a couple of months, PyG also supports explanations, or we had, like, GNN explainers before, but now we have a general framework for explainers, so we can also look into what influence the predictions and use different kind of algorithms. Now I kind of like want to know why and what influenced the decision that I would like the movie The Reader and would rate it that high. And what I can do in PyG then is use the explainer class. I would like pass that explainer class my model, and then also I would select the algorithm. Captum is also just like an open source library for explainability or interpretability. And we build a wrapper around Captum such that Captum can also be used with GNNs. And Captum has a list of different attribution methods, including integrated gradients. So we can just specify that here and use that integrated gradient attribution method from Captum. We also specify that we want to use, we want to get an explanation for the model output. What you can also do in case of a classification, you can, even if the model output is that it would predict class three, you can still ask for an explanation why it didn't predict class one. And then you would give here not the model, but the class you want to get the explanation for. specify some information about our model, such that it's a regression model. The task level is the edge, and the return type of the model is the raw output. There are different kind of explanation we can get. We can get information on, for example, node features, so how important was the title of the movie for the prediction, how important was the title of the movie that is like two hops away that someone else as a user has rated. And we can get information on the importance of edges. And here I'm only going to look into the edges. And we can just call the explainer with the data we have. And the edge label. And the edge label here in this case only contains the index from my user and the movie, the reader. So you get, like, one explanation for this specific link prediction case. It outputs a heater explanation object, which contains, again, the user and the movie information and also the original edge index information. But then additionally, we also have something called edge mask, which contains the information of how much each of the edge contributed to the final prediction. I mean, as there are many edges, like 90,000, we don't want to look into all of them. So what the next cell does is first what do we do here? So we output, we just concatenate both edge masks to one data frame, and we see kind of that the user, 231, rated the movie 3445, and then we see attribution of this link. So it's in this case negative and means that the case that this user rated this movie kind of like shifted away the prediction, meaning that if there wouldn't be this link, then the prediction would be higher, so then we would, for example, get 4.2 or 4.3 as a prediction instead of 4.1. To make it a bit more clear, we only look into the movies we rated, so I want to know how much did the two movies I rated, I rated The Schindler's List and Avatar, how much did they contribute to the final predictions, and we see that the fact that I rated Schindler's List contributed more than the fact that I rated the Avatar movie. that was the practical part. I can take now some questions if you have questions on the notebook, otherwise we have some a couple of slides left and then we can go later for the Q&A session.

Speaker 1 [59:31]

What is the estimation function? It can be a neural network.

Speaker 2 [59:42]

Maybe the person can clarify the question mean kind of like a typical business graph could be that you just have users and products and that would be already a heterogeneous graph if you have information about whether a user bought a specific product that is I would say kind of a typical business application do you have fun I mean

Speaker 1 [60:39]

It would look like where you have a crossroads as nodes, for example, and then these edges would denote roads, and then you would have edge features for how long it usually takes to travel a certain road and so on. And then you can use that, for example, for navigation and so on, and that is actually what DeepMind is doing as well. As far as I remember, there was some highlight last year or something like that, where DeepMind revolutionizes traffic prediction using GNNs. Yes? More general question. So is PyG actively supporting alternative SLRs like GraphCore IPUs, or do you know that? That's funny because we just came back from a London event where we announced our collaboration with Graphcore IPUs, and Graphcore published a bunch of notebooks on their end, and they have some paper space notebook experience where you get, like, six hours a day for free to use Graphcore IPUs with PyG. Okay. Great. Do you want to continue? No. Okay. So I'll quickly wrap this session up. And so what you have seen here is how the different problems of GNNs, how they are defined, and how you can practically use them. And I see, personally, PyG as a low-level library, like an engine where you can fully customize everything. But that usually also comes with certain disadvantages. So for example, I often get complaints about that one needs GNN expertise to use that. For example, you're still responsible for feature engineering parts and coding parts, how you actually define your graph and so on. And so if you're not fully into GNNs, it might be a steep learning curve, and we're trying to bridge that gap by providing many, many tutorials and so on. But in the end, if you really want to productionize it as well, there's an additional layer of complexity that comes into play. For example, currently, PyG only supports, like, CPU-based graph storages, and you would have to think about how you can connect to a graph database and so on. And if you want to write, like, daily batch predictions and so on, you may want to manage the life cycle of your graph so that you update your graph regularly on a scheduled basis. And then there are some additional questions like production orchestration and so on. And so what we're also trying to do is that we're trying to build a framework on top of PyG, which we named Kumo, that automatically tries to overcome all of these problems at once. here of this new framework is that you don't need to really dig deep into GNNs and just use them if you want to. And so what that looks like in practice would be that you can connect your raw data using one click, and so you could connect to S3, data warehouses, and so on. And that builds on the premise that all your data is kind of stored in a relational database. And so what Kumo then does is that you connect your tables into the Kumo framework, and then you specify how these different tables are connected. For example, using primary keys, foreign keys, and so on. And then we're building this graph from scratch for you in the sense that we connect foreign keys, primary keys, so that gives us direct linkage, but we're also doing some more advanced option here, such as bringing tables that are far away in the graph nearer to the table you want to make a prediction for. And then we think about how we can make formalizing the task as easy as possible. And we're coming up with a declarative language here, which we call predictive query, where You can just say for which entity you want to make a prediction for, for which time frame, and based on what target. And in that case here, for example, the symbol query language would look like that we are interested in the average amount of the balance between now and the next 90 days for each user account. And that's a product where a framework we're currently building that uses PyG as the engine, but gives you, like, a more high-level interface to get familiar with GNNs. And with that, I'm happy to conclude this session here, and we're both happy to answer any questions if there are any. Yes? So you mentioned that it's difficult to deploy these GNN networks. Yes. But could you give, like, a couple of examples, like, compared to deploying, like, deep neural networks? Like, what would be, like, an additional, maybe, challenge or something like that? So I would say the two most problematic challenges is scalability and updating your graph. So usually if we think about large-scale graphs, we speak about 15 billions of edges, for example, and then you can no longer store that in a CPU. You need like a distributed learning scenario and so on. So that is one challenge. And then the other challenge is to constantly update this graph over time, because graphs are usually changing, and you always, if you want to make a prediction, you always want to incorporate the latest information into your GNN. And I would say these are the two biggest problems here to overcome. Yes? Can you also model that somehow on the graph, like when you go like a link should never access or should always exist, so it's like you kind of don't want to learn that part, but we still want to be somehow able to feed in the graph completely? Or do you have to do this more like afterwards, like the post-processing step or maybe a pre-pairing step? Okay, you want to take that? Okay, I'm not fully sure I understood that question, but I may be able to give you an example. So there's one line of research which is called like signed graph neural networks where you can specify like is there's a positive edge Is that a negative edge and then you could specify these rules as part of your input graph? And then the GNN should be able to learn and generalize that rule across the prediction Does it answer your question to some extent? and you just know that certain product combinations are not possible, like if users like product X and Y, then you just know that it cannot have product Z, and that you can somehow... Okay, okay. So there are two adjustments here you would need to make. So if you train that model, that would be you're responsible for defining the negative sampling strategy. And if you're sure that, like, you never sample a certain set of nodes, then there's no point to include that in your negative samples. And then in the batch prediction phase, where you really want to predict GNNs, you wouldn't want, or you wouldn't need to predict on the set of nodes that are, like, not included in this, in the set that you want to make a prediction for. Like if you want to, for example, recommend a store for a given user, then it makes no point in recommending a store that is like in the USA right now. And so you would never just input that into the GNN. So technically, if you train it that way, it would give you probably a low score for these stores as well, but you would still need to compute that, right? So if you know about that in advance, you should include that in your preprocessing pipeline you're still not looking super happy but okay happy to follow up with that yeah the man in the green shirt You know, we know that graphs always change. So that's kind of a fact. But there are also specific libraries that I think also build on top of this that include this temporal dimension explicitly. So my question would be, what's your perspective on including this type of information explicitly into a graph or not? And I mean, it would make sense to also include something like that into 5G. Yes, so that's a very good question. So there exists indeed, like, extension libraries built on top of PyG, and one of them is known as Py2 Geometric Temporal. Sadly, I think it got really successful, but it doesn't look really maintained right now. So we're trying to move some of this to PyG native directly. And I don't want to spoil too much, and I'm not even sure if I'm allowed to talk about that, But I think we are on the Internet as well, but anyway. So what we're building in combination with PyG is like a temporal open graph data set that really drives or tries to drive the efforts of building more temporal GNNs from the community, and in combination with that, we're fully trying to support more temporal graph GNN models into PyG as well. And that is, yeah, it's not ready yet, but we're thinking about that, and we're making great progress for that. But just to finalize, there exists already some GNN models in PyG that can make use of temporal GNNs, and we also support, like, advanced data processing such as temporal Sampling and so on so that is all ready to use But I wouldn't say it's like 100% done probably it's never done, but we're working towards that Yes built around PyTorch, how easy would it be to swap the computational backend for something else to be able to say okay I have a tensor for my parameters for the edges and I don't want it to be a PyTorch model in the end but a model built with, let's say, tensorflow or PyTensor or JAX or whatever. Yeah, how hard would it be to... Critical question, okay. First of all, we have been moving away from our own external libraries, and so we're mostly fully relying now on native PyTorch, which helps us tremendously in the way we can export PyG models. So we can use, like, ONNX, we can use TorchScript to export our models, and that is all fully supported. From the perspective of supporting different deep learning libraries, I would say PyTorch is so coupled with PyG that it's, like, super hard to move away from there. And so we're really relying on the success of PyTorch here, I know. The one thing I would say is that it gives us many advantages to just rely on a single deep learning library, because we can use, like, new features introduced that directly into PyG. And so we're also working closely with the PyTorch team, and as a result of that, like, They heavily improved their sparse matrix support and so on. And that is something we cannot really get if we're relying on multiple backends. So then we are just limited by the union of features of all these different libraries. And I think that's like an important consideration to make. I probably wouldn't change that direction if I would start from scratch. And so, yeah, we're relying on exporting models if you want to use, like, TensorFlow or so on. Yes? Okay, very good question. So we had a workshop last September where we announced, like, PyG 2.2, and that release comes with a clear interface to connect to different graph backends. And so what we do is we provide interfaces for a feature store and a graph store, and And that basically allows us for independent scale-out, independent of CPU memory. And so if you implement that interface of GraphStore and FeatureStore, for example, for Neo4j, then that would mean that we can directly sample subgraphs and sample features from that graph database, and that would produce those PyG mini-batches. And so what we do right now is we are working with several graph database providers, such as TigerGraph or Kuzu. And they have been both able to implement that interface. And I'm not sure, previously I talked to someone from Neo4j, but we have that on the roadmap as well.

Speaker 2 [76:57]

So you've already mentioned the whole set up as broadly as this can be.

Speaker 1 [77:12]

using the graph neural network setup with all the cross-task kind of tasks, let's put it like that, being multi-target prediction or having no metadata, no description in pictures or any sort of other different non-tabular inputs to combine those approaches. Say, in order to predict not only

Speaker 2 [77:50]

of the year.

Speaker 1 [78:21]

So, that is like fully supported. It just needs to be written by you. And so, what that means is, like, you have full control, full customizability of the training loop. Like, if you want to predict, like, multi-target, then you just output two predictions instead of one and train that together, right? So that is super, super straightforward. Regarding your question about input features, that is a tricky one, because you always need to encode your features, right? And you can either then, like, if you want to include, like, image covers of all the movies, for example, you can either, like, use a pre-trained ResNet, compute the embedding, detach the embedding from the computation graph and just use them as input features, Or you could jointly train your CNN and GNN together. Yeah, you could just stack them. The problem with that is that this likely yields you a GPU memory error. Because what we see, for example, in this demo is that we are operating on, like, 1K or 10K movies for a single prediction. And that would also mean that your batch size in your CNN would be around the same, right? Yeah, but theoretically, I mean, of course, with unlimited GPU, practically, I wouldn't do that. I would split the task, but theoretically. Yes. With unlimited GPU, all of that's supported. Cool. Thank you. Yes. You have a kind of a practical question. So I've tried to practice before, and one thing that got me confused is, so I tried So I try to do, to split the data differently. For example, if I do three experiments, each time I split the data differently. Then I look at the graph representation, every time it's completely different. So I'm wondering if this is because, or maybe it's because my problem doesn't require a graph, or it's just something natural that's happening. So I'm just wondering if that can happen, Okay, not fully sure I can help you with that, just on a high level thought, but so the problem is that if you have a graph data set and you split that, your graph data set looks different afterwards? Yes. Every time you run the split? Yes. And that is with a fixed seed, so the number- Yes, but each time it gives different training data. And that is a node-level task or a graph-level task? It's a graph-level task. Okay. That should be the case. So maybe, I don't know, maybe the seed isn't respected by some random operator or something like that. Otherwise, I'm not sure. Happy to help you out on Slack or GitHub with that problem. I think it's solvable. Yes? I have also a question, but maybe more regarding QMAP, because more from a prompt engineer perspective. We had an example about this.

Speaker 2 [82:04]

individually

Speaker 1 [82:06]

and we had individual results but let's say we want to have a movie night and people rate movies differently but we don't want to end up with a trailer night so we just want to decide which movie to watch like a collective decision can we make this with the framework as well interesting question um so like the first thought that comes to my mind is that you're just predicting the top K movies for each user and then try to learn how to combine them, right? But if you think about something more fancy, I would say you can add these clusterings as part of your training pipeline. So in the sense that you have some kind of hierarchical graph, so like a user rates a given movie and then you have like a cluster of users that rate a given movie as part of a group or something like that. That would be something that would be doable, but I'm not sure if it's really practical to do that. Sounds like hyper-etch. Yes, so that would be related to a hyper-etch CNN, yes. So it's not just averaging, it's already... So the way you would treat that is that you somehow learn how to aggregate the information from each user into some kind of cluster decision. Exactly. And ideally you want to train that as well. Might work, might not. That's something we haven't looked into it, to be honest. So if you define the clusters as part of your graph, then we can do it, yes. Okay, then if there are no more questions, then thank you very much for your very interesting... Thanks for attending.