Teaching Neural Networks a Sense of Geometry

Speaker 1 [00:02]

Yes, I work at Ericsson in Stockholm. Ericsson basically builds the mobile infrastructure, so making sure that everyone's phones can connect to the internet. There's many interesting machine learning challenges connected to that, but I also pursue a PhD in the field of topological data analysis and that's what I will talk about today, because last fall I was feeling that I sort of needed to take a step back and be able to explain a little bit what I I was doing and what the field is about and how it fits into the larger machine learning ecosystem. And this presentation came out of that and has been so far a good way to sort of start questions and discussions around this topic with people from different fields. So let's see how it goes today. And I want to start by talking about baboons and geometry. So if we present a few pictures like these to a baboon and ask it to point out the outlier, it will not be able to point at this picture. Yet we as humans recognize that there's something different with this picture because it has a different shape, a different geometric regularity. But how do we even ask a baboon something? So researchers can test the intelligence of baboons by letting them answer questions on a touch screen like this. And baboons are pretty smart. They can ace pretty advanced cognitive tests and they can recognize basic shapes like like these ones. But somehow they seem to have a blind spot when it comes to recognizing geometric regularities, such as distinguishing the rectangle from the less regular polygons in this example. Whereas we as humans, on the other hand, may seem to have what could be called a native sense of geometry. And these researchers went to quite some length to prove that it's actually a native property. So they gave this kind of test to children in kindergarten. They gave this kind of test to people different countries to people from an indigenous people just to make sure that this is something native and not just something we acquire through culture or formal education. So if you're ready to believe that, that the geometry is something that sets the intelligence of humans apart from that of other animals and maybe long-term goal is to build human level AIs, then maybe geometry can be interesting. So what about machine learning? What if instead of training baboons we train neural networks. So what these researchers did was to take a pre-trained convolutional neural network and they passed these kind of pure geometric shapes through the network and looked at where they ended up in the internal representation space, in the embedding space. For example, checking if they fed similar shapes, would it end up in the same area of the internal representation space, for example. And that was not the case and maybe indicates that shape is not really used in the way at least those kind of networks organize their internal information. Maybe they don't have any concept of shape. And that seems to be corroborated by quite a lot of research. If we have a cat and we give it the skin of an elephant then, I don't know about you but I would still think of this as a cat that sort of tries to disguise as an elephant but the network is happy to believe that this is an elephant and that indicates a strong bias towards considering texture and not considering shape when making its decisions. Now, in many applications, that kind of bias is fine, but I'll try to argue that sometimes we actually want the machine learning algorithms to consider shape. And if we want that, then one way is to teach them a little bit of math. So in math, shapes are studied in the field of geometry that we may remember from elementary school. So in geometry, we look at properties like distances, curvature, angles, properties that are very sensitive to changes in the underlying object. So if we start with the ball in the top left corner here and we successively deform it in small steps, then actually all these steps will represent the change in the geometry of the object. But shapes are also studied in the field of topology, and in topology we look instead at properties that don't change even if we twist or stretch or bend the space. So we get properties that are very insensitive, very global, but thus also very robust. For example, in this picture, it's only in these steps that we have a change in the topology of the object, only if we tear it apart or puncture it. So geometry has been studied for a few thousand years, and topology for at least a couple of hundred years, before there were any computers around. But more recently, there has been a focus on actually developing algorithms to try to compute geometric and topological properties. And in the field of topological data analysis, TDA, that we'll talk about today, we look more specifically at spaces that arise in the context of data analysis. So in other words, we want to talk about the shape of data. And while topological data analysis has topology in the name, the methods used in this field really serve as sort of an interface between topology and geometry, so we can reason about both local and global properties. So at some basic level, we try to tap into this ancient knowledge in math and see what happens when we adapt it to data, to machine learning. And maybe that's something that could be interesting more generally. If I were to generalize a little bit, I would say that the algorithms that we use and love today are very impressive and built on very interesting math, but maybe built on math from relatively few fields, relatively few mathematical pillars. And it's kind of interesting to think what would happen if we include a little bit of math from new fields in this mix. And that is something that the WASP research program is exploring. It's a very ambitious research program that we have in Sweden around AI. And one of the things that they are exploring is this intersection between math and AI. So if you want to do research in machine learning, come to Sweden, check out this program. But for now, let's talk about what we mean when we talk about the shape of data. So in the introduction, we looked at pure geometric shapes. But of course, in reality, we would like to be able to say something about the shape of real data. What's the shape of this point cloud? What's the shape of the second point cloud? we compare them. Maybe to the left I have a training set and to the right I have a set of simulated samples generated from the training set and I want to compare them. Maybe intuitively these two point clouds to the left and to the right don't look so similar and these maybe look more similar because they have a common clustering structure, but how do I know which properties to look for and how can I compare point clouds that may not have the same coordinate system, maybe not the same number of points or the same dimension? So here's an example of a method used in TDA. So we'll start with a point cloud and we'll take it to be in two dimensions for the sake of illustration. Next we will select a threshold and then we will connect all the points that are a distance smaller than that threshold. Doing so we get the graph and then we just count the number of connected components in that graph. So let's take an example. If we start with a low threshold, then no points are connected yet, so we have nine connected components for the nine individual points. But if we increase the thresholds, then we have at some points are now being connected and we have in the graph five connected components. If we increase the threshold, we have more points being connected and fewer components. And in the end, there would just be one big component because all the points will be connected. And this information about the connectedness of the space at all thresholds can be summarized in the dendrogram. So if we looked at the lowest threshold that corresponds to the vertical dashed line here, and it intersects nine horizontal lines for the nine components that were alive at that time. For the next threshold, it intersects five lines. So the dendrogram allows us to see for each threshold how many components were alive, but also how smaller components merged into larger ones if we follow the tree structure. And connectedness is an example of a topological property that was one of the examples in the introduction. We had this ball and then we split it in two and merged it back together. And this is a fairly common recipe for methods in this field, that we take a space like a point cloud and we look at it at different thresholds. And for each threshold we build a graph or sometimes another kind of combinatorial object, but something from which we can read out the topological property. So we get this multi-scale, multi-threshold view of our space, where we see how topological properties vary as we vary a threshold. And maybe the dendrogram looks familiar, and maybe that's because it's also used in hierarchical clustering, where it's often used to select a threshold. And then once we have a threshold, we can go back to the input space, to the point cloud, and we can look at the corresponding components, only there we don't call it components, we call it clusters. we get a clustering for each threshold. And that can be useful, but in TDA we would rather want to say something about the whole space, so maybe let's keep the whole dendrogram as it seems to contain quite rich information about the connectedness of the space. And then maybe if we have two spaces we can sort of compare them, something about their shape, something about their connectivity, through their dendrograms. It turns out that we just have to take one more step before we can do that and convert this into another mathematical object called the barcode, which will track when topological properties appear and for how long they persist. For example, we had one component here that appeared at threshold zero, persisted for a little while until it was merged into a larger component, and so we get a short bar for that. Same thing here for a second component, and if we do this for the whole dendrogram, we get the whole set of bars, the barcode, and this is something we have a distance on an optimal transport type distance so that now if we have two spaces we can compute their dendrograms and then their barcodes and we can compare something about their shape, about their connectedness through the barcodes. And this is one example for one topological properties and the methods in TDA can also be parameterisable to capture different aspects of geometry, and if they are included in neural networks, then these parameters can be learned based on data. Sometimes we don't want to extract some shape information from data, but we would rather want to control the shape of some data, maybe of some embedding space, that can be done too. And finally, when we took the example of a point cloud, the methods can apply to more general types of inputs, such as graphs and images. But now for the remainder of the presentation, let's look at some examples where we can use these methods. So our brains are full of neuronal and glia tree cells, such as the ones in this picture. And neuroscientists need ways to compare the shape of these cells and relate their shape to the role they play in the nervous system. And for example, in the project to which I collaborated, that was driven by a team of neuroscientist at IST Austria and that we collaborated to from KTH. We looked at the shape of microglia cells in the brains of mice and whether, for example, that shape differs for cells that come from different brain regions or for healthy versus disease mice and so on. But to do all these things, we first have to take these physical biological things and convert them into data. Now, we could open the brains of some mice and make some measurements. Maybe I count the number of branches, or I measure how long they are. I could also put these things together in a feature vector. Or maybe I take a picture of the whole cell, maybe in 3D, so that I get a voxel representation of it. Now, the first approach might not be rich enough. Those measurements that I choose might not be the ones that best reflect the variability between cells. But the second approach might be too rich, it might be too high-dimensional, and it does not include the prior knowledge that the neuroscientists have about these objects. For example, that they are invariant to a lot of things, translation, rotation, etc. But also, for example, that the thickness that we see in this picture is mostly the result of the handling done in the laboratory and is not so meaningful biologically. So in theory a neural network could learn this aspect if it has a lot of label data and the right structure, but in general by incorporating some kind of invariances, some kind of topological priorities in this case, we can often make training more efficient and maybe even the resulting network more robust. But at an even more fundamental level, somehow even if we represent the data with a small feature vector or with a very large vector representing it as an image, we somehow still make the same assumption that the data can be represented as a vector at all, that it is Euclidean data. And that is not always the only choice. So in this project, we model the data topologically. So we look at how the branching structure changes as we move the extremity towards the root of this cell. So in the introduction we looked at connectedness of a point cloud as we varied distance and here it's the same but we have a tree and sort of the threshold is how we reveal this tree starting from the extremity moving towards the root of the tree. But we get the same kind of descriptor, the barcode for the tree and that is the lens through which we see the data. And it's a mathematical object but it's not a Euclidean one. And that's something that I found interesting moving into this kind of project that sometimes with some math we can come up with new types of data. So I think data is anything on which we can have a language, something on which we can reason in some precise way. So if you give me a class of objects and you can measure distances between those objects and maybe you can add two objects together, maybe even have a law of large numbers, things like that. If you have these things or even just some of these things then I would say that what you have is data, and if what you want to capture is sort of the shape, the morphology of some underlying object, then representing it with methods from TDA can be interesting. And in that sense, this can be related to the trend of geometric deep learning that sort of encourages us to look beyond Euclidean geometries. But once we have data, then we can do data analysis. And just to take an example, here we have to the left a dimensionality reduction of these cells seen as through their shape, and we can compare that to the physical layout of these cells in the different brain regions, just as an example. So let's stay in the brain. Here we have a picture of a neural circuit, and let's task ourselves with delimitating the individual neurons in this picture so in other words we have an image segmentation task. So in a standard image segmentation task we may have a bunch of pictures coupled with their segmentation masks so a human annotator has taken these pictures and colored the different regions corresponding to the different neurons with different colors. Then we can feed our pictures through a neural network can have different architectures but that will output a probability or a classification for each pixel saying which segment it belongs to. And that's something we can compare to the ground truth and construct a loss function on. But let's think about what the maybe classical loss functions will encourage the network to learn. So it's a purely local loss function in this case, like we go pixel by pixel, we look at whether it was correctly classified, and then we just take a sum or an average. So there's nothing in this loss function that can account for more global properties like the shape of these segments. So maybe the neural network will output a picture that looks like this, that it thinks looks good because it has a low loss and most pixels are correctly classified, but to our human eye it might not look satisfactory because it has this fragmented shape with a few pixels belonging to the different classes here and there. So what we would like is perhaps is a way to tell the neural network to not just care about things pixel by pixel, but also care about the shape of the segment. For example, that we should have a few consistent segments such as in the ground truth and not this fragmented behavior here. And that is something that can be encoded with the methods from TDA. So if we produced one of these barcodes for the ground truth and another one for the prediction of the neural network, then this can be compared and included in the loss function of the neural network so that the neural network can be penalized if it would produce outputs that deviate topologically from the ground truth. Maybe arriving at pictures that look more like this. So for last example, let's look now at the shape of maybe the internals of a neural network or an embedding space. So I think embedding spaces are everywhere in machine learning, even if we think of a classical classification problem, we can see it through that lens, like what we want is to construct an embedding space where the different embeddings forms clusters corresponding to the classes, or if we have embeddings of images like in the previous talk, then we want the distance between those embeddings to somehow correspond to how similar we perceive them with our human eyes. So maybe most or all problems in machine learning can be seen as sort of constructing an embedding space with the right geometry and maybe that can be a promising area for TDA as well because it's all about the shape of these kind of spaces. So if we have an autoencoder, then we have an embedding space in the bottleneck here because it will contain low dimensional representations of the input that will in some sense contain as much information as possible from the original input. And the classical autoencoder may be trained with a reconstruction loss, so we just take a sample, we pass it through the network, we look at what we get at the other end, and then we just compare the input to the reconstruction. And similar to the previous example, this will be a purely local loss, so there's nothing in this loss function that can account for global aspects, that can account for the shape of the embedding space. So if we would have a data set that would be in three dimension and just happens to have the shape of a mammoth, don't ask me why, and we would want to sort of flatten this mammoth, we want to reduce its dimension to two, then these are two propositions which are both legitimate and are optimal solutions even to some dimensionality reduction optimization problem, but the problem has been formulated differently. Maybe in the middle there is only this kind of local loss function and to the right a topological loss component similar to the previous example has been included to try to make sure that the shape of the embedding space is somehow aligned to the shape of the original space. In this case, maybe just making sure that the flattened elephant is still connected as the original elephant was. So whether this matters or not depends on the applications, but I think neural networks will never be smarter than the loss function we assign to them, but if we want them to actually consider something about the shape, we can tamper with that loss function, maybe with methods from TDA to encourage the network to consider these things. Or we can, already at the stage of modeling our data, incorporate some of these variances. But none of this would not be worth so much if we couldn't do it in Python. So let me just talk a little bit about some Python libraries where this can be used. So if you want to compute maybe barcodes of some data, I would recommend you to start with Giotto TDA, which interfaces nicely with scikit-learn. So for example, we can build pipelines where maybe we extract barcodes from some data, and then we feed them to some of the machine learning algorithms in scikit-learn. And doing something like the example in the introduction can be as easy as this, initializing a persistence object that's kind of how we see the data space for which combinatorial object and the distance that's how that's the distance we use to merge the points together including distance and then we can just fit transform away from there if we have a point cloud pc if however we want to include these methods in a new network then i would recommend to look at torch topological at least if you're using pytorch and something like the last example could be done in this snippet. This comes straight from the great documentation of this package. So if we assume that we have we already have an autoencoder model that we pass in to the constructor here, and then we also initialize our persistence object as in the previous example, and we also choose a loss function that is compatible with these barcodes. Then in the forward pass, we assume that we get a bunch of high-dimensional vectors, Maybe our data set or a large batch of it. We pass it through the autoencoder model to get the low dimensional Embeddings we compute the barcodes for the originals and the low dimensional embeddings and then we can apply loss and What is interesting here is that when we then call backward to get the Back the back propagation started it will sort of back propagate all the way through the barcode to the data so in this case to the low dimensional embeddings, so that in the next step of the optimizer, these will be changed to sort of have the right shape according to the loss function. So that's all I had. Thanks for listening. Happy to take some questions.

Speaker 2 [23:24]

Thanks a lot, Jens. That was a great talk and surprisingly a little mathematical algebra. We do have some questions. What can one say about the void around the points? Can one detect outliers?

Speaker 1 [23:47]

Yes, it's an interesting direction to maybe an outlier is something that sort of changes the topology of the space in some sense, and then these methods can be an interesting direction to detect that as an alternative to maybe other anomaly detections.

Speaker 2 [24:12]

Thank you. How do you take a segmented neuron image and get the points of interest to create the TDA barcode?

Speaker 1 [24:27]

Right, so that's released to this one. Yes, so so this in this case it's Well

Speaker 2 [24:34]

It is

Speaker 1 [24:36]

We can see that so it's a little bit different from the point cloud example, but we can see that These segments basically well we can see the images as a graph and with the Or under the different Segments as as graphs and seeing the connectivity of them in a sense

Speaker 2 [25:06]

A little bit connected to that, how do you handle these barcodes in code? What kind of object do they resemble?

Speaker 1 [25:15]

Yes, it's a set of points in the plane, basically, so it can be easily encoded, but the subtlety is with the distance, which is not the Euclidean distance, but something that has to work with these sets that can also have different cardinality, different number of bars, but that's something that these packages can handle, can compute these distances and so on.

Speaker 2 [25:48]

More questions coming in. Do you think it could be applied to medical images, e.g. x-rays?

Speaker 1 [25:56]

Yeah, I haven't seen really any examples in that area, but somehow, yeah, I think nature likes geometry and topology and likes to create these kind of patterns that are sometimes possible to capture with these methods.

Speaker 2 [26:13]

Another interesting one regarding use cases. How can topological encoders be used for human pose detection of the computer vision applications?

Speaker 1 [26:27]

I have not really seen any example for human pulse.

Speaker 2 [26:38]

Let's see, I think we can do some more is the data used usually in the form of point clouds voxels rasterized data

Speaker 1 [26:48]

Yes, so it can be point clouds, it can be images or voxels or graphs or basically there's a sort of a modeling choice there to encode these into these kind of combinatorial objects that we have a filtration on. So there's also a level of possible innovation at that level, but yeah, most common I guess is images or 3D images and point clouds, graphs, time series as well.

Speaker 2 [27:31]

Okay, now a nice one. How do you choose prop activation function for this case?

Speaker 1 [27:37]

Sorry, how do I choose?

Speaker 2 [27:38]

a proper activation function.

Speaker 1 [27:38]

Prop. So the activation function, like in the neural network, can be, this would typically be maybe the first layer of a neural network, either to extract something or to control the shape of the input, and then you can compose it to more classical neural network layers and have your...

Speaker 2 [28:15]

Last one. Would you be able to isolate, classify logical or physical structures, for example, in apostrophe composed of dots, laser detection, from a global noise? I don't get the question. Sounds interesting.

Speaker 1 [28:33]

Yes, please come and talk to me about it.

Speaker 2 [28:35]

Okay, so maybe you can ask it later. So thanks a lot for being here. That was a really interesting talk, and we now have a coffee break. And if you have more questions, please stay here. I think Jens will be around. And I think you can share the slides, can't you? Yes, sure. Okay, thanks a lot.