Gaussian Progress

Speaker 1 [00:04]

So hi everyone, welcome to my talk. I'm going to talk about this thing called Gaussian progress, which, a bit of a pun intended, is the most utmost normal topic I could come up with. And Gaussian processes, I think, are sort of this wondrous thing, but when I was thinking about it, the feeling that you often get is that, you know, when you're a machine learning professional, you know, there's so many algorithms out there. Like, this might be your feeling on a day-to-day basis, and there's this pressure that you might need to keep up with all this new tech, and that can be really, really demotivating. And what I want to do now is this talk is not about downplaying this feeling. I think it's just there. But I do want to show a really, really lovely hack and that is understanding a mother algorithm. Many, many algorithms tend to share a component and if you can just really, really understand what single component is being used everywhere really, really well, then everything else might become easier to understand. And the Gaussian, I think, is like one of those core concepts. Because it turns out that if you appreciate what the Gaussian distribution can do, then there's lots of algorithms that are just much, much easier to grasp. And this talk is an attempt at explaining the power of the Gaussian by kind of stepping up the ladder of complexity of algorithms. So today I will literally introduce Gauss, and then I will explain the Gaussian trick for classification, clustering, and outlier detection. I will then show how you can actually make neural networks just get properties that might make them better by using the Gaussian. And then I will attempt to life code a Gaussian process. This will be tricky, but I think we will be able to manage that. So to introduce what Gauss is, there's a bit of a story about a preschool. So you have to imagine there's a town, there's a preschool, and there's a teacher, and the teacher is lazy. The teacher doesn't really feel like teaching any of the kids anything, because the teacher really just wants to do reading the newspaper. So what the teacher then says is, you know, hey children, please add the numbers 1 until 100, and that way the children are busy doing arithmetic, and then the teacher can read the newspaper, and that was sort of the plan. But then there was one student who sort of looked at the task and sort of said, all right, this is number 1 and 2 and all the way up to 100. But what I can also do is I can just rewrite it. I can take the number 1 up until 50, and I can take the number 51 back to 100. And if I reorder the numbers this way, one thing that you then see is that, you know, 1 plus 100 is 101, 2 plus 99 is 101, all the way up until 50 plus 51 is 101. And just by reordering things a little bit, 101 times 50 is a whole lot easier than sort of doing all the arithmetic of all the additions. So there was a student in the classroom who figured this out. And once the student figured this out, the student got bored and figured, let's think about this. And the student said, ah, another thing I could also do is I can also say, take number 1 up until 100, and 100 up until 1, then you also get like 101 appear at the bottom. But now you've got 100 of them, even though you're interested in 50 of them, so then you have to divide by 2, and it's also the same number. And you can also say, does this generalize? If I add this all the way up until n, can I still apply the same trick? And the answer is yes. It's the exact same idea, but by telling you this story, it's a lot easier to remember, and then you sort of realize, ah, this is what mathematics is supposed to do. Math is kind of like a compiler, but for numbers in a way. And the reason why it's relevant to share this story is this actually happened. And the guy who, or the kid, I should say, who sort of did this, that kid's name was Gauss. This story is actually attributed to the guy who came up with the normal distribution. And the thing that I like about this story is just like how by thinking about this building block, you can sort of do arithmetic better, by understanding Gauss, you can actually understand machine learning a whole lot better. The one thing that I'm always a little bit bummed out by, though, is like, this is a really cool story, and this is inspiring, and it helps you understand what is happening. But when you open a math book, or a book on any topic that's like this, they typically explain it by showing you this dry formula, which is a bit of a bummer. So what I'm going to try and do now is I'm going to go in-depth into this Gaussian thing, literally, but what I'm going to try and do is omit this and focus on this. So that does mean that I'm going to sort of ignore a whole bunch of details, which in the end are important. But what I'm trying to achieve here is intuition. So there's a certain shape. This is the normal distribution. And the idea is like there's lots of stuff in the center and there's some stuff on the sides. But a lot of stuff in human nature sort of resembles this in some way. Not perfectly, but like human height. There's a sort of a Gaussian-like distribution and human weight, tons of stuff. And again, this is the shape. This is the formula. But why is this useful? Well, you know, because you can do some clever hacks with it. Suppose that I've got two of these Gaussian distributions. Let's say that I have the height of people, and there's one subgroup of people who have distribution A, another group of people who have distribution B. If I have some person who has height xi, then looking at the distribution of, or the difference of likelihood between one Gaussian and another, you can sort of attempt kind of a classification thing. And this is a concept we're going to have to remember. You can compare two Gaussians to say, hey, if this Gaussian represents one group of people in terms of height, and this other Gaussian represents another group of people, you can really compare the two groups by looking at the likelihood. And another convenient thing that you can do with a Gaussian is you can at some point say, you know, there's like a lot of mass in the center, but the outskirts, there's like at some point this threshold where we can say, hey, that's an outlier. At some point, there's a moment when the point is so far away from the bell curve, you literally can't hear it ringing anymore. And that's also kind of a useful concept. So let's keep these two concepts in mind. And what I'll now sort of try to do is just show you that with these two concepts and also in higher dimensions, you can really solve most problems. I should say, though, there's also like a two-dimensional distribution formula that's out there. So to skip this math, what I'll just show you is how this algorithm can sort of help you. When you have a couple of points, sort of in a point cloud, a Gaussian distribution in higher dimensions essentially says, you know, in some higher space, some blob of points, there's some sort of mean. There's always some sort of highest point. And this can be in higher dimensions, which you kind of see here, right? So it's like a mu1, a mu2. It's a little bit of mathematical notation, I apologize, but this is the idea. This is something that describes a Gaussian in a higher dimension. Now, what's equally important is not just the means, but also the spreads. So, like, there's a spread on the x-axis, one axis, and there's a spread on the x-two axis, right? That's also very definitive of a Gaussian. And the final thing that makes it very definitive is this notion of a correlation, or sort of covariance. The direction in which the Gaussian is pointing, that's also depicted in this covariance matrix. And the idea is, if I have these numbers, then I have this shape well-defined. And this is what a Gaussian essentially is. And, you know, the correlation can flip, so you can also be pointing in a different direction. And you can also say there's, like, no correlation whatsoever. But you can express a Gaussian by knowing the mean of the distribution and by saying something about its variance. That defines the Gaussian, if you will. So let's see if we can use a Gaussian to already make another algorithm better. And I'll just take k-means as an example. So k-means, if you're unfamiliar with it, the idea is I have some sort of data set, and I would like to cluster it. So the way that this typically works is you say, ah, there's probably three clusters. I want an algorithm to automatically find these clusters. And then the idea is you start with a couple of centroids. These centroids will then sort of go out there and look for its nearest neighbors. So each and every point is closest to one centroid. We allocate all of these points to a centroid, and then we take the mean of the points that are sort of assigned to the centroid, and then we sort of move the centroid in a direction such that it's sort of in the nice little middle of all the different points. You know, then the means shift a bit, and then you can repeat this, and at some point this algorithm is going to converge, and then you can sort of say, okay, all the points that are closer to one centroid belong to one cluster, all the points that are closer to another centroid belong to another cluster. And there's a couple of cool things about this algorithm, like it will always converge. At every time you do this thing, the total distance between the centroids and its neighbors will decrease, so you can prove that this actually will converge. But it's also a very nice two-step approach. Like, there's one step where you say, I'm looking for all my neighbors, and there's another step where you say, hey, given the neighbors that I have, I want to make, like, a good move in the right direction. But this algorithm doesn't make much sense, if you think about it, because there's a lot of stuff that's iffy about it. Suppose that I have this as a starting point, and then I have these two centroids. I mean, sure, what I can sort of do is I can say, let's divide them up. But for all of these blue dots that I have here in the middle, I don't think a hard allocation to either of these clusters makes sense. Because the idea is, it might be soft clustering at best, but at this particular point, I think it's a really weird idea to say, you, member of my population, belong to one cluster and one cluster only. It might make sense to say, no, you can be like part of one cluster for like 30%, part of another cluster for like 70%. So how about what I'm going to do is I'm going to replace the notion of a centroid and just put a Gaussian in there. So what I'm going to try and do is instead of saying, hey, there's a center that I'm sort of following and all the points belong to it, I'm just going to do the exact same thing, but I'm going to be moving a Gaussian in this sphere. And the nice thing there is, because I have a likelihood distribution, because, you know, we can map to some sort of likelihood, then maybe I can do soft clustering, and I get some other benefits from this as well. And there's some text saying what I just said. So the idea is you start out with two Gaussians, they can sort of be out there in space, and what you're then going to do is you're going to say, okay, we're going to sort of try and figure out for every single point to what Gaussian distribution the point might have belonged to. And in this example, I hope you can see from the beamer, but there's like a lot of red dots. Those are obviously closer to the red distribution than to the green one. And there's a couple of bright green dots, which are obviously part of the green distribution. And there's some sort of teal, turquoise, bluish dots sort of in the middle, where we're a little bit unsure. And then the idea is that now that I've sort of figured out which neighbors are sort of associated, what I can do now is I can say, all right, then using sort of a weighted moving average, if you would, that's what I'm going to use to sort of move the Gaussian in a new step. And it's, again, just the same as with k-means, except I'm moving a Gaussian around. That's the difference. And then the nice thing is what you can do is you can sort of move this around, and then at the end you can say, hey, again, I've sort of clustered my things. And what you can then do is you can sort of say, well, there's this one member that's not really part of any cluster. But this is a way of clustering that also allows for covariances to sort of say, hey, it's not a perfect circle. What I'm doing with this k-means is sort of a different shape. And this is basically k-means, but in my opinion, just better. And it's just a consequence of the fact that, you know, there's this distribution that we can go ahead and use. And again, this is it. The intuition is just this. Now, there's also a GIF that shows how this sort of works. You can get this from Wikipedia. But the downside is if you really want to sort of look for a Gaussian distribution or this is called a mixture model, if you're going to go ahead and Google what this is like, I believe it's on the next slide, what you're going to get is this formula and this formula. And if you understand Gaussian distributions, this is great, because this is sort of a deep dive. Once the intuition is there, you can use this to really fundamentally go on the lower level to sort of figure out what exactly is happening and how this algorithm might be efficient. But if you're trying to sort of learn how to do clustering, but better, and this is the first thing you see on your retina, like, not only is this going to be confusing you and demotivating you, but it's also sort of distracting you from what I think is the coolest observation about this, and that is that this is hugely applicable. So what I've just shown you is that we can use this for clustering, right? But this has so many ridiculous applications, you can even make neural networks better with this. And I'm just going to have a bit of a rant on this. So remember that I said this in the beginning, that you can sort of say, hey, if I've got like one Gaussian from like one class, and I've got another Gaussian from another class, you can sort of compare the likelihoods. And that way, you can sort of say, ah, it might be more part of one class than another one. Well, I can sort of fit some Gaussian mixtures on a class. And then using the Gaussian mixture model, which is typically used for clustering, you can also do classification. And the same thing will hold for outliers. So suppose I had this data set. And again, it's a bit contrived. But what I could do is I can say, I have some sort of group A, I have some sort of group B, I have some sort of group C. And then the idea sort of is, well, take everyone from group A, take them separate, train a Gaussian mixture on that, that will sort of give us an impression of, hey, how are people from group A distributed? Can we sort of define that? You can do that for group B, you can do that for group C. And that means that for any new point within the realm of X1 and X2, you can sort of say, what's the likelihood of it belonging to group A? What's the likelihood of it belonging to group B? and what's the likelihood of it belonging to group C. This is the one-dimensional representation of this. This is a two-dimensional representation of this. And oh yeah, by the way, if you do this, you kind of get outlier detection for free. The cool thing is, at some point you can say, you know, this point is so far away from any distribution of the classes I've seen that I can simply yell, this is an outlier. This is not something that I can do. And the funny thing is, this is a huge safety mechanism in machine learning. There are very few algorithms out there that have a way of describing, look, this data point is so far away from anything I've seen, I am now maybe saying that I don't want to automate this decision. Being able to do that, being able to say, hey, this is maybe an actual outlier, it's really far away from anything I've ever seen, that's a really, really convenient property. And this is just fitting a bunch of Gaussian system data, but already we see that it can be used for clustering, for classification, and outlier detection in one. There are very few algorithms that actually do this. And by the way, this is all probabilistic. So if you're into probability theory, you get all sorts of properties here that you can really use. And if you're into maths, because everything is a Gaussian here, you know, there's lots of convenient tricks that you can do, and the math is quite solvable. And the final detail, a Gaussian mixture can fit pretty general shapes of data. You might have to tweak the parameter that says how many Gaussians I'm going to throw into the mix. That's a fair problem. That's a bit of a detail. But very weird and arbitrary shapes can be fitted with this. If your data is sort of not linearly separable, this trick will totally work. And I'm not just saying this. I actually implemented this. So what I have here is an example of the two moons data set from scikit-learn. And what you can see is that, I hope you can see from the beamer, but what I've essentially said is, hey, I'm just fitting a bunch of Gaussians here. And once it's done fitting, this is sort of the likelihood distribution. So you can see that there's like some peaks here. So those are the peaks of the Gaussian distribution, if you would. and together they can form a really nice arbitrary shape but once this is fitted I can say there's some sort of threshold around it and please do outlier detection and a yellow point here is considered an outlier and a purple point here is considered good enough to make maybe a prediction and you can tweak sort of how far you want the spread to go you can sort of tweak when you can say it's an outlier or not again because it's a probability distribution you can quite easily do this and another thing you can also say well suppose that one of the moons was from one class and another moon is from another class, you can also get a nice classification boundary and you can also still work with probabilities. And because everything is, again, a Gaussian distribution, you can do math with this stuff afterwards as well. If you feel like playing with this, me and a bunch of colleagues and friends and I've even had a guy from Brazil contribute to this project now, we've made this project called Scikit-LEGO and the idea is that we just have some LEGO bricks that really fit well to the Scikit-Learn ecosystem but that might be missing. It's kind of an opinionated package, some kind of sort of... It definitely is opinionated. It's basically if me and Matthijs agree, then we add it. But if you want to play with the GMM classifier or the GMM outlier detector, go ahead. It's implemented. We'd love for people to use it. Definitely have a play with this. It is scikit-learn compatible. So it's not every day that you come across an algorithm that can do all of this stuff. And I think that's actually fairly impressive, if not inspiring. because if you can just understand the Gaussian, then you can pretty much already do half of what you want to do in data science. But these days, it's fairly, you know, hip and stuff. We should talk about deep learning. So what I just want to show now are just like two examples of how you can use this sort of Gaussian mixture knowledge to make neural networks behave maybe better. And one thing that you can do is you can say, how about I just glue some Gaussians on top of the neural network? What will happen? And here's an example of something called a density mixture network. The idea is just some sort of X goes in, But at some point before the final output, there's an intermediate layer where you say some of the output nodes are called mu nodes, and some of the output nodes are called sigma nodes, and some of the nodes are called pi nodes. The idea being that you have a mu1, for example, that's a mu of a Gaussian. You have a sigma1, which is the sigma of a Gaussian. And pi then says, for this particular x, does this Gaussian have a lot of influence on the prediction, yes or no? And by doing this, this neural network suddenly has like a multi-peaked output. It's not a single point estimate that it's giving. It's actually being trained to learn a proper probability distribution. Not a whole lot of neural networks can do this, but by just putting some Gaussian sauce on top of it, you get properties that you might want. Now, there's still like numerical downsides to this approach. Like I'm not suggesting it works for everything. But just the whole act of, hey, I'm just applying some Gaussian sauce to the mix, gives you properties to a neural network that you might need. And there's the mathematical details. Another thing that you can do, which I think is the cooler trick, you can also add other probabilistic properties with a mixture. So take an autoencoder. Typically what happens when you have an autoencoder, stuff goes in and then the same thing should sort of go out. But you squeeze it down to a latent state, and typically what happens is things that are different suddenly go into different clusters within that space. and if that happens I mean one thing you can then do is train a Gaussian mixture on that so what you can do is you can say hey there's just a bunch of Gaussians trying to sort of learn what the embedded state is like and if you then sort of rearrange what that means is that essentially means that when the encoder takes something and puts it into a latent state you have a low dimensional representation of a data point that you can give to the Gaussian you can do classification with that you can also do outlier detection with that but what's even cooler is you can also say suppose that I want to decode one of these circles because they're all gaussians it's also very easy to sample from it so it's quite easy to say I just want to have a random zero in this particular case could you please generate one for me and this sounds like a cool idea so I figured I'd just build this and what I did is there's this dataset called MNIST I use MNIST for this and there's this other dataset called Fashion MNIST which is sort of the same thing, but for fashion. So what you see here are just some sampled outputs. For every number, I trained a six-dimensional Gaussian embedding, and I said, okay, sample from a zero Gaussian and put it into the decoder. And here's some examples of stuff that I'm sampling. It's definitely not perfect, but, you know, close enough. And here's some fashion. I will say, I don't... Yeah, so there's a bunch of shoes, and at some point, I do want to mention it's not really a perfect rendering, but I'm definitely on to something here. And there's settings of the neural network that you can tweak here. And as opposed to a variational autoencoder, what I like about this idea is the encoder is just trying to do the best job it can do in encoding. And once that's done, then I'm going to introduce probabilistic aspects. Sort of a two-step approach, in that sense. But another cool thing about this is what I can do is I can say, hey, how about I sample now, not from the Gaussians, but just from that space. I'm just going to sample a random point. And when I decode that, what comes out is gibberish. So this mixture that's happening in the middle is definitely doing something. That's actually sort of maybe, I shouldn't say it's a manifold because it's not sort of the textbook definition, but it's a manifold. It's sort of a way to represent how data points are sort of distributed in space. So that's a convenient property. And again, the only thing I did was I took a textbook autoencoder and just put GMM in there, and suddenly I have all these very cool properties. And note, the shape, it can be a non-normal shape in here. Because it's a mixture, it'll just try to fit an arbitrary shape. But again, I have a better algorithm because I was doing something with a Gaussian. The final thing that's kind of cool, so suppose you have all the zeros in latent space. If you're going to fit a six-dimensional Gaussian mixture on that, then the mean of every single one of those Gaussians might represent a slightly different style. Because again, it might be the case that the zeros also have clusters within them. That's 15 minutes? Perfect. And then the nice thing is, by going over all of these means, you kind of also get a glimpse of different styles. Apparently there are a couple of zeros that are more skewed like this, and a couple of zeros that are like perfect round circles. And from an artistic perspective, it's also kind of interesting. It's kind of a way of saying, hey, I'm classifying and clustering and generating at the same time. And again, I know of no algorithm that can do all four of those things, besides this sort of approach. And again, the only thing I did here was just do some stuff with gaussians. So, we're nearing the part where I just want to do some live coding, but before I'm going to attempt to do that, I want to show like one, like, what I think is the weirdest property of a Gaussian, and this is sort of this weird meta thing, and I have about 10 minutes to explain it. So, if you have coffee, now is the time to drink it, because what I'm going to do now is I'm going to talk about like super high dimensions when you're doing something with a Gaussian, but I do want to already point out, like, just adding some Gaussian sauce made some algorithms better here, and that's a cool thing. but another way of looking at a gaussian is you can say here's a two-dimensional gaussian right if i look at one point from this two-dimensional distribution what i can do is i can plot that in a different way a two-dimensional point you can also say well on one dimension that's two points and the only thing i'm doing is i'm just making sure that the stuff is ordered essentially but this is just a different way of representing what i've what i've done here And, you know, there's, like, correlation here as well, right? So it's going to be really weird for me to sample an x1 over here and then an x2 that's super high. So there's going to be some, if there's correlation in this plane, there's going to be correlation on this axis as well. And it's going to be hard to draw, like, a five-dimensional Gaussian in sort of this space, but it's quite easy for me to draw a five-dimensional Gaussian in this space. Questions later. so this is the value so this is x1 has this value right so this is the value of x1 this value that's the thing that's on the y-axis here okay that was a good question that was a good question great thank you for that good question but then you gotta wonder okay so so how can I change maybe this because this kind of looks like a time series it feels like something is changing over time right so this is interesting and the main thing that you can sort of tweak is what can I do with this covariance matrix? Is there something I can do with that covariance matrix such that that covariance matrix is forced to have properties that I'm interested in? And how's about this? What I'm going to do is I'm going to look at the distance between the points, right? So these two points are way closer to each other than, let's say, this point and that point. So one thing that I could do is I can say, how about, this is really meta, how about I take the Gaussian shape, I look at the distance between two points, I'm going to take that distance, give that to this Gaussian, and I'm going to use that to fill in this confusion matrix. The question is, what will happen? And that is something I'd like to live-code, because I think it's very hard to explain it without me coding it. So what I've got here is just an instance of JupyterLab, and what I can do is I can type numpy.random.multivariateNormal, and then I can sample a multivariate point. And what I'll just do is I'll just say I've got some variable called mu, which is just np.zeros, and let's just say that k is equal to 2. So this is a two-dimensional, an array with two things in it, and they're both zeros. And what I'll just do is I'll do something with sigma as well, and I'll do np.i, give that a k as well. And then if I put mu and sigma in here, this is a multivariate normal, and I've just sampled one point out of it. That's the only thing I've done here. And what I can do is I can now type plt.plot, right? And then you can sort of, I can do this a bunch of times. But this is sort of the view that we had before. This is sort of the alternate view of sort of a two-dimensional Gaussian. And what I can then do is I can say, you know what, increase that. So now sample one point that's from like a 10-dimensional Gaussian. And it's going to kind of look like this. And if I do like 100 of these points, it's basically white noise because the covariance matrix that I've got here, that is independent. So all the different things that I'm sampling, they're being sampled independent of each other. But now how about I change that? So what I can do is I can type for i in the range of k, and then for j in the range of k. What I can do is I can say, hey, take that sigma thing you got there, right? Take that one part of the matrix. And what I'm just going to go ahead and do is I'm just going to sort of do something that kind of resembles a Gaussian here. So I'm going to divide i by 10 just to make it a bit smoother. But I'm basically saying this should be the distance, by the way. So this is the distance between i and j, right? So if the distance between the 20th index and 100th index, that's going to be huge. And when you square that and put that in the negative and give that to the exponent, you get something that's very, very close to zero. So let's just demo that with, like, maybe six points, right? So this is a different thing that I've sampled, but if I look at the sigma matrix, you can kind of see that on the diagonals, the stuff that's close together has a high number, and stuff that's on the outskirt has sort of a lower number. But what I'm going to do now is I'm just going to increase the number of points that I'm sampling. Let's just look what happens. Suddenly, this is still super random. This is a perfectly random thing that I've sampled. But there's also a pattern in it, a process. and that's the idea behind the Gaussian process the weird thing is if you sample a high dimensional variant of it one single point can represent something over time because of the way that I'm sort of changing this covariance matrix but here's the freaky thing what I can also do is I can say how about I just you know change this a bit and I look at the distance between the two points and I give that to like the sine function then a single point from a Gaussian distribution now represents a sine wave to me this was super unintuitive when I saw this the first time but by programming this with just a couple of lines of code I already was sort of in this moment with okay, I'm at least on to something here so why does this happen? and I kind of made a picture to try to explain it but the idea is if I have a covariance matrix that sort of looks like this then essentially points that are close to one another, they will influence one another. They have a high covariance. That's what that means. And if I have a sine function in there, then the high covariance is basically going to be, well, you're going to co-vary with something that is maybe 10 steps away, and every 10 steps away, there's going to be high covariance for it. So by defining, I guess, what people like to call a kernel, like the thing that I put in here, the function that I give it, that will force properties on the time series that's being sampled. And then I figured, let's do this in a really, really meta way. So I've made an object here called a kernel. And it's just an object, but the nice thing about this object is what I can do is I can say, take the kernel and take, let's say, like a linear function. And these are just examples of kernels that I came up with. So this was the kernel I typed before. Here's the function that does like the sine wave. I have another function that sort of returns a constant. And I have another function that just takes the difference between the two points. and what I would just like to do is just show you if I just take a linear function like this like the difference between the two points that's what's going to define the covariance matrix this is what it looks like and if I take the RBF thing that I had then this is what it looks like but the funny thing is, what happens if I say take the linear kernel and then multiply that by the linear kernel they get a polynomial so the weird property here is by giving a function that defines the covariance matrix, you can sort of force that the covariance matrix takes a certain shape, but the function itself can be used as a Lego brick, because one thing I can now do is I can say, oh, I just want the sine thing to be added to the linear thing, and maybe I should zoom out a bit, but I'm multiplying here. Sorry, this is a bit of addition. My bad. But now there's like a linear pattern being added to something that repeats, but the cool thing is what you can also do is you could take that linear pattern and multiply that by something that repeats. And then you get the seasonal aspect that sort of changes amplitude over time. So the funny thing is, you can do stuff to these functions, and they're kind of like Lego bricks that sort of have a cookbook, if you will. And there's actually a PhD who wrote a thing called the Kernel Cookbook. And the only thing it does, it explains all these different kernels and how you can multiply them together to sort of get arbitrary shapes. But then, how do you make predictions with this? Well, that's the really mind-boggling thing. If I have a normal two-dimensional Gaussian like this, if I have an xi value over here, then this will constrain what values this xj value can take. If my xi value is over here, the xj is not going to be down here. There's a constraint on that. That's probabilistic. That's something that the Gaussian will provide for you. But this will also happen on this view. So what you can sort of say is, suppose that there's this point that's given, then you can sort of say, well, then that determines sort of the probabilistic shape that all the points here can make. And then as a final demo, because of time, what I've got here, I'll zoom out just a bit, I have this kernel that I have made, essentially, and I've got some data points that it's trying to fit on. And you can see here that I can say, look, these are just some data points I want you to fit on, and if you're going to fit on these points, I do want the kernel to have like a linear increasing relationship and something that's seasonal. What I can now do is I can say, well, I don't know about that sort of linearly increasing thing, but I do know for certain there should be something of a sign in there. And this is the best thing that it can fit. But you have fitted the best sign function here. And what I can also do is I can say, just take the linear thing. Oops, I like demos, they're bummers. Wait, I think I can do this, right? Yeah, there you go. So you can do linear regression this way. So what did I just do? Well, I'm sort of running out of time. But what I hope I've just quickly been able to do is hopefully I've been able to give you some intuition on some things that might not have been obvious. And that's that the Gaussian is actually sort of everywhere in data science. It's this distribution, it's a thing, it's basically everywhere. And knowing about it really helps out. And it can be mind-blowing to sort of witness the full applicability. But understanding this mathematical Lego brick really allows you to recognize algorithmic parts in other algorithms. So if you're sort of being overwhelmed by all of these algorithms that you can learn, one thing you can also do is just spend an afternoon in a notebook and really trying out all these freaky things that you can do with a Gaussian. Because just understanding that to its fullest extent is going to make everything else a whole lot easier. So this might be a better investment. But if you feel like you don't fully, totally grasp this, that's super normal. I've skipped a lot of details in favor of intuition. But what I hope is the reason why I skipped those details is because the details are the things that are really, really demotivating. If you type in Wikipedia, this is how Gaussian processes are explained. If you go to a book, then this is how a Gaussian process is explained. The first thing you see is that a Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution square. No pictures or anything. And if you read the papers, it's worse, because you get these algorithmic explainers that don't really help you. So what might help, and this is the sort of thing that we might want to think about, mathematics is super useful when there's context and intuition. But it's really downright terrible when it's just a bunch of symbols being dumped onto your retina. And it's worse if this occurs to you when you're young. But the best advice I have here is when I was lucky, when I was a kid I was really lucky, because my parents actually found a really convenient way for me to prevent this intellectual paralysis from happening to me. So, the story about Gauss that I told you in the beginning, the very first story, I read that from a book called The Number Devil. And it's the weirdest thing. It's a book meant for kids. It's meant to sort of for eight-year-olds. If you've got kids, trust me, best book ever. Really is still one of my favorite books. It's about a boy who's afraid of maths, and then in his dreams, he gets a visit from the number devil. And there's all sorts of fairy tales. One of them is a story about Gauss that I just told you. And if you really want people to understand, like, hey, maybe focus on the intuition first and then on the details, there simply isn't a better introductory book than this. Everyone in my class that read this book as a kid got straight A's on their calculus exams. And all my math professors, they just see this as a pattern. This book is something that can really help you sort of gain this intuition of what is actually a number, and this is going to help you a whole lot. Thanks for listening. Okay, so any questions? Question, question, question. Here's one. Thanks for the talk. Maybe it's a stupid question, but if you go to the parallel coordinates and you say there's a distance between the dimensions. This stuff. Right. Yep. So there is a, you said it looks like a timeline, but if it's not a timeline, if the distance between dimensions is not, it's arbitrary. So technically, it could also be spatial. You can have, now it's just one axis, it can also be spatial. The intuition still applies, but then you've got higher dimensions upon higher dimensions, but the trick still applies. The idea is you have some sort of a kernel that says, given the distance sort of between two points, what can I do to the covariance matrix such that they behave in a way that I deem important. But this also works for higher dimensions. I will say, though, I chose not to discuss that due to the serious amount of confusion I might introduce this by doing so. But distance is A interpretation, and time is sort of an easy explained thing. But it can also be other things than time. If you're reading stuff on Bayesian optimization, then that distance is just distance in hyperparameter space. But it cannot be random. No, I think... The thing here is we're talking about math, so there might be someone in the audience who says, ah, I know a detail that you don't, so I'm kind of afraid to say that. But no, I would argue that you need to have some sort of meaning going into the kernel in order for a covariance matrix to really mean anything. Thank you. If you have a cool counterexample, come to me afterwards. I would be interested in that. Yes. Thanks a lot for the talk. I really appreciate the intuitive approach. And my question is about slide 51, I think, where you have... That's oddly specific, but well done. This one? Yeah, exactly. The general problem, I think you said you have 16 Gaussians. Or six. I have to count them, but some number. Oh, it says 16, so it's probably 16. I mean, that is 16 means 16 standard deviations, and you've got maybe a few hundred data points. Ah, right. So that's a bit of a mathematical detail, but here's what happens. For every single point here, I can sort of ask, what's your likelihood value, right? And because there's gaussians in them, there's some likelihood value. And this gives me a distribution over the likelihood values. I can set a threshold there. Okay, thanks. There's a documentation page on the open source package that explains this in more detail. But this is one of the ways you can do this. Good question. hi um the gaussian autoencoder business that you did you mentioned very very quickly this stuff half a sentence yeah yeah uh variational autoencoders this is not a variational autoencoder could you say a little bit about the relationship of yeah so to variational so the actual history of this was i kind of figured like hey i want to add something probabilistic to my encoder and then I came up with Vincent's autoencoder. Unfortunately, that name was already sort of taken. That's part of the story. But the thought here is a variational autoencoder changes the cost function, right? So the steps that sort of the gradients are taking are being influenced by the fact that the center space has to be a Gaussian distribution. Note that I believe typically it has to be a standard normal distribution, so not a mixture. It has to be a single Gaussian. That's what a variational autoencoder typically does. What I'm doing here is I'm saying I don't want to constrain the autoencoder anyway, so let's just train that. And once that's a given, then take that latent state and then train an autoencoder, sort of a Gaussian mixture on that. So it's more of a two-step approach. I'm not adapting the cost function whatsoever with this idea. And I believe this approach has merits to it. There might be some potential benefits, but yeah, the Vincent's autoencoder was not going to be a good paper. Okay, any more questions? Questions, questions? No questions? Okay. Yeah. Thanks for the talk. It might be a stupid question, but you nicely showed how the sine function is approximated and the linear function. Could you give more intuition on the RBF kernel, which is something I've always been missing, not just in this talk? So in the end, it's just a shape, right? And the main thing to take away, whatever function you give it, the one thing that that function will do is it will make sure that the covariance matrix just behaves in a different way. And RBF stands for radial basis, and it basically just means it's kind of like a hump. And there's different humps you can do. You can normalize stuff. You can add extra parameters. But the gist, first and foremost, is whatever function you can have that takes distance between two points and then fills in a coordinate of a covariance matrix, That's already a kernel, and there's a huge cookbook of stuff you could do. So the thing is, it's not too important that you have the exact right RBF. It's more the general shape that matters. This answers the question. Okay, so we have one more question. Time for one more question. So for people that are interested, actually Google this thing called the kernel cookbook. This character, David, he's actually kind of clever. He writes about some cool stuff. but he actually wrote sort of a cookbook on how you can combine all these different kernels and what properties that they have. It's actually kind of a good read if you're a nerd, but make sure you get the intuition before you do. Hey. Yo. Can I go back maybe to 71? You folks are really good at remembering slides. So maybe you want to go before 72. Ah, 72, right. So here you show how you actually take out the X1, X2 But later you need the distance between x1 and x2. So this is a two-dimensional distribution, which I can draw. A five-dimensional distribution is kind of hard to draw on a two-dimensional plane. But this red dot, it has a very high x1 value, and it has a very high x2 value. The x2 value is slightly higher than x1, so x1 is slightly lower here than x2 is. Green is the other way around, and blue, they're super, super low. So the interpretation of the values here is, whoops, you can see the mouse, right? So the interpretation of the values here is whatever value I have here on the x1 axis, that's the value I see there. And the v value was? Speak up a bit. And the v value was? I mean... And this v value, so this is the value of x1 here. That's this value. So it's just when I have this dotted line here, I have some sort of value on the axis. That's the dot high tc here. The only thing I'm doing here is I'm just sorting x1 up until x2. I'm only sorting on the x-axis here the actual value that you have here from here to here this value that hits this axis that's the value you have here okay okay thank you very much Vincent for the great talk, another round of applause please