Your Model _Probably_ Memorized the Training Data

Speaker 1 [00:07]

I have a lot of inconvenient realities to get through, so I'm going to stop you there. So my name is Catherine Dremel. If you don't know my work, I work in data privacy, which I like to think of as inconvenient realities as a service, because that's what we offer in data privacy. If you ever encountered data privacy, you've probably heard a lot of nos, and I'm here to tell you exactly why. So today we're going to talk about one inconvenient machine learning reality, which is your model probably memorize the training data okay whoops so i mean maybe you think like what's the problem man like ai is pretty dope it's really cool and that means it doesn't really matter if we memorize or not i was on a panel recently it was a global machine learning conference online and the actual moderator of this panel told me like so what humans memorize things all the time humans learn things, like now AI learns stuff. Really, this is not something that is that much of a problem, right? But then I asked him about copyright, and he didn't really have a very good answer for that. So this is, for example, one of the exhibits that the New York Times put together in the lawsuit against OpenAI and Microsoft, where you can see that output from GPT-4 directly quotes a New York Times article. So perhaps, yeah, learning things and repeating things as humans, this is something that we do. But we definitely don't do it at scale. And we definitely don't do it usually word for word. And so this is one of the problems that we have in machine learning. And we're going to figure out how it works. So for the most part, we like to think of deep learning or machine learning we say it doesn't over fit, it generalizes. So surely we haven't actually memorized anything if we can generalize well on the testing data or on other unseen data, validation data and so forth. And I'm sure some of you are familiar with kind of SVMs and so forth. So I'm going to borrow some reasoning from SVMs to talk about some of the ground theories that we need to understand to move on to understand memorization. So one of the things we can take from SVM is we can take kind of different class boundaries or decision boundaries, so in class, out of class, and we might want to map them in some sort of multidimensional space, right? And in SVM, we borrow this idea of margins, and in case you haven't been in school for a while, I'll remind you, the margins are the space between kind of the support factors or the observed data from our training sets, so the nearest points to the decision boundary and the other kind of decision boundary, the other class. And what we try to do both in SVMs but also in deep learning is we try to actually maximize the margins. And this is because this allows us to have better confidence that the prediction is accurate. But it also has some other interesting properties. So this margin analysis, it was studied, papers about it started coming out of Google around kind of 2017, 2018, and some more recent work shows exactly why was Google Research or Google Brain studying this problem, and numerous other researchers were studying it as well. Well, these margins, especially in deep learning, so you might say, well, deep learning distance is intractable. It's an intractable problem. But one of the things that they actually notice is if we take the data representation in these intermediate representation layers, so to translate that, that means sampling the data point at the different layers, deciding due to its activation, where does it exist in decision space, and then essentially taking an approximation of the distance to the next class, or where we might say is out of class, so in class and out of class, that when you actually sample those distances across numerous layers, you actually can predict generalization. So you can, with this is doing those, you know, sampling and then taking the distribution of those margins, essentially they took a few moments of the distribution, and then they were able to create basically a linear predictor of the margin space and the generalization. So this means, I'm going to translate it in case you don't know all the words that I'm saying. I'll translate it quickly and say that this means for the bigger spaces between class boundaries, we generalize better. And when class boundaries are very near each other, we generalize poorly. Okay. So, Catherine, why are you talking about this? I came here to hear about cool privacy stuff. All right. I'm getting there. I'm getting there. I'm sorry. I'm trying to build some of the understanding. So how do margins relate to memorization? Why does this matter? And one of the things that we can think about when we think about margins and when we think about class boundaries is we can kind of think about what does our real-world data look like? And one of the things that our real-world data looks like is we have these beautiful examples sometimes that we get that are kind of common examples. And here, I'm really into bunnies. I don't know if you have any animals at home, but I have two little bunnies. And so here is kind of a classic example of a bunny. And if we wanted to create a bunny class and how we might differentiate it or space it away from other classes is we might say it has two ears, it has eyes on the side of its head, it's fuzzy, right, it's cute colors, it's set against green or grassy backdrops and so forth. So these are all things that if we trained a deep learning model in computer vision, we might learn, so to speak, about bunnies. But that's my bunny. And Fiona doesn't care that bunnies have two ears. And Fiona doesn't want to look at me from the side. And Fiona doesn't want to sit in grass. She sits on carpet. it. And so we can see, by the way, I just got to say, in case you're a fan of bunnies, you need to check out our murder buns. So it's a Reddit community and she definitely looks like a murder bun, you know, but she's actually really happy. Like the ears back mean she's happy, I promise. Anyways, but so Fiona is still a bunny, right? But the problem is, is that Fiona would be a really weird example. And when we start thinking about this, so I'm going to go back, when we start thinking about this space, this is also some of the behavior that we start noticing also in adversarial examples, is we're finding areas where perhaps we are not near any class boundary or perhaps we're near a false class boundary. And when we start looking at how data is distributed in the real world, we end up with a lot of rare or uncommon examples that exist, like the Fionas of the world, but they are severely subsampled or undersampled compared to their class accomplices that are maybe more commonly or more overrepresented. And it actually gets worse. So when we add multi-class complexity or in language, when we add many languages or we add emojis or in video, when we add different kind of lighting and so on and so forth, we not only have the complexity within class, right, where here we can see the visibility pattern of a bus, you know, looking at a bus from the front or the side is quite a common way to take a photo of a bus, but you also have all those annoying captures that Google has made you fill out now for years to identify buses in weird positions or where exactly is the motorcycle so we don't hit somebody and kill somebody with our self-driving car. So what we have is not only the uncommonness of a particular, let's say, example within the class, but then we have some classes that are overrepresented in the entire population. So when we're dealing with these multi-class problems or in language when we're dealing with many different languages and we want to try to have one model that can do all of it, which maybe is a bad idea, although more on that from Ines later this week. We also have this exponential problem of what we call these zip-f distributions or these long-tail distributions where we have a lot of these rare examples that may or may not be representative of real data situations, but they come across quite uncommonly in our training data situation. And so what does that exactly mean? Well, I have an inconvenient truth for you, and that's that it's actually mathematically proven from Vitaly Feldman's research at Google, and then later he went to Apple, that it actually is impossible to do overparameterization or deep learning on such distributions without actually memorizing the training data. Why is that? I mean, I really wanted to work you through a proof today, but we definitely don't have time for that. So what I can promise you is, A, this paper is an amazing read. B, there's many papers now based off of these papers. And C, I will be also doing a long post series on breaking down the proofs and the theorems of this paper but essentially if I were to summarize it the the essentially the information that a model loses or the cost of not memorizing these rare examples is greater than because of the way that the distribution works is greater than never memorizing them at all or never seeing them at all and this is because again when we go back to thinking of margins we can think as obviously when we're in high dimensional space and we're in we're within the complexity of a deep learning space we can also create class pockets and what they've actually studied is this creates a pocket that actually ends up helping us generalize well because for every fiona that's in the data set then if we see something similar to fiona and we're even within any type of boundary around the fiona example then we can classify the bunny correctly and if we get it wrong we're probably not that wrong right a human could maybe even make that mistake and so we we we be via the way that we optimize these models we actually entice the model when it has massive information gain from a single data point and it hasn't seen anything like it to actually store that information and it's both mathematically proven and proven in practice. And so you might have already kind of started thinking about this if you've been in machine learning for a while, and I don't want to date myself, but we kind of used to talk about overfitting by saying, okay, we have too many hidden inputs or we have too many parameters. We may overfit, especially if our data set is small. And that's very true, especially when you're dealing with kind of simple or less complex machine learning models. And what we ended up seeing with deep learning is we actually created what is sometimes referred to in theory and practice as the double descent. And so as we added more parameters and more hidden inputs, we were actually able to generalize well. And everybody was like, wow, that's crazy. We have more parameters than we have data points, but somehow this model is generalizing well. It must be deep learning magic. Unfortunately, now we know a few other things, and what we do know is that it actually memorizes. That some of this generalization is because of these kind of pockets, I'm going to call them, multidimensional space of understanding. And then, you know, it's great if you generalize well on something that doesn't reveal somebody's personal information and is maybe less cool when you just randomly dock somebody. So in this case, this is from Google Research. They were actually able to extract quite a bit of sensitive and repeated text from ChatGPT, as well as from a variety of GPT models and from Google's own models. And so if you want to look into this research, I will cite a few, but Carlini and NASA are some pretty amazing researchers who are working in this space. And what did they do here? Well, they asked ChatGPT to repeat pom, pom, pom, forever. And then eventually they got these tokens essentially from a memorization that happened during training. Any guesses as to why this happens? Okay, I will go further. So one of the reasons why this happens is when we create POM, POM, POM, POM, POM, right, the chance that the GPT model or also whatever reinforcement learning they've added has actually seen this token sequence before is quite rare, right? It's quite improbable that it saw the word poem for 20,000 times. So what we're essentially doing is we're creating instability in the network. We're creating instability in the context window. And then because of this instability, we can uncover some of these lesser visited parts of the model, let's just say. So some of these other things. And I've tried some interesting attacks. I'll be publishing some blog posts on it. Obviously, when you have smaller models, it's harder to extract this because there's also maybe less of them. More on that in a second. But obviously, for this person, this is not exactly, I think, consensual use of a person's phone number, email, and so on and so forth. So when else does it happen, right? So from research, we understand, and also from practice and careful extraction attacks, we understand some of the reasons why this happens. One of them is repeated examples. So, in the latest research that was published on this, we believe that 30% of repeated examples are memorized. So, you can, for example, extract, I don't know if I have any Beastie Boys fans here, but I got Beastie Boys there, because yeah, okay. If you don't know who Beastie Boys are, then I'm too old for this conference and I need to stop coming. Okay. Anyways, so repeated examples, we know that that's definitely a problem and that's something that we can reproduce. For example, you can extract the entire Apache software license. Is that a big deal? Probably not. Is it a big deal maybe if you're writing text that's cited often? Probably so. Or let's say your code is often used for something like Copilot. This happens there too. So repeated examples we definitely know, especially when it's repeated frequently in relation to the other classes, right? So the other examples that might be seen. Then we have the uncommon or the rare examples. So this middle version was taken from a stable diffusion attack, and this photo was only shown twice to the training model. And then the model was able with stable diffusion to then also produce this given the prompt of the person's name. So we show that it can happen also in a variety of other deep learning models, not just in generative, although I guess stable diffusion is generative, but not just these huge, massive, over-parameterized, but also smaller generative models. And then the research that kind of inspired all this is an amazing paper, well worth a read. That's from 2018, I think, or 2019, so quite some time ago now. That was from Zhang et al. And this graphic is from them. And what they did is they took CFAR 100, so they took one of the common computer vision data sets, and they completely randomized the labels, like completely. So, of course, the data set is just full of error, and they thought, like, it'll never converge. It shouldn't converge. Well, actually, it converged quite well. Of course, it didn't do well on test data sets at all because it had memorized, you know, completely random labels. But this paper, which is kind of a seminal paper from the deep learning space, is called Understanding Deep Learning Requires Rethinking Generalization. And so the hint was kind of already there for us five years ago, six years ago, and now we're kind of uncovering the fact that this memorization happens and that has a play at why this generalization appears to work and yet also why machine learning models converge even on erroneous data. And so if you have mislabeled examples, again, the cost of not memorizing that example, the model doesn't understand what's real and what's not real. Hallucination much? It's just, you know, there's no such thing as reality in math, right? And so for this reason, things that are mislabeled, things that are errors, those will also most likely get memorized because they will most likely be far from their normal neighbors, so to speak. And therefore, there's a good chance that you have erroneous data that is memorized. And research in this space is still relatively new. So it's quite possible, if not probable, that there's going to be more types of attacks in the future. And perhaps there's more of this that I can't tell you about today, but I might be able to tell you about in a year or two. so how can we evaluate memorization if we're dealing with sequence-based models one really cool way that we could do this was presented also by google researchers looking at k-adaptive memorization and the idea behind k-adaptive memorization is we want to take samples especially anything that we know is already repeated samples we want to give the sequence to the model and we We want to see how many of the sequence in a row we can actually extract from the model by giving it a long enough prompt. And here we can see the prompts to the left. And we can also see the behavior of overparameterization on both this generalization, right? Because we know that the 6 billion parameter model performs better usually than the 125 million model. But it also memorizes a lot more. And so there's some pretty cool experiments. I'll also be releasing some things on my GitHub so that you can play around too. But if you yourself also do fine tuning or anything in the language space or any other sequential space, I recommend maybe you set up some testing to try to extract some of the training data and see exactly what has been memorized and how much. I want to make a quick note here that if you have some clever stuff happening with Beam search or any other intermediary between let's say the language model and the user, you may not see exactly the same behavior, but it doesn't mean that the model hasn't actually memorized it. So there's some caveats to doing this testing, as in it's not full proof, there can be examples that you don't catch. And there's also other things you can do. So I'm a big fan of security work in the machine learning field, and the term red teaming we borrow from the security world where we say we want to run attacks against our models and see if we can do things like extract information from them. And so one of the common things I would recommend is membership inference attacks. You can usually run membership inference attacks just also like as part of your normal CICD. What might that be? Well, it means that you take an in-training set example and you take an out-of-training set example, and then you take a look at the confidence intervals and you start measuring the distance. And if you can predict whether something was seen or not seen by the distance between the confidence intervals, then you probably have some information leakage in your system. There's also extraction attacks you can perform other than just the k-adaptive one. This is, for example, an example of a model inversion attack. This is something that you can run on your computer. It's already on my GitHub in the repo for the book that I wrote. And what you can do is you can train a facial recognition model and then using just the class, so just saying like class equals three, which is this person, you can actually extract a representation of the face, basically optimizing kind of similarly actually to how stable diffusion works and taking a noisy image and then optimizing towards essentially where we have full activation and where the confidence is high. And so these types of things might be fun for you to play around, and if you do machine learning at work, it might be cool for you to set up a security team that occasionally, it doesn't have to be every day, but occasionally takes a look at these problems and tries to attack your machine learning setup and you try to then also defend against such attacks. There are many ramifications of this problem, right, mainly in the legal social space. so first and foremost we have copyright and creators rights where people of course who are paid to create work they expect to be paid for the work that they create and if we like accidentally oops we scraped it and we might have memorized it that's maybe like not so cool for the creators and so maybe we have to think about how do we build systems where if we're going to memorize we reimburse or we side copyright and creators rights or we create ways that they can own their own models, right? Another thing and kind of related to the idea of scraping data from the web and just using it for your own purposes is contextual privacy or what we call contextual integrity in the privacy space. And that's a theory from Helen Nissenbaum that takes a look at when we use technology and we're in the humans in the real world. In the real world, it's pretty easy for us as social creatures to kind of understand this is private and I'm sharing it just in a small way or this is public and I'm here being videotaped. You know what I mean? We can all kind of understand this. But when we take that stuff and we translate it into technology, it can be easy for that context to get lost. And I think we all have known somebody who accidentally shared something in a context online that they weren't familiar with, that accidentally put stuff on public or that thought, you know, who's going to go to my random website other than my friends, family, and maybe people who I know? They definitely don't think, oh, ChatGPT is going to go to my website and scrape all the data and then spew it out randomly to another person. And so this loss of the context of how information was shown is an important aspect of privacy. And finally, we also have security for persons and for democracies. So if you're a famous person, now it's quite easy to create a deep fake of you using any number of the generative AI platforms. And that's, of course, going to be used for misinformation, disinformation, and so forth. And also the fact that these things can repeat. And even if you're like a smaller, more rare example, perhaps as we add more parameters, it also becomes easier to use against quote-unquote normal people. And this is, of course, a risk for our democracies and for the way that our society works. So what are we going to do about it? I got some ideas. So first and foremost, who here has heard of differential privacy? Who here has implemented differential privacy? Yes. Oh, I know you. So differential privacy is, I'm going to give you a mini primer in case the concept's new to you, but it's one idea of how we might be able to think about reversing some of this memorization, or at least being careful about what we memorize and how. Differential privacy was invented now many decades ago by Cynthia Dwork and then developed further by many researchers. And the concept of differential privacy is that we should be able to measure privacy. Essentially, it's the idea that privacy kind of in practice is kind of this social and legal thing, but can we create a scientific, rigorous, mathematical way to measure privacy? And when we do that, we might say that privacy is how much information a person loses or another person gains about an individual when interacting with a data system. So this was essentially the premise that Cynthia Dwork used. And let's go through, let's talk about it. So in differential privacy, in the classic definition, and I'm going to use the classic definition, but there are many other definitions now over time. And in the classic definition, we have a database, and then we add or remove a person, and then we get another database. And we say that the person who wants to query this, they're allowed to query the first database and the second database. We don't restrict when and how they can query. And what we say is we want to restrict the amount that they can learn about whether a person was in the data set or not when comparing the first query to the second query. And so we can take like a really clear example. If I run a count query on database one, and then I run a count query on database two, and presuming, you know, my select is select all, then I will see a difference of one in the result. And I can make a pretty good guess, especially using something, some clever Bayesian reasoning, that that means a person was added to the data set. And what differential privacy tries to do, and I have the equation down below, but you can also look and read about it online. But this vanilla equation, this very plain definition of differential privacy says the amount of information where I could guess whether it's database 1 or database 2, that should be bounded by this term e to the epsilon. And epsilon is a parameter that I can tune. So I can decide, as the person running the system, exactly how rigorous I want these bounds to be. Do I want them to be really tight, do I want them to be really loose? And how do we do this in practice is we actually put a bounds on the amount any one person can contribute, so we literally create bounds, and then we add uncertainty. And a lot of people think differential privacy is just adding noise, yes of course, but we add very careful noise. And we can tune the noise based on what type of problem we have and based on what type of distribution we have. And what we're using the noise for is we want to make the attacker uncertain if the answer is true or not. And so if I give you a count and let's say you query one day and I give you one number and then query the next day, I give you the same number. Then you query the next day, and I give you a number that's less or that's more by a little bit. And then the next day, I give you back that same number that you saw. The chance, even using, you know, advanced Bayesian reasoning, that you can really actually infer, is this person in the data center or not? I'm adding some uncertainty to that dimension. And again, I can tune that. So how does it work in machine learning? So differential privacy, again, it gives us basically this uncertainty. It gives us what legally we might call plausible deniability. I wasn't in the data, right? Maybe I was, maybe I wasn't. Anyways, it gives us this probability-based mechanism to talk about what information people contribute. And it gives us also a way to think about when should we not contribute anymore? When should we stop a person from contributing? And so when we use it in machine learning, we have differential privity, stochastic gradient descent, and what we want to do is we have our mini-batches or our samples, whatever you're using, and when we have the gradient update, and we want to bound this by the people, so we need to know if people are going to appear in many batches or just one, and we want to try to keep people in known batches because, again, we have to measure on an individual basis to some degree. and then we have our training update which is how much privacy loss to the people in this have and if the gradient's really huge then people have a lot of privacy loss so what do we do we clip the gradient there's a maximum amount of information that we can learn in each batch to this tends to also be a good regularization mechanism more on this in a minute and then we add of course some carefully chosen noise right and we can choose this from gaussian noise we can sample this from a variety of different types of distribution. And what we're trying to do there is we're trying to protect the contribution of each individual via this noise. It can also work in other ways. So cohort of experts is like all the rage now in LLMs or whatever. And so cohort of experts we can also use in differential privacy. So now I'm taking you way back to some of the first research on differential privacy and machine learning. And that was the Pate model that was written by Nicholas Paponaut while he was at Google studying. And so what we do is we train a bunch of little models, smaller models on smaller examples. And we make sure that individuals can only ever be seen in one model, right? Or at least if they overflow to another model, we either undersample them or we say, okay, this person contributed these three models or these two models and so on. And then we show some public data, right? Of course, this was like back in the days where Google is like, well, we scraped it from the internet, so it's totally fine. Anyways, but we take some data that we think we can use, and we have the cohort vote on it. And now, because we can say that each model represents that subgroup of individuals, we can also decide to leave one out. Or we can, based on the different contributions, we can decide who gets to vote, who doesn't get to vote for the labeling. And if we believe that we have a bunch of uncertainty and randomness in other models, but one model is really certain, then maybe we just don't answer that, right? So we do lose information. And then that operates as our differential privacy mechanism. The math of it all works out. And we can either give a differentially private response, although that'd be kind of expensive to always run this many models, or we can train a new model based on essentially saying the model is differentially private. And the reason why I keep hinting at generalization and I keep hinting at its relationship with differential privacy is when we do this in practice, we have trade-offs. We do lose information. We definitely lose information. And we tend to lose information for difficult examples, for outliers. And for easy examples, for kind of the easy bunny, we get it totally right. It's totally fine. And this has been known for a while that we just get kind of weird examples right, or we get things very close to the decision boundaries improper when we train with differential privacy. And this is because these ones leak a lot of information in compared with our peers. They contain a whole lot more of what we would say in privacy research. Privacy, you know, we have to protect them. They're the oddities. They're not the norm. And yet, at the same time, depending on the problem that you're trying to solve and how your data was collected, you might have to learn those. And so this is a real conversation that I think and I hope that we should be having in our field of saying, when is it okay to get the outliers wrong and when do we actually need that information? Because the trade-off is huge in terms of people's privacy. I'm going to show you a few other examples. There's a big field now called model unlearning or sometimes it's called selective forgetting research where we also essentially train a leave one out example and then we compare it or we do distance approximations to the model after it's gone through some sort of unlearning algorithm and there's an unlearning challenge i think right now active at google if you want to participate i think you can win like money or maybe you just win google cloud credits i don't really know but um to essentially start to say how can we unlearn things and why is google investing money in this well because gdpr right because they actually have to forget things that have been memorized in these models and it's worth investing money and time and research into looking at these things and so perhaps this is i would say this is still very much in research mode it's not practical yet but this is an interesting area to follow to start to see how and where can we do this and how can we use this leave one out principle to figure out what information might have leaked about any one individual into the training data set. In addition, regularization, so differential privacy is a great regularizer. So if you need regularization for your data set, I recommend differential privacy. But if you want to do it another way, and if you want some of the privacy benefits, some other regularization methods such as pruning, dropout, or quantization can also add some of these properties, and there's active area, particularly in the field of federated learning, into looking at these things and figuring out how can we make sure that what we prune is what we want to forget about, let's say, a particular individual or smaller group. The same type of method goes through model distillation. So in case you're unfamiliar, model distillation is kind of a semi-supervised way to train a task-based model based on an unsupervised model like an LLM, and Ines can tell you all sorts of details about how to do that with Prodigy and with Spacey, but the idea is to create smaller models. And again, this helps us. Why does this help us in privacy? Well, because we're actually, of course, pruning areas, which means we're probably losing some of the memorization that has happened. And also if we're doing active distilling where people are actually labeling the data, we can actually have human oversight and say, you know what, it's probably better for us not to repeat this person's name or not to repeat this person's address or this other piece of misinformation. But I'm going to propose like two wild ideas here at the end, and this is stuff I'm really excited about that I'm going to be spending my summer trying to experiment with. So if you're in Berlin, please stay in touch because I might be running some crowd experiments what if we just said it's cool that models memorize like what if we just said it's fine to memorize as long as everybody participating knows that we're going to memorize so crazy wild idea you get a base model you get a training apparatus and you build your own gpt based off of how you write or you meet up with some friends and you create a little ai rig together and you train your own GPT that you'd like to share with friends and family and so on. And then you actually own the model. So both you have added to the data, right, and you've also probably more normalized the distribution of the data. You don't have as much of a long tail. You have some, but you probably have less than just publicly scraped data, and it's your model to use. That would be one way. Or what if we did that at scale? What if we formed data trusts where groups of people could donate their data with the knowledge that maybe some memorization can occur, or maybe with saying, here's the differential privacy mechanisms we use to build these models? And what if we did that so people could donate their data so we also fought a little bit of this long tail, so we had more representative data with more different persons involved, and maybe we can create more accurate and fair and ethical computer vision? and maybe those people get a piece of the open model or maybe those people get to use the model for free for many years and what about if we did that in community levels so I don't want to make anybody mad but what if Berlin decided to train their own gender model and send it to Bayern we could do that right we could send Söder a gender model based on our community orientation based on things that we believe and we could also all have it to use too right perhaps this is a really interesting way and maybe if we take balcony craft work if we take energy from people's balconies and we donate it to the cities in the states that we live in perhaps we can create energy efficient models and i'm willing to have berlin memorize how much and where my balcony is in order to get climate change, so in order to have climate-correct politics. And that's something that at least I'm willing to do, and I hope maybe others are too. And so the future of AI doesn't belong to AI bros, and it doesn't belong to plagiarists. I think the future of AI actually belongs to all of us. And I hope you've had a little bit of time learning, fun learning about data privacy and inconvenient realities, but I also think it's really empowering to think about how privacy and the principles of privacy can create different incentives for machine learning and for AI, and I hope that maybe you're inspired too. Thank you very much for your time. Thank you. Alright, thank you for your talk, Catherine. There are a few questions. The first two questions are touching, tackling the privacy issues. You already mentioned that regularization can improve privacy problems. And there is a question regarding masking the training data prior to training the model. What do you think about that? So, yeah, I've thought about this problem for a while. I think there are some ways that masking can work well, but we probably won't catch everything. So let's say we're using masking where we want to do pseudonymization or tokenization, we might call it in privacy, of named entities. So the first question is, how do we know whether a named entity is allowed to be known or not? So if I ask, like, who's Olaf Scholz, I expect an answer. But if I ask who's Catherine Jarmel, maybe I don't expect an answer, right? but this is very difficult because this implies knowledge of who's, you know, who should be learned and who not, so that becomes difficult. I used to work in the field, and I'm still excited about the field of encrypted machine learning, and if whoever asked that question wants to talk a little bit more, what we would do in encrypted machine learning, and there's a chapter about it in my book, is we would just encrypt all the data, and then either you got access to the to the resulting model or we have different ways with certain types of encryption where you can essentially just send your data encrypted get a response back and decrypt it locally on your device and so there's perhaps other ways to solve the same problem in a different way all right thank you the next question is regarding federated learning so do you see problems like that in federated learning so federated learning has the same problems as centralized machine learning when we think about privacy so we still have gradient leakage and in fact usually with federated learning we have gradient leakage on an individual level which means we need to be very careful okay so in case you don't know federated learning like might happen on device, and each person contributes their data. So when you send me your gradient update, I can learn a lot about your model and your weights, because I know the general model, I know the global model, and I know your gradient update, and I can essentially reverse engineer properties of your local model, right? So we want to avoid that. But the cool thing is, is because federated learning, we don't centralize the data. I think it has a lot of other promises. So like it's some of my ideas behind these types of things. I think federated learning can play very well in this space but we also have to be careful with the gradient updates and we have to be careful how we run what's called the aggregation software so if we want to do truly private federated learning we still have to add differential privacy but what we do is we do this at the aggregation step and there's a whole chapter on federated learning in my book if you want to read more all right thank you the next question is the following like why not adding a layer that penalizes output that is too close or too identical to a sample yeah i've heard that copilot does this so i've heard that copilot literally has like an observer that's running that tries to make sure that it doesn't repeat a copyrighted code that's extremely expensive right and when we when we increase the data space it becomes even more complicated right so maybe one reason copilot can do this is it knows what language you're using And it knows, you know, it has a fairly limited vocabulary in comparison with massive language models or massive image models. But this becomes very hard to do hypervisor, we can say. And the question also becomes, is there a way around that? So this is the traditional kind of guardrails approach. I don't actually think guardrails solves this problem because I spend a lot of time jumping guardrails and having a really fun time doing it. So I don't really think that's, like, the long-term solution. But I'm curious to see how the different, you know, kind of interplay with how do we optimize these systems. Maybe there's some cool ideas in there that eventually gain merit and are less, like, susceptible to attacks. I think that's a really cool area of research. All right. Thank you. And a final question or rather a comment. There was a slide mentioning, like, we are adding random noise to the training data. And the person just gives a comment like, well, that might sound not so good to the customer if we are adding a random noise to the data. Yeah, I mean, so it's not truly random. So usually you sample from either a Gaussian or Laplace or other type of distribution, which means the probability that you get an answer near to the real answer is quite high. That's why it works, right? Because we do have some trade-off between accuracy, but we essentially only want to add a little bit of uncertainty we don't want to add too much and this can also be trained so I'm happy to talk with you about how to talk to about differential privacy with your customers for better or worse I've been doing it for some time but for example this is you know this is used at scale for example at a variety of the large tech organizations so I don't think it's so scary and far from the truth anymore in fact differential privacy is named directly in numerous EU regulations now and is also named even in U.S. regulation, so in upcoming U.S. regulation, so it's not such a wild idea that it was maybe 10 years ago. All right, thank you again, Catherine, and maybe we can wrap up with another round of applause, please.