Unforgettable, that's what you are: Evaluating Machine Unlearning and Forgetting

Speaker 1 [00:08]

So let's get started. Machine unlearning. Today we're going to talk about how unforgettable all of you are, and we're going to do that by exploring the field of machine unlearning. Now, just by show of hands, how many people came to my talk last year at PyData Berlin, PyCon Berlin. How many people were there? Okay. That's kind of useful information for understanding machine unlearning, but we're going to give a very, very short introduction to what I presented last year. You can obviously find the talk online, but I also wrote a very extended blog series on memorization, which we'll talk about in a second. But machine unlearning is what we're going to explore today, which is one of the ways that we can address problems like memorization in deep learning. I'm Catherine Jarmal. I go by KJam. I've been doing deep learning for a while and privacy for a long time now too. Okay, so how unforgettable are examples in machine learning? This is one of my pseudonyms on the internet. So when I have to give a profile and I don't feel like giving my own information sometimes I use my buddy here I generated this face by asking for a cool guy who's very good at AI and this is what I get so you know if I need to be a cool guy into AI then this is who I present to the world and what is useful if you want to use visual pseudonyms for yourself online is to do a reverse image search so when I reverse image search my cool guy profile I get glasses models I get other people using pseudonyms this is from some Russian hacker forums so I get cool other people into security and privacy and I get other glasses models why do I get so many glasses models any ideas Glasses look smart right? We have a bias there in our training data. Why are they like this like rugged, half-shaven look, you know, all of this type of thing? Does that have anything to do with being cool and smart in AI? Probably not, right? But we can already expose maybe not only some of the biases in our training data, but the fact that we can very easily reproduce faces like this, and that the closest faces like this that we can find that are not AI generated are glasses models that probably, unfortunately, don't know much about deep learning, right? So perhaps maybe in the future we want to unlearn this, and we'd like to represent better what cool guys who are into AI look like, probably more like people in this room, right? Which may or may not wear glasses, which may or may not have different skin colors, different hair colors, different hairstyles, different beard styles or no beard styles, right? Might have different outfits. And so when we think about what we're trying to learn, when we think about unlearning or forgetting, we're also talking about what we're actually learning. And I think that's a useful exercise. So let me give you the super, super TLDR of my talk last year. There's a blog series that I also have on it on deep diving into deep learning memorization or how we define memorization in deep learning, how does it happen? Why does it happen? Well, one of the things that we memorize is common examples. If you have a famous person, or if you have a famous piece of text, or if you have a famous song, or something like this, this will most likely be learned. And by learned, I mean verbatim able to repeat, or per pixel able to repeat. Now, in a lot of ways, this is what we want to do. This is what we want to learn, right? This is the goal of our machine learning system. But it's important to know that these common examples are there. We also have novel examples, so particularly what we might call complex examples in information theory. This means these are outliers. These are kind of people that are not in the norm. These are sometimes personal data as extracted from chat GPT. These are things that, for a variety of reasons of how we train machine learning models, particularly very large deep learning models, these are artifacts of the training process because we reward the model when it sees something kind of similar to this in the future it's worthwhile using some parameter space to store that information this all goes back to information theory if you remember that from your uni studying years and finally why does this happen well we used to have this nice divide we'll talk about in a second between overfitting, which prevented memorization, and generalization. But now with overparameterized models, we can do both. In fact, it is actually useful for us to do both. Because of the information that we get from these novel examples, plus the common examples, we can both have enough parameter space that we both generalize well, we learn common things, and then we also can learn and store uncommon things in the chance that later this information will be useful for us. And here we have an example of text completion where green means the text exactly matches the training data, and you can't see it, but it's 6 billion, 2.7 billion, 1.3 billion parameters. And you can see the distinguishing remarks there. So who cares? Catherine, you just said, like, memorizing data is useful for learning. This is something that we need to do when we build large-scale, especially general-purpose deep learning systems. Why do we care if data is memorized? Anybody? Privacy, thank you. Thank you. That's what we're going to talk about for a little bit of this talk. When we memorize sensitive data, we expose those people to what we call privacy attacks on machine learning systems. We're just going to talk about a few of them today. There's many more. but today we're going to talk about two of the main ones that are relevant for machine unlearning and the first is a membership inference attack so we have a model it doesn't matter if we have access to the underlying weights or not we have just a model we have a target example or we have an idea of a target example we don't even have to have a fully formed idea of this we could do something near what we think and we could move closer and this target example we want to determine this is us as the data scientists or machine learning hacker and what we want to do is we want to use the query to determine whether the model has seen the target example or not was the target example in the training data or not and we do this actually by training the best way to do this is we actually train a discriminator here that we can use locally to predict if the target example is in the training data set or not, and we can actually do this with fairly good accuracy and high fidelity. How do we do this? This is now called an MIA distributor, or there's different varieties of LIRA attacks and other things you can read about, but we basically take the training data, our own training data, we take training data that we think is similar to the training data of the model. It doesn't actually have to overlap, it just should be in the same domain space. We shard it. We randomize it into subsets, but we retain the information about those subsets. Then we train some shadow models. Because of the properties of transfer learning, these can even be smaller deep learning architectures. They don't have to perfectly match the architecture of our target model. But of course, the closer that we can get, the better. And then we take essentially the logits or the outputs or both of these models, and then we store them. And the information and the labels that we store with them is whether certain examples that we want to now train for are in or out of the training data. And then we essentially train our discriminator with that information, whether the example was in training, out of training, and with this as the sole feature space. So now we have this feature space of the model outputs that we can then use to determine whether an example was in or not. Why does this matter? Why does it matter if I can determine whether your data was in the training data or not. Is there any risk for you for me knowing whether your data was trained upon. What if you want to remove it. What if you'd rather remain anonymous. Or what if I can use that information to actually exfiltrate your data. What is data exfiltration? It means I can steal your data from the model. I can take the data out of the model as if it were a database query. How does that happen? Well, I'll give you one example. There's many exfiltration attacks, but I'll give you one example called an inpainting attack. How many people here have used diffusion models before? How many people here have maybe used inpainting? Like make the picture bigger, color in the edges, take this out, and replace it with something else. This is a very easy feature to use on diffusion models. Nicholas Carlini and other group of researchers from Google Research were able to take and train their own large diffusion models. They had their targets. They put this mask to the model, say, please inpaint. If the model had seen it, we'd get a pretty nice rendition. If the model hasn't seen it, we get a pretty poor representation. And we could actually do this in iteration. So that's why I say we can start with a fuzzy example, and then we can get closer and closer until we get higher fidelity. And this is because we're using things like the membership inference to give us information about how confident or what levels of confidence, what levels of information the model can give us about what it's learned. And then we're reversing that to try to extract high-quality information of what it hasn't learned. And the reason why this is so pixelated is they were able to do this on much, much smaller data sets on CIFAR and stuff like this. So if you do it on CIFAR, you could do it on many other data sets. This is a big problem, right? Because if we can exfiltrate data, this means that in your code programmer, you're maybe repeating somebody's code. In your creative writing course, you're repeating somebody else's creative writing. In your image generation, you're showing a model's face. So all of this is maybe something that we need to think about. So what if I want to avoid this? So I'm in charge of the machine learning system at my work. Maybe I want to think about how we can avoid this or what to do if somebody wants their data removed. We're going to talk about imaginary level of effort right now. So we're going to re-come back to this thought later. But first we're going to talk about what is the level of effort? Retraining from scratch without the information, right? That's pretty high level of effort. If we had to do this every single time, this is a lot. And can we even fully find what we want to unlearn? That's another question for later. We have differential privacy. So if you haven't seen me talk about differential privacy, you can look it up. You can look up many people talking about differential privacy. Differential privacy will not reduce the impact on the common examples that are memorized, but it will reduce significantly the impact on the novel examples. And yet at the same time, these novel examples have information, and we also lose that information. So differential privacy may be a medium level of effort because also first we have to build the differential privacy infrastructure, then we have to be able to train it, and then we also lose something. So maybe that's, like, then we have to figure out how to get that information back. And then we have this magical field of machine unlearning. So machine unlearning research, you can look back, kind of started around the time of GDPR, surprise, and then slowly gets more and more traction and also continued traction now because we have these massive, massive generation models, and we have the problem that the generation models continue to output copyrighted text, text that it shouldn't know, people's personal information, end, end, end. And so enter machine unlearning. What we're going to do is we're going to do something that is much easier than differential privacy. We're going to do much easier than retraining from scratch. We're going to unlearn. But what does that mean? Does anybody, like, how would you define machine unlearning if you had to define it? Any ideas? Yeah. Just like remove the sample. Maybe that's good enough. Okay. We'll start with that example. So maybe one thing of unlearning is we can use what an idea that is in research is model distance. What we essentially have is we have some forget set. Maybe it's one example, like you say. Or maybe it's many examples in this forget set. Then we have a model that's never seen the forget set, and then a model that has unlearned the forget set. What should we expect? Should these models be near, far? Maybe it was some noise, yeah. So what we're going to do is we're going to project them into some sort of space where we can compare them, the weights to each other, maybe we decide some sort of projection space that we use, and then we measure the model distance. Now and this model distance we want to say is small because we want to say that never seeing the example is similar to unlearning the example. Does this work? So how many people here remember from linear algebra theory permutation invariance? Yeah. Why doesn't this work with permutation invariance? We can have different initializations. We can have a variety of different transformations, permutations that happen at every layer, all the way to our ReLU or whatever it is that we're using, and the model can actually learn the exact same linear algebra, the exact same structure of transformations to end up having very different weights, from a distance looks good, looks like they are very different, And yet they've learned exactly the same function at the end, right? And so because of the way, by nature, the stochastic nature of machine learning, and the way that we initialize weights and so forth, and also the permutation invariance that is inherent in linear algebra with certain problems, we end up with this problem. That this sounds like a good idea, but it just doesn't really work. If so, then we have to do a lot of other work to make sure that it works and that we don't have time for, right? the whole point of unlearning. So what could we do? Maybe we take this privacy attack, right? And so this is a different way of looking kind of at the membership inference is this likelihood ratio attack. So I told you Lira, this likelihood ratio attack. What's the likelihood that an example was in the data set or not in the data set? And when we take these likelihood ratios, what we kind of use is some sort of metric we get from the model like scaled logits, like confidence intervals, and we try to discern was the example in or out based on these outputs. And to do so, we make some sort of hypothesis line. We can use Bayesian reasoning. We can use machine learning. We can use any number of things to decide to learn where we draw this line, and unlearning developed their own unlearning LIRA, so likelihood ratio attack. And essentially your goal is then to decide what is the model that never saw the example outputs versus what is the model that unlearned the example or examples outputs. And you have to kind of take these distributions and you want to try to separate them. If you can separate them easily with little overlap, that means we're leaking a lot of privacy, right? We're leaking a lot of information between the unlearned and the learned model. It's hard for you to do if your hypothesis line actually ends up getting a lot of, let's say, false positives or false negatives, then we have succeeded in building a response that is almost like we never learned the information, which is our goal, right? So maybe this is a cool metric that we can use. Maybe we can start to look at these responses and have a measurable attack ratio on forget set examples. But we also need to take into account that this will be different on different examples. So if we average this across many examples, there are maybe some examples that are easier to unlearn and more unforgettable examples that are harder to unlearn, particularly complex or novel examples. Or we could just define it in a totally different way. So if we don't want to make it repeatable, if we don't want to make it so that it's generic and works on every deep learning model, we could zoom into the task space. What are we trying to do? And in this case, this was Microsoft Research, who wanted to try to unlearn Harry Potter, Just as a, so they could publish something. I'm sure they were trying to unlearn other things too, but they said they were trying to unlearn Harry Potter. Okay, so they're trying to unlearn Harry Potter, and they have this token distribution, this baseline is the original one, for next token probabilities of the prompt Harry Potter studies. And what some of their idea was, let's take some LLM-specific types of behavior, and let's unlearn just this LLM-specific types of behavior. and what they decided their definition of approximate unlearning is that it would unlearn enough about Harry Potter to make us a little bit unsure how much it knew about Harry Potter as a user right and what we can see here is at the very baseline so what it knew about Harry Potter in the beginning is Harry Potter studies magic and at the end it says Harry Potter studies the or a Harry Potter studies law, right? And this is after like 20 steps, 40 steps, all the way to 120 steps, we can see the distribution of probabilities in kind of the most likely next tokens, right? And so we could also take a particular task example and say, we're going to zoom in on this, we're only going to solve unlearning for one particular task example. Of course, there's a lot of other interesting bits of this paper that I don't have time to present today, but there will be a blog post soon, so yeah. But what I hope that you're seeing is that we need more standard definitions to compare results. Like right now in unlearning, kind of like some people are using model distance, some people are using this, some people are using that. How am I, as a machine learning practitioner, how am I supposed to decide what unlearning works for me? And there's deeper questions, too, of what unlearning means. does removing Monet mean removing his influence on the Impressionists? Like, how can we really prove that we unlearned Monet? I asked GBT to help me remove Monet from Impressionists and to give me a painting, and this is what I got. So, I don't know. I'm not an art historian. Maybe we can debate. If you know an art historian, we can have some fun together. But what do we mean about influence? What do we mean about the fact that if you unlearn one concept, maybe there's many many many related concepts and there's many things if we study information theory which i think we should probably study more often than we do we also understand that when i first learn something i learn more information than when i learn it the second and third and fourth time and also when i learn about one concept it can teach me about other concepts right so information is inherently linked that's what we're trying to do with deep learning And so it starts to become difficult if we take away one example, what information do we remove from others, and also what do we retain still of that initial information? And in a related problem, if we remove just, let's say, copyrighted photos or something like this, it doesn't actually remove that information. That information can come from other places. If we remove Batman, it doesn't mean that people aren't going to be cosplaying as Batman in our training data set. So what does that mean? It's even a non-trivial problem to find duplicates and near duplicates. So these are all kind of like problems that are related that we have to define if we want to define unlearning. But we're going to move on because we're not going to solve the problem of unlearning definitions today. So I want to show you what is happening in unlearning research. How can we actually unlearn? well one is to actually change the training and so we have this approach that i'm going to show now called sisa unlearning but many approaches kind of build on this idea and sisa unlearning says essentially shard your data then train your models each model has only seen a shard of data then you use inference aggregation some sort of mixture of experts we like to call them now some sort of inference aggregation to take in inputs from the many models. And what you can do as you train is you can actually also make checkpoints and you can introduce the epics in a way that they're more structured and therefore you know when the data was first seen and when it was last seen. And this might help you roll back, so to speak, a particular model. So this means worst case scenario, you have to retrain one of these models from scratch, but you don't have to retrain all of the models from scratch. An other worst case scenario is like, or the next worst is you can roll back to a checkpoint and then train from there. So these are things, this is one way to change the training. Another way is to use gradient-based approaches. So normally we try to descend the gradient, right? Normally we'd go in the inverse. But what if we took the forget set and we actually just gradient assented? We follow the gradient. Any problems, any thoughts? What could go wrong? Yes. Yeah, we could forget really important information that we need. never be able to draw cool guy who knows about AI again. And then I wouldn't have any good pseudonyms online. So we maybe don't want to do that. So we can take our forget set and we can take a retain set that we pick. This is now a level of effort is growing because we must be a little bit clever with how we pick the retain set, probably same class or nearby or in generative, it should be similar enough in information. Then we do DPO, direct preference optimization, or we do some combination of loss maximization for the forget, loss minimization for the retain, and we hope that in the outcome we have retained enough information that we still know what cool guy with AI is, but maybe this time he doesn't have glasses, right? And then another idea, because this, of course, starts to add a bunch of extra work to pick a good retain data set, is to take the forget set and to then move it towards some sort of pseudonym. So we take pseudo labels or we take generic labels or we take pseudo classes or generic classes and we try to optimize the forget set towards these pseudo classes. So unlearn the original data, push it towards some sort of generic definition. And this, as you can imagine, works for some problems well and other problems not as well. And my question is, does it scale? So what if you get, so all this fine tuning, you're putting in all this effort, maybe you have to make retain sets, maybe you have to introduce new types of preference optimization for your workflow. What if you get 100 delete requests a week? That means you're going to have to be doing this fine-tuning training, then you're going to have to be testing, did we actually unlearn what we wanted? Then you deploy, then you get the next week of requests, right? So this is actually a lot of compute time. What if your data expires about every six months and you're actually already retraining models every six months? Then you're already retraining. It doesn't even make sense, right? And this has to do with your data retention periods, which hopefully you know about. And my question when I read this research is, who has the actual know-how to do privacy auditing at scale? There's not a lot of good libraries for this that are automatic. There's a lot of domain-specific knowledge that happens. So who can run this operational infrastructure in any manner of scale, even though we haven't even fully defined what unlearning is? And would you be able to even test for memorization and prove that it is or isn't happening? Have you yourself run your own Lyra attack? Probably not. And it's not your fault. It's that it's not set up to do it this way. And the other problem that we have is unlearning actually hasn't been a very holistic field of research. So unlearning methods that we have today actually leak other privacy attacks. If I can take a model and it is different by one example, you've unlearned one example like we did. Then I can actually reconstruct pretty well the gradient of that example. And then I can reconstruct the data pretty well. That's actually the exact same way what was called back in the day a model inversion attack works. So this is, for example, a very similar type of attack that basically just tries to optimize the gradient for a particular target class. And also, unlearning, if done inappropriately, leaves all these weird artifacts in your confidence and your outputs. And these act as information side channels. I can gain a lot of information about what was unlearned, how it was unlearned in those attacks. The other problem is there's something called a privacy onion effect. This was discovered by Google Research that, again, these novel examples, they hold special areas in decision space. And when we project layer by layer on a model, we can find things that operate a little bit like margins and support vector machines. We can actually use this to study generalization. And when we remove those data points that are novel, that are special, that are near decision boundaries, we actually end up exposing new points. And this happens all the way down until we basically have no information. When we remove at-risk examples, it Bi automatically puts other examples at risk. And what I want to repeat is this happens for outliers more than other people. This happens for at-risk populations more than general population. This is who is being exposed. So I think we need to rethink unlearning, and I think we probably need to rethink learning. We need to have real conversations. This is a multidisciplinary problem. It needs a multidisciplinary solution. It needs people from product, people from privacy, people from risk. We need to understand and evaluate risk and reward. We need to do that as part of our work. And we need to build infrastructure so that it's easy for all of us to do the thing that we wanna do, to do our work in machine learning, but make security and privacy a normal part of our workflows and make it easy for us to opt into these types of things. Finally, I wanna end today with talking about privacy as a human right. Privacy is a human right. When we expose at-risk people, we are exposing people that might be targeted. That happened with the IBM Lockhart in here, and is happening today. And so I ask of us, if we don't take the responsibility as a machine learning community, if we don't take the responsibility of thinking of privacy and security as part of our work, there will be people that exploit the fact that we don't do this. And so I would like for us to do something about this. Thank you very much.