PPML: Machine Learning on Data you cannot see

Speaker 1 [00:03]

Thank you so very much for being here and for the nice introduction very much appreciated I appreciate this is getting towards the end of the conference so I'll try to make it as small fun as possible interactive so as we were saying that the title of my tutorial is preserves are in machine learning and I put in the slides the further the cover one the the link to the slides this deck actually on speaker deck and the repository we're going to use in a second I'm going to start soon I just want to tell you what's the plan for today I'm going to start with this presentation and I'll tell you what we're going to talk about what's the plan and blah blah and then we move on to the hands-on session and we'll try to do whatever we can do together so the sorry the aim of this tutorial is to provide you an overview of the emerging technologies in the in this area which is known as previously preserving machine learning and in particular we're going to focus on the Python ecosystem there's lots going on actually around around the the data science world but we're focusing on Python and the the approach I'm going to use is the design test approach so we want to always privilege the development and practical aspects of the thing so So whenever I'm going to throw you notions which might be new to you, I don't think I will have the time to go very much into details. I'm very happy to talk about it later, but not during the tutorial. Otherwise, it will be 19 hours, not 90 minutes. And the perspective will be the one of the researcher. And by that, I do mean I want to give you references and further readings and stuff you can read. And there also will be some live coding. Actually, that was the intention initially. We're not going to do live coding, but we'll do coding together for sure. And why I'm doing this, this is an announcement I wanted to put into the slides. This is the very first step of my plans for a fellowship with the SoftSense and Ability Institute in the UK. My plan is actually to give awareness of these technologies. And this is indeed my very first tutorial. So you're the first audience I have, and I very appreciate your time today. And which means, from my perspective, whatever sort of feedback you might have, for me to improve what you're going to listen to, that would be awesome for me. So thank you very much for that in advance. Since I didn't know the audience, and let me just start with a question. How many of you are familiar with PyTorch and deep learning terms? Okay, so for those attending online, it's the majority of the audience. Is it me doing this? Sorry. and that's my roundy faces getting in the way I will be very quick then on this warm-up since I know really the audience on I'm gonna give you just a very quick overview of the main things we I assume you know already so some of the terms in the deep learning things epoch stances parameters hyper parameters loss optimizer all these terms should be um shouldn't be new to you if they are please let me know we'll be using this like uh bread and butter later and in in case uh you want another view of what's going on with the deep learning thing in the python ecosystem well uh in the in the python zen in the zen python we can say that there's one and there should be preferably one way to do it we know that in deep learning frameworks, this is still not the thing. We might say that the machine learning with scikit-learn is essentially that. Although scikit-learn is not the only tool, of course, but it's essentially the standard, or the most popular one. In deep learning things, there's still plenty of solutions, although I want to highlight what, this is actually a slide which is old, more than three years, it's 2018 actually. And this thing is actually improving, and we indeed had a very interesting talk in this conference about data APIs, standardization of n-dimensional arrays, and this is exactly one of the important use cases that standardization might be useful. So going towards having the same different frameworks, but indeed the same API, which makes transition from one to the other very cool. And in fact, this Twitter from François Chalet, a creator of Keras, was essentially showing, showcasing four different frameworks to do exactly the same thing. And you have to really, really be careful in understanding the very details in it to really understand which is which, essentially. So they look very, very similar. And it's going towards that direction of standardization then. In the deep learning, we have two main family frameworks. we have static graphs dynamic graphs study graph you have one static graph in memory and it goes the graph is important because it will be part of of the computation we're going to run and so you have to keep in mind that whenever there's a deep learning model going on there's a graph and a computational graph which is which is being sent and processed somewhere dynamic graph frameworks are actually creating this graph for every epoch as in the animation And PyTorch belongs to this family. And so the backward propagation works slightly different. And the static graph is just like you have the graph. You can back propagate it. In the dynamic graph, you typically record whatever you're computing, and you replay backward to apply the optimization in gloss. Very quick rundown on main features of PyTorch. NumPy-like API, again, very good of having standardized API for NumPy-like objects, like tensors. You can convert tensors and ndArrays. You can have either way. You can have CUDA support, which is brilliant. And whenever we're going to define a model, we're going to have a class, which is extending nn.module. So whenever you have to define a model in PyTorch, you need to extend nn.module. You have to implement the forward method, and that's it, essentially. And so in the constructor of the class, you have the definition of layers, which are the tensors. And then to complete the learning step, the training loop, you need to define the optimizer, zero the gradient, and define the loss you want to optimize. And last but not least, you also have support for dataset and data loader, which is brilliant, fantastic, on encapsulation in the Python data model. We had a talk about that later yesterday, indeed. This is an example of Cypher using TorchVision, and we'll be using something like that. good now finally let's talk about privacy very quickly and and what's the deal with privacy the thing is that ai models are indeed data hungry and so the more data you have typically the better to you are the better the model and the thing is that there is indeed a push in in the community for high quality and curated data set the problem is when this data set contain data you're not allowed to see so data that you cannot process because they're sensitive data and so there might be different solutions the the first thing and most obvious thing you can come up with is okay I have data and I can put data and models in silos they will be on locked out locked up and will be controlled whatever processing we're going to do with that this is actually very impractical to be fair and also one of the most popular thing to do is to apply what is called anonymization techniques or maybe data should account for privacy the thing is that normal anonymization technique might be leaking information and this is just one of the many example I can give you are the very popular one on the right-hand side of this light actually left-hand side for you is the robust the anonymization of the Netflix bite it set is very popular paper actually and the thing is that this isn't a showcasing of an example of what is called linking attack so you can even if the data set is indeed anonymized by itself you can link external data set to leak external information and that was actually part of a competition very interesting and so advanced more advanced techniques are actually needed and we'll try to talk about it later so on the very first idea I want to throw in here is why don't we allow analysis without moving data from their own silos at all and this is essentially leading to what is called federated learning. Federated learning is a wonderful paradigm actually, it's a protocol of communication and the idea of federated learning is that you have data not moving at all from the different owners. So you have nodes and this network which owns the data and all you need to do is essentially sharing the computation you do on your own data. So there's this accumulating cloud there, which is getting all the computation from the many, many clients. And honestly, this part of it worked not specifically for federated learning, sorry, for privacy initially, but it was the model that Google was using to train multiple different mobiles. That's why you see mobile phones there. And so now the question is, is that it? So is further learning the solution to rule them all? The problem is that the answer will be no, of course. Otherwise, we wouldn't be here talking 90 minutes about further learning. And the reasons are manifold. I'm gonna show you two examples we're gonna see later, practically. One is that models are vulnerable. And one of the most popular example you might probably already know is what is called a Versailles example. So you can perturb the data, you can change the input data to the model in order to make the model to make mistakes in predictions. And the reason why you want to do that may be many, actually, depending if you're a malicious user or you just want to test your model. It doesn't really matter, to be honest. The thing is that models may be tricked. And this is a very simple case. It's kind of a popular one. As in, you have this input image, which is correctly classifies as panda. And then you modify with some noise, which is actually carefully selected. And then you come up with the last image that you cannot perceive as changed, actually. And this is the trick with the adversarial examples. Humans cannot see those being tampered, being changed. But the model will, of course, notice that. And it will classify it as a gibbon. This is another very interesting one. And this is called model inversion attack. We're going to work on this specifically. This is taken from a paper. And this is also exposing another quite important vulnerabilities in models. This is taken from a paper. And let me be clear, this is not the generative model. So the picture on the right-hand side is the original photo, the figure on the left-hand side, or probably the right-hand side, sorry, the blurred one, has been extracted from a trained machine learning model, which was not a generative model. So it was not a model trained to generate data. It was a model trained to classify faces. This is a popular one. This is back to 2015. But this showcase you the fact that you can actually leak sensitive information. Imagine if this image was actually a medical data set, something you absolutely don't want to share with anybody because it's super sensitive. It's the definition of sensitive data, all right? And so we have to be aware about those things. And there's also another bit. Fiddler learning requires encryption if you want to implement it because encryption actually is everywhere. It's in the communication on the channel because if you if you transmit the data on an insecure channel actually you don't want to do it but we want to do an actual step further and we want to make the encryption to be smart in the sense that we want to to get encrypted data you we don't want to decrypt the data because they're going to be sensitive but at the same time we want to make computation on this data and this is actually a technique called the amorphic encryption we're going to see that in in action later on, but just throwing you definitions you might find familiar or not. Just let me know in case. Olmorphic encryption is an example of astymatric encryption in which you have someone holding a private key and a public key. The public key is the one used to do the encryption and the private key is used to do the encryption. And this is essentially the interesting bit here is that you can process encrypted data without deciphering it, and you can actually do computations. This is the interesting bit. So whenever it comes to machine learning, it's always a matter of multiplication, summations and stuff like that. You can do that on encrypted data, and the decryption of the result in the ciphertext is going to be exactly the same as it was in the plaintext. This is the brilliant one. And last but not least, in the context of previously presented machine learning, we cannot mention differential privacy, which is nowadays the standard for machine learning method, which is based on a different assumption. Not going to go very much in details of this, but the assumption is that instead of doing any encryption, you start is based on the idea of plausible deniability. And differential privacy is a system for public sharing information about the data set by describing the pattern of the groups without sharing the information about the individuals, which is most of the time what exactly we're aiming at. And to give you a very practical and effective example, let me just introduce you to differential privacy in a way I've seen done in this video, which was actually brilliant and very on topic because this relates to something we'll be seeing later on. It's called differentiated privacy on PyTorch. It goes like that. So imagine you have this data set, and this data set is essentially lots of pictures, and you have a very high level of details in this image. The thing is that you can use this image as a whole, but most of the time you're interested in using, like on a single picture, but most of the time you're interested in the aggregate of the data set. And essentially, what you're actually interested at into using that as a whole, as an aggregate. The thing is that these aggregate contains sensitive information because it contains the faces of the people in this. So this might be private information. What if we just blur this information? And so if we do that, will that affect the aggregates? as you can see not much idea of using differential privacy most of the time we're not even interested in looking at the individuals we just want the aggregates for the model training but if we do if we do get the individuals we're not allowed because they're sensitive information so why don't we do get a sort of noise version of this data so that we can still compute the the aggregates without leaking any private information. And I'm telling you this as a story, as a fairy tale, which makes sense, more or less. But the thing is, this framework has a mathematical proof. So you have mathematical guarantee that you're not leaking any information. This is why it's becoming the standard. And so aggregate can be counter data, computing mean, or it could be complex training and machine learning model. And actually, you can apply differential privacy at different steps of the pipeline. And the gold standard now is to apply it during the training. You will be seeing that, what is going to be the effect of that. Cool. So this is it. Before we jump into that, do you have any question? Expectation or things not entirely clear? Good. Okay, then. So, this is the repository. I'm going to open it now. I don't know if there's any... Oh, yes. Oh, very quickly. What's the agenda? Three parts. We'll try to do our best. Probably, we can jump through different bits. First part, model vulnerabilities and attacks out of CyOx samples, the panda thingy, and the model inversion attack. Then we move on to further learning, we talk about further data, further machine learning, so split, and then differential privacy and deep learning with model training to counter attack model inverse, model inversion attack. First block, just PyTorch. Second block, OpenMind, which is a wonderful community, defining many tools in this context. And the last bit is about Opacus, which is another tool in the PyTorch ecosystem. This is the main three blocks of technologies we're going to use later on. Fantastic. This is the links, again, to the repository and to the slides, if you want to. if you're free to catch up with me later on Twitter get a mail whatever you like good so first things first let's grab the code on the repository and let me know if you have any any problem there should be I'm going to show you you directly. That's the wrong one. There you go. Okay. Now, I'm going to take this, make it very big. Is that big enough? Okay. So, in this repository, you may find the get ready notebook the get ready notebook is the notebook that shows you the instructions on how to set up the environment and to test that you know the environment is working I don't know if you're using anaconda Python which is the suggested one if you're not you can use the the there's the beep requirements equivalent to install all the packages in your virtual environment is that highly recommended to set up the environment first because the especially for the second the third part of the specific package we may need all right All right, so let me just start with that. I should have probably already going. I was wrong. Let me just start the Jupyter server. Are you all familiar with Jupyter Notebook, by the way? Is that a tool you would use to do this kind of thing? Brilliant. That was just the wrong port, I think. Why 8282? It should be 88, 888, right, there you go. Good, so if you open the get ready, is the font size big enough for you? Brilliant. So first bit, set up the environment, and essentially this is the command you want to run. I tried to test it on multiple platforms, not entirely sure Windows is gonna work, Should be, tried, not entirely sure, I can't guarantee it, and I'd say then in the rest of this notebook you should execute the following cells and there are a couple of cells to run to test your environment. So if you're using conda, there are two main commands you want to run. Conda mcreate first, and then conda activate ppml, which is going to be the environment called as reported in the file. The file looks like that, in case you're wondering. That's the one. A few dependencies. Kind of a standard one. I think the most important one to report here are packers, of course, as I mentioned. PyTorch. PyTorch is going to be this version specifically because it's a dependency of SIFT we're using. We're using version 0.5, which is currently actually not very up-to-date. There's lots going on. For what I wanted to present to you today, I decided to go with this version, although I will give you reference to what to do next with the new version. There's lots going on with this PySIFT project, and changes are being applied is there anyone who's having troubles in setting up the environment I can help you with anything I can help you with oh really me yes that would be interesting to report what kind of yes that one yes this is the the fact mm-hmm that actually I didn't see that coming I am I'm being honest 1.8 point one is not available in condo porch okay for silicon I'm sorry about that that is something I will keep in mind for the future I'm sorry I didn't this time I don't have an m1 to be fair as you can see it's an old-fashioned one and and so I haven't experienced this kind of problem myself sorry I tell you I tell you what for those of you not willing to this is something okay this is something thing also to keep in mind for me sorry about that in case okay I'll tell you what one thing you can do you can run everything on colab that should work the problem I didn't do it in the first place is that if you go but I will very open to suggestions about that about the organization sort of thing if you go to the for example in the very first bit and model vulnerabilities in the model inversion attack. I prefer not to saturate the notebooks with Python code, so I used external module. Sorry about the audio, which is weird. And so when you want to run this in Colab, you have to import the files manually. Normally if you want to run notebooks from Colab, sorry, from GitHub to Colab, you have to have everything there, otherwise it's boring and you have to drag and drop new files. If you're fine doing it, you cannot do any setup, it can run everything in code up. Not all the notebooks will have this issue actually, but you can do that alternatively. It is, it is. You can, yes, yes. But, oh, you can git clone it within Colab? So, oh, yeah, that's something I never tried. Okay, so you don't need to drag and drop anything? We do it for large data sets and data. Oh, that's brilliant. Okay, okay, that's even better. Good. So, yeah, you, yeah. And does the git clone set up the environment of anything, so, no, so you have to do it manually, okay. So, I tell you what, for those of you starting to work in Colab, I'll tell you what are the packages needed for different bits. For the first bits, you just need PyTorch, so don't worry about it. If you weren't running PyTorch, that's fine, and that's already available in Colab. Good. Let's get started then, if you don't mind. So I'm going very quickly about the fast sign gradient method. I want to talk more extensively about the model inversion attack, which is more interesting to me. me. Just going through this, and to show you the example here of the panda thing, there is an exercise in the end, and you're very welcome to do it on your own later. It's not really important. I just want to tell you how easy it is to implement these things and also to test the vulnerabilities and models. So normally here what we're doing is the plan with this notebook. This is adapted from the original tutorial in the PyTorch tutorial. I didn't invent really anything here. I just want to point you the interesting bit. We're using Linet, pre-trained on MNIST, classical example. And there's something you should be able also to run on a laptop. Nothing is computational heavy here. So this is a very small convolutional network and we download the MNIST dataset, you can go through and run all the cells, it should be working and in the end you get an accuracy on the test data of 98%. Then we deliver the attack and the attack works in a very sneaky but clever manner. The idea is that we want to change, so let me go back to the idea here. So the The idea, we want to change by a certain amount, which is this epsilon parameter here, the original input. And the epsilon controls how much noise you're adding. And the more you add, of course, the more it's visible to the human eye. The interesting part of this is that it's actually very easy to implement. Because all we need to do this, the sign of the gradient is essentially cleverly selected to fool the optimization. because this is essentially taking the sign of the gradient, which is the opposite of what the optimizations do. The optimization normally goes in the opposite direction of the gradient, because we're minimizing things. Here we want to maximize, and so we want to pull, let me say, pull the optimizer. And so we are getting the gradient. And PyTorch makes things very easy to do, actually, from an API point of view. So once we have the prediction on the test data and we run the attack, what we do is if the prediction is already incorrect, it's fine. We skip it. Otherwise, we take the gradient of the data. And so we run the attack. It's a very simple function implemented by an epsilon here. And then we rerun the model using the perturbed data. And then we essentially calculate the accuracy. And if you run this attack with an epsilon of 0.5, what you get in the end is just 94%. So you're already ruining the accuracy. And this is the adversarial example, five adversarial examples generated. So for example, this was originally a six. It's been predicted as a four. This was eight, predicted as nine. so on so forth this is just with one epsilon value the exercise then tells you to try different epsilons and in the end also seeing how can you uh how can you detect the the fact that these epsilons have been tampered originally so the the bigger the expectation is the bigger is the epsilon the more the damage on the accuracy but also the bigger the epsilon the more you identify by the fact that that's a perturbed and changed data, essentially. You can run this on your own. It's not absolutely important if you don't. And this requires some time, so if you don't mind, I can just close this on this point and then move on to the next one, unless you have particularly interest or curiosity about this. okay any anything from the audio from the online audience yes glad yes yes that's a very good point so the point is is there any way I'm should I repeat it for the online audience okay yes the the question was about is there any way to use this knowledge of other side example in a clever way to to counter-attack essentially this is what you're asking isn't it so how can you now that you know there's vulnerability what can you do and the plenty of things with the the short answer is yes you can use the the other side examples the thing is we're not talking about countermeasures we're just showing you in this case vulnerabilities but of course the next step would be okay I should do something about it yes very good point let me just move on to the model inversion attack which is way more interesting and actually even more intriguing to do and very quick so the reference paper is this one model inversion attacks exploits confidence information and basic countermeasures I've reimplemented on my own two of the three models presented in the paper and The experimental settings we've been using is exactly the same of the paper. Good. So let me just show you what's going on here. So the very idea of this, the first thing is we're going to train a model to recognize faces from the AT&T Faces dataset. I wrap this up into the PyTorch data set object. And this is called All Faces. And you should be able to download it on your machine and get it. I don't get the download because I already have it on my local machine. But this should start to download and to get it. If you have troubles, let me know. Then we set up the data loader. OK, a few notes about the data set. The data set contains 400 images of files corresponding to four different subjects, 10 photo each. Each photo is that dimension and grayscale and is taken at different times, varying the lighting, the facial expression, open and closed eyes, smiling, serious face, and the facial details. The train test petitions I applied in the training exactly the same the authors are using in the paper we might argue this is not very clever from a classification standpoint because this is not very good petitioning actually but for the sake of the model inversion is very clever the what the the paper is doing is randomly picks for each of the subject seven faces out of ten those will go into trading the rest the three remaining ones are going to test set all right of course from this classification point of view is not very clever, but this is where it is. Good. Okay, so let me just show you these faces and also have, so this is example faces from the new set. Let me also show you another note, which is a sort of a question for you. Would you please run this and also tell me that you see exactly the same face. Those are randomly selected because I skipped this, but I should focus a little bit about this. This is taken from the training loader. The training loader has the shuffle set to true, so they're randomly taken. And when I run this, you should have exactly these faces. Do you? Brilliant. Anyone having any guess on why this is happening? Yes, it's very up. And this is just the side information I want to give you. Always remember to fix the seed when you have random things, basically all the time. So if you really want to make things exactly the same, regardless of the machine, fix the seed and you'll be there. So that particular case set the seed to NumPy and Torch, and this is the reason why you're having, despite the shuffling, you have exactly the same faces displayed. Good. So if we move on, I already implemented the soft map. So So the models in question are very, very, very, very, let me say it again, very simple. It's called softmax regression as an extension to logistic regression. It's 2015 paper, so the terminology is not super up to date to what we have now. And then it's multiple layer perceptron. And let me just show you, oh, the learning rate is just 0.1. Let me just show you what I mean by softmax regression. This is the model. It's just one linear layer ending up with a logarithmic softmax in the end. So we do train this model. And then we save the best model here. You can run this. It's very, very quick. You can also run the MLP if you like. But it is essentially taking more time, so I'm not going to do it. I already saved the checkpoints. And you don't need to run this if you want, because later you can download the checkpoints already. So ultimately, this model is going to have a 94% of accuracy. The multilayer perceptron is going to be slightly better in classifying faces. So again, the classification deal here is to classify which is the subject the face is belonging to. Very simple. Oh, by the way, this is the best validation accuracy we got. And so this is the model we've been saving as a checkpoint. Good. You can run this. You're not really required to do that. And the multiple layer perceptron is a linear hidden layer and prediction layer like that. Now that we have the... What you're having here is that you're using models has from an API. You're not having any knowledge about the model. You haven't you're having any knowledge about the data and you don't know anything at all about the model is a black box In fact this particular kind of attacks is called black box testing. Sorry black box attacks. You don't have You don't need knowledge about the internals of the model. You just use the model as a black box So far so good Let's keep the watch under control so now the point is we have the train model we want to deliver the attack now so in this notebook essentially we're saying you can keep completely the training is really not important you can just reload in the model on the best on the best training this is exactly what we're going to do so let's try to run torch okay we get the dead so this is the size of they sent 280 samples in the training 120 in the test set something something up to 400 or make sense and we get the training data which is the big one good okay let's deliver the reconstruction attack and so let me just show you first what where is the attack oh it's probably downstairs in there right this is the idea of the reconstruction attack very quickly the thing is we want to pass in the label we were aiming for just the class so the the only piece of information we need is is that we are dealing with a classification problem and we're dealing with some classes identified by numbers, essentially. So 1, 2, 3, 4, depending on what's that in. Well, that might be. And then we pass some parameters which controls this attack. Alpha, beta, gamma, and lambda. Lambda is a learning rate. Alpha, beta, and gamma essentially are some threshold. So alpha is the total number of iterations we run the attack up to. Beta is the maximum number of iterations without any improvement. This should be without improvements. Gamma is the threshold of the cost, so the maximum number of loss we can calculate during the attack. And this is the very idea. So this is the general attack. For this particular case, we're not using any auxiliary term, so this is just to emphasize we're not using any external information. So the auxiliary term in the original paper is specifically to refer to any additional information you have on the attack you're dealing with. We have no clue whatsoever what's going on. So this is just to highlight that the initial cost is the cost generated by the model. So it's the actual prediction of the training model. And then we keep on going up to alpha iterations by processing the data we get from the model and trying to optimize the generation of these tensor we're creating by knowing which is exactly the label the model is producing. This is the information we're actually stealing from the model. So we're getting the prediction from the model as a black box, and we're using that as using the optimization in the other hand. So let me show you this in code. So we start with a tensor of zero. This is the MIFACE implementation attack. Yes. We start with a tensor of zero, so absolutely nothing. which requires gradient because that's the trick. We start with the model which has been pre-trained and so we get the prediction from the model actually the full tensor of logits here. Then we set up a loss for the attack and we iterate up to a number of alpha iteration. We reuse the model. We zero the gradients of the tensor we're modifying. And then we get the target class generated by the target class is the one we are aiming to, so the one we want to steal. And we calculate the loss between the prediction and the target class. And then we use a stochastic rate descent optimization on this tensor. And the last bit is this process function, which is very generic. Actually, it's not doing anything at all. This process function is just normalization between 1 and 0. It's like dividing by the maximum. So it's the tensor minus the minimum divided by the maximum minus the minimum. It's very normalization. And I'll tell you a second what's the effect of this normalization. And we keep on going until some sort of threshold optimization has been reached using purely stochastic gradient descent that's the another probably another thing so I should say in the original paper they're using grade descent I'm using stochastic gradient descent but essentially the same thing so this is the attack function I didn't want to spoil you anything sure let me just for a second let me just clear the output here sorry I didn't want to so the gradient you compute for the attack you do on the cost function that and then all the loss that you define yes very good point yes not at all the model is actually a black box that's a very good point yes let me just rephrase this this is a very good point so we are creating our own criterion which is the classification loss but the model is being used just as a black box this loss which is this one here is between the target class and the output generated and so we're trying to identify which is which is what is that the model is actually producing and if we run the attack on this very simple model that's because I haven't run everything sorry about that let me just redo it. This is the pre-trained model, functions, and let me find it. So we start with an empty thing, of course, and this is what we end up with. We're just after one iteration. The reason for this is because the model is overfitted, of course. And in fact, there's an extension paper. This is in the slides, to be honest, the reference to that paper. There's a follow-up paper after this saying that there's a direct connection with model inversion attack and overfitting. This particular model is very overfitted and so you just need one iteration to steal this kind of information. So just without any knowledge of the model we've been able to reconstruct some information that has been used to train. And so this is still an example but you have to imagine that things might get really complicated and even worse if you're using sensitive data. I hope this clarifies a little bit the importance of doing this sort of things is there any other question so I can keep going and by the way another thing I should mention here is that the quality of the reconstructed thing is is not very high just because I didn't do anything I just apply normalization that process function is just to do some sort of normalization to the data you can in the paper they're using a technique which is called is a normalization of images I can't remember off the top my head and moan sorry but they they cleaning so to speak they're normalizing the results so that they can look more clear and more evident what's going on there very similar to what I showed you in the slide earlier and I tell you even another thing if you're now intrigued by this and this is just starting let me just rephrase a little bit this we're just taking the first 10 class out of 40 and what we're passing to the model is the pre-model the the pre-trained model and the class we should aim for so I want class number zero this is what we're saying okay and and as you can see is always like we have no clue whatsoever what's going on but all the time is a different face so we we don't know exactly if that phase will correspond to the actual class, but nonetheless we're stealing sensitive information within the model. This is called model memorizers and stuff like that. So if you're intrigued by this, you're very welcome to try the second bit of the exercise if you want. I can give you 5-10 minutes if you like. And run this with the pre-trained multilayer perceptron. That will take a little bit more time on your laptop presumably it did on mine and the code is very similar it's actually the very same thing but you will see that the the quality of the reconstructed images is even better with exactly the same code and this is because the model is more complex and so more parameters and so more space for memorization that's the takeaway message i'd say do you do you want to to do this exercise and try it out it's really copy and paste from above i can do it um with you live if you like um so mlp we instantiate mlp we get the weights i from this link and I should probably have it already let me try the luck here and all I have to do yes please thank you all I have to do is to get the code here There you go. We load the weights. This time is called MLP model inversion attack. This should be MLP, and then we print MLP. That's the model. Brilliant. Now for I or CL in range 10, and my face MLP CL what yes sorry Piper yeah good it's taking a little bit more time but as you can see something different and sometimes it requires more iteration so it keeps on going this is taken 152 iterations. That makes sense actually because the model is more complex so the optimization has more parameters to optimize. But they're not the same so in some senses it's actually even better quality and there's plenty we can do but nonetheless we are leaking information the model has memorized. Okay. Was this convincing enough? And again, this is not a generative model. Good. So let's see what we can do. Good. So maybe in the interest of time, since we are talking already about model inversion attack, let me just use a different order of topics I had in mind. Let's just keep now to the part number three. So all of this, I don't have to repeat things because you already have it very fresh. And let's move to the differential privacy bit. And let me just introduce you to Opacus. First things first, if you're looking for a very nice reference, I found this book online, which is online and totally open source in terms also, not just the source of the code attached with the book, but also the source of the book, is really wonderful. It's called Programming Differential Privacy, Differential Privacy for Programmers. So it's exactly what I was looking for. And it is a very, very interesting book. From the book introduction, it says, the book assumes a working knowledge of Python, as well as basic knowledge of the Pandas and NumPy libraries. You will also benefit from some background discrete mathematics and probability this is all about basic and rather cause in this topic should be more than sufficient and these are the references you'll find in the notebook is very very highly recommended and and and the striking things for me was also that the logo of the book was programmed in Python which was brilliant and this is the logo of the book and if you're up to a shameless plug already had a small section of the first chapters from this in another tutorial I gave, which is called Privacy Perceiving Data Science, and in particular in the section of Programming Privacy, you'll have notebooks explaining you the difference between moving from key anonymity to differential privacy. You'll find it there. And then, in this section, we're finally introducing APACUS. APACUS is the PyTorch framework for differential privacy. And if you're wondering what APACUS is, Opakos is a new high-speed library for training PyTorch models with differential privacy. That's more scalable than existing state-of-the-art methods, and in fact it is. Differential privacy is a mathematically rigorous framework for quantifying the anonymization of the sensitive data, and it's often used in analytics with growing interest in the machine learning community. And let's have a look at how Opakos effectively works. And to do that, I'm opening the link pointed here introducing Opacus, which is pointing to an article on Facebook. This is from a group from Meta, so from the Facebook research. And the reason why I'm here is because there's this wonderful animation, which is really nice and gives you the idea of how Opacus works and where differential privacy is used. So, essentially, this goes like, okay, I have a normal training loop, and you start from forward pass, and so you get parameters on the layer one, layer two, you calculate the loss, and then you back propagate. And normally, the way you do it, you go layer by layer in the backward direction, and then you update the parameters on the optimizer. the way your packers does is that whenever you have to do this optimization that the value in the back propagation are clipped then aggregated some noise is being added and then those parameters are then used in the optimization so to make it very simple the idea is that you can add noise during the optimization to the gradient so that the the model won't retain any information because it's going to be and the way of course the noise will be headed it's according to the differential privacy strategy okay so long story short the model will be trained on perfectly cleverly perturb data with noise so that the model won't memorize presumably and no privacy information will be leaked and if you're not believing me, which is fair enough, you might want to see it in action. So let's now train the very same model, softmax regression, using Opacus. And I think that the most remarkable takeaway message from this notebook is how easy it is to do it and how much, how less is impacting the existing code. In fact, I can import the same things and this hasn't changed it. This is actually copy and paste from the previous notebook. It's just creating the data set. I'm not doing anything fancy here. Same data set again. Blah, blah, blah. Again, same faces because the seed is fixed. Good. Now, setting up some parameters for the differential privacy. So the maximum with the gradient norm. The epsilon makes sense for the differential privacy. It's the epsilon of differential privacy. And the number of epochs we want to train the model. Rule of thumb with differential privacy it's recommended to increase a little bit the number of epochs you want to train your model because otherwise you're not presumably getting the same performance. By the way something I forgot to mention is that yes we're going to train the model on this noise mechanism in order to not leak any information but of course we don't want the model to worsen the performance too much. So the deal is, we want to change the model to save privacy, but at the same time we want the model that works, otherwise it doesn't make any sense. Okay, and so rule of thumb, just increase a little bit in the number of epochs. So we, importing the same model, from the same module, creating softmax regression. Opacus comes with this validator which says there's no errors in this model if you look at the tutorials this is showcasing you on ResNet and essentially this is just to tell you that sometimes some models using for example backpropagation sorry no backpropagation using batch normalization apologies for that it contains contains layers that are not very compliant with differential privacies, so you cannot apply differential privacies on batch normalization. So it gives you warning saying you should change the model actually. Also in addition to give you warning, it also comes with utility method that fix your model for you automatically. That's very brilliant. So again, same learning rate, same model. And now it comes to Opaco's bit. So from Opaco's we import privacy engine. And the privacy engine is this object which we instantiate here and we create this privacy engine by passing in the model we want to optimize the optimizer sorry the model we want to train the optimizer the data loader and the number of epochs and all the parameters needed for the differential privacy we run this which of course is not working because I forgot to run something else and I I apologize for that. Let me just do it again. Okay. Now, I have everything on my hand. So I have this Softmax regression, which is a new model with differential privacy. I have the Softmax SGD, which is a new optimizer with differential privacy. I have the new loader with differential privacy. The loader is also passed in because there's a new feature they added in to add some different sampling, which is also another thing We might want to talk about extensively from previously preserving perspective, but just take it for granted for now it's not really the point and Now I am importing the very same function and I promise you it's the very same function from the same model module The same Python module so the same train function I defined before and I didn't show you this because it's really boring It's just like the normal training loop in which you have iterate over the different epochs and then if you you step the model and the step if it's training you stop the gradient or not blah blah blah it's the same thing it's nothing nothing particularly interesting here to see so I'm importing the very same train function but this time I am passing to this function the previously preserving model the previously preserving of optimizer and the new loaders with differential previously and I run this and it works I hadn't have to change any bit in my code so if you remember this this is very highly overfitted so I I won't expect 90 98 and 94 whatever that was accuracy on this model but let's see where we go it's taking a little bit more because there's more epochs and there's much going on the back end of course there's much more computation going on so of course this is affecting a little bit the competition time as well time is it good you have a good half an hour to do the rest of it fair enough why this is going till the end do If you have any questions, curiosity, any feedback, please go ahead. I'm kind of surprised if you compare this just to stochastic gradient descent where the individual batch components are selected stochastically. So just the fact that now you're adding noise at a different stage, right, not in the selection training batches but rather in the vector of the gradient itself right yes so I'm surprised that this is going to work at all yeah okay I mean good so first of all why doesn't stochastic gradient descent itself introduce some kind of privacy or maybe it maybe I should ask does it introduce privacy and does the optimizer that you use change the privacy of the model that you learn Essentially, the algorithm which is in the back end is called differential privacy stochastic neural descent. That's a very good point. And this is the algorithm they designed to create Opacus. And the strategy of adding noise to the parameters during the optimization is what differential privacy stochastic neural descent is doing. I just showcased you with the animation to give you the idea of what's going on. The documentation is very thorough so you can actually find more details there long story short the the sampling and the methods of noise is being added your optimization is not just random noise it's just very selected noise using different mechanisms the most popular one is Laplace this is in differential privacy but there's extensive literature in that actually in the meantime thank you for the question just a second I come back to you so we have a 71 percent of accuracy so of course they related I don't know if we can keep on going if we have any better accuracy but fair enough at the moment this is what we have doesn't the noise being added during the back propagation become some kind of a private key then if you have the noise edit you can yes that's a very good point and that's the reason also why a packers is is applying specific sampling on the data loader as well because the data selection during the training given limited amount of things you can do especially with shallow models like this might be might be leaking some particular information yes that's very true so I'm showing you this as a black box and I promise you there's lots going on back end and documentation is fairly acceptable in my opinion but of course it's open source so you can contribute and I'm personally willing to be to be honest. So now the thing is to close this block now that we have the model let me trust it's not not trusting me now Now that we have the model trained with Opacus, we can see if this has had any effect on the model inversion attack. So calculating, instantiating the Opacus pre-trained model, re-doing the same thing, MI faces, running running the attack here and this is what we get absolutely nothing at all so the same attack is not being able to reconstruct any sensitive information here and you can keep on going you always have the same thing okay does Does that convince you a little bit? Yes, please. Brilliant question. If you look at this, what we're doing is essentially a form of regularization. So indeed, that's, for example, why batch normalization is not required. That's why you typically don't do any extra regularization on the model. Differential privacy is already a regularizer to the model indeed. Yes, very good point. Thank you. I do mean. I can't remember off the top of my head what's the role of dropout is. That's a very good point. I can't tell you in an hour, I'll have to think about it. But that's certainly something I should remember for the next time. Thank you for the questions, they're brilliant. Any other feedback question on this part? Just out of interest, has this worked with me showing you the attack and then the solution? Okay, I should probably change the structure later. Brilliant, okay, so now we have sort of 20 minutes, and let me just now show you something we can do with federated learning, which is the last bit I wanted to present to you. So let's introduce federated learning very quickly. Federated learning is, to be fair, one of the most intuitive things in this scenario, And despite the fact that, again, you have multiple clients contributing to one big cloud of server, which is aggregating the things and then distributing the updated information. So I didn't explain clearly how federated learning works. And the very idea is that the standard representation of federated learning is every single client has its own data partition. and their own model. So every single client has capacity to train a model locally on their own data. And then every single client communicates on a secure channel with this cloud, which is gathering all the different gradients, if you're using gradient-based methods, is gathering the multiple gradients from all the clients, averages it, which means that it's taking into account all the different clients, and then distributes the updated gradients back to the clients. So that in the end, every single client will have the updated model which has considered implicitly the data from the other clients through the aggregation. Does that make sense? This is a standard representation of further learning. To be honest, there are two things that nobody tells you initially, and I think they're actually paramount important. First is that to make all this to work, to have everything to work here and to make any sense, the features you're having in this scenario should be the same for all the clients. So, and technically speaking, this is called horizontal for data learning. The idea of horizontal and vertical, of course, because if you have horizontal, there's also vertical for data learning, is that imagine the classic representation of data in machine learning. You have samples and features. So you have the matrix of samples on the rows and features on the columns. If you have horizontal for the learning, you're essentially taking this matrix and slicing horizontally, which means that the features are the same, so the columns in this data set are the same, just the rows will be split through the clients. okay this is horizontal for the learning vertical is the opposite so you have the same set of samples from multiple clients but each client has different set of features and if you're wondering whether any of this scenario is real you might think of so the case of mobile phones is the case of horizontal phillips learning because it's the case of we're having the same features that different samples in they said okay the case of vertical in all the other hand is as well realistic imagine that you're having some fancy health data set you're allowed to use from one hospital but another hospital is holding it labels that's the typical example of vertical for learning so you have one client one node in this network holding the piece of information relating to the labels and another client holding the piece of information about the features and you can keep on going actually you can you can have one data sets containing some set of features and other data sets containing another set of features but still the same subjects does that make sense this is vertical further learning we're gonna see example of this awesome please do so much right go ahead okay Thank you. Are there any metrics to measure privacy or how can one measure privacy achieved? Could you repeat the question, please? What was the first? Are there any metrics to measure privacy, that's the first question. Any metrics? Yeah. To measure privacy. I don't understand what you mean by measure privacy. Yeah. Or how can one measure... Measure? Yeah, measure. Okay, sorry. How can one measure privacy achieved? I'm not entirely sure I got what you mean. Sorry about that. I'm puzzled by the use of the metrics. I think, I think the question is, um, is there any way for you to be able to measure on the, maybe the level of success or failure achieved in the privacy you're looking at? Oh, yes. Certainly, yes. There are tests you can run on your data to see how much information you're leaking or whether, yeah, I skipped this part, and I appreciate the question. It makes sense, actually. And if I understood correctly, this goes towards the data anonymity thing, as in how much information we're releasing. And this is the very idea of k-anonymity, for example. So k-anonymity is built on the idea of how unique is a record within a data set. So how easy is to spot one, and it's not just about one single column of value, it's also the aggregated columns. So, for example, if I have in one data set the only one sample which is the only representative a representative for one specific set of combination of values for three or four columns, that is very identifiable, and this is what k-anonymity tries to solve, I think. In case there's the section about privacy, data science, I mentioned before as a reference. There's a question here. Could you compare differential privacy with federated learning at the end again, and then they follow up, when should one be used against another? Absolutely. Or which one would you prefer? Yes. So this is a very nice question. And to be honest, I haven't provided an answer to that yet because I haven't specified what further learning can do. I just showed differential privacy, to be fair. Although, the two things can be combined. And further learning is just a framework that you can use in combination with other things. This is differential privacy. Further learning is another framework in which you're essentially having multiple actors in this communication and involved into the computation. If you're having data sets with with one single silos or database, you don't need further learning. Essentially, further learning is coming into place when you have multiple actors, so multiple data owners or multiple silos of data you want to access. And of course, you can combine further learning with differential privacy. The two things don't go separated. But further learning has also some sort of cons. So first things first, is further learning similar to distributed computation? Too long to read? No. Client devices have limited network bandwidth, in case of the phone examples. And there's also another deal in further learning. The different nodes holding the data, since they're owner of the data, they also should be allowed to do computation on this data. This is a major, huge requirement for the learning is imposing, which sometimes may not be attended. Something like, you're the hospital holding the data. I won't never access those data, because I'm not allowed to. I just want to set up a framework to do computation on this data. So I just want you to send me the result. I'm asking you to calculate. But at the same time, and that's the deal, don't want you to steal my calculations because I'm I'm it might be my ownership IP yes thank you that was looking for that that's my idea so I have I have my solution I don't want to leak my solution to you because I might be giving away information private information as well not data information but private information and so this is the deal with further learning with encryption the encryption and the privacy preserving should be guaranteed on both ends this is very roughly and so this is where we're going yes is 10 minute oh you're right thank you fabulous so I think I already answered this question so on morphic encryption I told you already what's the deal with it and and and I probably also give you why you want to do that so a morphing eruption again is enables computation on encrypted data without decrypting the actual data and moreover any operation made on ciphertext generates results with then that's when decrypted correspond to result in the plaintext which is amazing we're going to see an example of that in a second and as you You may imagine this is going towards the cryptography field. I'm not touching it, not even close, but there is a huge literature about it. And I will give you in a second a very nice reference to where you can go if you're interested in exploring more about this, everything I've told you today, before closing. So what we're going to do here, we're using a package in Python which is called PHE, standing for Partially Omnimorphic Encryption. And this example has been actually adapted from what has been found in the repo. We're doing logistic regression using omnimorphic encryption. Nothing particularly fancy here. we're using Wisconsin breast cancer data set we're also taking a context timer to see how long it takes to do computation we have its security bit so again Alice and Bob the lack of fantasy here is overwhelming we have Alice which has some model which is the logistic regression model and then Alice is the one holding the public and the private key so Alice is the one who is in charge of the crypting things and Bob will be the one having to get the public key from Alice and using this public key to encrypt whatever Bob wants to encrypt and send it back to Alice so in fact we do get the weights and the intercept is just like the parameters of the logistic expression and we use the public key to encrypt the parameters of the model. What I was looking to is this. So we We generate the pairs of keys here. And then, let me just run this so it makes more sense. Very quickly. So this is the data set. This is Alice, and so Alice has the model which fits the model on some data, makes prediction, and encrypt the weights. Yes, that's the bit. There's Bob. So we have data set with some robust scaling normalization. We initialize Alice. And so Alice generates the key for Bob to receive. And then we have Alice fitting the model, which takes some time, no more than two seconds, very quickly. And so this is just to test the model performance on the test data to see how it works with unencrypted data. And this is it. And then now Alice encrypts her trained model parameters. So we have the weights and the intercept. So we're sharing now with Bob the information about the model. So Bob takes the public key, halice has, halice holds, which is the one Bob needs to use to encrypt the thing Bob is calculating. And so essentially, Bob, what he's doing is taking the parameters from Alice, updating according to his own calculation, and giving it back to Alice. This is the deal. So set the weights with the one encrypted sent by Alice. And then we calculate the encrypted value. We take some times on the test data, by the way. So he's running on the test with encrypted value. So I'm using the encrypted model from Alice. I'm not using any model on my own. And then Alice is able to get the scores from Bob and decrypt it, leading to this error in the prediction. So this is just a very simple example showcase how a morphic encryption actually works on making basic computation like standard machine learning computation and the completion we're doing here logistic regression so very basic if you wanted there's a more extensive example in the end showing you how to implement for the learning and morphic encryption in this or the notebook before wrapping up I want to show you two things very quickly one is the plug about resources I promised you and this also connects to open mind which is this community I reach out to last year year and a half which is very active in the context of previously was only machine learning and defining many tools in this context the most popular one is sift or PI sift which is now at the version 0.6 and open mind is offering the private AI series courses which is totally free online these are the courses which have been produced so far and foundations of private computation is where I adapted part of the materials in this section which unfortunately won't have the time to go much into details I should better calibrate for the next time apologies for that but I'm going to showcase very quickly before the end and also introducing introduction sorry to remote data science is the is the latest course released in the last year in which there is the new version of PySift 0.6, which is slightly better from the version you will be seeing in this repository, which is using a completely redesigned mechanism for privacy using what is called AutoDP, Auto Differential Privacy. So it's very focused on differential privacy and allowing to do computation using the same framework I'm going to show you in a second, which is based on furthered learning and encrypted communication. The example reported here is called Duet. And Duet is the first prototype they developed to showcase how you can do data science on data you're not allowed to see. To do that, you have to have two notebooks. We'll be pretending we're being on two different platforms. You can actually be on two separate machines. but I'm showcasing you here on my own laptop, so I'm doing the client and the service entry, so at the same time, the data scientist and data owner. But, let me just close some things, otherwise it's too much. I just want to keep the reference open for later, to open mind. So, pretending I am a data owner, I have SIFT and I'm launching duet I'm launching duet so now there's some server running on the open green network which is another package now integrated in SIFT and this is waiting for someone to join in I now move to the data scientist notebook so the data scientist is the one who wants to do data processing on data they don't have so I'm open sift and oh sorry this is data owner I messed up so this is data owner which is the one that should run first so I am the data owner waiting for connection okay and this should be already connected because data scientist now is connected brilliant so now data scientists and data owner are connected the data owner should make some data available so in this case we're having the iris data set the the fanciest example I could think of we are having data split nothing interesting here normally data can be any format if their torch tensor is way better so they're ready in the form they want it to be so we converted it in tensor and then we put some description thanks to PySift which is extending torch a tensor adding tags and description and then we can put the data to the data store and this is data store we have a data set of data and target and to make things very easy for competition we are just setting the to accept from the data owner every possible request uh in the in the very top of the notebook there is reference to the original notebooks where these things have been adapted from and you will find extensive examples about this but again this is part of the the private AI series course in particular foundation of private computation. To finalize, now I am the data scientist, the data owner has made some data available for me and is accepting any request I'm making. So it's trusting me totally and completely. So first thing is let's see if the data set is available. I can read the data. I can access this data set from the store. An important bit to mentioned here is that we have pointers we are not having data we just have pointers to some data we don't know anything about we just know the tags we know the description and we know the type nothing else um i'm running out of time so i make it very short essentially what we're doing is to define a torch model here which is slightly different from the standard torch model because It's a SIFT Torch model, which essentially needs to have reference to the remote Torch reference in the data owner bit, because this model is being created from the data scientist and then sent to the data owner to do the computation. This is the deal. In fact, we do create the model here. It's a local model. Sorry, I forgot to import Torch. but then we send it to Duet. So now the model becomes remote. And so if you keep on going, I don't want to steal any more of your time and to your lunch, you can do all the computation considering this pointer to the remote model. And you can actually run it. And this computation, which is now running, is actually running on the data owner end So while this is completing, essentially what I'm doing is getting some data. So I'm passing in the remote model, the remote torch, the optimizer, which is local, and the data pointer and the target pointer. So all these pointers are actually living in the data owner end, and the data scientist is using it to do some computation, which is going somewhere, actually. is performing some regression. And then there's notebook goes like you can download the model and do the test on your local data. So this is one example. The other one, not having the time to talk about, unfortunately, is another case of, and I'm wrapping up with that, is SplitNN. SplitNN is the next steps in further learning, which is about I'm not just splitting the data I'm also splitting the computation of the models so I may have some part of the network leaving in one client some other part of the network leaving in on the client and we're sharing the gradients and finally aggregating all together and and duet allows you to do that please feel free to complete there's not quite easy to do to go on your own i think for any kind of issue problem feedback suggestion feel free to reach out i would be really appreciative uh and thank you very much for for the extra time and for being here.