Privacy-preserving Machine Learning for text processing

Speaker 1 [00:03]

So hi everyone, thank you for coming. Sorry I've got a bit of a cold so I hope you can still understand me well and I won't cough too much. So yeah we're here to talk about privacy preservation techniques and with a focus on text data. So I'm a data scientist, I work at the company called PeopleDoc and we are dealing with human resources documents and so as you can guess there are some sensitive information in that kind of documents, in those kind of documents. And so we've got this very strong privacy by default and privacy by design concept, which are very important to us at PeopleDoc. And that's something that's already interesting me before, but that's kind of the focus of my work. But we're not going to talk about human resources document today because that might not be, might seem a bit boring, so we can do something more general. So let's say you had a master's board in school, so your students, they receive a lot of messages from friends and relatives, and there has been kind of a problem recently. There has been a high increase in advertisement and unsolicited messages, messages, things that the student doesn't want to have, don't want to have, and it's getting really big of problem because at breakfast time there are so many increase of income of messages by whole, yeah, so it's a wizarding boarding school. So the thing is that you want to be able to distinguish between legitimate holes and so called spam and you want to do that but of course you want to preserve the privacy of the old communication because you don't want to spy on your student and you definitely don't want your student to think that you are spying on them so you try to ignore the problem at first because you're a headmaster so you're like really busy you don't want to take care of that but But it's really getting out of hand, there's so many holes, it's a big problem. So at first you try some heuristics, something which is based on rules, it's not really working that well, it's not accurate enough, and also it's not flexible enough, so it's very easy for attackers to go around your rules and still deliver their unwanted messages. So you don't have a choice, you're going to have to collect some information, and since It's messages, text messages. You know that they will include some private information and you don't have any other choice than doing that. So that's the first rule of doing privacy-preserving machine learning. It's just like, don't collect private information. If you don't have private data, it's privacy-preserving. Thank you for our attention. If you don't have any choice than, you know, collecting some private information, then you are becoming responsible of the safekeeping of this data that means that the bare minimum is to use encryption at rest that is any hard drive any drive containing this data should be encrypted when it's not used that's like low level and of course you should enforce best security practices so go talk to the security folks in your company if you're in a company They're here to help you. They're not just here to give you annoying advice like change your password and use strong password and stuff like that. They're actually here to help you keep your data safe. And even more, the data which is not yours, your users' data is safe. That's good at rest. So when you are not using the data and it's stored, but when you want to do actually like machine learning, you're going to have to train models. So you have to do something more. So trying to get back to the first rule, which is don't use private data. You can try to extract the private information from your data set, and then you're back to square one, and you can do machine learning as always, and that's easy. I'm going to very quickly introduce two main concepts, which is the one of personally undertaking information, So, which are things like first and last name, usernames, email addresses, but also like IP addresses, MAC addresses, session IDs, all that kind of things that link uniquely to one individual, one-to-one. And the other thing is the sensitive attributes, which are things which are not sensitive in themselves, but they become sensitive when we are linked to one individual. It could be things like medical information, financial information, knowing that there are some people who are wealthy, some people who are not so wealthy, that's not like a privacy leak. Same knowing that some people have some illness, well, it's sad, but everybody knows that. But knowing that your neighbor is earning that amount of money or this person is suffering from this specific illness, that's a major privacy leak. And these things make more sense in structured data, so tabular data, and we can make a quick detour by this structured data idea to really grasp the concept and see how we can apply this concept back to actual text, so unstructured text data. There are several ways to remove this personal information in structured data. The easiest way is just masking, so that's like removing the columns containing the information. So that's easy because you know that the email address column contains email addresses, so you remove the columns, no problem. It's a great way if you know that each line in your tables corresponds to one user, one individual, but there are some cases where, like in your table, each line is a different event. And in that way, you might still want to know that some events are linked to the same individual, even if you don't know anything about this person. So in that way, you can use things like hashing. That's a destructive way of mapping some personal information to an ID. And what I mean by destructive is that you cannot go back. From the hashed ID, you cannot get back the initial value. and that's done on purpose because if you cannot access back the initial username or email addresses, that means that if your data is ever leaked the attack account cannot go back either so that's what you want and well, basically that way you still know that lines 2 4 and 18 are linked to the same individual but you don't know which individual it is In some cases, in some applications, in some data pipelines, you might need to have something which is more plausible. If you just have random hashes, it won't work. For testing purposes, you won't be able to test your preprocessing pipeline, so you can use fake data. There are some libraries, like for example in Python, there is a library called Faker. They generate plausible data, so you can tell them, generates me a possible first and last name, and you can give them a locale so that this thing matches the localization you are using. If you're Germany, you'll have German-sounding first and last name, and so on. And also, it's really important for street addresses, which are very country-dependent, so you can get something which is plausible. and the last thing is maybe that in this big databases you as a data scientist you don't need this information but this database is shared with other people like for example the finance department and the finance department they also use this data for billing purposes and so they need to know to whom they should send the invoice so if that's the case you should split your data between a non-private table which contains everything with an ID and then you have this private information table which links the ID to all the personal information and this private information table is really accessed on a need-to-be basis and has a special security measure set up and that's like the bare minimum which is recommended to be GDPR compliant for example so that was for tables data. So as I said, this was easy because you know that first name are in the first name columns and last name are in the last name columns. But when you've got like big chunk of text, the first thing is which of these tokens you got, which were the sensitive ones, which are containing any private information. So this is still like a very open question and research question, but there are some solutions you can try and they are all very very data set dependent and also application dependent but here are some things you can try one idea is using a dictionary based removal so if you're working for school if you're working for a company you might have like the names of all a database with the name of all the company employees and you at least want to remove all of that from the text data so that you remove some sensitive information. It's not perfect, so for example in the spam use case messages, if you're removing just the student's name, you're not removing the relative names or the friend's name, so you still have some personal information. But it's kind of easy to implement, and it could be a first step. There is another approach, which is using an untied dictionary. So an untied dictionary would be words you know which are not containing any sensitive information and you only keep those words in your text. The problem is that means you get like all common words in a language and also all declined versions of these nouns and all conjugated versions of the verbs so that gets quite huge and you only keep things which are in this huge dictionary, so that's not really scalable, and also you're losing a lot of information, how do you handle typos, how do you handle things which are both common names and first names, for example. But depending on the sensitivity of the data you're manipulating, you might want to consider something like that. The other solution is if you are handling text data, but it's not really unstructured, it's what I would call semi-structured. I'm thinking about anything like forms, curriculum video resume, anything like reports, instant messaging logs, or any kind of logs. You kind of get some sort of structures. And so anything which is following an email column and then you got a token, it's probably an email address. I mean, in the form, you probably fill your email address. So you can use this underlying structure, try to extract it with regular expression, and then remove the associated personal information. regular expressions also are great for things like emails and social security numbers any kind of ideas, company ideas so before you tell me that's not a totally correct regular expression for matching emails there exists like one canonical form, it's like 25 lines long you can find it on Google if you need it but you get the ID. If you know that you're working for a company and the employee ID in this company is like 11 numbers long, you might want to match every string of 11 numbers and remove that because it's probably an employee ID. The other solution is, of course, to use things like named entity recognition, which are available in the shelf library like spaCy, for example. They can catch a lot of different things persons, location, organization and you may want to replace them with a given token or and if you got any possibility to like tune this name not a zero connection you definitely want to have a higher recall rather than higher precision that means you rather have some false positive and removing some things which are which could which are not private and to let some private information. And last thing, when everything else has failed, you can try to have your own pseudonymization algorithm. Things like name entity recognition that we train on regular text purposes. And so if you're handling anything a bit more exotic, so insurance claim, human resources documents, medical report, you probably want to have something which is more fine-tuned to your needs. Also, there are some tools which are domain-specific, in particular for medical reports. There are some existing libraries which are more suited to this kind of thing. The advantages of using such an algorithm is that, of course, you hopefully get more accurate on your use case, on your domain-specific data. The cons is that it requires annotation. And that is, it's time-consuming, and also, that means that someone has to look, to go through every document, and then annotate every private information. So, since we are trying to enhance privacy at the first time, it's kind of a trade-off that, at first, someone has to look through everything. So, we've removed everything. So, here again, we go back to the structured data example. We have removed all personal information. That's great. Are we good to go? So that's an extract from the school nursery. In your school, you know that Harry has just come out of the nursery, school nursery, and you want to know, hey, you're a bit curious, what kind of medical condition did he have? And you know that he's 15 years old. And you know that he's in House Gryffindor. Okay. So we've got only two 15-year-old boys, only one in House Gryffindor. So just by knowing public information about Harry, you're able to leak this medical, his medical condition. So that's the sensitive attributes here. So pseudonymization is like the first step to avoid, to leak directly some private information. but it's not enough because some what you call a pseudo-identifier attributes, some combination of these attributes may be unique. What are pseudo-identifier? For example, their age, their gender, zip code, affiliation, that kind of things. Even in a very large data set, some of this combination end up being unique. And the thing is most of the time you don't even really need these exact values like you don't care that someone is 15 or 14 you just want to know that they are under 25 so we have this possibility to like aggregate these attributes so that for any combination you can guarantee that you have at least K rows so K different individuals matching this combination. So, as I said, you use ranges of aggregates, so we go back to the same tables, but now we've aggregated ages, so we've got two ranges, below 14, above 15, and we've also aggregated houses, so that we get this true anonymity, which is, of course, true is a very small value, but this is a very small dataset. So now we are good, we say, okay, we've lost a bit of precision, not too much, but now we are not going to leak any sensitive attributes, right? Well, why can't you tell me about Luna? You know she's 13 years old and she's in Ravenclaw? So let's have a look. There are only two lines, there are two lines, because it's too anonymous. But they both have the same sensitive attributes. So actually, even if everything is grouped, you can still gain some information. That's why we need to introduce a further concept, which is L-diversity. So that in each of these care anonymous groups, we've got at least L different values. Okay. Keep going with the same table. A bit different because I've only kept the people who are under 14. So a bit more choice. and I removed entirely the mention of a house because that was L diverse two diverse L's and can you still tell me something about Luna maybe so I'm just going to highlight there are five different lines which could match Luna and out of these five lines, four are the same disease so you know with high probability that Luna is suffering from levitation sickness so L-diversity is not protecting you for statistical attack there are always kind of other kinds of attacks which is like if you are dealing with not categorical but numerical value you might have different numerical value but they are all in the same range so you still know that someone has for example a low income even if you are not sure what exactly their outcome is, income is but still get an idea So there is this notion of t-closeness, that the distribution of each sensitive attribute is roughly the same in the group and in the overall data set. But here you see that typically why we need machine learning is to discriminate between distributions. So if you enforce t-closeness, you kind of lose a lot of utility in your data set. So there is a risk trade-off between what value you as a data scientist who does not want to break privacy, you get out of the data set, and how you can ensure that malicious users won't get too much information. So that was structured data. Now we are back in text. And as you can see, there are some ways to remove things like Minerva McGonagall. so first plus last name if you've got a good name entity recognition it will probably match but if you've got a phrase like head of a Gryffindor house it's much harder to anonymize that, to remove that information it should probably remain even after pseudonymizing your dataset and if you think about it would you anonymize as a human would you anonymize this phrase Would you remove everything and just say it's a person? Would you say it's someone from the Hogwarts staff? So you keep the information that it's a teacher. Would you say it's the head of some house, which is not just any teacher, someone who has some responsibilities. But then there's only four houses, so there's a very high risk of being able to untie and dotify the person with other information in the text. Once again, it's a really hard question. We don't have a complete solution yet. The one idea might be to extract every pseudo-identifier and treat them as a personal identifier, the same. Or at least aggregate them. For example, if you've got a number, you think it's an age, you remove it and you add an age bucket instead. If you are dealing with a resume and someone is coming from Harvard, you remove Harvard and say they're coming from an Ivy League university, and that's probably going to be the same in your general process. Once again, it's really application-dependent, but there are a lot of information you can remove or aggregate without using utility for you. Yeah, and then you must aggregate them. well is it enough there is this application in natural language processing which is called authorship attribution algorithm which aims at discovering who wrote a certain piece of text so as you see it really depends on your application but in some case you might not be able to really remove all private information from your dataset without destroying the data entirely, so something to think about. You've tried to do something on your dataset, but you are not going to expose your dataset to your end user anyway. But you might expose your model, you might send it to a browser or to a device, or at the very least you're going to have an API which which users can be able to interrogate your APIs, and maybe attackers may be able to have access to this API too. So you want to have models, and you want them not to leak any information, even if you have some private information in your data. There's this concept which is called differential privacy. and it's a mathematical framework to measure how much privacy information you are leaking to the outside world from a process. So process in general, it could be a machine learning process, it could be a query mechanism. We have a very nice mathematical formula. Basically, the idea is that you've got two data sets, they are the same, except there is one more person in the second data set than in the first one. And then you've got a query mechanism or another model, not model, algorithm, which is K on this data set. And they got a probability distribution which is quite close from whereas it has been trained only on D1 or it contains the additional individual in D2. so without any math, the idea is that if you train your model on one dataset if you add one person and you train your model again, you should have roughly the same model that way you have no way to know for sure that the individual you added is in the dataset or not that's a way of preserving privacy let's take an example which is not differential private That's a very naive thing, maximum a posteriori. So as long as you're seeing only spam messages, you classify everything like spam. So that's a really stupid algorithm. But if you start seeing one sample which is not spam, then your algorithm, which will output something in probability, so you've got some randomness in the output, sometimes you'll start to output not spam that means that if you have an attacker who can query your API can query this algorithm and they know which was the last sample which was added which is also something which is possible then they know as soon as you're starting to have not spam responses that your last line added was not spam. And for spam or not spam, it's not something sensitive. Now, imagine that it's not about spam or not spam, but VIH status. You definitely want to protect the information that the last user, or even if it's the last couple of users, might have VIH. So, how can we do things better and be differential private? let's say you want to do a survey and if your students cheated at the test and you just put out the survey and if it's anonymous and everything you're probably not going to get very honest answers because you know nobody wants to say yes you cheated to the test if it's ever leaked you get into a lot of trouble so instead you ask your students to do that they take a dice they throw it if it's between 1 and 5 they answer the truth if it's 6 they throw the dice again and if it's odd they say yes I cheated if it's even they say no I didn't cheat that way if the survey results are leaked with identifiers and somehow you can get that to which student answered what they can say no I say I cheated on the survey but that's just because I was unlucky with the dice but I did not cheat so you can preserve plausible deniability for everyone and since you know the value of P and K and you only care about the aggregate value you can still recover the number of students who cheated with a limited loss of precision so that's like win-win that's when you are collecting but if you have already a dataset which is not differentially private of course you can make a copy and with the same kind of mechanism make the copy differentially private and then delete the first data set and then you're good to go differential privacy as i said is not an algorithm it's like this general mathematical framework so it also applies to when you are training the model and in particular if you're using stochastic gradient descent and a lot of things are using stochastic gradient descent to start just to start with deep learning and so it's always the same idea you want to never be able to tell for certain if a point was included in the data set or not so you're gonna first use gradient clipping when you are doing your updates so that each individual points as a limited influence on the global model, and then you're also going to add some random noise so that even if someone gets access to all the updates, they cannot be reverse engineered these updates to see if one point was included in the batch or not. If you don't want to re-implement that from scratch, there's a a library for tons of law, which is called Privacy, tons of privacy. And they implemented that. They also do all of the nasty accounting stuff to measure what is your privacy budget. So that is how much of privacy you lose by training your model. You always lose some bit because, as we said, well, we're trying to extract patterns, so you obviously lose some privacy. You've got three additional parameters to tune, but other than that, I mean, it's all implemented, so it's not that hard to set up in practice. If you're not using deep learning, or not using anything using stochastic gradient descent, there are some other solutions. One of them is called private aggregation of teacher ensemble. So the idea is you're going to partition your data and train several models, and there you've got two different possibilities. Either all of the teachers are agreeing to the outcome, and these models, these teacher models, they are trained on different data. So you can imagine that there's a very strong consensus that means you're not leaking any privacy information, because in the yellow and in the red, they are totally different individuals, and you still get the same conclusion so that's okay you're not leaking information the other case is well the teachers are disagreeing so probably somewhere the whereas you add one individual or not in the data set it changes the model entirely and here you have got a privacy leak so what do you do you you make them vote but you add some noise if there is a strong consensus the noise doesn't matter because, anyway, it's lower than the consensus. If there is, like, two and two, the noise ensures that you say sometimes yes, sometimes no, and you get something which is probably deferential private. That's good, but if you only expose the prediction, but if you want to deploy all of that, then you've got to deploy the individual teacher, and the teacher are not deferentially private, so they can leak some information. So what you can do is, instead of deploying the teachers, you use the teachers to label some public data. Why is it public? Because you get some data without the sensitive attributes. And the teachers are providing the sensitive attributes. And we've seen previously that with this noisy voting mechanism, what they are providing is differentially private. So basically what you're getting here is a privacy-preserving data set, a data set containing no private information, which goes back to the first rule. If you don't have any private information, it's very easy to do privacy-preserving machine learning, and then you can train the student model, and you can expose the student model to the world. It contains everything it needs to do predictions, and it cannot leak anything because it has never seen any private information. The last thing that you can encounter is that you want to train some machine learning models but you cannot do it in your own control environment. So we've seen previously how you can anonymize data, how you can ensure that your model is not leaking any data, But if you cannot train your model in a control environment, you have to ensure that the training process is not leaking any data. There are two main use cases. The first one is that you've got a lot of people having a bit of data, not enough to train a complete model, but they still want to collaborate to train a model. one solution is you got a central server which is updating a model starting and each client is sending small updates and the main server is averaging these updates and sending back a new model and then in different clients try improve on the data they have this model and they send that there are updates and so on and so forth. You can do it in two ways. One way is you want to send everything in clear because you think everybody is of good faith. The problem is that if your central client is compromised, then they can possibly reverse engineer your updates and gain some information using, for example, a generative adversarial network. So, okay, you say, okay, I don't want to send my updates in clear text, so I'm going to encrypt everything. And there are some solutions, and that way, since everything you are sending is encrypted and the update is done on encrypted data, the main server is not able to know what you're sending them. The problem is then that the main server is also not able to assert the good faith of the client, and then it makes the whole system more vulnerable to backdoor attacks, for example. So it's an interesting solution, but you still have some security issues and you have to be aware of what kind of environment you are working in. Another solution is called homomorphic encryption. So homomorphism, just like there are functions which are preserving operations. so typically addition and multiplication and the idea is really really neat because you just kind of encrypt your data and then you can send it to a third party provider and they train your model for you and then you get the encrypted model and you decrypt it and it's great because the third party providers in the cloud they never see your data and also you never see the algorithm they are using so if it's like a private algorithm and they don't want expose it, they can still train models on your data without seeing your data and without you knowing what they are using. So, in theory, it's just perfect. In practice, in order to have this very nice thing in order to have this very nice thing where every other data is really encrypted, you want, for example, something which is called semantic safety, that is to say that on the ciphertext you cannot get things like which are the most common bytes or that kind of thing so you're going to have to add some noise and so you've got a bit of noise on each operation and after too many operations the noise drawn out the signal so that means you need to reset the noise level there is a technique called bootstrapping which is like basically you encrypt with a new key and then you decrypt with a previous key and that way everything stay encrypted but there are not many algorithms which are fully I mean homomorphic algorithms which are also fully bootstrappable that's kind of holy grail also we're talking about addition and multiplication which are linear operations but for example in your neural network you've got activation functions and these are typically non-linear operations So that means you've got to send back the value, the encrypted value, to the client. The client has to decrypt it, perform the nonlinear operation, encrypt it again, send it back. So a lot of encryption and decryption and a lot of communication costs. And even without that, the homomorphic encryption algorithms, which are available right now, the operation on ciphertext is much, much slower, several order magnitude slower than on cleartext. so it's like a really neat idea but not right now it's not practical maybe in the future we'll get more research on that I hope we get something but yeah I wanted to talk about that so in a nutshell that's always this awkward part when you say things you already said again but maybe not clearly enough thing is if you want to anonymize your data it's hard and it's hard on On structured data, on tabular data, it's even harder. On text data, unstructured data, it's hard to do it perfectly. That does not mean that doing it imperfectly is a waste of time. I mean, anything that can, you know, you've got encryption, and then if you removed as much information as possible, and then you ensure that your model is not leaking too much information, that's better to have three layers of protection than only one, which, you know, might be failing. The other thing is all these edge cases. So whether you're using a dictionary-based approach or name-notity recognition, things like typical American names are probably going to get recognized pretty easily. But anything which is like non-American-sounding names, it might be much harder. And these people also might come from minorities where leaking their information might be more dramatic for them than for someone which is more privileged in the society. So you should really see how you're less privileged and rarer users are protected by your algorithm because you're also doing that for them. That's my thing about your data scientists. you should still do proper coding because even if, in theory, your algorithm is perfectly differential private and you've done all the math and you're like, regret, small privacy projects, perfect. And you've got a stupid bug that's going to rinse off your effort. So do testing and write readable code and do refactoring and, you know, just do good software engineering. And there's always this trade-off between, well, actually, I think, protecting privacy and kind of, well, getting things done fast and getting things done accurately. If you just start to worry about how you anonymize your data set, well, you lose a bit of time in your pipeline. So obviously, there's a trade-off. If you're dealing with medical data, you need to do that. If you're just dealing with the number of likes on Snapchat, maybe you shouldn't. I mean, find the right trader for your application. But yeah, there's no golden rules. Very quickly, some open questions. So the impact of the use case, we already talked about that. There's starting to be something like on medical reports. they estimate the loss of clinical concept to under 3%. So it's not entirely negligible. It's quite small. Once again, it depends on your application. There is this whole broad area of research of generating privacy-preserving synthetic data. But then what are the guarantees? If you guarantee that what you've seen in the original dataset is never in the synthetic data. It's not really privacy preserving because if you generate a lot of synthetic data, you see in the hole where the true data was. But then if you generate some data which is too similar to the true data, is it still privacy preserving? And there is always this question like you want to preserve privacy but that means if you want to have really a metric to do that you have to have some annotation at some time so you have to have at first a human looking for through private data to ensure that the future data you're going to get is going to be protected but there is this tension between that so to, we haven't talked a lot about the spam problems but maybe you want to use them, buy a very good price I don't know if you have time some questions, the slides are online and I've got a bit of bibliography if you want to know more about the subject. Thank you Thank you very much Sarah Great practical case especially for privacy which I think everyone should appreciate very much. So Q&A hi thank you for the presentation do you sample noise from any distribution or do you have any other methods for sampling the noise in your implementation the noise you mean for you show some methods these are for annotations right the methods which my question is how do you sample your noise how do you define noise in your system how do you get noise in like differential privacy stuff yeah differential privacy stuff yeah you usually generate noise using how do you generate your noise Using a pseudo-random generator, which is provided by NumPy. Is it possible to generate noise from any discrete or continuous distributions? Yes, but you choose if you want to. Typically, you use Gaussian noise for continuous distribution, and if you have a discrete distribution, Well, I guess you have to set some threshold or maybe do some permutation or that kind of thing. But if you're using stochastic gradient descent, you're working on continuous values anyway. Okay. So you can add some Gaussian noise. Okay, thank you. More questions? What? Ah! In methods like this, I'm always really curious when you have written a model or written an algorithm that does something that a human can do perfectly. Do you have a metric for written text of inter, how is it called? Inter-annotator agreements. Yes, exactly that. Just something rough? Well, I don't have any answer right now. I know that there are some metrics that exist. it's not something i i well i don't have any uh specific algorithm to recommend or sorry anyone else oh tough one can you catch um when dealing with um when dealing with um obscuring information and text how do you deal with longer core reference chains uh well um yeah that's that's a good question actually I didn't talk too much about that but actually yeah you can say when you're doing like name entity recognition just say this is a person and this is a person but probably you might need to know that this is person one and this is person two and this is person one again one solution is if the name are the same the tokens are really similar just to keep like a dictionary while you are doing this name entity recognition and then replace it if it's synonyms so like first Mr. name, surname and then you just have Mr. Surname or some other kind of synonyms like Mr. Director of Things it's harder you have to, one solution is to add some semantic information so you have a knowledge graph outside of it so that you know that the President of the United States is Donald Trump and then you can match that it's the same person. Yeah, that's a tough one. Okay, final question. Someone who's not sitting way over there. No one? Also good? Community space. What? Where? Ah! Come on. just a question so i mean as a data scientist we all know that we should use this like anonymization and like privacy preserving approaches in your experience like are the companies that do that or like is it something that's still like more an objective than actual i think it depends a lot of companies uh i know a lot of companies are like doing roughly pseudonymization and they think it's more than enough uh because like they're gdpr compliant so that's good and some other companies uh well some of us say that privacy is our main concern and that's just marketing talk and some are really caring about that and yeah it's something I mean like all we got this great keynote this morning about ethical rules we should all implement I think it's also from us data scientists to say okay this matter and I won't deploy this in production as long as there is not this minimum level of work which has been done on the topic obviously it's not possible at every stage if you're like a very small startup but you should still consider that as soon as you're using some data you're like becoming responsible for this data and if there is a major data leakage how would you feel can you still you know get up in the morning and think you're a good person or not and you should not say that it's far away and it's diluted responsibility yeah sure but it's also So, due to responsibility, there is no responsibility. So, I guess it's a very personal question. Perfect, perfect ending. Make sure that when you wake up, you are a good person. Thank you very much, Sarah.