Machine Learning Testing Ecosystem of Python

Speaker 1 [00:03]

so thank you today I will talk about quality assurance in machine learning so which means basically I will concentrate on machine learning specific vulnerabilities and how we can audit machine learning models or in other words how we can test them and for what so let's start with vulnerabilities of machine learning. We all love machine learning. It is great. It achieves tremendous success in many fronts, but especially the recent literature showed us that our machine learning models are susceptible to different kinds of attacks, and they are somehow vulnerable to some flows which I will cover later on. So basically, what I mean is that we have kind of adversarial examples that are very problematic for machine learning models, which is mostly related to the security of the machine learning models. And I'll also touch a little with on the privacy issues with machine learning models because they are, especially the big models, likely to leak private information, the information that is present in the training set. We all know that, especially when we talk about deep learning, we all know that they they are black box. And for some use cases, we really want to understand and explain why an ML model comes up with a specific prediction or decision. But in this talk, I will mostly ignore this one, and it doesn't fit exactly to the testing concept, but you can keep that in mind. And the last thing is the fairness issues of machine learning models when it comes to decision making so for six or seven years we know that people can really easily tweak examples or observations so that they can trick or fool the machine learning models so if you look at the stop sign at the left hand side the most left hand side image, okay, it is for humans, it is a clearly stop sign, and for many of the computer vision models, it is with very high confidence a stop sign. But if we put spatially crafted little stickers on that sign, on that shield, as humans we We still recognize that it's a stop sign, but our computer vision models really fool about its class, and it predicts that it is something other, like a turtle, or a dog, a cat, or whatever. And the bad thing is that we can craft inputs or stickers so that we can manipulate the machine learning model however we like. We can make that model to predict that it is a 200-speed sign so that when an autonomous driving car sees that sign, it should increase its speed to 200 or something like that, which is actually possible, as you see at the right-hand side. So a couple of years ago, people really tricked or fooled the computer vision object detection or the traffic sign detection system of Tesla, so that they just, as you see here, they played with the tree there, and now the Tesla autopilot recognizes it's increased speed to 85. So it has really serious security considerations. And it is not just computer vision. I mean, it's not just computer vision. You can really fool any kind of model. Models that are trained on text data, models that are trained on tabular data, models that are trained on audio data, it doesn't matter. You can craft these inputs very easily. That is the problem. Other than that, okay, so you may conceptualize the adversarial examples like the security aspect of machine learning robustness, but we also have other problems like privacy issues. So sometimes we train our models on the data that contains private and sensitive information, like the patient's information, their specific diseases, or we train models on emails, on people's CVs, you know, any kind of that. But we learned that, especially the large language models, like GPT-3 to birth-based models, really memorize that training, that sensitive information from the training data. And to be able to memorize a sensitive information, some very large model requires it to see that example only a couple of times in the training set. and when you craft a special prefix to a let's say a language generation model it spits out that private information like the emails of a person or the past illnesses or credit card number whatever so i put this image here from a paper from last year because it doesn't contain very serious sensitive information because GPT-2 is trained on public data. But if you fine-tune these kinds of models with the data that contains private information, then you should make sure that they don't leak that sensitive information during inference time. and it is not just an academic toy or game you can craft all of these attacks into the real world for example at the right hand side you can see that our well-loved yolo algorithm recognizes the guy at the left hand side as a person but the other guy who puts a sticker okay a special sticker in front of him. For that guy, the YOLO algorithm, version 2.3, really doesn't recognize him as a person. So I can print that sticker for you in a t-shirt, you can wear that t-shirt, and you can just bypass the security of an institution if they use such a system and any kind of things you know it's it can be really transferable to to the physical world another thing is the fairness so we are now quite knowledgeable about that the machine learning models really easily learn the biases in in their training set and probably you heard some of those news about how biased a machine learning algorithm can be. There are a couple of examples here. Recently in Holland, an algorithm wrongly accused thousands of people, fraud on childcare data. Just based on their ethnical things things or things like that yeah and we all know that gpt3 is has some serious bias problems as well so this is very concerning if you are developing nlp solutions based on large models which should you should do right this is the start state of the art at the moment so you somehow you should get a large language model, fine-tune on your use case, on your data, and use it. You cannot train a new GPT-3 or BERT or whatever. It's too costly. But bear in mind that you are transferring all these vulnerabilities into your systems. Yet another problem with the machine learning models is their generalization capability. Okay, we train our models on the train sets and ultimately we test them on the test set, right? Why? Because we believe that test set is a good proxy for the real world. I mean, when we deploy it to production and that model sees the real data that's coming from the real world, we expect or we hope that the distributions of the two, the real world distribution and the test distribution is the same. So that is why we test it on the test data. But it is usually not the case. I mean, this is a very edge example here. When the COVID breaks, no one would expect any model to predict that such a very hard shock would change everything. But you can just think about our generalization issues. You are predicting your sales, and you are introducing a new product, which can be complementary to your existing products, which means a distribution shift for your forecast, for example. So, how to make sure? Thank God, some communities maintain some good resources for you to convince yourself that machine learning models can be problematic. So for example, AI incident database, you can go hit that webpage and see what kind of things occur in real life. report some AI incidences from the last month some police officer needed to stop an autonomous car because the car didn't put their lights on so you can find some good examples of that Okay, so machine learning, in the end, needs red teams and blue teams. So I talked about attacking models and their vulnerabilities. I didn't mention about the defenses, how we can make them better or something like that. Let's forget that at the moment and just appreciate that the fact that we need red teams to attack machine learning models for security front or on security front or fairness front or data distribution you know anything all those kind of vulnerabilities and we need blue teams to fix those vulnerabilities for us hopefully but at the end of the day we don't have enough resources we expect everything from our good old data scientists or machine learning engineers so they should develop state of the art models to solve a business problem at the same time they need to make sure that everything is fine when you deploy them in production The good thing is that governments across the world started to appreciate this fact, the vulnerabilities of machine learning, and they started to think about putting regulations in place. So the most recent one is the European Commission's proposal on regulating AI systems. In this talk, when I mean AI, I mostly mean machine learning. You can assume that as well. The whole idea of the European Commission's regulation is that we can classify machine learning models into high-risk systems and the others. What we mean by high risk is that if your products or services contains a machine learning part or itself can be a machine learning service, and if it relates to the security health of the real persons or the human rights, then it is by definition high-risk AI system. So of course, at the moment, it's a proposal, and in a couple of years, in two years, we it to be in place all across Europe, all across European Union. So if your ML model serves, or ML model is one of those high-risk systems, then you need to properly test your models before you put them in production. You are not allowed to put them in production without providing your test reports. And in some cases, even, you need to test your systems to the external parties, independent parties. It's like how you test your cars to some external or TÜV in Germany. It will be something like that. So this fact or the upcoming regulations really makes us to think about how to properly test machine learning models and how can we make sure that everything is fine when we deploy them in production. So there is a definition of the high risk AI system in the proposal, but it's likely to change in the course of the time, but it will be around the one that I just explained. So what matters is whether it relates to safety of the people or the human rights. So in addition to that, there are some special things, categories, which by definition goes into the high risk category. And if you are, for example, working in a public institution, for example, gas supply, water supply, electricity supply, transportation, then you will most likely be in that category, high-risk category, and you need to test your machine learning models. But there can be others, like CV companies, because if you automate this first selection of the CVs, using some ML or something like that, you should make sure that your algorithms are not biased. So there are some other examples as well. But the regulation boundary is actually like this. So you need to validate your machine learning models on privacy things, security things, And once you deploy them, you need to monitor your ML models with human in the loop approach. Okay? So which means that you need to log every prediction of your model. And you need to detect if something serious happening in the production. And if something serious happening, then you can, you should be able to interfere with the system. Okay? you should fall back to the humans. For high-risk AI systems, this will be mandatory. So you kind of need a good logger and anomaly detection or an ML firewall on your monitoring stack. And another part of that is the explainability, which is kind of, you all know, familiar from the finance domain, which is already regulated to some degree, And so for some use cases, you need to explain your models. So what does this mean? So this is the classical machine learning pipeline. We gather the data, make some data mumbling, data cleaning, data transformation. Then we should come up with some features and train some models on our train set. And using the test set, we should find out which model performs best. We also look at the generalization by looking at the difference between train set and test set or whatever. So this is roughly the classical machine learning pipeline, but this is where we are going. Now, you need to, when you train your data, you now not just only look at the performance metrics, but you should also look at some other metrics regarding fairness, data drift, security, and the others. And you should also produce some technical documents to provide to the regulator authorities and a user instruction and user manual that explain what users can expect from your machine learning models. And once you deploy those models, you need to put in place a full-fledged incident response with human in the loop approach. So let's concentrate on the testing part of that. Yeah, we should somehow change our classical pipeline to something like this, which integrates ML testing as a first-class citizen, okay? Now, what we need to do is, as I said before, not just performance metrics. We also need some additional metrics to test our models. It is some metrics from fairness domain, like fairness metric, or some metrics that measure, hopefully, the data drift, or some other metrics that hopefully capture the security level of your machine learning models, but in the end, we should look at that. And this is a best practice, not just because of regulations or whatever, okay? Because we know that when you increase The security, when you remedy security problems of your models, you also tend to increase the generalization capability of that as well. Okay, so in any case, this is something that we can appreciate as a machine learning community. And human in the loop, when monitoring that system is very critical. If you are familiar with cybersecurity, This is something like SOC teams, they just monitor the environments and get into play when needed. Okay, let's talk more concretely on how to test machine learning models. I will not go very much into details of the everything, so I will just give you an overview. So we have very limited time, but yeah. This is the four pillars of trustworthy or robust AI. So you should concentrate on security, privacy, which we know from my explanations before, and reliability regarding the distribution shifts and the explainability aspect. So before going more detail, some general remarks. As you all know, as a machine learning engineers or data scientist, we all know that data transitions from stage to stage. We get the raw data, we somehow prepare it to the analysis, and we extract some features from that data, whatever. So this is a data supply chain. And as a general rule, if you play on your data early enough, then you are more likely to prevent the escalation of the problems. So the earlier in this data supply chain, the tests are the better and easier to apply. The second is that we should consider several threat models, okay? Whether black box access is the only way to access your ML systems, or some gray or white box access is possible. This really changed the security aspects of your models. So when you are working in a company and developing an ML model, you should now specify your threat models and take actions in that respect. And now put security, fairness, and data drift, at least these three of the kind of tests, into your ML pipeline, and put in place a monitoring infrastructure, incident response infrastructure for your production environment. So, let's talk about security issues. As I told you before, we know that machine learning models are susceptible to adversarial examples, and they are easy to create, actually, for many of the models. So we call this kind of attacks as evasion attacks. And the goal of the evasion attacks is to manipulate the predictions of your models. And the bad thing is that we can go to your model however we like. We make your model to predict cat as a dog, specifically. Another type of attacks that is related to security is the poisoning attacks. If you are constantly getting data from external environments and regularly train your models using that data, then you are opening a poisoning attack surface for your models. People can put specially crafted inputs that you don't notice when collecting the data, and when everything goes fine in production, people can trigger a specific example which manipulates that model according to their wishes. So it's like a time bomb or something like that. And another thing about the security of your model is the stealing attacks. Yeah, people can train student models using your model as a teacher student, as a teacher model, and at the end of the day, they can steal your model, which means an intellectual property of your company might be at risk. How these attacks work? At the left side, you see how the evasion attacks work. It just query your model, okay? If they access to your predictions of your model, that's enough, that's enough. They don't need to know the inner details of your model. just generates, even sometimes just put random examples, random queries to your model and see what kind of return, what kind of output it returns. Especially if your models produce probabilities, returns outcome as probabilities, then people can train their own student model. And when they apply adversarial attack techniques to that student model, that examples most of the time also work on your model. This is what we call transferability of the attacks. And it is what makes everything very dangerous. People really don't need to get inner details of your models in text models. I mean, if you are developing an LP model for some task using BERT or something like that, people can attack your models easily because the attacking points or the attacking examples that work on pure BERT would most likely work on your case on your fine-tuned model as well. So, the Python. Python offers great tools, open source tools, to test your models. So, my favorite tool for testing a model in this direction is the ART, the Adversarial Robustness Toolbox. It offers many state of the art algorithms for you. It offers many attack types and defenses as well. I mean, if you would like to play in this area, you can just start doing that using ART. It was initiated by IBM, now it's managed under Linux Foundation. And another thing is the clever hands. It is now managed by the Vector Institute in Canada. It's also a good toolkit for adversarial attacks and security testing of your models. When it comes to privacy, you can test your models. As I said before, the larger your model gets, the more likely it memorizes the information in your training data. So be careful with large models. And also keep in mind that transfer learning transfers the vulnerabilities to the fine-tuned models to models as well. So be careful on what kind of pre-trained model are you using when you apply a transfer learning scheme. Unfortunately, we don't have good solutions or defenses for that. The only viable solution seems to be differential privacy, but it is really impractical if your data set or model gets too large. its compute costs are very high. And you should also keep in mind that differential privacy costs some accuracy for your models as well. So yes, this is just an example of membership inference attacks. Membership inference attacks means that, similar to evasion attacks, people query your models and try to guess whether a specific example is in your train set or not. So they can, with a great success, they can determine that, okay, this example is part of your training data, which means your training information can be stolen by the external parties. It doesn't only related to text models. People can also apply model inversion attacks to steal, for example, images from your models. from your models please yes right now I think I have time to 35 yes it was five minutes for questions but so I have 15 minutes right okay I mean according to the agenda I have, but yeah. Okay, let me quickly wrap up. So, okay, you have some tools to test your privacy issues as well, so I will not go over that, okay? I will share the presentation with those who are interested. And yeah, you can apply some testing using the Python ecosystem. So it has a great ecosystem when it comes to machine learning validation interesting. Okay. You can, especially for fairness, you can make use of AI fairness, which is also a part of the Linux foundation. Even though you don't go through the previous show, it's like a couple of more decades. I think you're right. Yeah, I guess. But if you get bored, you can just get out of the class. It's fine. So, but anyway, I speed it up. So, yeah, about data drift, I mean, there are some couple of data drift concepts. It's about changing the distributions, okay? Changing the distributions, you can test them using some statistical methods. Yeah, when you get the presentation, you will get the packages and the links. You can just follow so that you can use by yourself, okay? Okay? So what to do? So the first thing to do, okay, change your way of doing ML. Keep track of the versions of ML models if you don't do that at the moment. You can do that using an MLOps tool, but if you are not using an MLOps tool, make sure that you need to keep track of the versions because you need to be able to associate the the production predictions with the specific model versions. So it is something mandatory. As I said, test your models for security fairness, privacy and data drift, along with the performance metrics. And for defenses against the adversarial attacks, usually adversarial training is the only way to increase the robustness of your models. So put some adversarial examples in your training set. It is like using a data augmentation technique, okay? And be prepared to write good documentation, so it is something that you need to do as well. So I will introduce quickly the validator that we are developing at KIT, Kastro Institute for Technology, which we will make open source during this year. We are still working on that. So this is the tool, an autonomous tool, end-to-end tool, that automates everything, every testing for you. You just give your machine learning model, saved models and data sets, and write your test as a config. So you don't have to learn the nitty gritties of every testing by yourself. So when we open source it, you can just start using it, adopting it at your company, and so it offers testing and monitoring in a single place. So, as I said before, it covers both of them because there are some feedback loops between testing phase and the monitoring phase. Because to detect, for example, problematic instances or examples in the production, we may need some information from the test set. That is why this system follows a holistic approach on this, okay? And it can integrate with your current envelopes tools if you are already using them. Yeah, it keeps versioning. It runs tests on your behalf. It is like a CI-CD tool. So if you are familiar with software development lifecycle, like Jenkins, Travis, or whatever, you just push your models to a repo, and all of your unit tests run, right? You just need to write those unit tests only once. So this is similar here. You just need to define the tests as a config, and it will just run it every time you push your model in repository. So you will get the reports, you will integrate it with your production pipeline, so if some thresholds are not met, you can prevent your model going into the production. So we don't expect ML engineers to know everything. So it is kind of a new field in machine learning, and it requires its own expertise. But you should somehow integrate ML testing in your workflow so that you don't spend too too much time on that, but can concentrate on what you need to do, which is developing or solving a business problem. And this field is changing rapidly. I mean, it is almost impossible to catch up with state of the art. If you want to apply these things by yourself, every time you need to spend really good amount of time to catch up with the literature. So that is why we are developing this one at KIT, with the support of the Baden-Wurttemberg government, by the way. Okay, so this is all. So we are looking for beta users. If you are interested, you can just write me, and I will somehow share these slides with you so that you can get to know those great packages in the Python ecosystem, so you can start using that to ensure the quality assurance in your machine learning endeavors. So thank you. Thank you, sorry about the time confusion. It's fine. Does anyone has any questions? I can't see anything on slides. So if you have any questions, please raise your hand. Thank you very much for the interesting talk. I wonder if there are similar regulations for software systems, or is it now really a new thing that just applies and is expected from AI systems? Because software systems have similar vulnerabilities, not all of them, obviously, but probably. Yes, but from the regulation point of view, AI is something different because it makes decision-making. Like it reads the MR images and diagnoses an illness or something like that. Or it is used in autonomous driving, which is directly related to the safety of the people. So that is why high-risk AI systems are something that should be treated something different. It's like we humans, because they are to do what we are supposed to do. That's why these regulations put focus on high-risk AI systems. So it's kind of different from that in terms of their consequences. Thank you. Any more questions? okay I'll come to him and then you are next sorry I think I'm first yeah so just short answer for my previous question depending on your sector yes there are definitely regulations in software development as well coming from healthcare, I know that pretty much. But I have another question. You mentioned in the middle of your talk that white box model would be able to prevent or circumvent some of the issues that occur with a black box model around the security of the model. Could you maybe give an example of... No, no, I didn't say that. Oh, sorry, I didn't understand that White box model, if you can apply white box attacks, it is more problematic because you have much more data than black box attacks, that's for sure. But at the end of the day, the gradients, if you know what I mean, the gradients are very dangerous things, and if you can access the gradients of a model, you can really, really reveal almost anything, even the training data, from that gradient information. Can I now? Okay. So, thanks. You mentioned earlier this poisoning attacks. Do you have an explicit example, like, how that works and what we can... And explain again, please. I will just give a toy example. Things like that, you are automated your credit card application or credit applications using an ML model, and every time I put my things, my information, I got rejected because of my salary or whatever, but I just tweaked my example, so every time I requested a credit, I increase my, I don't know, my square feet of my home or something like that, and that information persists in your database, and if you are using that information, okay, and if I crafted those perturbations on the examples carefully, I can invoke a positive credit thing by just making my salary from 30,000 to 29,000, okay? And I get credits accepted. It is because of the continuous learning flow of your model. So if you are doing online learning or active learning that you train periodically with the data that's coming from production system, you should be careful. Thank you. Any more questions? No. Okay. One more question here. I was just curious if the kind of extra layers of work involved with this testing will have an effect on the viability of machine learning in different cases due to the cost associated? I really couldn't understand. Can you please more? So what I was curious about is because of the extra work involved with the testing, that will affect the viability of models in different cases because it will be more expensive to develop the machine learning in some way. Yes, I mean, yeah, it is costly in terms of time. You should invest in testing. But it is something that you need to do if you are developing a high-risk AI system. Okay? This is the price that you should pay. But people are developing tools to make your life easier. that is why people like us in the university are developing tools for the community so that you can use it and open source it because we follow the literature we know the best practices we know the state of the attacks and we know the state of the defenses but no one expects you as an ML engineer in the industry to know or to be up to date with the literature So that is why you need to somehow use a quality management system in your organization. Otherwise, it would be very costly, and it would be, at this moment, impossible to find good ML validators or hackers. Thank you very much. Thank you. Time is up. You upload the slides. Thank you. Thank you very much. Thank you very much.