Accuracy Is Not Enough: Building Trustworthy AI with Conformal Prediction

Speaker 1 [00:07]

Today we'll be talking about building trustworthy AI. This is not just a buzzword, like we are actually going to deep dive, look at code, look at math, everything, so it's going to be a really hands-on session and yeah let's get into So what you will learn today is how to have, for example, 95% confidence on your predictions with control and prediction, and that 95 number can be whatever you want. It can even be 100%, but we can talk about this later. You'll learn how to detect hallucinations and get guaranteed probabilistic answers from your language models, even from ChatGPT, which is closed source. You'll learn how to calibrate your model to output probabilities and not scores, which is very, very important. But even more importantly, you'll learn not to stick to the basics, you'll learn how to make informed decisions, and you'll learn how to build useful models, trustworthy models, and not just models sitting in a notebook somewhere in your local environment. So, who am I? I am Chris Aivazidis, I am a data scientist, I currently work for VML and Satalia. I have worked on multiple high-risk applications, namely an AI auditing framework. I've done research into explainability, in natural language processing, I participated in machine learning competitions, I'm a speaker, as you can see, and I have a maths background. And feel free to scan that QR code or go to that link so that we can connect on LinkedIn. Now a really funny and appropriate quote I found online is that if your model says there's a 50% chance it's going to rain tomorrow and it never rains, congratulations, you just built yourself a weather app. So all the code, all of the methods, rather, that we'll see today are available on this GitHub repo that I created for the talk. You can scan the code or you can again visit that link. So let's get straight let's get right straight into the first use case which is machine learning in healthcare and as you can imagine this is a talk about trustworthy AI so these two go hand in hand and here the risks and the stakes are very high. Wrong conclusions can literally mean death on on certain situations. And some of the use cases are predicting diseases from blood markers, identifying tumors from scans, which is on the vision domain, or offering medical advice on the NLP domain, which is not advised by the way. And let me give you some context first before we jump in by talking about immune-related adverse effects. So in modern oncology, Basically, immunotherapy has emerged as a leading way to combat and treat cancer, certain cancer types at least. But these treatments can often lead to adverse effects, which can range from mild to life-threatening. So here is where machine learning comes to the rescue. And essentially, clinicians try to find the probability behind a patient actually developing this adverse effect so that they can make an informed decision. And you can see on the bottom right, for example, that a patient and a clinician must both be informed about what is the probability of actually having an adverse effect given this current treatment, this immunotherapy treatment, or what's the probability that I'm going to get cured given this treatment, or what's the probability I'm getting cured with another treatment. These are all very important. So let's talk about John. So, John is a data scientist who works at a hospital. And one day, a doctor comes to the hospital and says, listen, John, we need to build a model, we need you actually to give us some probabilities that a patient will develop an adverse effect given the immunotherapy, so what can you do about that? And he also gives John this data set which contains patient data from cancer patients and it also has a label which says whether the patient actually developed an adverse effect or not. So John looks at this data set and he's very happy. He says, we're good to go. I can build something for you, don't worry. So John gets straight into business doing all of his data science stuff. He pre-processes the data, he applies transformations, he does feature engineering. all of the good stuff we data scientists do, and he manages to build a model that is 80% accurate. Now, he's very happy. He thinks, wow, I built a good enough model. But there's still a problem. He's built a model for binary classification, and that model only outputs zeros and ones, right? The original task was to actually estimate the probability and not get a prediction. So John thinks, wait, scikit-learn actually gives me a way that I can get probabilities from my model. So he calls predictProbo, which is a method that supposedly gets you probability, sorry, gets you probabilities out of your scikit-learn model. So with that, he's very happy and a new patient comes in. He passes his data to the model And the model predicts that it's 90% likely that this patient will actually have an adverse effect. So the doctor says, you shouldn't really get a treatment because it's 90% likely that you'll have something bad happen to you, right? But the patient still decides not to listen to the doctor and goes for the treatment. And guess what? He doesn't develop an adverse effect. John thinks this is a bit strange, but there was still 10% chance that he wouldn't develop it anyway, so all is fine. Now is where things start to get fishy because another patient, like the previous one, with the exact same symptoms, the exact same data, and the exact same prediction comes along, and then more patients come along. So now John ends up finding out that out of the 20 or 19 patients that came in and had the exact same symptoms and the exact same prediction, only 10 developed an adverse effect and nine didn't develop an adverse effect. So even though he expected 90%, he actually observed 50. So what did John do wrong? And here is the intro to calibration. So this is called a calibration curve. And what it means or what it tries to do is it tries to show you how well your model actually outputs probabilities. The line here basically shows perfect calibration, which means we predicted 20% and it was actually 20%. But as we can see from this plot is that our model, when it's saying 90%, it's more like 50%. So this is exactly the problem that John was facing, right? And basically, in order to get this plot, by the way, we beam the data into really small bins, let's say 10% probability, 20% and all of that, and we just calculate the exact number of positives we have in that bin. Now, how can we measure, actually, this calibration? This is a very neat formula, which is called the Breer score. It's actually very simple, Essentially, we're just measuring the difference between the predicted probability we have, which is the score that our model gave, and we subtract one if that event actually occurred, and zero if it didn't. So you can see an example here. If our model predicted 90% and ground truth was zero, this means that we'll have a really high Breyer score. But if our model indeed predicted 90% and the truth was indeed that he would develop an adverse effect, we get a really low Breer score. So if you do that across all your samples, you get a Breer score for your model, which essentially shows how well calibrated it is. Yeah, and just a small tip that this score is a bit inadequate for very rare or frequent, or infrequent, sorry, events, because it doesn't sufficiently discriminate between the small changes in forecast. So, what can we do to fix calibration? One thing we can do, which is a bit obsolete at this point, is isotonic regression. So, what does isotonic regression tries to do? Essentially, again, we have our data, We rank them by the probability score, and we're just trying to find the minimum of the sum of squared differences between the observed probability and the score we have that is like a probability. And we just apply a monotonicity constraint. We say that this always needs to go up, And we have this as the isotonic regression plot and the curve that is supposedly more calibrated than the previous one. But there's actually a better way with Venn-Abers. So with Venn-Abers, we actually fit two isotonic regressions, one assuming that the label was zero for all the points and one assuming that the label was one for all the points. And what that actually does is it creates an interval that has statistical guarantees that the true probability will actually lie in there. Now, that algorithm is actually a part of the confirmed prediction family, which we'll talk about in a sec. And it can also be used to estimate uncertainty because the bigger the interval, the more uncertain we are. And we can also get the calibration curve for Van Abbers. We don't only get the prediction interval, so that we can compare with other methods, right? And let's see what happens when we apply it to our previous example. On the blue line, we see the uncalibrated model. On the orange line, we see the calibrated model with isotonic regression, which is still very bad. We see that it's overconfident in its predictions. Essentially, when it predicts 90%, for example, it's more like 60 or 70%. Whereas with Venn-Abers calibration, we actually are very, very close to the true calibration, to the wanted calibration. But as you can see, I mean, there are still a few spots that are a bit misleading. So, what other methods could we employ to deal with that problem? One way that we could have dealt with this problem with providing some uncertainty quantification for this domain is conformal prediction. And what conformal prediction does is it turns point estimates into prediction sets or prediction intervals that essentially guarantee, with a user-specified probability, that the true label will actually lie within that interval. So we see here that a prediction interval, a prediction set, rather, can be label four and label seven with 95% probability, whereas a typical machine learning model would just output label four, for example. So this is the guarantee we get, as we talked about earlier. And what are the assumptions, like what is actually needed in order to achieve that guarantee? So I think the assumptions are very, very mild. There's only one, essentially, that the data is exchangeable. And it's a model-agnostic, task-agnostic, distribution-free kind of coverage guarantee. How can we get better prediction sets? The better the model, the better the nonconformity score, which we haven't talked about yet, the tighter the prediction sets. So in an ideal world where the model was 100% accurate, you would get singleton sets. And yeah, as I said, because there was a talk about conform prediction yesterday, and I think it wasn't very clear, the parameter A, which is the probability that the true label will lie in that interval is user-specified, so you can make that whatever you want. You can make that 95% if that suits your case, you can make it 90%, 50%, whatever suits your goals. And if you make it 100%, for example, you can, but you would just get the whole set of labels as a result. It's like saying I'm throwing a dice, there's a 100% chance that it will land on some of its side. So how it works, essentially we have the nonconformity score, which is the biggest thing, the biggest part of how it works, which measures how strange a candidate label is for the given input. And that measure can be anything you like. It can be negative likelihood, it can be one minus the predicted probability, whatever you like, whatever you think is the best way to describe strangeness. And the good thing is that the guarantee holds regardless of what you choose as a nonconformity score. We also need a calibration set to compute those nonconformity scores. And what we do afterwards is we just calculate the quantile, the smallest 95th quantile or whatever we have set with our alpha. And then the prediction set is just those labels that are less than this quantile that we have set. So in our example, let's assume that we had three classes instead of one to make it a bit more intuitive. And the three classes were no adverse effect, mild adverse effect, or severe adverse effect. And we wanted a confidence level of 95%. So let's say this is the first case where we have a prediction set that says no adverse effect with 95% probability. This is a very certain decision for almost everybody, right? Like we have 95% chance that nothing happens to us, so we should take the treatment. Case two, however, says that the prediction set is either no adverse effect or severe adverse effect at 95%. Now we're uncertain. We don't know whether this is actually a good option for us or not, right? And on case three, we have either no adverse effect or a mild adverse effect. Again, that's a rather certain decision, right? Like even if I get a mild adverse effect, it's still better than having cancer, I guess. In case four, we have mild adverse effect or severe adverse effect. Again, I would say this is an uncertain decision because if I have a mild adverse effect, then and that's fine, but if I have a severe one, yeah, that's not so good. So let's jump into the second use case, which is machine learning and trustworthy AI for natural language processing. Here, things are not so bad. I mean, wrong decisions can lead to loss of money, for example, and some of the use cases are chatbots, knowledge systems, sentiment analysis, whatever you like. Before we get into it, let's take a look at this article that basically says that Air Canada had this chatbot, and that chatbot basically told the user that, yeah, you can refund some of your money because we have a policy somewhere, and that user tried to actually refund that money. And after a couple of months, they told him, we don't have that policy, actually. It made it up. so the user then took them to court and he actually won and he got refunded even though it was the chatbot who said that yeah you you're eligible for a refund so what if we could for example ask an llm what is the capital of france and we could get an output saying paris with a certain guarantee right or if we had some summaries of news articles and then we get back a set of articles, that at least one has a Roos score of more than whatever, or we have a bunch of reports, medical reports, and we say that at least one matches the expert labels, and et cetera. I mean, you already get the point. So here comes conformal prediction again, and more specifically, conformal language modeling, which essentially extends conformal prediction for language models by sampling the outputs until a calibrated stopping rule guarantees with a given probability, like we said, which is user specified, that at least one of those answers is admissible. So in conformal language modeling, our nonconformity score is comprised of two main things. One is this quality threshold, which takes the logits of the model and measures the quality. And then another diversity threshold, which can be whatever, but I think they use Roussel. And they measure that, they filter samples before adding them to the set, and they stop when they exceed the given confidence. They also have this other rule, actually, that breaks long generations into subcomponents, subphrases, and with that way, they can actually identify whether there has been a hallucination or not. So this has actually been implemented in the tortscp library, which is a Python-specific library for deep learning and conformal prediction that has implemented tons of these methods for every kind of domain that you can think about. Vision, NLP, classification, regression, and all the rest. So let's see with an example how it actually works. We are given this X-ray image to the language model, and the first output of this generation, draw multiple outputs, like I said, says something which is correct, but then says something that is totally false. It's totally different than the reference point we have, which says no signs of edema. It actually says that there is pulmonary edema. So we actually reject that sample. Then another sample comes in, again, saying some stuff that is correct, some that is incorrect, But it also says, oh, it says there is no frank edema, which is, I guess, okay, not going to think about it. Anyways, and the third sample comes in and says there is no pulmonary edema, which is correct. So at this point, the filtering rule and the quality rule pass. So we accept the whole set. As I said, with this method, you can already identify which points are hallucination or not, so it's very, very good. But one thing we actually need to note is that we actually need to access the logits of the model in order to use that method, right? So it only works for open source models. Now, somebody will come and tell me, hey, Chris, I don't use any of that open source stuff. I only use ChatGPT. How can I use that? But conformal prediction has your back again with this paper called API is enough and some other stuff, that essentially tries to mimic what the previous paper was trying to do, but it does that even for closed source models like ChatGPT and how it does that is by drawing samples from the prompt, it draws multiple samples and calculates the frequency. We define the base nonconformity score as 1 minus that frequency but we also introduce two more scores that will hopefully help us create a more robust nonconformity score and we calculate the normalized entropy and we also calculate the semantic similarity where we use the most frequent class as the anchor. Then we do the rest of the typical conform prediction steps, which is having the calibration step, having our calibration data, calculating the quantile, removing the answers that are nonconforming and creating our prediction set. And on the bottom, you can see how that nonconformity score looks like, as I said, minus the frequency plus the normalized entropy minus the semantic similarity, but we have also introduced two weights that you can use to control how much each of these parameters actually influence the nonconformity score, so you can really tailor it to your use case. and here's the link to the paper but the bad thing about this paper is that they didn't include any code so if you go to my github repo i actually created the code for that paper so i don't think you can find it anywhere else online and hopefully you'll find it very useful so as i said typical process we have a calibration set of Q&A pairs. Let's say the first one is which American born Sinclair won the Nobel Prize in literature in 1930 and the answer is Sinclair Lewis. We calibrate the model using the nonconformity measure we showed earlier. We also find the weights L1 and L2. As I said those are more of like hyper parameters you don't actually find them with this method and and you get the typical conform reduction guarantees and the set that we have talked about so far. Now here's an example of how that works. So I ask chatGPT, what is the capital of France? It returns Paris, Paris, Lyon, Paris niche. And now we can start calculating our nonconformity measures. One is frequency, the other one is normalized entropy, and the other one is semantic similarity. Again, I repeat that for semantic similarity, we use the most frequent word, which is Paris as the anchor. And for normalized entropy, this is actually the same for every candidate, but we still use it. And this is the final conformal score. And essentially we have all of the outputs that the LLM have, all the unique outputs. We pass them through the nonconformity score, and then we exclude the ones that don't pass this filter. And we have a prediction set that says Paris with 90% probability. Yep, so this is some independent testing I did with my code, with this method. So here on the left, you can see some examples that are good and where this method performs as it should have. I ask it which country is Europe's largest silk producer and then it comes back with Italy. I ask what is Bruce Willis' real first name. It comes back with Walter and I check. This is legit. And then I have two more questions that are not really good, but I just wanted to see how it works. Like if I ask a question that is one of those questions that are trying to trick ChatGPT similar models, which is, what is the number that rhymes with the word we use to describe a tall plant? A tall plant is a tree, of course. So three would be the correct answer. And it comes back with three sets of answers at 90% confidence level, three, three. This is probably because I used GPT-40 mini, by the way. And also these outputs are actually normalized, so it gave me both the word that is used to describe the toll plant and the word that rhymes. We also have the correct one, which is three, and we have another one which for some reason also says number three instead of, yeah, three. And I also asked, because I'm Greek, what are the top three ways to fix Greece's financial situation? And it came back with four answers at 90% confidence level, saying economic reforms, tax reforms, investment incentives, and economic reforms is a general thing, apparently. We should really look into that. And there are some other things like tourism promotion, debt restructuring, stuff like that. So let's get into our final use case, which is machine learning for robotics and trustworthy AI for robotics. So here, wrong decisions can also lead to death and loss of money. And some applications are self-driving cars, control systems, I don't know why sentiment analysis is here, my bad. Anyways, so as you figured it out so far, I first give you examples where we failed doing that. So one recent example is actually from Tesla where a guy had his autopilot on and the Tesla crashed into a wall and killed him. So Elon Musk, by the way, said it's too late for that. Yeah, sure. So, let me show you an example from a paper from Google and DeepMind that uses conformal prediction to make robots smarter, safer, and ask for help. so as you can see here the human asked hey robot could you put the ball on the small counter in the microwave and the robot first gives us his next step prediction with certain probabilities, and we see that the more likely answers are put metal ball in microwave or put plastic ball in microwave. So with conform prediction, they actually have the prediction set which says it's either put ball in microwave or put plastic ball in microwave. So there are two decisions, both kind of equally likely. So the robot is uncertain, and instead of just doing some random thing, he asks. He says, should I pick the metal ball or the plastic ball? So now a human, okay, that went from the start, but a human can now tell him, you should pick the plastic ball, please. And the robot manages to complete his task. So we already kind of talked about how that works. If you want, you can take a look at their paper and they also have a GitHub repo and a website where you can play around with different decisions. They also have simulated some of this stuff. I think it's really nice. But the main thing you need to understand is when that robot has a prediction set that is more than two, more than one, sorry, then it's uncertain and it asks for human help because that's the only way that they're sure that they'll take the right action. This is an extension of multiple choice question answering. And as I said, they use conform prediction to guarantee that the prediction set contains with probability one minus alpha, the right answer. So what are some tools and resources you can use to start your journey in conform prediction? In terms of libraries, I already told you about TorchCP, which is a control prediction library mainly used for deep learning, but you can find stuff about tabular data too, vision, text, whatever you want. There's also MAPI, which is very, very standard for tabular data, although lately it has faced some criticism. there's also creps which basically is a library that can create conformal predictive distributions which might be a topic for another conference but it's very very interesting you should definitely take a look and if you really want to get into conformal prediction look behind the math make sure that those formal guarantees indeed hold and it's not just stuff that comes from my head, you should take a look at this paper, which is from one of the guys who's like the leading researchers in control prediction. And he has multiple examples with maths, with code, with everything, and all the proofs to make sure that what he's saying is legit. There's also this control prediction blog, which comes from Valerie Manokin. So Valerie is one of the earliest adopters, let's say, of conformal prediction. He's very, very good. He's also on LinkedIn. He writes a lot of content. You should definitely check him out. And some takeaways. So this is a really famous quote, right? All models are wrong, but some are useful. And this was my point originally with this talk, that you should go behind or you should go past creating models that are just 80% accurate or whatever. You should go past blindly trusting your models and blindly trusting predict proba, for example, from scikit-learn. You should actually calibrate your model's probabilities because that's the only way that they're useful. If you don't do that, what good is your model? You can't really use it. you should use control and prediction because it turns point estimates into probabilistic guarantees with whatever coverage you want, whatever guarantees you want, right? Which means that you have a way to quantify uncertainty, you have a way to make smarter decisions, to account for all the risk in your given domain. And you should also learn by now that you should build a robust decision-making pipeline. Regardless of your model, regardless of your domain, your model will make some predictions. How do you actually use them? What's the risk behind trusting your model, trusting your prediction? What can go wrong? What can go right? And finally, uncertainty quantification saves lives, quite literally sometimes, but it's very, very important to account for the cases where your model might be wrong, where something might be really off. So yeah, that was my talk. Thank you very much.

Speaker 2 [34:29]

Great talk. Thank you, Chris. And now we have a couple of questions. So, not a couple, really. First of all, given that problem predictions intervals are often wider than necessary, are we solving the uncertainty quantification or just, I think it is hiding ignorance behind conservative bounds?

Speaker 1 [34:56]

So I don't think that's very accurate. First of all, there are multiple flavors of conformal prediction. Like what I've shown you today are only pretty basic stuff. There are multiple flavors that account for making your prediction intervals narrower. But the unnecessary parts, I'm not so, yeah, I'm not really buying that. because if you use any other methods, like Bayesian methods or Bootstrap or stuff like that, you don't actually get the coverage guarantees. Here, we have mathematical guaranteed coverage, which means that whatever happens, as long as the assumption of iid holds, you'll get that kind of coverage. It's not like we're saying, here's the whole output space, this is your prediction interval. It's still, I think it's still narrow enough. And given that it's, I think the only way that we can guarantee that coverage, I think it's the best we can do.

Speaker 2 [36:04]

Okay, good answer. So the next one would be about LLMs.

Speaker 1 [36:10]

LLMs.

Speaker 2 [36:10]

LLMs can also pretty consistently answer questions bluntly wrong. This conformal prediction doesn't really help in rating the transforthiness, doesn't it?

Speaker 1 [36:26]

Well, I think it does, but I mean, it depends on the domain. So if you wanted to be trustworthy in an open-ended domain, so I ask it a really general question like the one I had before, give me a second. so yeah if I ask it for example what is the oh sorry yeah what are the top three ways to fix Greece's financial situation there is not really one answer right like regardless of what method I use to to measure the transworthiness there's not really one answer it's open-ended so if your task allows it if your task is close form I guess then this really makes sense but yeah it's it's dependent on the task I think

Speaker 2 [37:21]

I think I think just to introduce a little bit flavor on here there is some additional things that when open I also tells that the LLM is absolutely sure that it is answering wrong so that's one of the reason probably for this question the next one is also about confirmed prediction if our prediction focuses on prediction intervals rather than probabilistic forecasts aren't we missing information about the shape of uncertainty

Speaker 1 [37:59]

That's a good one. That's a good one. Yeah. I think you can still extract some of it from the non-conformity. Actually, you can extract a lot from the non-conformity scores, but here's why I showed both things. So, in the first example, I show you how to calibrate your probabilities, for example, and next, I introduce you to conform prediction. So, I think, let me go back to the cases here. so if we were on the second case right where we're uncertain we either predict no adverse effect or a severe adverse effect I think this is where we can jump to the calibration techniques like we can extract the probabilities the supposedly probabilities which won't be 100% right but it's the best we can do it's as close as we can get I think to uncertainty if that doesn't work

Speaker 2 [39:03]

Okay, then let's talk about classification tasks. What about them? Where the set of possible outputs is limited? Does they influence our approach for confirm, predict, for example, semantic similarity?

Speaker 1 [39:21]

So I think I got lost on the last part, but in general, if we're just talking about classification, this is exactly what we have right here, right? Like we have three classes or that can be whatever, can be two, can be 10, 20. And what conformal prediction does is it creates those prediction sets. Sometimes those can be big. If the input is hard to predict, predict you know in certain cases for example if you had 10 classes you might get eight classes back if it's really uncertain really hard but if it's something really easy for your model it will probably be one element or two elements and that's all dependent on your model and how good it is like if you have a bad model of course you would consistently get really wide prediction If you have a really good model, on the other hand, you would consistently get single element prediction sets. I hope that answers the question.

Speaker 2 [40:26]

Okay, then the next one is about confidence intervals. So what is the reason it is not possible to use standard confidence intervals?

Speaker 1 [40:37]

The reason, okay, so standard confidence intervals come from statistics, and usually they refer to a single variable to do that, but you make a lot of assumptions in order to have those confidence intervals. Probably you're assuming normality, you're assuming tons of different things that definitely don't hold in real life, so this is why we go for prediction intervals.

Speaker 2 [41:07]

instead that's a reasonable answer so the next one regarding the coverage the

Speaker 1 [41:07]

instead.

Speaker 2 [41:13]

courage guarantee may not depend on the choose non-conformity score but the specific set of labels does right

Speaker 1 [41:23]

the specific set of labels. I guess maybe this is referring to whether the problem is imbalanced or not or something like that. Because if it's about the domain, then it doesn't matter. Like if we're in healthcare or robotics, it's the same thing. But I mean, even if it's imbalanced data, the only thing that it will do is that on the on the class that is more dominant you will probably get single set predictions whereas when it's more uncertain so in cases where it's actually the minority class you would probably get more like wider prediction sets i think that's the

Speaker 2 [42:14]

That's the thing. Okay Then the next one, how do I properly choose the distribution of the predictions for the conformative theory? This is very important as Everything is dependent on it

Speaker 1 [42:31]

So, as I said, we don't make any assumptions for the distribution of the data, nowhere, with conformal prediction. So, whatever works, like, whatever works, whatever distribution generated the data is fine. It still works with conformal prediction.

Speaker 2 [42:51]

Okay, the next one would be also about conformative predictions. If it guarantees marginal coverage but struggles with conditional coverage, isn't it reinforcing biases by providing misleading confident predictions?

Speaker 1 [43:12]

That's a good question. I think that's a really good question. So as I said in one of the earlier questions, actually, there are multiple flavors of conformal prediction. I only showed you like the basic original version in the beginning. and there are other ways other control prediction flavors where you can actually combat that and you can either go for marginal coverage or conditional coverage or whatever you like so there are ways to fix that if that fits your use case

Speaker 2 [43:47]

Okay, and let's have the last one here. How to get the ground truth on how confident the model should be?

Speaker 1 [43:58]

I don't think there's a ground truth for that. Getting the ground truth for how confident... I don't know how to answer that.

Speaker 2 [44:09]

I'm sorry. Neither do I.

Speaker 1 [44:11]

So

Speaker 2 [44:12]

Okay, so in this case, let us thanks once again Chris for his great presentation.