Conformal Prediction: uncertainty quantification to humanise models

Speaker 1 [00:07]

Hello Leute! I'm so happy to be here at PyCon Germany in Darmstadt and tell you about conformal prediction which is a statistical framework for uncertainty quantification which can help us humanizing any machine learning model. Just a few words about me. I have a background in theoretical physics and I'm working as a machine learning engineer and data scientist at the National Institute of Geophysics and Volcanology in Italy, where I work in the space weather domain, and I'm also one of the organizers of the local chapter of Pi Data in Rome. That's the agenda for today, that's what I prepared. We will discuss the importance of uncertainty quantification and the role of conformal prediction, also the importance of having a human in the loop and some of the applications and some more advanced topics of conformal prediction. So let's start. Well in general we should consider that the outputs of machine learning models may sound a bit pretentious and also some someone would say oddly precise and they might not be very useful in the sense that they provide just some point estimates and thing is that should we really trust the outcomes of work and what can be considered a black box model especially if you are dealing with high risk settings like healthcare finance or even the space industry. Just imagine that you are a physician, okay, and you hang a lung CT scan to a model that is supposed to help you identify cancerous tumors. Traditional approaches tend to provide raw model scores or binary predictions, for example, and they do not provide information on the model's confidence confidence in his prediction so they may sound a little bit sibling and this problem is related to the fact that in general models are trained just to predict accurately and not necessarily to provide a prediction set but let's take a step back and see how do we as humans express uncertainty well when we face uncertainty, we tend to naturally provide alternatives, expressing how confident we are in our responses. So if we are unsure, we tend to signal our degree of uncertainty. So just imagine that you are seeing head-ridden digits, okay? Someone might say, oh, it's clearly a four. Someone say, I don't know, maybe it's a four or it's a nine. Someone else might just say, I can't even read that. So being a human, after all, means that we are allowed to be uncertain. And so, can we try to make a model more human? Can we humanize it in order to also make it more trustworthy? But how do we estimate uncertainty? Well, uncertainty quantification is certainly a broad topic. we can divide this in two main branches made up of intrinsic methods and extrinsic ones also called poor stock methods the intrinsic methods require some retraining of the underlying machine learning model while the extrinsic ones do not and just to give you some examples you can think of bayesian approaches and quantile regression as examples of intrinsic methods, while we will focus today more on calibration and conformal prediction, which are good candidates for extrinsic methods. We start with calibration, and that's the definition. If you consider, for example, a classifier, we say that a classifier is well calibrated if a prediction with a score p, say 0.3, is correct p percent of the time, so 30 percent of the time. And that's the mathematical statement for that. Similarly, you can think of calibration in terms of a regression model if you reason in terms of the quantiles of the output distribution. But can we see it more visually? Yes, we do with the help of the so-called reliability plots which essentially plot model scores, raw model scores on the x-axis and the fraction of true positives on the y-axis. That's the case for a classifier and quite similarly one can reason for regression models as well. For example, you can see that model scores on the x-axis of 0.2 for this particular case happen to be real positives roughly 40% of the time. So you would expect that a model score of 0.2 would translate into an actual 20% fraction of positives. So a well-calibrated model would lie on the straight dashed line. And in general, machine learning models tend to exhibit under-confidence or over-confidence. And calibrating the model is crucial towards making good and reliable predictions. We can extend this framework thanks to conformal prediction, which essentially turns any point estimate to a prediction set. And the beauty of conformal prediction is that this approach is almost distribution-free in the sense that, in general, we just have to assume exchangeability between test set and calibration set. I will tell you more about calibration data shortly. This approach is more diagnostic, and there's some mathematical guarantees of coverage of the true outcome. So that's a recipe for one experiment of conformal prediction and you can use it for classification and regression tasks and it's quite easy to achieve, it just requires a few lines of code. What do we need? We need a model that is ready to be trained, after all it's a postdoc method, an heuristic notion of uncertainty, an S of X and Y, an error rate alpha that is related to the confidence level that you want to set, which is 1 minus alpha, and just a pinch of fresh unseen data for calibration. So you need a separate set from the testing and training one, and you have to reserve some calibration data. How do we proceed? You have to define a nonconformity score as a level of disagreement between the X and Ys. Then you can compute those S-scores on the calibration set, and you have to find a score value at a specific quantile, q hat, which is related to the confidence level that we set at the beginning. You learn this q hat on the calibration set, and then you can use it during inference to form your prediction sets. And the beauty of these prediction sets is that now you have the mathematical guarantee that the probability of a new unseen point belonging to the conformal set is related to the confidence interval that you set. And with this in hand we can conformalize all the predictors. But let's see how it works in practice. Let's imagine that we have a model that is assigning scores in classifying images. So what you do basically is sorting the nonconformity scores which are evaluated like that with the inch loss from certain to uncertain and essentially you evaluate q hat which essentially discerns certain enough for conformal instances from the uncertain ones so with the learned q hat you can now move on to inference and evaluate all the nonconformity scores for a new unseen image. And essentially what you see is that there are possible results that can make to the threshold to the Q hat which is controlled by alpha. And essentially what we are saying is that for this particular case, the prediction set for this image is either a cat or a lion. So, if you move alpha, you change the Q hat quantile and you essentially can control the error rate. These are just some Python packages that you can use to perform some conformal prediction. I'm not necessarily saying that those are the best ones, but just to give you some examples. For example, here you can see how you can apply MAPI to perform a classification task. Essentially, what you do is wrap your pre-trained model around the MAPI classifier class. You evaluate nonconformity scores, set an error rate, and perform inference on unseen data. Similarly, you can use scraps for regression tasks, again you calibrate your nonconformity scores, you set an error level, and again you perform inference, getting back prediction intervals. Let's now move to the concept of having a human in the loop. Well, in general it's very nice to have some mathematical guarantees, but the thing is How can we deal with set-valued predictions when we have to make some decisions? Well, I hope to convince you that prediction sets are perfect when you have human-in-the-loop pipelines. And these pipelines can have a human which is fully in the loop or partially in the loop. And just as an example of fully-in-the-loop pipeline, you can think of a doctor that is trying to diagnose a patient with breathing difficulty. So the doctor might order a chest X-ray, and the prediction sets, the conformal sets, narrow the focus of investigation and potentially speed up decision-making of the doctor, because a set, after all, is more useful than a point prediction, especially if multiple conditions are present for a patient. While, as an example of partial in-the-loop pipelines, you can think of mortgage approval, for example. Simpler instances can be fully automated with little risk, while trickier ones are better left to humans. And through the lens of conformal prediction, with the conformal guarantee, you can tune the acceptable error rate. And actually, there's some research showing that it might actually be beneficial to provide conformal sets to humans. Some researchers actually designed three independent tasks to represent a variety of real-world settings, where in general, human decision-makers might benefit from model assistance. Those are image classification, sentiment analysis, and named entity recognition. And they actually proved in a statistically rigorous way that the human performance, measured in terms of accuracy and response time, is actually boosted by conformal sets as compared to no conformal guarantees, which is the control, for example. So, compared to the control, the Top-K approach provides just a fixed number of likely alternatives to the human, while the conformal sets, in this case, also quantify the uncertainty. And so they actually prove that humans do leverage this uncertainty quantification information. And I hope that you agree with me that we do really need an ethic for AI in general, which emphasizes the role and the centrality of anthropos, of the human. And that we have to direct any technological effort for the good of people and society. And as a famous German said before me, having the human as an end and never just as a means. So we have to make sure that the values and goals that are entrusted to the machine are actually aligned with those of the human. And that's the importance, the concept of having a human in the loop. But these values in general, and this is a subtlety, are not absolute and universal, but they actually depend on the space and time, in the sense that they depend on the reference society and on the historical contingency. Let's now move to some further applications of conformal predictions, and I want to start with time series, because, as I told you at the beginning, we need exchangeability between the calibration set and the test set so you might wonder can we still apply this framework to time series when where essentially we have a preferred direction of flow for the time time series in general can be non-stationary so in general you don't really expect the exchangeability condition to hold? Well, the answer to this question is yes, we can still use conformal prediction provided that we use the appropriate extensions for that. And I want to mention some of the algorithms extending the assumptions of conformal prediction and I want to start with MBPI which is historically the first algorithm extending conformal prediction with time series specifically in mind. So essentially, in order to use MBPI, you have to assume this data generating process. So f is an unknown relation between the x and y's, and we have to assume that the noise term is identically distributed. And on top of that, you perform some bootstrap ensemble predictions. And what you get, and this is actually proven, are the mathematical guarantees that are still holding infinite samples. Another powerful extension is the adaptive conformal inference, which makes use of this quite simple online update of the significance level, of the confidence level that you set. So it gets updated from time step to time step and this is meant precisely for online and real-time settings. The only drawback is that the coverage guarantee here is asymptotic. It means that you get the mathematical guarantee provided that you wait enough time for the algorithm to adjust for different level of uncertainty. And last but not least, I want to mention conformalized quantile regression, which is the quantile regression that we know, an old statistical method, in its conformalized version. And essentially, this has been proven quite recently, that if you add a coverage penalty, that eta term, you can still use it for non-stationary settings, for time series in general. And you get non-asymptotic guarantees for a lot of non-exchangeable processes. And here's an example of time series. So what you see is the half-hour electricity demand in Victoria, in Australia, and what I did is introduce an abrupt change point at some point during the, within the test sets and I changed the y-axis by two gigawatts so a quite important effect. Well as you can see the training data does not contain such a change point so the model which here is a very simple random forest regressor cannot really anticipate that change point and if you do not update the residual the non conformity scores, the prediction intervals of the MBPI method, which are built upon distribution of the residuals of the training set, do not cover the true observation. They essentially continue straight and you get a coverage decrease after the change point. But if you keep updating the non-conformity scores in an online fashion, you can see that you can introduce an element of adaptivity in the bottom panel and essentially when you update the conformity scores, you now allow the method to capture the increase of uncertainty. You can also think of conformal prediction in financial settings where usually you can employ value risk or expected shortfall as metric for risk management across financial institutions where it's paramount to have accurate risk estimation and essentially this amounts to quantify in a robust way the quantiles of the so-called profit and loss distribution and here conformal prediction can definitely help us and there's some research showing that you can get sound risk management if you use for example adaptive conformal inference and this is very helpful especially within highly volatile markets where you have condition changing widely from hour to hour and this can allow firms to have an efficient liquidity edging or a good compliance with regulatory frameworks in general. And last but not least I want to discuss what we are doing at the National Institute of Geophysics and Volcanology where we essentially deal with space weather. Space weather, I don't know if you are familiar with that, but it's It's essentially the study of the physical and phenomenological states of the natural space environment, including the sun, which usually ejects charged particles through the solar wind, which travels the interplanetary medium and reaches the planetary environments like the Earth environment. And it's very important to study space weather and its effects because space weather can affect the GPS positioning, it can induce radio blackouts because it affects radio communications and a lot of other effects. Here's an example of what we are doing at INGV, which is trying to forecast GNSS amplitude scintillation, which essentially means that based on the ionosphere condition and the space weather condition you can also get a loss of luck with GNSS receivers and that's very impactful when it comes to GPS precision positioning and so on. Here we used a simple CatBus regressor wrapped around a conformalized quantum regression approach and in general at INGV we studied the ionized upper parts of the atmosphere the so-called ionosphere and the effects that space weather and especially the one that comes from the sun forcing induced on the ionosphere so we developed some predictive models and a lot of them are based now on machine learning approaches and we do so with real-time operations in mind and we tend to like conformer prediction in general because it really allows us when we have properly calibrated probabilities or quantities in general to reason in terms of user defined risks when you want to reason in terms of cost-sensitive decisions made by the users. And that's pretty much what I have for today, and I'm open for questions.

Speaker 2 [23:12]

Thank you so much Vincenzo. I think human in the loop an interesting topic especially with the EU AI Act and responsible AI usage framework. So I've got a couple of questions in the slide. So the first one. So what are the advantages of calculating prediction intervals by conformal prediction versus bootstrapping?

Speaker 1 [23:33]

Okay, the main advantage I would say is that you get mathematical guarantees of coverage, which is something new that the other methods do not really have. So when you get the prediction intervals from methods like vision approaches or bootstrap, Essentially, you don't really know if you aim, let's say, for the 95% confidence level, you are not assured mathematically that that interval really represents the 95% confidence level. While you are assured by conformal prediction, provided that you respect some very basic assumptions. And the same also works for vision approaches, for example, which work perfectly in principle, provided that you know the exact data generating process, which is almost never the case. I hope I answered.

Speaker 2 [24:48]

Yeah, hope that answers that question.

Speaker 1 [24:48]

Yeah.

Speaker 2 [24:50]

And the second one, if I understand correctly, conformal prediction depends a lot on sensible uncertainty measurement of a given input-output combination. So is this the challenge?

Speaker 1 [25:03]

I don't know if they mean the dependence to, let's see, to the non-conformity score, I guess. I don't know if there's someone in the room that wants to add some more comments on that. I think you mean the dependence on the heuristic notion of uncertainty. to a certain extent, yes, it's a crucial part of having a conformal prediction algorithm working. And yes, the results, well, I would say that the coverage guarantee, it's there, you are guaranteed. But if you select some S of X and Y's, some non-conformity scores instead of others you might get some better adaptivity for your problem so it depends from this respect but in general the mathematical guarantee is always there

Speaker 2 [26:10]

Thank you. The next question is the uncertainty quantification aggregating aleatoric and epistemic uncertainty. Is there any way to separate out these two or estimate just one of them?

Speaker 1 [26:24]

That's a very good question. And yes, it aggregates both kinds of uncertainties. And to be honest, I'm not aware of any method yet to discern and separate the two kinds of uncertainties, unfortunately.

Speaker 2 [26:41]

I've got last two questions. So how do you determine alpha?

Speaker 1 [26:46]

No, it's you don't have to determine alpha you have to set it so if you are facing a business challenge and you want to Set an empirical risk of five percent You set that alpha which is zero point zero five and you get the confidence level of zero point ninety five So this can be related to the business decision making making it's the business that can tell you this is our exact acceptable error rate the error rate which is deemed acceptable for our costs so you said it and you have the mathematical guarantees that the prediction interval the prediction set actually respect your user requirements

Speaker 2 [27:41]

And the last question, how would you extend conformal predictions to sequence problems like next token prediction in LLMs?

Speaker 1 [27:51]

To be honest, I don't really know about that. I'm not much into NLP, unfortunately. But for sure you can find some... This is a very good review, the last one. Well, in general, this is kind of the Bible right now for conformal prediction. And I'm quite sure that you can find the answer there. and also the basic theory is explained and a lot of things are there so go and check it out

Speaker 2 [28:29]

So thank you so much, and also thank you for patiently answering all those questions. I think we have a question here. Sorry, I forgot my credentials on Slido, so I have to do that again. For the alpha recognition, I think my first question was already asked, that you have to intuitively...

Speaker 1 [28:51]

Intuitively.

Speaker 2 [28:52]

get this alpha parameter but do you consider any bias that you need to take care of like for example this comes from biogen statistics that one needs to be experienced to in order to get a precise alpha prediction but if you're doing it for the first time and if you have no priori

Speaker 1 [29:13]

No priori.

Speaker 2 [29:14]

In your experiment, what do you consider, how do you really...

Speaker 1 [29:18]

How do you really

Speaker 2 [29:20]

account for bias in this case.

Speaker 1 [29:24]

I don't think that you have to reason in terms of bias, because it seems like you are thinking from a Bayesian perspective. And I will just rely and tie that alpha level on the monetary costs that businesses are facing, if they get an error or not. So the prior, so to say, is just the business knowledge. and that's the way I would interpret that, no more than that really

Speaker 2 [30:00]

Thank you so much. Thank you. Please give a round of applause to Vincent.

Speaker 1 [30:04]

Thank you very much.