The Day the Agent Started Lying (Politely)
LLM agents often suffer from silent degradation, where the model continues to provide confident responses despite a shift in the underlying data distribution. This problem is particularly acute in non-deterministic systems where ground truth labels are unavailable for real-time validation, rendering traditional accuracy metrics useless. In a customer support ticket routing scenario, for example, the launch of a new product can introduce new vocabulary and shift the meaning of existing terms, leading the agent to misclassify ticket priority while reporting zero system errors.
To detect this drift without relying on manual labeling, a multi-signal evaluation framework is used. This approach monitors six distinct metrics: Shannon entropy (calculated using all available class probabilities to measure internal uncertainty), fallback rates (acting as a canary signal), vocabulary drift (measuring the distance of current word vectors from a stable centroid), human disagreement (tracking when users override agent decisions), LLM-as-a-judge (using a model like Claude Haiku to verify if the agent's reasoning is faithful to the input), and trajectory (assessing the logical flow from input to outcome).
The key takeaway is the implementation of a tiered action plan based on the number of triggering signals. A single alert, such as an increased fallback rate, suggests observation. Two or more signals indicate a need for investigation. When multiple signals across different layers—internal confidence, external human feedback, and logical verification—trigger simultaneously, it provides a high-confidence indicator that the agent's prompt or model requires updating. This "nervous system" approach allows operators to identify and fix silent failures before they impact business operations.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track MLOps & DevOps and was classified suitable for intermediate domain / novice python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
In this talk, we will walk through a concrete production-style example of an LLM-based agent that automatically classifies and routes incoming customer support tickets. The agent takes raw ticket text as input, predicts a priority label, and routes the ticket to the appropriate support queue. A human override is possible but expected to be rare.
At deployment time, the system performs well. Classification confidence is high, fallback usage is low, and manual corrections are infrequent. Over time, however, the environment changes: new products are launched, outages introduce new failure modes, terminology evolves, and internal definitions of ticket priorities shift. Nothing crashes, latency remains stable, and traditional service-level metrics stay green; yet the agent’s decisions slowly degrade.
This talk focuses on how to observe, measure, and act on that degradation.
Using recorded ticket data and a demo, I will show how to instrument an LLM-based agent with continuous evaluation signals, including:
- Tracking class-probability entropy over time to detect increasing uncertainty
- Monitoring the rate of “unknown” or fallback predictions as an early warning signal
- Measuring embedding distribution drift between historical and recent tickets
- Quantifying disagreement between current agent decisions and historical routing outcomes or human corrections
I will demonstrate how these signals can be computed in rolling time windows, visualised on simple dashboards, and connected to alert thresholds. Rather than relying on a single accuracy number, the talk shows how multiple weak signals together reveal silent failure modes that would otherwise go unnoticed.
The focus is deliberately not on training new models or tuning prompts. Instead, we concentrate on operating LLM-based agents safely after deployment. You will see how to build a continuous evaluation pipeline, how to distinguish normal variation from meaningful drift, and how to decide when intervention is required whether that means retraining, prompt changes, label redefinition, or temporary rollback to human routing.
By the end of the talk, attendees will have a clear, practical blueprint for monitoring LLM-based agents in production and for detecting quiet, confident failure modes before they affect users or business operations.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:00]
Welcome to the last session of the second day of the conference. How is everybody doing so far? Okay, thank you. I hope you have some energy left for the next session, which has a very interesting title I wonder why wouldn't ancient lie to me, but we will find out please give a warm welcome for Asher Melnick And if you have any questions, please ask them through the app. Yes
Speaker 2 [00:32]
Well, yeah. So, let me start with a quick question. So, how many of you recently, or not very recently, just deployed an LLM agent? Could you just raise your hands, please? Uh-huh. Uh-huh. Uh-huh. Uh-huh. Okay. Okay. Okay. So, and now could you please raise your hand if you're still completely sure what your agent is doing and it's doing completely what it's intended to do? No one? Ah, one. Oh, okay. One, two. Okay. There's three. Okay. I want to talk with you later. But this is significant difference. So this is actually this gap between first group of people and second group of people is actually what I'm going to talk today. So how to actually catch when your LLM agent is degrading and starting to lying to you. but still it will sound very confident. But if we will be able to move our... It's not moving. Help! We need someone. Does anyone know how to move it? Because it's not clickable? Oh, sorry, it's not just, oh, it's off, oh, my gosh, probably need PhD to do this, yeah, guys, yeah, so let me, no, it's not working either, no, it's not working, I'm sorry, it's, guys, might be 2,000 years later, you'll know what's going on there. Do you have another USB-A slot on your Mac? you you you Thank you. Thank you. won't be able to show to go to another screen so this is exactly isn't like it's memory mirroring the screen So, it went to the, it went to the shore, okay. so if I try to go to another screen, so it's not visible in there, so this is not what I intend to do, but I, like, it's not... Okay, at least we, but to another screen we can, I mean, this, it does work, and I can click, ooh, yes, wow, thanks to the guy. Sorry, it's not the most common thing to do here. I will try to speedrun as much as possible. Now. Yeah. So if you want to stay in touch with me, this is my LinkedIn. So in short, I work for Blue Yonder. I'm not going to tell the story how I end up there. But in Blue Yonder, currently what I'm doing, I'm doing AI joint development. are working in AI agent evaluation, and I should written here also AI mentor and educating teams, but it would be too much AI agent words for one slide. So let's just look, before starting with the solution, we need to first understand what actually is wrong, what's going on. So let's say you have some traditional solution, agent, whatever you call it. It could be ML. It could be whatever. It's just some sort of algorithm model that just does something. It runs a prediction, and you know the ground truth in the end. So you actually can label it. And if you know a ground truth, you can label it. So that means that you can actually calculate some metrics, and if they look wrong or something is going bad, you can just improve your model and just repeat the cycle. So this is just a cycle. It's a loop. Works fine. Great. Unfortunately, it works only for deterministic systems. So if we talk about LLM agents, so what's happening? So you have your agent, same as before, right? it does something, right? It gives you a response, but I mean, what's right? I mean, what's actually the ground truth? I mean, there is no ground truth unless you actually really sit down and define it somehow. But it might be too strict or something. Or you just see the person that actually clicks right, wrong, right, wrong, right, wrong, which actually might be, for our case, happening like in two, three times per cent, but if you do it like for 100 per cent, it's not automation any more. So if we don't have actually grand truth, so how can we calculate metrics? So this loop is broken, and you have no idea that your agent is doing what it's supposed to do. Yeah. But, yeah, this is one of the most important things, lines that I want you to take from this talk. So if you can't improve what you You cannot measure, and you cannot measure if something has no label, so if you don't know what to measure. So here is the concept of my talk. So we have an LLA agent that reads customer support tickets, assigning priority, like it's high priority, super high priority, medium, or low, and assigns all tickets to some team that should solve them. So everything should happen automatically on scale, and humans shouldn't actually be involved, but could be. Two things, two disclaimers, not disclaimers, two things I want to say before we actually proceed later on. So first, all tickets that you'll see here are actually synthetic. So those are generated. I couldn't bring real customer support tickets here for obvious reasons. But failure patterns are actually real. So I just synthesized the data so that they will copy the pattern that I observed in reality. And the second one, I will mainly explain what's going on, just show you some plots and explain what they mean, and so on, so on, so on. But all the code, and if you want just to look at this demo toy setup, will be available in the GitHub, and the link will be in the very, very last page of this. Please don't go there right now. Otherwise, I will lose you completely. So, what's going on here? You have your agent. It's deployed. Here you have your timeline. Let's say in our setup, right? We deployed in January. Okay. So, you see this nice number, like, 100% accuracy, which actually means that just only you managed to, I mean, your agent, not you, managed to actually to give a priority to the ticket and send it somewhere. Yeah. And zero errors in your logs. Yeah, because you actually don't really know what's track. So, everything looks actually really cool, and this, like, numbers what you probably want to show to your manager. Yeah. But there is something happening in June, month six. So, what's happening? Actually, company launch new product. In these terms, like, for example, purpose, we will talk about authentication. So, new authentication method. So, it's launched. Okay. So, actually, some new words are coming. Some might be changes are coming. And your AI agent is actually not aware of what's going on. So, some strange signals come from outside, but your agent doesn't know what to do with this. So, it's continue doing what it's used to do, like just giving priorities, routing, and actually showing the same numbers to you. Until the one moment when you just see that your engineers actually come in to you, like ops engineers, and say, like, okay, something is wrong, like, for the last five months, we got 113 wrong tickets to us, like they were marked as urgent while we're not. What's going on? So, nothing crashed. So, everything was green for five months. No alerts. You just missed it. Here is an example of the ticket so you can understand what's going on and how it's working. So, ticket. This is still unresolved. Log in with Microsoft. Broke after our Azure AD tenant was renamed. Do we update something here? Our SLA requires resolution within four hours. Wow. That's really strong. So what your agent will think, actually, it says, like, okay, okay, okay, I see login, I see Microsoft, this is what I was trying on, this is what in my prompt, I see SLA, maybe urgency, okay, fine, fine, fine, so this is like actually emergency so wrote it to authentication team as emergency. But truth, actually, this is not it. Truth that this is actually medium. This is not how. This is related actually to the change that company did. So this is a new authentication method that launched in just configuration question which just requires 15 minutes fix. So here might be some numbers what you might see or would see. So confidence is actually below the threshold, so actually it will not trigger anything. So fallback will be not triggered for this case. And this is your agent reasoning. So it's treating as a high priority of emergency. Why this happening? Why this actually... While if you would measure entropy, you already you could spot that something is completely wrong going on. Your agent has no idea what it's actually doing and what it's reasoning about. So, let's see what's going on here. First thing. So, this is, again, our timeline. Here. So, this is a stable phase. Here, this new authentication method came, your product, right? So, this is a transition state, and this is our drift, where we see everything is significant. But let's just dive in a little bit and understand from the agent perspective how it looks like, what's changed. Actually, four things changed. First is a new product launch. So, new vocabulary is coming to your agent and your agent is not aware about this vocabulary, so it actually doesn't know where to route it, what to do with this. There is no actually description. Yeah, it never saw it. So, second, it's like new failure models. So, now, difference between urgency, like act with urgency and something like normal is different. So before this launch, let's say, let's go back to our case, example case, so before if something really bad happened, it would be saying something like, okay, we have, like, we cannot log in, for example, our password is failing, whatever. For now, it's completely different model how failure will look like for the model. So third one is terminology shift by itself. Before our change, login, just word itself, it actually means something different. Before this modification change, login meant that something is broken. User cannot actually use our solution. User actually cannot reach our solution. So something is completely bad, and we need to fix it right now. After this launch, actually, logging changed completely different, the meaning. So now, logging, it's just completely different word. It doesn't mean that customer cannot log in to our system, that it cannot reach it, because other words they're using. So it's not anymore the right priority, and meanings of the words is different. So now the first, the last thing is priority label shift. So we remember that we have new wording, that some of the words actually change the meaning, and we have different failure models, right? And we see that from our prompts, actually, like how we prompted this agent. So if we see login, so usually it should be high, but login and authentication, it should be medium, but it will be labelled as high, because we didn't change anything in our prompting. So yeah, it was coming for quite a long time, completely slipped, no logs, nothing, so just silently coming, and the model is completely confused, so your agent is confused, but on the surface, everything looks perfect as it does what it's supposed to do. So here's a little bit of the architecture that we're going to take all about. So we have some tickets, tickets going to some LLM classifier, LLM classifier defines what the urgency and which team it belongs to, then it might fall back occasionally, and then we just go to support queue, and here, this page, at some point, occasionally human can override it, whether it's completely wrong or not completely wrong, and so on. So at every step in this architecture, we go into log, what agent is doing. So and that will actually evaluation process possible for us. So if you look at the code later on, so this is a decision DB, this is where you can find actually everything we write. So, predicted priority class probabilities, confidence entropy, and so on. Not going to read it all through. So, now about signals. What we're actually measuring. Our metrics and signals. Yeah. Might be look scary because a lot of text, a lot of something. Yeah. But we will go a little bit deeper into all of this. So this is in principle six matrices that we are taking in, six evaluation signals. So like Shannon entropy, fallback rate, vocabulary drift, disagreement. Disagreement is the only one where a human is actually involved. Like judge score and trajectory, if you want to review this later on. So, this is actually what takes it as input and, like, formulas below how it's calculated so that you can easier understand the code that you might want to see. And here is actually how to interpret it. So, like, for example, for entropy, this is, like, higher is entropy, obviously. So, it's more confused it. Just important note here for entropy. And this is probably the most important thing in this talk. So please, if you just forget everything, if you're already asleep or whatever, please hear me out only for this one thing. Please, for Shannon entropy, just use all available probabilities that you have. Let's say for our case, we have like four different classes of tickets, like extreme Extreme urgency, high, mid, and low, right? So this is like four. If we just look at this formula, so then it will be just we have like the highest entropy will be like two, but actually it returns not one number. It returns probability for each of these categories. So please rather use this one, because if you can look only on the lowest number for one of the categories, it might actually fail you. And you might skip drift. So, okay. Later on, we will talk about those, like, in pairs. But this is pretty practical plot, what it shows you in our timeline. So, when we go from the stable running zone, running time, actually introducing our new authentication method, as for our example, when our metrics that I just showed you roughly will actually be triggered. When they will detect that something is wrong, something is wrong, something is going on. Yeah, here you can see that actually fallback rate, it's like it's so-called a canary signal, so it shouts first when something is wrong, so that means that but from specific of this metrics, it's actually kind of a shows point that something is wrong, but it cannot say what exactly is wrong. So S1 and S2, like entropy and fallback, those are uncertainty signals. Both measure just internal confidence, and neither of them needs a ground truth, so this is the point of all of this talk. So here you can see how it actually changes with a drift, how it detects it, how slowly it's going. So you see that in a stable time, it's actually, I mean, it's pretty good. There is no shift. So you remember, lower is better. And at the area when actually all customers are talking in terms of new product, it's like has no clue. I mean, agent doesn't have, like, has no clue what's going on. So it's actually kind of not even guessing. Yeah. So for these two, like, vocabular drift, this is a distance from a stable centroid. So it's actually calculated, like, window, usually a week, and batches of all tickets that are actually building the vector of words that are used within this week. And then it's shifting and shifting and just measuring, like, which of these words shifted to comparing with some stable error that we defined as a stable error. So disagreement, as I already mentioned, is the only one human written. So external signal. Might be one thing to say here is actually for stakeholders might be one of the most important. Why? doesn't require knowledge of any statistics, so what it actually does if a person is looking at the ticket decision, let's say, it overrides it, but for disagreement, it actually allows you to, for the person who is overriding this ticket, to specify is it completely wrong or is it just adjustment, and if it's completely wrong, well, I mean, this is a pretty strong signal that your person is saying that something is completely wrong. Yeah, and you can see that for our case, at the end, we had, like, one, half of the people, one, two people said that this agent was wrong. Yeah. This is raising signals. First one, like, S5 is LLM as a judge. I used Claude Heiko for this. It just looks on the cached reasoning that we're recording and verifies it with actual tickets to see was it actually faithful or not. And trajectory is actually kind of comes together with the lemmas judge because it shows how How logical is our outcome? So this is, again, you can stop me here and say, okay, I see why you're talking about all of these numbers, why you're talking about all of these metrics, and blah, blah, blah. So what should we do? So here's your action points plot. So you can see how all of these signals are alerted, like signals means metrics. So if you have only one signal that is saying that, okay, something is wrong, it probably will be a fallback, because it shouts the first. You can just watch, maybe observe what's going on. So if two, okay, investigate what's going on, very likely something is broken, something is wrong, might be you need to, yeah, S3, S4, whatever, yeah, just might be you need to roll back or completely fix your prompt or, yeah, something is going completely wrong. So this is live demo, and I really, really hope that it's going to work. This is the dashboard. I mean, if you copy the code, you will get it as well. Yeah. So let me go, like, very roughly, I'm skipping this part. So this is actually an example, I'm not sure how visible it is. I'll try to make it maybe a little bit bigger. So this is how our model, like, AI agent is actually processing the tickets. So here we can see, like, all of these metrics and tickets and reasoning in the same page. So this is table error. And we have the first ticket that's coming to us. So ticket states. Our service account password was rotated by IT, and now all automated processes are locked out. Wow. That sounds serious. What agent says, well, it understands it, yes, it's definitely blocking, so it's rotated as high, and to the right people, okay, this is good. Truth, so truth is actually high, and so this is actually correct. Our confidence, as you can see, it's actually also really good, entropy is fine, perfect. What will judge set? So let's look at our reasoning. So in here. So account assess problem affecting user productivity. Needs immediate resolution. So judge looks at this reasoning, looks at the ticket, okay, sounds good, yeah. Trajectory, if this reasoning was actually pretty logical to the ticket itself, and yeah, looked fine. So everything is good. Now we're going to the unstable error, where we already have the drift in all our wording and so on. So another ticket which is actually indicating something, yeah, can't log in since SEO migration. So getting invalid grant after entering credentials, IT says it worked last week. Okay. So if not tuned, so we have the same agent, right? It says, wow, it's high, yeah, because, I mean, person cannot log in, something is really wrong going on. But truth, it's actually not, it's medium. So because it's just like change in a setup. So confidence, confidence dropped significantly, but still there is something. But if you look at entropy, it's just like nothing. Yeah. Judge, you can also see it's almost zero. So it just immediately catches and see that actually your agent does something completely wrong. So yeah, like all of this matters. let me go back to the presentation. So, Blueprint, if you go to the code, this is everything you need to know. Again, this is what we log, like text, class probabilities, confidence, reasoning, human override, so this is what we log. From all of these logs, this is what we can what metrics we can apply, what signals we can track. And here you can see your action plans. So if one signal just triggers and just might be don't start panic. And this is actually by GPT. But I really like this phrase. So you don't need a single accuracy number. You need a nervous system. I really like this. Yeah. So just accuracy doesn't really shows that your agent does what it's supposed to do, I guess. So, just use more metrics even if you don't have ground truth and you cannot define it by design. And this is the link to the code that I just mentioned multiple times. It has really nice vibe coded comments, so it's self-explainable. Yeah, here just description might be where you can find what. Yeah, I guess that's it. Thank you.
Speaker 1 [32:04]
Yes, thank you very much, and I'm very sorry for the beginning, but you did very very well We have a lots of questions, so I hope fear We cannot answer every question and the first question is how do you establish? Relative thresholds for alerting coming from a stable word pre-deployment
Speaker 2 [32:24]
This is actually all coming from these numbers, so you can get it from these formulas. And we know, for example, for this case, we know what is stable error. So we see the drift actually after new wording are coming, let's say, to us, right? So we can just divide it, and, yeah, after that, just simply, I mean, there are actually thresholds that suggest it to do, like, for example, like for entropy, this is like around 0.67, so this is like a stable. If it's like high number, again, number, it depends on how many classes you have. So there are actually suggested versions, like thresholds which to use, but you have to apply looking at your data, at your tickets, or not tickets, whatever you use. So just try to identify which is stable error, which is transition, and which is complete
Speaker 1 [33:33]
Okay, thank you. The next question, will this evaluation signal work for a multi-agent workflow rather than one agent or classifier?
Speaker 2 [33:41]
Ah, that's actually a tricky one. So the main thing is, maybe I'll go here, lock whatever you have at any stage. Otherwise, yes, you can do it straight away. But if something is actually triggering and saying, OK, there is something wrong, if you build it on top, how would you actually understand where exactly it broke? So you need to look at every step.
Speaker 1 [34:15]
And then one question, is it really so different than ML? Data drift exists there as well.
Speaker 2 [34:21]
Sorry, sorry?
Speaker 1 [34:22]
Is this really so different than machine language? I'm actually learning sorry Data drifts exist as well. What's the question?
Speaker 2 [34:31]
Sorry, I'm a little bit confused.
Speaker 1 [34:34]
Yes, we'd like it like it was written. So the next, just a moment, please. Next question is, is the judge classifying every ticket? This makes your cost double, or only for a testing evaluation set?
Speaker 2 [34:49]
No, actually every ticket, yeah, so if you log, I mean, for this case, actually for this demo, I mean, we don't have, like, a big agent that does something, like, properly, so we use, like, just cached this, like, reasoning, so, yeah, for each ticket.
Speaker 1 [35:07]
Maybe the last question. How do you get the model confidence of model or judge here? Getting a good confidence value from LMMs is usually difficult.
Speaker 2 [35:18]
Well, it is difficult, yes, I mean, what can I say?
Speaker 1 [35:25]
Maybe then one more question. I have an alert problem. I have an alert equals problem. What do I do now? How can I use the six metrics for derive actions to get the LMMs to stop lying again?
Speaker 2 [35:38]
Well, this is exactly the recipe that I already gave you. So you have, like, multiple metrics. I mean, don't act when something, like here, for example, this recipe. So how many signals you have, like how many metrics is alerting that something is going on. So then just act with urgency. See if it's only one or two, might be just check which of those, might be just something, miscalculating something, yeah. So there is no, you don't need to jump immediately and shout fire, fire. Nothing is working, yeah. But if several, because you measure it from different point of views, you measure it from external point of view, you measure it also from internal point of view, and a little bit deep dive into what's going on underneath this agent. So I guess if you have like different agents, different metrics from these different layers actually saying that something is wrong. So, I mean, this is a very solid sign that you need to change your agent. Is it answering? I don't know. Oh, yeah, I mean, you can do. I mean, this is the purpose of all of this, right? Because it actually can indicate where to search, what is failing, at which stage.
Speaker 1 [37:09]
Okay, I think we have to stop here because the lighting talks already have started. Thank you very, very much for a very great presentation. And please give a big applause.
