Generative-AI: Usecase-Specific Evaluation of LLM-powered Applications

Speaker 1 [00:08]

Thank you everyone and a very warm welcome from my side as well in this final hour of the first day. I cannot be more than appreciative. Before starting, I would like to also acknowledge the collaborations of my colleagues Patrice Valera, Ekaterina Weindel and Alex Burres. Let's delve right in. Let's see how I'm going to spend the next 25 minutes together with you. Here is an overview i would first start from some introduction to some lingos some basic concepts around evaluation of general applications then i would talk and work you walk you through a workflow that we developed for evaluation of the of the general applications and then i would walk you through some some use cases of how to consume these evals in products as well and conclude with some takeaways. Let's just jump right in with introduction of some lingoes. First of all, when we are talking about LLM-powered application or agents, what are we talking about? In order to give you an intro, let's focus on a very simple use case, single-purpose agent. We drag retrieval augmented generation. It looks something like this, that the user would send a user query, and then you have the agent that gets triggered. This agent normally has a router, which is basically a planner on how to use the agent's skills and a set of skills. In this specific case, in this very simple example, the router first classifies the query, then if it's applicable, it actually searches the knowledge base, and then it goes through search of the chat memory that the user might have together with the agent, and finally generates a response, and outputs that response. And this can be an interactive back and forth between the agent and the user. So a typical agent structure would be this router, set of skills and memory and state, which basically is the shared information for different skill sets of the agent. Now, even in this very simple example, what you might notice is that these agents are very complex to evaluate. there are simply way too many options and design choices that can be made. In this single example, there is the choice of different LLMs. There are the hyper-parameters of LLMs. There are the prompts to LLM calls. There's the agent design itself, and there are the different tools, whether it makes sense to add to or not. And there's a choice of embedding models specifically for RAC. So on and so forth, many design choices, how do we want to go about it? These are highly non-deterministic systems, And there is compounding error coming with every different component that we are using, which means the testing will be very complex as well, and it would necessitate qualitative and open-ended metrics for evaluation. Now, for delving forward into the evaluation, I would like to also distinguish between two different topics here, LLM evaluation versus agent evaluation. Because if we are talking about the evaluation of LLMs themselves, this is actually available already. So the open and natural language processing community, they have actually started looking into evaluation of LLMs with certain benchmarks and with respect to general skills. This has led to many different leaderboards, so to say, these leaderboards are listing the performance of different LLMs from a specific perspective and with different benchmarks. Examples are artificial analysis, hogging-face leaderboards, C-leaderboards, etc., so you can easily look it up. However, they are tied to some benchmarks which are basically evaluating the language model with respect to generic tasks such as language understanding, coding, and math. When we are talking about agent evaluation, these LLM evaluations do not suffice. Why is this the case? When we want to evaluate the agents, we are measuring the performance of the agent with respect to its intended use. And there are two limitations of the LLM evaluation. Firstly, the LLMs are not the only component in the agent to be evaluated, and secondly, the requirements of the use case are normally miles away from the benchmarks and the evaluation criteria of the LLMs, which raises the question, how do we want to go about evaluation of our agents? So there are different dimensions to take a look into, and it's important to decompose the different dimensions. As data scientists and AI engineers, we are normally looking into the quality of the outputs, but that is not the only dimension to look at. There is the price and speed of the LLM agents as well, which become much more important when we are talking about or when we are putting the agents into production. Now if we stay with these three basic dimensions, price, speed, and quality, for the price and speed we have different proxy metrics that eases the evaluation. For price, we can take a look at, for example, the average token consumption of the agent. For speed, we can take a look at the end-to-end latency of the agent. But when it comes to the quality, this is where our trouble starts. Because basically, we would need to see what is the requirement of our use case. And most of my talk would be around this quality and the use case specificity of this quality. Now, I would also like to tell you what is the end goal here. So our real end goal is to create, similar to LLM leaderboards, to create agent leaderboards, meaning in our development and release cycles, I would like to actually collect information about the performance of my agents with respect to these different dimensions and different criteria that I talked about, such that we can make more informed decision about our designs and we can actually direct our effort in agent development in the right direction. Now, let's talk about that quality that I was talking about. What we want to do is to basically evaluate the text outputs that are coming out of the agents. And for this specific task, there are different, really a plethora of different metrics that you can use. I will categorize the metrics into two categories. One is code-based, rule-based, and classic NLP evals. are short for evaluations. But here these metrics have limitations in capturing human judgment really, when it comes to free-form text specifically. There's a different regime or rationale in creating these metrics and that's by leveraging LLMs themselves to judge the output of the, judge the text that is outputted by agents. And here again the idea is that we would use an LLM, we would instruct it to take a look at the output of an agent from a specific criteria that we define as the metric, and the LLM agent would provide a score. And here again, there are openly available repositories, and there are different tools that one can use. For example, DeepEval is one of them, but really there are many different openly available libraries that one can actually leverage from so that's to set up this stage now let's see if we want to perform use cases specific evaluation what are the two basic components that we need starting from the agent that I was talking about we have basically the user request agent and the generated response what we want to do is to evaluate the generated response the first component that we need are these evaluating agents which are eval tools mostly leveraging LLM as judge and they are basically assessing the quality of the output of the agent aside from these eval tools you would also need to define and collect or better curate use case specific benchmark data sets so similar to the leaderboards that we have for the LLMs and the benchmark data sets that are created for the for evaluation of the LLMs. Now for the evaluation of our agents, we would need to also curate such benchmark data sets, but these needs to be specific to our use case and need to encompass the different requirements that our use case has, the different failure cases, etc. So basically all the different cases that we would like to test our agents against. Now once we have this use case specific benchmark data set, then it becomes actually quite straightforward how to go about the evaluation. We would just send the queries from this benchmark data set to different agents that we are designing and we are collecting the output from the agents and sending them into the eval tools. Out of the eval tools come the different quality metrics that we wanted to have in order to have an aggregated quality metric of our different agent versions and we have cost and speed as two other criteria as well. So that would provide me with the the agent leaderboard that I was talking about. Now, what I would like to share with you is the workflow of how to create and how to develop these two basic components. Really here, I would remain tech stack agnostic, I would remain agnostic to any use case, etc, with the purpose to provide you with the lessons learned that we had when we were creating these evaluation tools. Let's go forward. So this workflow starts from the curation of the benchmark data set, and here is where we are performing human evaluation, actually. And the second step is we would continue with creating of our eval tools, mostly with LLM as judge, as our evaluator. And the third step is something that is normally overlooked, but is actually quite a necessity if we want to perform reliable evaluation, or as reliable as it can get. And that's the supervised alignments of the eval tool with the human evaluations. And this step is for me the most important, and as I said, it's normally overlooked. Once we do that, then we have the two components that were necessary for our evaluations. Now, I would go a little bit deeper into each of these steps, share some tips with you and walk you through our kind of workflow. The curation of the benchmark datasets for me starts with identifying subject matter experts. People who can critically look into the LLM output, they are actually my users, and I would like them to evaluate my output. Really, I cannot stress enough how the knowledge of an AI engineer or a data scientist like myself is not enough to address the gaps that there may be in the output of the agents. As a second step, I would jointly define the KPIs or the key performance indicators as the different dimensions of the quality that is necessary in order to see whether my output makes sense or not. This joint definition should be between AI engineers and data scientists together with the subject matter experts, and also the business stakeholders, such that we can actually take a look at what makes sense to evaluate. It doesn't make sense to evaluate everything that comes to our mind. Rather, the KPIs should be definitive and should be revealing the limitations of our outputs. As an example, I would normally advise for binary KPIs, so pass and fail, because these are more informative for design choices. And for example, when it comes to content generation, two KPIs that we found quite informative are the compliance checks of the outputs as well as the factual correctness of the outputs. So these were kind of two things that we identified that were quite informative for us. Now, once we have the KPIs on the SMEs or the SME committees, the data collection and data curation follows, and this looks like this, that we have our agents. our agents are providing the generated responses and this is where we bring our human into the loop so we asked our smes to evaluate the outputs with respect to the kpis but their job doesn't end there we are on we are also instructing them to provide the reasons for for their evaluation so with every specific kpi to provide me why they are assigning the specific score to the output response and also to provide edited text based on the generated output this is nice to have for some metrics it's not a necessity for every metric i would say and then based on the collected data set from the subject matter experts we have our data scientist team who would create benchmark data sets from these interactions and evaluations i would also like to show you a little bit give you a rough idea of how the structure of the benchmark data set would look like. It's something like this. So we would have the input queries, which are the user query for the agent. We have agent final output, but it's also my recommendation to collect agent trace, meaning all the intermediate outputs that we have from the different skills and the different tools that the agent has these are of utmost importance when you are doing root cause analysis and want to reveal the limitation of your agent that these are from your agents then but the next part would come from your SMEs so SMEs would have the edited text as I mentioned the different KPI scores based on the KPI definitions and also the reasons for providing their KPIs and this would be really a valuable data set in order to really lead you through your evaluation and through your agent design. Also some tips when it comes to benchmark data set curation. Initially it's really hard to give your first agent to your users, therefore here you can actually create data sets based on your historical data or even think about simulation of your data sets and if feasible provide those simulated data or those created data sets to your SMEs for retrospective evaluation, if you will. Also, you can augment your curated data set to have more data at your hands, and you can leverage text processing and LLMs themselves. However, if you are doing that, do remember to sanity check your augmented data. Now, once you have your agents in production, I would highly recommend to strive to integrate these human evals into your product. This is feasible, for example, by integrating user feedback forms in the UI of your product, and I cannot stress enough how valuable this has been for us to collect some data. When you have your agents in production, also continue monitoring the human evals and continue curating your benchmark dataset and improving them with their feedback. Now, this was the first step. Let's go through the second. As I mentioned, the second is to create those eval tools. With every KPI that you've identified, there should be one eval tool, but the question is, how do we want to develop it? So I want to start by creating what I call a base eval tool. This can be based on the openly available libraries, et cetera, so it's really dependent on your KPI. you can leverage a lot from what is openly available. And this should be tied with your KPI, meaning that the eval agent would need to score the output with respect to the KPI. Then out of this base eval tool, you can actually create multiple variations of the same base eval, so that you can perform some experimentation and iterative optimization, and thereby create or develop your optimized eval tool, if you will. For creating those variations, I have two tips maybe. What you can do is to explore your SMEs evaluation reasons in order to retrieve patterns and bring those patterns as human evaluation criteria to your eval agent. So this can help you in better prompting your eval agent, if you will. And in such pattern recognitions, you can actually use a lens themselves for clustering the SME reasons in pattern retrieval. So once you have these different candidates for one eval tool, then what you can do, or here's some tips before we go forward, For your eval tools, what I would highly recommend is to instruct your evals to also provide reasons for their verdict. Meaning not only they would give you a score of pass and fail in case of binary KPI, but also they would give a reason why they came up with this pass and fail score. These reasons are highly valuable, both for the development of the eval tools themselves, when you are aligning your eval with your human. And also for the consumption of your eval tools, they become highly valuable as well. Why? Because they would be the main output to really sanity check whether your eval tool is generating something meaningful or it's actually hallucinating. Also, what I would really recommend is that regardless of what sort of KPI you have, whether it's binary or not, always have the option for the eval to provide the answer of, I don't know. This would decrease the probability of hallucination. Let's move forward to the supervised alignment of the eval tools. So here the idea is we have our candidate eval tools and we would like to align them with the human scores that we collected in our curated dataset. And the benchmark dataset that I was talking about now here becomes our main main source so the inputs to this evaluation would be the benchmark data the models that we would like to evaluate are the eval tool candidates and the ground truth for us would be human provided kpis then we go about iteratively optimizing our eval versions by by optimizing the convention's criteria which is the agreement of the eval tool with uh with the human and here one of the things that I normally observe is that out of the box eval tools normally give me 30 to 50 percent of human agreement but already with a couple of iterations I can actually increase this agreement to higher levels some tips again do remember to use their reasons of the eval tools for for debugging your eval tool and also for optimizing them now You have these different candidates based on your benchmark data set you have these evaluations You can actually have an agent for a leader board for your eval tools as well and thereby Select the best performing eval tool out of this process Some tips here again. We are using LLM as judges. They will remain highly stochastic And you would need to strive to reduce this stochasticity as far as possible by setting the hyperparameters of the LLMs, but also by quantifying the randomness that there is to the scores of your eval tools, and also considering this randomness when you are coming up with the best performing eval tool. Also, be mindful not to overfit your evals to your dataset. If possible, create a holdout dataset for testing your evals and select your best eval based on that holdout. For that was regarding developing of more reliable eval tools. What I would stress is really this human eval alignment is a necessity, I would say. Now, having those eval tools, how do we want to actually consume it in the products? First obvious example would be for continuous development and monitoring of the agents. So we would consume basically our benchmark dataset, as well as our aligned eval tool, in order to evaluate each version of the agent, either in product or in the development cycle. And here again, some tips or hints to be mindful about. If you have a small data sets for human-eval alignment, be careful because your human-eval alignment would remain limited in terms of performance. Also that LLMS judges have their limitations. So it's highly advised that your AI and test engineers would review the outcome of the evaluations and the outcome that the eval tools provide. Here, the reasons that the eval tools are providing are one good way to go. Also, I get this question a lot. Is data leakage a concern? It's actually not. So we are developing our eval tools based on the benchmark data set. We are using the same benchmark data set for evaluation of the agents. Is there any data leakage? No, there is no data leakage because the only thing that is common between these two are the queries the output responses are actually changing based on DLLM. More interesting use of evals are actually as additional features for the product itself and this would increase the reliability of the agents. What I mean by that you can have your generating agent alongside your generating agent you can have reviewing agent. These reviewing agents are actually the eval tools and you can provide the output of your eval tool to your users such that if there is any reliability issue with your output the user can be notified and this would increase a little bit of reliability to your agents. This was a lot of content. This brings me to the recap and my final words and takeaways. Here's what I think. I think systematic evals are a necessity if we are talking about continuous development and monitoring of LLM agents. More often than not, LLMs themselves are leveraged as judged to evaluate the output of an agent, which means that these evals would have the same limitations as do the LLMs. To increase the reliability of LLM as judged, this human eval alignment is necessary. We can apply best practices in data science and supervised learning to develop more reliable eval tools. However, they would never be reliable enough. So human is always required in the loop. It's basically the LLM restrictions. I would also like to invite you to, alongside the scaling of your product, also continuously revisit and scale your benchmark dataset and your eval tools. And if you would like to get started reading a little bit on this topic, there's this article, Who Validates the Validator by Shreyna Shankar that I would highly recommend. Thanks so much for bearing with me. If you like what I talked about, find me in the ZEISS booth downstairs. I will be here for the next two days. Also, feel free to connect and send me a direct message if you have any questions. Thank you, guys. Okay. we have few questions on slido i will read out quick because we are very strict on time so first question what was your what was your agent use case that you integrated this evaluation approach into uh unfortunately i'm not at the liberty of disclosing uh our specific use case but you can think about it in terms of content generation so creative content generation okay second one how to do data augmentation with agent traces given that they could be quite complex and lengthy so it really depends on the specifics of your use case but when we are talking about data augmentation is basically text processing and also you can leverage LLMs to come up with different versions of the same responses for example in your benchmark data set but really two takeaways here would be text processing and also leveraging LLMs themselves did you have to deal with imbalance in your curated benchmark data set example distribution of some evaluated KPIs may be skewed to specific values extremely good question and yes this will always remain a concern when it comes to practical settings you would go about it how you would go about it with the imbalance data set when you have a data science and machine learning problem I would say so take a look at your metrics write them etc how do you deal with the changes in your knowledge base and the need to update your benchmark data set That's also a very good question. So the changes in your knowledge base, again, that change needs to be categorized. Are we talking about a change which is in contradiction with the previous knowledge base, or are we talking about appending our knowledge base? Each of those can have different consequences for the benchmark dataset. Such changes is exactly the reason that I mentioned. It's very important that your benchmark dataset curation and also your eval tools, are continuously revisited if you have a product at hand and if you have a lifecycle for your product. Our LLM-as-a-judge metrics often tend to judge to optimistic or not strict enough. Any tips for tackling this issue? Human agent alignment, human eval alignment are exactly meant for this purpose to see whether they are optimistic or not. To my experience, providing clear instructions and really retrieving some patterns from your human evals and providing them to your LLM as judge would help to some extent. At the same time, you can also think about swapping your LLM between your generating LLM and your judging LLM. You might have some freedom there, meaning that if you are using a less capable LLM, a smaller LLM for your generation, can use a more capable llm for your judge as your charge okay this will be the last question in data science we usually work with train validation and test disputes does it really make sense here to split into val and test set to avoid overfitting the eval data uh this was one of the points that i mentioned so be careful about overfitting your evals this will always remain a necessity. However, when it comes to using the benchmark from eval development to actually monitoring the agent, this is not a concern because really it's not the same data that you're using. The agent output would change in that case. Okay. Thank you, Dr. Roma, for this wonderful talk. Please give a round of applause for this wonderful talk.