Responsible AI with fmeval - an open source library to evaluate LLMs

Speaker 1 [00:07]

And for this session, I'm going to share your Responsible AI with FMEval. I will start with this open source library. It's easy to understand and navigate, and hopefully it would inspire you to do more on Responsible AI and also large language model evaluation. Myself, I'm Mia Chan. I'm based in Berlin, Germany. Some folks probably also from Berlin, yes. and I work in AWS as a GNI specialist solutions architect which means I work with the different companies and work on the AI machine learning workloads with them so for today's agenda I will share you a little bit background about large language model evaluation key metrics and also in practice so before I move on to the next one how many of you work with generative AI workload every day yes happy hand and keep your hands if you already move your project to production that's good some of you still struggle on the way of POC some people already moved to production happily I believe there's also a really interesting like struggle when you are experiment the different large language models so for example when I looking at the LLM evaluation I were definitely starting from the lifecycle of the development so probably some of you already familiar with develop environment and production environment the same mindset applies to previous software engineer developing now also for generative AI workload. How did it usually start? Start from your use case, right? You probably want to generate some posts for your upcoming, let's say, newsletters or your emails to your colleagues. There are some use cases out there. It could also be an external face website that you are thinking of to generate a product description for your user. So once you identify the use case you might start a model selection and also evaluate how good or how bad model is and then you decide if you would like to move on production or you are looking for a better way to improve the model. You can improve it with engineering, rack or fine-tuning. Afterwards you can also move to production if you are satisfied with your result then you could integrate with application monitoring the application and to see if you want to do another iteration so to me the reason why we want to run the evaluation it's not only you put the large language model experiment just with the gut feelings every I would say every week or sometimes during a week like three models that were just released so I found a lot of people would say how would I do my roadmap planning for my project it's so struggle right if you don't plan your roadmap you are just ending up keep doing PLC keep doing experiment and when your manager asking you what are you doing yes I'm busy but it's not your production so I hope you won't get stuck in like the the endless PLC face using the evaluation as a data point to support you to go out on production if it's not good enough you know there's a baseline you can improve in the future. So don't stop at the POC, start together with evaluation and move to production. Another aspect I want to ask how many of you already know this book? Not many? Okay I want to share a bit about why I see the responsible AI right now is resonate to the generative AI. So this book from a Jonathan Hatt actually are focusing telling a story about for people who born after 1995 during their time growing up especially during teenager phase because there are just so many social media just around there so people's mental health got rewired and got impact and how this resonates to us as builders actually in the last chapter of this book in mention about if you feel like the anxiety of the digital product brings you now as a builder what you can do is also bring those intention when you build the applications so what I would do when I build a generative AI application I will think about the users who is going to use this application would that be child would that be a shopping center but then probably some child just looking at their parents are shopping and then also they see the monitor together what would be the user experience for your product And bring that, I do see the necessary to bring along the story for responsible AI dimension together with large language model developing. Of course, there are several pillars, but today I want to focus on two, privacy, security, and safety. I want to share you examples where large language model that people will say is hallucinate, right? how would you make sure it's not hallucinate or how would you make sure for the data or domain that you cared it responds correctly for your use case so later on I want to jump to the key metrics so starting from the use case of large language models use first the user might have thinking about the different use case like open-ended generation, task summarization, Q&A, or classification. For example, like you're probably already familiar with different type of the task. If you are not still familiar with the task, I could share you one of the website I would like to visit a lot is Hockingface website, and then you can find a tab called task. If you don't know which task that your large language model is performing, just visit the page and you will find the Resonate task metrics and also the data set you could use it. I found that it's really helpful because they are just endless tasks and it's impossible to identify them all. And of course, today that I will focus on the question and answering use case, especially I see that in the education use cases, if you build education platform, then you want the large language model generated response based on the truth. When you ask some factual questions, you want a large language model know something. If it does not know, you could add another layer of RAC to pick up with the documents, or you can think about fine-tuning with a different document that you prepared. But first step, you need to understand how good and how bad for the question answering use case it behaves. So when looking at the benchmark selection, I would say it's easy to start from the task. Another aspect, I would say it start with the domain specific. So some of you might work with financial use case, some of you might work with the legal use case. And also, of course, there's also dedicated benchmarks for healthcare. So I just put a list here, you don't need to remember them all, but just give you an idea of when looking at the benchmarks, use case, industry, and particularly if you are looking at safety and bias, they are also data set and benchmarks, you can leverage it. Okay. I will pause it here for a while. like to check if there are any questions I also put the the slider code there if you joined a bit late then you still can put your questions in there cool so right now I want to share you the evaluation in practice which means I'm going to share you with a little bit demo before I start with the demo I want to share you how do I usually start a whole evaluation process first I will select which model that I would like to evaluate together with the data set so I want to ask here how many of you already prepare your own self curated data set before good that's quite a lot yes so probably when you start to prepare your own created data then probably the first of struggle part is like what would be the data format look like how would the large language model understand the task and also the result I wanted to perform so I would suggest for people who hasn't hasn't had the project to start it yet, then you can think about to start with the data set that is the public one. And then you can look at the format and then later on adjust to your use case with your own hand-picked data point. Then you can run the evaluation. The evaluation itself could come along with the algorithms, task, and also the metrics. And it could also run together with the different models. It could run on the providers like on the Bedrock SageMaker or HackingFace Open Source the model for this libraries that I mentioned. And then you will run an evaluation job as a result that you might want to parse the result to PennDice.fm or you might want to have a graph report so that it's easier for you to present for your stakeholders. different ways of sharing the results with your stakeholders. So I want to stop the slide here. Okay, good. So the project, the library that I want to introduce you is called FMEval. So why did I choose this library to start? Because it's easy to navigate. If you're looking at the root folder for FMEval, then you can easily to see their examples, source code, and also the test folder. Within the FM eval, if you want to understand how they actually compile the whole evaluation job, you can actually dive deeper with data loaders, evaluation algorithms, model runners, reporting, and transforms. What I like this structure because it's easy for you to understand either you run the examples just follows what the data resource it has or you prepare your own data loaders to connect with your own data source. And then within the eval algorithms it provides four different approaches for open-ended generation, text summarization, QA and classification but then if you are thinking about the use case outside of these four use cases you can also learn from these folders how it actually curated how it actually run the evaluation. And also the model runners here it also provides you like a wrapper for SageMaker models, SageMaker Jumpstart models or Hugging Face models or Bedrock model. So I would just quickly show you what's the structure for this repository look like but then I also want to share you the deep dive of one of the notebook so I will just close the navigation bar and then I will just run through this whole notebook in general this one okay so right now what I want to share with you it's starting with a a model initiation. So start with the model itself and then you might want to start by asking the factual questions. In this case would be like who is Barack Obama. You can also replace it with other leaders that you know in different countries and later on once you're making sure the model itself is available then you could also test it out with your own data set. So the next one when the FN eval setup you can actually see that the import classes they're coming along just the workflow that I just described the first come in the data config the second come along with the model runner and later on to think about what are the algorithms that you will check and also the evaluation job configuration so it's quite straightforward and easy to follow if you are going through this code snippets. Okay and during the time when the data config that is showing I would like to dive deeper with the sample data that I'm preparing for this one right. So for this factual use cases you need to prepare question and answer pairs and then you also need to tell the model the knowledge category is the capitals. So the previous the one like who is Obama That one will fall in with the question, answer, and the knowledge category could be like country leaders or area leaders. So with this dataset preparation, then you can easily to check how good that your model is grounding and stay with the truth. Because as I mentioned, in the use case of education platform, you want the model itself to provide something trustful for your user. So let's go back to the notebook. So when you finish running like the data loader, then you probably want to set up the model itself and run the evaluation job. The reason why that I started notebook already when I first opened this tab, because that actually run through a few minutes to go through them. So let me show you what are the details here. So actually within the evaluation job itself, it also leveraged a library called Ray. So some of you probably working in like a compute area, you know the Ray library already. So it leveraged Ray to accelerate the compute so that actually when you break down the calculation, it actually run a quite decent time with the breakout with 1.07 seconds. one is under one seconds so then you could see the aggregate with them and this one probably would take two minutes right and then the aggregate with them from my experience for the notebook will take about three minutes to finish all of them and then sometimes it's this one okay so this one will take a saving the data set will also take another second as well so you can see like a from the whole end-to-end evaluation job is split to the data set as small chunks and then try to calculate and also compare with the ground truth that you provide. So after that you can parse the evaluation result as a JSON format or as you want to see like in the Pandas data frame. So in the Pandas data frame you could also see the input and output for the model response and also the calculation of the score so if it's correct that eval score which is one if it's failed then it will show the eval score as zero so then you can use this pandas as a result if you want you can also run a graph on it to analysis for your own user and also to them know the result okay so I will go back to the slides right like when you run evaluation job what are the expected steps preparation evaluation and check the result so I will just keep like repeating this one like you will remember it when you walk out this room and then we already saw the demo so the last one of course would be like a call to actions right so for some of you haven't start doing yet start doing a project learning by doing for people who already used large language model and evaluation and moving things to production you might already find your own way to navigate the whole evaluation things. You can also use the time, could be, I don't know, this afternoon or weekend or next week, to think about how you could contribute to the evaluation framework out there. The FMV eval, I would say, it's like a really simple one to start because just the one that I show you, right, like several subfolders, it's so easy to navigate and dive deeper into the code snippets. But then if you are thinking about not only the model eval, you are thinking about about the RAC eval, agent eval, of course, there will be other libraries out there. So I would just like to encourage you to start coding, contributing to the open source world. Yeah? And thank you. This is my linking code. Any of you interested, feel free to connect. And I'm happy to take questions if there's any.

Speaker 2 [18:40]

Thank you so much Mia and now please if you have any questions please write it on slider we're moderated and We're starting with our Q&A session So the first question are there alternatives to FM eval

Speaker 1 [19:07]

That's a good one Alternatives for example, like this one is the open source one, right because I work in AWS I also will buy us from our offerings as well. So on bedrock and SageMaker. We also have that offering but I'm also looking at the open source libraries that I were also thinking about like Light light LLM. I think it also provides some capabilities to eval for the different workload okay and any of the comments I think you could also put it in the QA so that also other people will learn if there are also other evaluation framework that you work with

Speaker 2 [19:50]

The next question Did you experiment with Regus for question answering it could be handy? Here or as an addition in evaluation suit to also capture rack specific components

Speaker 1 [20:05]

Yeah, I think ragas is also another popular from work Especially I think the use case that are coming along rec evaluation and when I'm looking at the direct evaluation I think that will come alone with two parts one if it's the generator result is stay with what you retrieved and Another one if it still stay relevant because sometimes when you get some most relevant document But then when you create and generate the result it shows the totally different thing is not answering what you ask so I think it's important to leverage the let's say regs or other rec evaluation from work to help you to stay close with your use case

Speaker 2 [20:49]

The next question How do Evaluations work for instance toxicity evaluation? How was toxicity? Determined is there some default implementation?

Speaker 1 [21:03]

I think this one is really good because I think for all the evaluation as I mentioned there are benchmarks their data set right when looking at the benchmarks and the data set it also come alone who curated that data set and who defined what is toxicity and what is not for example like some people that they shared experience with me sometimes when you use a chat bot and then the chat bot I would say, sorry, I'm not able to answer your question. Some people got offended by this sentence already, right? So you will see the same sentence will bring different level of the discomfort or the sense of security when you use a system. And how you, as a developer or a builder, to find where is the threshold or where to put the line, I would say definitely get closer to your target users. For example, if you are developing a website, probably some of the users, they are checking your website every day. They probably will be like the golden members of your services. And then try to spend some time with them to talk about their experience as well. And not only just talk with your VIP users, but also in the platform, provide a way for people to provide the feedback, like a thumbs down or the complaint button. Because some people that you didn't aware of, probably they are also using your platform, but you just don't know. So provide a feedback channel for your user. That's also a way to collect if people find your system are offensive or toxic in that aspect.

Speaker 2 [22:43]

Would you recommend evaluation like this for every AI project or are there exceptions?

Speaker 1 [22:49]

Yeah, I found evaluation conversation what comes alone For both internal use case and also external use case for internal which means within a workplace Let's say within the workplace Usually the users are adult Right and then even sometimes it's like toxicity or random pictures pop up You could kind of like ignore it. You won't impact your mental health so much but then from it comes to like the public service like public facing service then I would say it's definitely critical for you for you and your stakeholders to evaluate before you release because that would also go back to the book that I call out right once you build some public services it actually a pack everyone who use it yeah so I would say external one over the internal one of course it would be great if you evaluate all of them but apparently not usually we don't have enough time and resources

Speaker 2 [23:55]

How does the library handle ambiguity in the answer for example different spellings or capital cities

Speaker 1 [24:03]

Yeah, I think also when you double-click on the algorithms, people would also use different ways to calculate the differences, right? Like previously, like when I do the comparison between the name or address, you could actually calculate if you could accept a 102 characters spelling error. And you could also think about how you want to evaluate the different things. so so that the question was about evaluate the names or the different ways can you repeat again I just lost it for

Speaker 2 [24:53]

Sorry we lost the question

Speaker 1 [24:59]

Oh ambiguity the one that I called out was from when a response is clear then you calculate the differences but then when there's an ambiguity in the responses as well that I think that would also to look at the use case if you are using like a question and answering pairs it need to be clear it need to be exact but then if you are looking at a summarization you are looking for like Generation content that you will use the different algorithm to calculate how close it is to address to your question So I would say it depends on the task and also the response

Speaker 2 [25:41]

We have a little bit more questions here and a little bit of time also. So are there any good other resources to get started?

Speaker 1 [25:54]

Yeah, resources, I would say there are various of resources out there, and I personally also feel overwhelmed for every day just look at the linking stream. Like, every day there are new things released. So, to me, what I like is always go back to, like, the classic one. For example, there are, like, universities like Stanford, they always release the courses. Of course, you don't need to go through all of them. You could just looking at people in general how people would like to review them and then go through those contents and Then myself sometimes also go through archive to find some interesting papers I know papers are boring But I will just look at the figures the figures always will tell you a lot of story The figures were telling you how many models they are comparing what are the evaluation benchmarks they are looking at so I would say I were looking at the archives papers together with the figure as a really quick snip and Then to identify probably the top three things. I want to read among the day and of course I would say the good resources always starting from something that is Relevant for your content that you are building if you are building the education content then probably ecommerce one probably won't fit for your use case that often and then so try to find the most relevant content I think that's my definition of good resource to start

Speaker 2 [27:28]

And the last question. Is the evaluation an inclusion check or how does it confirm if the LLM answer is correct?

Speaker 1 [27:41]

What is the inclusion check what can also comment on the details? What is that inclusion check about?

Speaker 2 [27:55]

Can the author of the question clarify, please?

Speaker 1 [28:07]

Okay, but then probably I would just start from what I kind of like understanding from the question, right? So if I understanding it, for example, the inclusion check, like my understanding from large language model response. So the example that I shared was about the capital, right? And then previously in a notebook was country leaders, right? so in that case would be like a question and answering pairs and then for other tasks that say generate content that say just not only generate content but also summarization I think that would also come along with the definition from you and your stakeholders I will take example of the summarization previously I work in a case for summarization itself and some people were looking at just the word count down to let's say a hundred word and some people were looking at if they 100 word include a top three important thing from that paper or not Or something some people would ask for the summarization for if there are top three call to actions from this paper or not so actually I would say depends on the use case and also who is going to use your response to Let's say the downstream business. You need to ask them what they are going to use the text and then you can define What's your metrics and how you are going to measure them accordingly?

Speaker 2 [29:51]

Okay, I think all questions were answered and thanks the speaker