Do you know how well your model is doing? Evaluate your LLMs
Evaluating Large Language Models (LLMs) is critical for benchmarking performance, ensuring safety through railguarding, and verifying that fine-tuning actually improves model capabilities rather than degrading them. This process mirrors software testing, where systematic evaluation prevents the deployment of buggy or toxic outputs and ensures the model meets specific hardware performance and response-time requirements.
The technical approach centers on the Hugging Face ecosystem, specifically using the Transformers library for model training and LightEval for benchmarking. To demonstrate these tools, a small GPT-2 model is fine-tuned on math logic data to improve its reasoning capabilities. The workflow involves loading a GPT-2 tokenizer to process question-and-answer pairs and using the Transformers training pipeline to create model checkpoints.
LightEval provides a framework for measuring model quality through built-in tasks and metrics. Available task categories include knowledge and reasoning, question answering, chat and instruction following, coding, and multilingual support. Evaluation metrics range from simple multiple-choice accuracy and log-likelihood to advanced methods such as using a second LLM as a judge to score the primary model's responses.
For specialized business use cases, the framework supports custom tasks and custom metrics. This allows developers to reserve a specific test set from their training data and define precise scoring logic—such as a binary correct/incorrect point system—to measure success against real-world data. While LightEval is optimized for open-source models via the Hugging Face Hub, it can technically evaluate closed-source models by comparing model outputs against defined targets, provided the user manages the specific API prompting requirements.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Natural Language Processing & Audio (incl. Generative AI NLP) and was classified suitable for advanced domain / advanced python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
We will begin with an essential revision of the Hugging Face Transformers library, covering basic LLM inference and fine-tuning. The core of the workshop will introduce and provide deep practice with Lighteval, an efficient and powerful LLM evaluation framework. Participants will learn how to leverage Lighteval to compare various LLMs available on the Hugging Face Hub using a range of pre-built tasks and metrics.
Finally, we will delve into advanced evaluation techniques, focusing on creating custom tasks and metrics tailored to unique, real-world application requirements. Participants will learn how to prepare custom datasets on the Hugging Face Hub and integrate them into Lighteval for precise, domain-specific evaluation. By the end of this workshop, you will possess the practical skills to rigorously evaluate, benchmark, and fine-tune your LLMs with confidence.
Prerequisites:
- Have experience coding in Python (with Python installed in the local machine)
- Basic understand of machine learning and LLMs
- Experience with Hugging Face Transformers preferred but not necessary
- A Hugging Face Hub account (sign up for free)
- A modern computer that can fine-turn small LLMs locally
Preparation:
Clone and follow setup here
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:10]
It's sad.
Speaker 2 [00:19]
and chair the session.
Speaker 1 [00:22]
The idea is to ask questions in talks.pycon.de. I'm going to do my best to
Speaker 2 [00:39]
include everyone if more the remote ones so yes and ask them whenever you want I'll be very very aware of it and we will be very aware of that
Speaker 1 [00:56]
Hey
Speaker 2 [01:18]
all the setup instructions and all the exercise on the link so Yeah, you can It's actually uploaded a bonus point if you have already started installing and you know, but if not you can do it now Because I would do a little bit of yapping at the beginning. So I think it's a good time that you started the Downloading process then, you know, you can pay attention to me. So Yeah Oh, also, I think, okay, I think, you know, my style of workshop is quite flexible, so I'm happy to run around. I know the camera person will hate me for that, but if you, let's say, oh, I have an error, and then just raise your hand, I will try to come over, or you can, you know, just interrupt me any time, ask questions, I'm okay with that, because it's a workshop. We share the space together and learn together. That's what I believe in. so if you're watching online sorry you can you know we have a very good host here that can help you to ask the question but unfortunately i can't see your screen so maybe can't help you as much so okay i hope all of you got the link and yeah good okay for folks who just come in maybe your neighbor can help you to access the link it's on github type my name there and then i like eval exercise. Cool. Yeah, I hope all of you got it. If not, you know, ask your neighbor. They are your best friend today. So first of all, I want to talk a little bit about why we need to evaluate our error M. So I'm going to, you know, I love telling stories. I'm going to give you a story time now that why is that like looking into this right so a while ago I am trying to write a blog post about fine-tuning a LRM which is GPT so I was trying to you know see if I can fine-tune it so we get like you know because gb2 is the the open you know open source model it's not like very sophisticated yet so i'm trying to fine-tune it to see if i can make it to do something more specific for example like understanding math logic because um unfortunately gb2 is well i mean like now if you use the newest you know model by entropic or open ai it would be very good at math reasoning but for those older and oversource smaller models they another question i'm trying to see like okay i don't have the power to create a model that is you know general purpose you know i'm not competing with those companies so um i'm just trying to see if i can train a small model to be good at specific tasks so that's the story of you know oh that's why i read the blog post trying to um you know train the small gbt2 model on my local computer so it's you know quite challenging so um if you have a you know cluster at home or whatever like way you can you know feel free to use those resources if you are comfortable with doing that but today we are also aiming to train a small model hopefully it will work on your computer and mine is not new as well like mine is like a few years back like when I bought it so if you have a new computer maybe you know you're better off than me already cool so so today we're gonna just try to do it again but with the help of other tools from hugging face hopefully we can retrain the model today we can learn how to use light email also provided by having face to compare models and then maybe compare our refined model to the original model to see if it's any better. So yeah, evaluating the area is very important. I think, so first of all, how many of you are data scientists? Yes, you probably know the importance of evaluation, right? Like, because you can't just be like, oh, push our model and say that it is better than the other one, because you don't have a mean to measure it, right? So, it's very important for comparing. Also, when you fine tune it, is the new one better than the old one? Because, again, like data scientists, you must know that sometimes keep training on something doesn't mean that your model get better. It could be worse if you have a task that is, if you have a new data that's not in your tree set. So benchmarking is very important. Another thing is rail guarding. It's another thing that, you know, maybe we don't think that much, is that, so, is the response from your model being weird and give something that is, like, not what you expected. So, you know, it's kind of like, I would compare it as, like, testing in software. So, if you write code, we all know, you know, we have to test it to make sure there's no bug before we deploy it. So, you know, if we're testing the software, we should also evaluate our RRM. That's as simple as that. So, sometimes also performance as well, because we may have hardware limitation, we may have other limitation, like how much response time we're trying to achieve. So, you know, having that evaluation is also like, it may be something you want to measure, right? The time that it's, you know, got some, you know, good enough result. So yeah, again, fine-tuning, I already explained it. Sometimes more training doesn't mean it's always better, so we have to measure it. How many of you have used HuggingFace? Very familiar with HuggingFace, yes, yes, good. So okay, good. So you'll be able to do something nice today then. I love HuggingFace. It's kind of like they have a lot of resources there, kind of like I like playing with what they offer, including open source model, including Transformer, which we will use today, and also like Eval. So, yeah, Transformer is, you know, I guess I don't have to explain too much, a lot of you are already familiar with HuggingFace, is basically you can use it to get a trained model from HuggingFace Hub, and then you can use it to get some prediction result or fine-tune it by training it. And then LightEval, again, is the tool offered by HuggingFace to do some benchmarking and evaluation. We will dive a little bit more into the detail later today. Again, I feel like I have to really dive deeper, because a lot of you are data scientists. So LightEval allows you to set up your own tasks and metrics, which is really good because, again, a good evaluation needs to have the right metrics. So it's measuring the right thing. So again, we will look into that later. So don't worry about it right now. Yeah, rail guarding, again, we don't want the LRM to respond to something that's super weird. We want it to be, sometimes you can do some evaluation to see if there will be toxic comment or toxic text generated. So that's also very helpful. So nowadays, if you build an agent, I know there's a lot of guardrails you can put in there, for example, some human in the loop thingy. But again, we are not building an agent today. We're just training a model. So we don't have to worry too much about that today. So coding exercise, I hope you have cloned the repo and have set up all the dependencies. So now we have some exercise that we would have to do. So let's look at mine set up. So I have mine here. I've cheated a little bit because I've created the answer. I'm not that good at live coding, so that's why I would panic if I'm live coding right now. So I already have it created, so cheated a little bit, But again, we have a lot of things to do today. There's like three parts in this exercise. If you are, oh, I'm like a super user of Hugging Face, I know a lot of things already. You can thrive through it. You can just finish all of them or jump to the part that you're interested in. Again, everything is kind of, I try to put everything in writing. So in case today, I know everybody have different speed of competing exercise. In case today you can't finish it, can always finish it afterwards so but hopefully today we have time to at least like look through all of them but even if not again like for the materials in your pocket so you can do it later the first exercise we're trying to fine-tune the model and so it's kind of like a warm-up for you so we are now trying to fine-tune the gp2 model and then this basically trying to to do what I've done for the blog post, is to try to see if I train some math logic data with it, then you will be better at math problems. So, there's already a file there for you, however, I want your brain to start warming up and working, so there are some to-dos that is not done, that we'd like you to, for example here, you know, it's not completed, So just for you to remember how to use the tokenization, like from Hugging Face, the tokenizer from Hugging Face, and also the, yeah, and then we will use some transformer here to train here, see? So this is the tool to do, for you to do. Again, like all the instructions are in the README, so after you have completed your code, and it should be start training, And then it would take some time, but the good thing about Hugging Face is that it would create the checkpoints, and then you would have this, and we will use that in part three. So I'll give you time now to do the exercise. What time is it now? Okay, so maybe I would give you half an hour is too much. I would give you 20 minutes to try to go through it. Oh, again, you can also use AI to help if you want to. I mean, nowadays, who doesn't use AI anyway, so, you know, 20 minutes, try to click on it, you know, get your hands on it. Again, like, if you don't finish, it's fine. We would, you know, you can do it at home. Cool. All right, we'll be back here in 20 minutes. Thank you. Thank you. Thank you. Thank you. Your computer went off. Thank you. Come on. I'm streaming this to visualize you. Because you know, you need to see here. Something, otherwise people lose interest in you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. All right. Thank you. Thank you. Thank you. Okay, how's everybody doing? Yay, I want to use the last few minutes to show you the quote unquote answer. You can use other method, but this is kind of working, so I'm going to show you in case you are kind of new to this so yeah so this is okay let me check back into what we were trying to do right so we have the fine tune here so first of all we need to tokenize the data right so what happened here is that so we got the tokenizer of gp2 here loaded in and then afterwards then we want to tokenize our new training data so we are doing it here so yeah we process it a little bit because the format of the training data will have the question and answer there but we want to extract the text from it so that's why we do this thing so we can have a pair of question and answer that so it's just kind of reformatting the data and afterwards then we would just do it in a loop and for each of them we would organize it so yeah I can also upload the answer for you afterwards but I always encourage you to try it yourself first and also this is not the only way to do it so don't worry too much about it right now so we would move on to what we're trying to do next so let's go back to part two so most bad at navigating my windows sorry about that okay so what are we doing next Yep. LightEval. So far we have only used the Tokenizer and the transformer from HuggingFace. So we haven't used LightEval yet, right? So let's look at what is LightEval. So I really like it because, you know, it's always good to have a good evaluation tool and HuggingFace is also providing it. So, it has a lot of tasks that's already there, so you don't have to write your own task if one of them works for you. And also, I'm sure that on the Huggyface Hub, there's also a lot of things you can use together with LightEval. For example, you can use a task that someone already uploaded, or a metrics that someone already uploaded, or some kind of evaluation data that someone already uploaded, so you can also get the resources from there as well. Right, so all those building tasks that it's already provided to you and you don't have to write it yourself, they are the common one that, you know, a lot of researcher will need. For example, knowledge and reasoning. There are more specific names for it, but this is just a category that is available. So knowledge and reasoning, question answering which is you know like our math you know logic training set is question and answer there's also chat and instructions following so if you're building a chat bot maybe that's for you coding and math also something we can consider for our own use in this specific exercise and multilingual if you are training something other than English then that could also be helpful to you so so that's the task so what you want to so this is very specific right like you're expecting an answer from a question or you're expecting a translation for the multilingual ones so these are the tasks to measure how good the results are when you get back from the model there are also different ways to measure it just like any other you know machine learning algorithm you have to of course check the answer but um you know if it's not perfect then how good it is right we also need to give it a score so there are different ways to do it um so for example you can do a multiple choice so if it's just incorrectly then yeah you know you know one point with that maybe you know if it's wrong and zero point so it's kind of like a log likelihood kind of you know once you have a lot of question you can kind of you know check how good your model is at this multiple choice exam so another thing is like you can also have more advanced metrics um so this could include a lot of more complicated things so for example um you can tokenize the result and then maybe compare the the likelihood of it to your perfect answer that's also a way to do it. Another thing is to use LRM as a judge, which is you are using an LRM to mark the results of this LRM, which is kind of funny, but that's more complicated. So I would say that try the other first because that's more straightforward. And if those doesn't work, then maybe you can use LRM as a judge, but you have to find a reliable LRM to do it. So, yeah. Next, we also have another coding exercise that would be, we are starting to use light eval so there's actually two ways of using light eval let me show you that um so here right so um you can use light about once you have pip installed i'm sure like if you do you missing in the repo you already have it um you can use as a command line tool which is the most straightforward way to do it however sometimes we don't want that right sometimes we want to maybe rerun the evaluation then you may want to store it as code so to do that you can also write a python script and then you know pull in light eval so you can run the evaluation multiple times or you can run it later with the same settings so you can do that again all the details are put in the readme and we do have a script that again is kind of like you know you have to fill in the to do so just to make make you familiar rise with what um like to build is available so make sure you have the documentation handy so you can you know check the detail um again like i'm going to give you 20 minutes again it's not enough to complete everything but um or try a few of these commands just just try a few things so it's again like it's time for you to explore to learn you can You can always do that later if you haven't finished it. And yeah, so you can get a feeling of what Lighting Bell offers you. Cool. Questions? OK. OK, so now let's have 20 minutes of exercise. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Can I do this? I don't know what to do with them. I guess I keep them as live as I can, so one way they can. What else will you do? Because it will show the people there. I don't know what to do with them. I don't know what to do with the people, because then it's clean. As long as nobody is coming to your office. No, I don't think so. So how is it structured? Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Any questions about life eval? Is it working? Is it not working? Is it time to complain if it's not working? Thank you. Thank you. Thank you. Thank you. okay yeah quickly again I show you the script if you are you know trying to get the script running evaluation model so yeah so we use pipeline so I didn't talk about pipeline in part one but I hope you know you you're familiar with it if you're familiar with hugging phase but basically pipeline just let you to put everything together and just get it running so it's very useful if you run the evaluation because then you can kind of you know run it multiple times with the same configuration so yeah you know that's the file again I would upload for you later if you want to check it so the last one the last one is yes okay so it depends on the task that you have chosen so like you well you could use a task that you have chosen and then use a metric that you have chosen so the task will be actually what you can think of it as the evaluation data that you use to or test data I don't know it depends on how you call it but the data that you use to test the model that you have finished training and then the metrics is how to score your model so if you put it in like machine learning terms yeah it will be the test set that you have you know reserved for testing after you train your model and then that that would be the task and then the metrics are the metrics that you use to measure the result of it. Does that make sense? Yeah you can choose the task you can choose to use a different one because we are using the built-in tasks at the moment and LightEval provides a lot of built-in tasks that you could use so you can either use that or part 3 which we would use a custom one so yeah any more questions Okay, right, so, okay, sorry, let me, I lost my train of thought, okay, here, so last, we are talking about how to have made a custom task and custom metrics, so, you know, it's nice to choose from the built-in task, but what if you know you have something super specific that for example you are training something that is specific to your business use case but the information are not perfect so you want to design something that's custom for it that it's not open to everyone to use so like I said before it's kind of like after you train a model, even if it's, let's say, a machine learning logistic regression model, you would probably want to see how it does with some real-life data or some data that you haven't used in training. We are probably doing that. Let's say when you fine-tune your model, you also reserve some of those data that you used to refine your model and you want to create a custom task to train it, to measure, evaluate it to see if you have achieved your goal, then custom task would be the way to go. Custom metrics also, if your task is super specific and the current metrics doesn't really measure what you want, then again, that's kind of a potential for you to do something yourself. So it's quite straightforward. I think I'll just let you do coding exercise and so let's look at part 3. Part 3 is here. So again, there are some to-do's that you have to fill in. I like when people are really getting their hands dirty and writing code. so here we are trying to define a custom metric first so this is we want to you know this is how we measure the score how we check the answer so for example we are making one that is like you know it just check if it's correct or not if it's correct then yay one point if it's not then zero point right so very very simple but again like it's an exercise and you can try to see how like she felt custom metric works another one is just have your own task so there are some parameters for you to do it and then we are using a test set here so yeah we're putting in the test set and you know to set it up and works we We should have the test set loading in somewhere here. Yeah, so we are using the same data that we have, but we would create a test set out of it and measure it. So, yay. Yeah, we will use another 15 minutes maybe to finish this Okay, 15 minutes maybe to finish this and then we'll wrap up and maybe I'll just tell you now because some of you may leave early so I don't want you to escape me without helping me at the end because I want you to help me with this little thing. You may see this image in the GitHub repo as well because you know once you finish your exercise I also want your help to help me through in the survey because i want to know how you use ai or maybe you can tell me oh you use ai in this completing the exercise and you know those things as well so there are just not too many questions you can probably do it in two minutes so the last two minutes of the workshop please help me to do that then yeah but anyway okay uh i don't have more
Speaker 1 [60:12]
I'm still in the website, checking out your questions.
Speaker 2 [60:27]
but if
Speaker 1 [71:18]
So we have a question, okay How this Sorry how this tool differs
Speaker 2 [72:25]
the tokenizer, the, you know.
Speaker 1 [72:53]
compared to
Speaker 2 [73:33]
So it's kind of like another benefit of LightSphere is open source. So if you love open source, if you love to contribute your own metrics or own tasks, then you can do that. You can upload that on Hacking Face Hub and people can use it. So I think that's one of the things I like about it is because it's open source.
Speaker 1 [73:57]
With this custom made.
Speaker 2 [73:58]
metrics
Speaker 1 [73:59]
Could you also do more targeted evals on models that are closed source?
Speaker 2 [74:07]
right so I guess you mean close okay so it depends like what do you mean by closed source yeah if you want to use it to evaluate those model I think so technically well technically you can if you have a custom model you can of course you can't fine-tune a closed-source model because you don't have the tokenizer you can't do that but with the light eval with the I'm just clicking on it right now so with these tasks that you set up so it would just you know you you know, put in the input of the model and then get it out and then, you know, do some calculation of the result and compare them. So, technically, I don't think you need it to be open source but again, like, I can see the challenge there would be like because you are, like, how you send your prompt to, let's say, Claude is different from how the way that, you know, have been faced in sending information to transformers so you can't use the pipeline those things so it may be a little bit more challenging than that I think that's the issue that I would have in my mind and I have to sort that out before I can attempt to do that Thank you I hope I answered your questions so I So I guess there are other tools for those close source model. But again, LightingValue is very customized to work together with everything that Hug&Face offer. Yeah. Right. Cool. We have four minutes until, is it four minutes until the end? 14 minutes, sorry I didn't calculate it correctly, you can tell it's very bad timing, I thought it stopped at half past, but if you, so you have options, you can go have coffee,
Speaker 1 [77:22]
We thank you very much.
