Small Language Models for Tool Calling Are Better Than You Think

Speaker 1 [00:00]

All right, so I think we can get started with this session and Yeah, I think you all know by now if you have any questions for the Q&A we have the talks So put your questions in there and then I can introduce you to Gabi Katlesova Who's gonna tell us something about the small language models for tool calling are better than you think So give her a please a warm welcome

Speaker 2 [00:35]

Thanks for the introduction. So I'm Gabi, I'm a researcher at Distilabs. We're a startup based in Berlin but remote. Before I did my PhD on automated machine learning and now I am working on small language models and on tool calling and a bunch of other things with small language models. So I will first give you an intro what are small language models and then on tool calling and also on more complex cases of tool calling. And then I will tell you how to make small language models better on tool calling by using synthetic tool calling data. And last, at the end, unless I talk for very long, I would like to talk to you about challenges in generating good synthetic data for tool calling. So what are small language models? I don't think there's a formal definition i'd say it's something that fits on your computer if you have a good gpu it might be larger if not then it's usually under 4 billion parameters um i think for us it's uh yeah under 4 billion but it could be also under 10 billion under 40 billion anything larger gets harder to train and also possibly to deploy and we know that LLMs are much much larger than this and when you want to create a small language models then you can either train it just like you would train an LLM or you could create it from LLMs so you could like prune them into a smaller model that's one option or what we are doing is that you distill the knowledge from the larger model into the small one so that it can do roughly the same as the large one. And that's what we do. The question is why would we even use SLMs when we have the great, amazing LLMs that can do almost everything nowadays. And as we know, they are large, which has also some disadvantages. So already it's been mentioned here that the energy costs are quite large. Also, the latency can be large compared to a small model. So I'd say there's like two classes of advantages with small models. First is that usually they can be local, and also they are small. So the local part goes into data privacy. If your data shouldn't go to a foreign server, then you want to use either a smaller LLM or directly an SLM. And you can save a lot by using SLMs because they are small. And another thing is more like with the stability and data. So when a new LLM version gets released, if your, for example, prompt engineering pipeline relied on that, then it may be no longer holds because the newer LLM was trained a bit differently. So you kind of want to use the older version, but maybe you can't. So that's where you want to have the local model that you can just somehow freeze in time. And then the last part is that sometimes you want to fine-tune because either it's more convenient for you than prompt engineering or you just really want to make sure that the model gets your data format right. So small language models can handle all, well, not all, but some kinds of tasks pretty well. For example, classification on the routing, I think it's a good task for an SLM or even like an encoder model could be. Then extracting structured information, that's also a pretty good task. And then question answering, I think like in general test generation is also a good task for an SLM, but if it gets more and more complex, let's say with coding assistance, if your SLM has only a few hundred, let's say 300 million parameters, it gets harder and harder to make it do the task, so maybe there's also a trade-off between the task that an LLM can do and what an SLM can do. And today we will be focusing on function calling, so that is when a user has a request, it should respond with a tool call. So how it works is that the model gets a list of available tools, functions in the system prompt and it can call some of them. So the user has a request and then the model should select one or sometimes more functions that will satisfy the respect. And ideally it should fill in also good function arguments. So for example if you have here the user is asking to amend something to the latest commit, then you call git commit, you should say that it should amend, and then you put the function in there, the message in there. But ideally it does that, sometimes it doesn't. So just a disclaimer, this was generated using QN3, 0.6 billion, which I think is a pretty good SLM in terms of how you can train it. Nowadays you've got the newer QN 3.5, also Gemma 4, didn't have time to benchmark them yet so I can assume they are a bit already better with function calling. Maybe they are not perfect, let's see, in like a few weeks. And usually, when you have a data set and you evaluate an SLM without training on it, it sometimes just forgets to add a good argument there. So one of the outputs I got, there just wasn't any amend, just a message. Or when it was on bash function calling, when you told it to go up one directory level, it instead wanted to stay in the same directory. And sometimes it just doesn't select the correct function. So when you have tasks like this, technically you could do prompt engineering, but I would maybe later get to the parts where I think this might be more appropriate to fine-tune that. This was a pretty simple case because you had only one user request and one model response. But you can have more complex workflows. So you might want to select the tool call based on previous conversation history, including the tool calls. So if you're working on a problem that's a sequence of tasks that go one after each other. If you also include everything, the performance may be better. But the disadvantage is that the errors hurt a bit more, or like more a property. So if you have a single tool call accuracy that's like 95%, then that's quite good because only five times out of 100 you are incorrect. But if it's in multi-turn tool calling and you respond with five tool calls and you have to, like, if you make at least one mid-sake in the sequence of five tool calls, then almost like a quarter of your workflows is incorrect. So getting it right is really important here, and that's perhaps why many SLMs weren't or aren't usable out of the box for some of the agentic tasks, because you make one mistake and then it could be a crucial mistake and you cannot really use it. So there the bar on accuracy is quite high. Just also to say there are more complex setups in multi-turn and sometimes you need to use them because for example if you want to reason on contents on a file or on what the names of the directory are, you also need to include the tool response and that again makes it a bit larger, and also you have to somehow deal with it. Similar, when you have more complex user requests, you might want to have multiple calls, so it's no longer one function, but multiple functions, you have to get them right, and it's no longer select one function out of the available ones, but it's multiple functions, possibly with repetition. So, like, It's quite a hard machine learning problem in general. And the last part is that you could also, like, have it like a chatbot mode. So not only it will call the tool call, but it will also summarize it somehow or aggregate multiple tool calls into one content. So it can get quite complex. I will try to have a simple use case so that we know how to train for the simplest use case, and then at the end we can talk about how it gets more complex with those cases. How to train a model for tool calling? So the thing is, maybe you don't have enough data, because to get tool calling data you You need users interacting with your functions. You need to create a large data set and have a sufficient coverage of what functions you have. So for us, the answer is to have synthetic data and use knowledge distillation. And how you can create the data is using an LLM. And knowledge distillation broadly is that a student model mimics outputs of a teacher model and it can be either very simple or a bit more complicated. Either you can just predict the same text or you could try to align the logits like the outputs of the two models or sometimes when you have two similar models you could even try to align their layers. For our case, we have the simplest case. So we just create a synthetic collection of texts, a synthetic training data set and it's basically we show real data to the teacher and tell it to create more data like this. And then we train the student on this data and also on the real data and as you may think the challenge here is making the data good. So to look into the whole pipeline, so we want to create the data and so we need to instruct the teacher well. So we take task description so that the model knows what are we solving, maybe there are some constraints, stuff like that. Then we collect training examples. Of course, probably the more the better, but we had cases where we had only like 10 or 20 training examples. Of course, for tool calling, it should somehow cover the available functions, even though you might have them in the task description because the list of tools is also a part of the task description. Even though you may have them, it would be nice to show their usage. But you can also supplement that with additional data. You could, for example, put in their man pages and something unstructured, no longer tool calling, to make it better. And then you get to data generation, so you generate a batch of data and then you look if your data is good or not. And you throw out the data points that aren't good, keep those that are, and repeat until you have a sufficient size of your data set. Then you fine-tune, and then you have your SLM. So I will show you the pipeline on a simple task. It's been adapted from a nice benchmark leaderboard, BFCL, that's on function calling both single-turn and multi-turn, also some agentic parts, but to kind of test the pipeline, I created something more simple, because for multi-turn there are parallel tool calls, and that overall just gets more complicated. So as a first step we allowed only one tool call per assistant turn, and so I kept the same tools, and then we generated a simpler conversation where we respond only with one tool call. And if we want to generate data from that, we will install the LLM. So generate an example on function calling. You have these functions, and it should follow, and then you make a list of JSON formatting, create environments, how it should look like, et cetera. And finally you say it should look like those examples. And you put in the conversation that's like user tool call, user tool call, and then the final tool call. And then the LLM generates something, which is sometimes correct, sometimes not. So you want to throw out obviously misformed examples. So if the JSON is not correct, if there are some hallucinating tools, you throw them out. Sometimes but this is more like for smaller LLMs, it's like though certainly I will generate something for you. Okay, let's just either toss it out or regex out your tool call just to make it correct. But then a really important part is to remove duplicates. Because sometimes the models tend to generate similar examples, especially towards the end. So if you just have a data set of a lot of similar data, then it's not going to be that good. We, by default, just have a simple Roosh comparison, so if it's on token level too similar, we throw it out. But you could get really creative here, like if, for example, the sequence of the two calls that are called is always the same, then you maybe want to instruct it to do something different or just throw it out and hope that something will get generated next time. So to repeat, we generate a batch, validate, skip something, and go on until we have the data set. Usually we end with something like 1K to 10K examples, depends on the task, but this is already enough. Some of the really simple ones were even okay with 100 examples. So it doesn't need to be like 1 million of googling scenarios to make it decent. And then we fine-tune, and that's just classical supervised fine-tuning, so we make it mimic the assistant responses only we do LoRa, and again, only a few epochs are necessary, and then we can just, it's still like a public model, no architecture changes, so it integrates well with VLM or LMCPP, anything you like. So for this simple case, the base QN, that's the bottom row, started with 85% accuracy, which gets to less than 50% examples correct. But then after the fine-tuning, it got to perfect or near-perfect accuracy. So you can get pretty far if your task is simple. And the nice part is that you can also... Some function calling models supported only single-turn tool calling. That was with Gemma 3. Gemma 4, by the way, can do multi-turn tool calling. And when we did that, we were able to make it learn multi-term function calling. So that was a nice use case of the approach, like learned a new way how to solve tasks. So the third model is liquid model. And yeah, it's, I mean, it works. It works, but there may be some challenges. This was already quite simple, which is okay, because some practical tasks just are simple. But then there are some challenges that you need to solve. So identifying when a JSON is not a correct JSON is pretty easy on a code level, but then identifying if the synthetic data is actually good? Like, does it make sense? Like, is it something that the user would ask? Or is it similar to the distribution that you're going to get in practice? That is a good question. And if you have another answer than this, then please tell me, because it's, I mean, it's something that probably everyone who who works with LLMs is asking yourself, I would just look into the generated data and see if it makes sense and see if there are any gaps, like is there something that the model didn't capture, or you can maybe use LLMs to judge like is this example okay or is it like conflicting with the job description that you had. An important part here is coverage because if you don't generate a function at all then and you're just not training on that, simple as that. And if an LLM thinks a particular set of functions is good because it saw it a lot in the training data, then it might just generate more of that. So you want to somehow maybe stratify a sample and say, hey, please generate this function now, or please generate this sequence of functions. So that's one thing. It could also forget to use some parameters. When I started with the simple demo, the function ls wasn't called with a, like list all files, at all. And then there was a single error on the test set. So you need to make sure that the LLM generates everything that you need to have there because of your fine tuning. And if it's not in the fine tuning data, then you're relying on what the model knew beforehand and also your fine tuning. So maybe it's already forgetting what it knew before. But another point is that you should also vary what the user requests are. So sometimes the user will say, list all files. Sometimes they will just say, show me the directory. And you should probably have different variants there so that it gets more robust to that. And I think you have to compromise here, because you can't, or maybe you can, but usually you can't generate 1 million data points. So you have to somehow make the data representative enough, and then balancing both the functions and maybe the wordings is not trivial. And you could do that, for example, via some clever mutations of the prompt. So you would know that you want, in 50% of time, focus on function coverage. In 50% of time, change just the user messages. And then hopefully that helps. We have an intern working on that. So hopefully, next PyCon or something, we'll have a talk about that. So let's see. Then there are specific challenges with multi-turn. These are really fun. So just because there is a specific number of turns and you have to train it for your task, that may be the SLM or even the LLMs I have never seen before, you have to cover all the realistic scenarios. So if you can have conversation up to 50 turns, then you should make sure that you have the 50 turns. Because likely, if it goes beyond that, it's not going to be good, or you need to do some windowing or summarization of the conversation. So that's definitely one challenge. It's also a question whether you should train on parts of the conversations. Because when I started, I didn't train on sequences of length 1. so the model was really bad at the beginning, and then it got better when the conversation got longer. Another part is the tool outputs. If you generate them via LLMs, then they may be just completely hallucinated. It will hallucinate the contents of your file if you want to do that. So you could connect it to an environment, but then it might take a lot of time because you need to generate them tool by tool rather than just spitting out the whole conversation. So unless you are caching, that's also going to make the generation last longer. And then there's a separate part on the tool calls. Actually, they have a really nice blog post, so you can check that out. If it's more agentic or if there are parallel tool calls involved, it gets much more complex, because different sequences of tool calls can lead to the same result. And then you need to somehow reconcile that so it's no longer compiling the function to function, but it's also the state of the environment and it gets more complicated and fun. And the very last part is that when I started working on that, I was a bit surprised because there weren't that many benchmarks or suitable benchmarks. I mean, there's the BFCL, there's also some TauSquare that's useful because most of the other ones, and including some part of BFCL, test, for example, the performance over a lot of APIs. But we are focusing on one specific task. So usually you would get, like, ten test points or 20 test points per one function family, and if I want to evaluate if I fine-tuned my model well, that's not exactly a lot, because, like, okay, I got better by one example. Yay. So it's, I think, hard, and we are creating some of our data sets, and maybe there will be even more, or, for example, there will be some production traces that you can evaluate on. But I think it makes sense to focus on this problem because, in practice, a lot of the workflows are on the limited function sets, so it's not like we should just study what LLMs can do across all possible APIs that we have, but it makes sense to also look on specific tasks. And then this is like a selling point for why we should do it like this, even though other approaches are totally valid for some specific use cases. So let's say you add a new function. So you've got a nice set of 20 functions, and you add a new one. So if you're using a good LLM, then you mostly don't care, because you will just add it to the prompt, and then it probably just works. Then if you can handle the costs, then okay, you don't need to listen to this talk necessarily. But if you want to use something smaller or an open-weight model, you probably now need to either do the prompt engineering, but I think that can be a bit unstable, because if you prompt engineer on a new function, then it may break everything that you had before. In that case, I think just fine-tuning an SLM where you just add it to the existing data set, okay, you have to pay some training costs, but then you have something small that hopefully works and that you can control. So thank you for listening, and I'm looking forward to the discussion. We can also talk afterwards after the talk, and you can check, we have other blog posts and things. So thank you. Yeah. Thank you.

Speaker 1 [25:39]

Thank you very much for your talk. And there's a lot of interest in this room. There's a lot of questions, so that's good. We won't make all the questions. So yeah, then talk afterwards, and we'll probably move them to Discord. So the first one, how do you make sure that you do not overfit to the synthetic examples all produced from the teacher model?

Speaker 2 [26:01]

So, we have a test set, so the test set should be different from the training set, so hopefully that makes, like, it should be sufficiently different from the training set and the synthetic set so that you capture it, otherwise it's just like any other machine learning you try to combat it with all the machine learning techniques that you have for overfitting. But I would say also, like, if you generate just more data and try to make it more diverse, then you can somehow make it better. Hope that answers that.

Speaker 1 [26:36]

So then we have another question. What was the motivation that you had to use an SLM instead of LLM in your use cases?

Speaker 2 [26:45]

So, that company called Distill Labs, so the distil is distillation, so it's a company that works on SLMs, so that's like the use case, but it's also like, I think it's interesting because difficult, like, there are some tasks that are difficult, so those obviously need those LLMs, so you need to, like, when you are coding, then you want a really good coding agent that will just do it well. But a lot of the tasks are simple and using an LLM on a really simple task is just like you could just go and burn those trees yourself. I mean, so I'm a big fan of using a small tool for a small problem.

Speaker 1 [27:30]

Then in which sectors or settings did you test this fine-tuning approach? Are SLMs suitable for rack systems currently?

Speaker 2 [27:42]

Which one is it?

Speaker 1 [27:44]

Oh, intersectors, okay.

Speaker 2 [27:46]

Okay.

Speaker 1 [27:46]

So it's like in which sectors or settings did you test your fine-tuning approach and would you say SLM are suitable for the rack system?

Speaker 2 [27:54]

rack system? Yeah, as I said, you can check out the website. There are other demos that you can look at. So I focus on tool calling mostly. My colleagues, I think there were some rack cases as well. In general, we do like many text generation cases. Yeah, I think this would be like more a question to our colleagues because I don't also remember everything that we do.

Speaker 1 [28:21]

All right, we have another question did you try to adapt an SLM to a specific agent harness?

Speaker 2 [28:29]

Oh, no, not yet. Easy answer.

Speaker 1 [28:34]

Let's see the next one that we have I think we have one time for one more How did you handle missing information and some of the teacher generated variants data set issues have caused my SLM to hallucinate quite a bit

Speaker 2 [28:54]

Like I think those tasks were still pretty constrained. So I don't think I saw much hallucination there Like obviously if it's gonna be more complex, then it gets more Like more problematic. So I haven't seen the hallucination after training Especially even it's still calling which is like a JSON output. So it's pretty easy to do that Of course if there would be like open form parameters, that may be an issue, but the idea here is to just fine-tune for the simple cases where the models still make mistakes, so then that is like a step three or four.

Speaker 1 [29:33]

So thank you so much, I think we're running out of time, but we might move the other questions to Discord. And so another applause for Gabi, thank you for sharing your insights.