Making bad CLIs fun with Small Language Models

Speaker 1 [06:38]

somehow. And then, of course, we need to see how well we did, right? So that's roughly what I will walk you through. I said, okay, four months, that's a good timeline. There's maybe some coincidence with the overlap between the call for paper and today, but it's totally, totally coincidental. And then I said, okay, I don't want to spend too much money. So 50 bucks, because I really dislike projects in which you're kind of forced to spend too much upfront, or you kind of have to invest too much, I think that makes this not very satisfying. So 50 euros is fine. I do use this laptop here, so that's there, and I did have a co-pilot license that was quite essential. So, we kind of, of course, I mean, you probably thought about this as well already, we can just use a large language model here, right? So, we just, the large language model doesn't know about our internal CLI, so that's a bit unfortunate. And if I speak, looking at the slide now, if I speak about large language models, usually we think about hundreds, billion parameters, I don't know, like 500. We don't even know about for the frontier models, but many, many billion parameters. But they don't know about your internal CLI, so you can do something where you just provide the help text as context, and then you say, okay, this is how my CLI works, here is what I want, make me a command, please. But in general, I find that it is a bit unsatisfying, because I want you to use my CLI and I don't want you to have to provide an API key for that. You depend on the cloud, you need internet all the time, it doesn't run on your machine, it costs money, or you need a really big GPU. Really unsatisfying. It gets the job done, but it's like one of these combined tools here for eating. I mean, you need those for camping, right? You need something that does everything. It does the cutting and the sporking. I don't know what that's called. spoon and fork. But it's not really what you want. What you want is individual tools for specific things that you need to do, so like cutlery. What I would use there is a small language model, and we've heard there have been a few talks about small language models, and I was surprised about the variety of definitions. I told myself, okay, let's try with a really small one, so less than a billion parameters. And then, what What we can do with this really small model is we can have fast on-device inference. We can just use the computer that we already have, no internet, no nothing, to just prompt our CLI. That's the plan. And how do we do that? With fine-tuning. So with fine-tuning, we can just add this information into an existing model, and then that model knows about our CLI and can generate the prompts that we need, or the commands that we need. All right. All right, so I started with this internal CLI that I showed you. What did I do here? So the plan is, I already mentioned, we want to generate some data. Then we want to do a supervised fine-tuning, so SFT, that's called. And then we want to evaluate, like, how does it actually work? I know I mentioned this already, but just so I'm not losing you. Okay, actually it turned out that, and that was surprising to me, the dataset generation is really the hardest part. So, the mapping, or this is, I came up with like three different steps that we'll walk through, and you might notice some similarity to the established literature or other talks. I was really quite surprising how similar this is, but yeah, I mean, let me walk you through that. So, where do we start? We first need to kind of generate some basic understanding or some basic pairs of natural language-ish instructions and our command line arguments. And so what I did here is I used a coding LLM. I just took Cloud Opus and basically prompted the coding LLM to map out that space. You give it the code as a context, and because it has such a huge context window, no problem, it can just take your code and convert that into a little program that just maps out this individual CLI instruction. So I have some examples here. Say my tool, this is a plotting tool that we use. I should have maybe mentioned, but it doesn't really matter. This is a switch called customer, right? And you can have different customer names, and so you just randomly draw names for that customer, and you create basically your command line strings. Similarly, it has a switch plot events, and then you create these command line things from randomly drawn. Sometimes your instruction have that events thing. Sometimes they don't. But, of course, you also need a natural language equivalent. And so what I came up with is just like having small snippets of natural language, like for customer XYZ or with events. So you have an equivalency here. And that gives you the ability to just chain these little snippets, these with events together. I will have an example on the next slide if you maybe can't imagine what that looks like. But it already sounds kind of natural language-ish. and what is nice about this is that it's not that much code because your data set generation the first step that you want to do it should just not be that much to review because I spotted a few bugs, this happens, coding agents they make mistakes so you have to check out that everything really works and thousand lines still totally doable and so what does this look like we have now a command here, this is our plotting tool we have the first switch. I already mentioned the customer switch with some, actually we have fixed customer names so we just bake those into the data set and then we have some products. Here I use some placeholders and some countries that I want to do the plot for. Again, we do demand prediction so that's just what you would plot here. My colleagues are smiling because they know that tool. And so, what you end up with as a natural language ish, again it's not really a natural language but as equivalent is this kind of thing. So that can, And my simple 1,000-line tool can easily generate that. It just puts together all of these little snippets, the width events, everything like that. It just concatenates that. All right, so what does this look like if we have something more complex, right? Because I promised you the CLI is complicated. So that's what it looks like. You have a command like this. That's already really annoying to remember, I can tell you. So doing something like this, it's not so easy, and it maps to this kind of text here. So I think as a human, you can kind of already read that. That already sounds vaguely natural language-ish. But actually, if we just make our data set like this, just like 3,000 pairs of this kind of thing, and then our command, and we train our small language model with that, it would totally suck. Because as soon as we use something that's not generated as the first word of what we want from the small language model, it would just totally break. I mean, it's not that surprising, right? So this won't work. We have to do something a bit smarter here. So what we do is what I call varied instruction, or I think technically it's called prompt mutation, but I don't know these words, it's just what I came up with. So what you do is you just take a bigger LLM and just reformulate your sentence. You just give that sentence and some extra instructions, come to what this then looks like prompt-wise to something a little bit bigger, and that little bit bigger can then rework your sentence so that you have a nice varied data set. And we need to pick something here for the bigger LLM, for the teacher, right? We want something that's a little bit bigger than our, like, less than billion parameter model. And ideally, we want something that's not super expensive because, remember, 50 euros, yeah? So that's not that much, actually. And we want it to be instruction-tuned, so we want it to actually follow the instructions that we give it. We need to tell our teacher what to actually do, how to make its variants, right? And I chose not to use reasoning because reasoning burns through tokens like crazy. So we want to save some money here. And so I settled on, at first I used GPT-4 here, but it's like one of these data set instructor generation steps. It's like 8, 10 million tokens, input and output combined. So that already with GPT-4 prices already quite expensive. So I went to my inference provider, Together AI, and I picked out this GLM 4.5 FP8, so it's like a quantized model. And that one, really affordable, so it took me like one euro for this entire 10 million, so that's why the euro's here. And that worked really well, so I really enjoyed using that model. But then something happened that I think really illustrates why you need small language models. The inference provider just stopped providing me with that model. They just decided, no, it's not worth it, it's too expensive, whatever. I don't know, there's probably some reason. They sent me a nice email, but really then they told me just use this five times as much expensive model. So it didn't really work. So then I switched to this QEN 3.59 billion parameter model. It's like, yeah, it's not really a large language model, but it is quite a bit bigger than less than a billion parameters. So it worked out okay. So how do we tell the large language model, the teacher, what do we want? We give it the reference manual. So I just let it generate from the code. I mean, we don't really have docs, so you just give your coding assistant the code and let it generate some reference, easy. And then you give it some examples. You need examples, otherwise you will get really bad results. So you have your fixed natural language instruction, your command line argument, what it would look like, and then some variations. And then, of course, you need to tell it how many variants you want. That has to be part of the prompt. And then at the end, so that's the only thing you vary between each of these pairs you send to your teacher, you really only change the instruction and the command that you have to vary. So then you do that 4,000 times or 5,000 times, however big your initial data set is, and you do it for 20, you tell it, okay, I want 20 variants, so you already have then 100,000 pairs of really nice, varied sentences and command line instruction. So I realize at this point that this is mostly synthetic data set generation, it's not really the NLP track, but stick with me. So we have to filter now these, or I forgot, we still have first our instructions actually here. So that's an example, right? You see here that you have our really simple instruction, and then we have our boilerplate sentence that we show to the SLM later, and then a bunch of variations. And you can already see that it really helps to put in some context. So, for example, here the The varying, the teacher LLM really knows that what it's plotting are time series forecasts, right? I mean, it doesn't know that a priori, and it doesn't know our tool, so we really need to provide some context for it to understand what's happening here. But, unfortunately, we often have, like, failures. I mean, the LLM is, the teacher LLM that we use is not that great. It's not, it does make mistakes. We all know we get sometimes garbage data, and it's really easy when it's broken formatting because I ask it to give me JSON. If it's not JSON, just throw it out. I mean, that's easy enough. But often, I notice something like this. So you have a command line or a command where you have a customer name and two dates, right? And then you have many variations that perfectly match that, but you have a single variation that forgets crucial detail. So for here, it just says for the specified customer. As you might notice, that variant just doesn't contain the information what that customer name actually is, right? It's just missing. And if we have that in our data set, that's horrible. Because later, we want to fine-tune with the data set, and so we are forcing our small language model to basically hallucinate this information, because hallucinating some customer name usually is better than not having any customer names at all. And so we really need to filter those out. So what we do is, in our initial data set generation, we just also persist some important identifiers that must be present, and that are basically arbitrary strings that must be present in both the command and the varied instruction. And using that then to filter out variants works quite okay. It's not perfect, especially for dates. There are so many different ways you can write dates, like this date. You can just write it like this in ISO format. You can say the 1st of January, a lot of these different variations that exist in natural language. And so filtering is hard here. So that's basically an unsolved problem. But you can use a better LLM, but then you're 50 euros. All right, but we have the missing customer here. Yeah, I forgot about that one. All right, so now I promised you some fine-tuning. We need a model for that. What do we use? We use the smallest one I could really, that is good still. So I opted for this quarter-billion parameter GEMMA3 model. We know GEMMA4 is now out, but they are quite a bit bigger, so they started like 2 billion. We want to stay below 1 billion, so we use that one. And it's really, really tiny, and that means it's really fast enough to run on a phone. And Google has a really, really nice demo where they use this to control, where you use the small language model to control a game here in your browser. You just type in or even speak some natural language instructions, and that gets then sent to your small language model, and it does something in the game. It doesn't really matter, but it really shows that you can need basically no resources for a model like this. And what it still can do, though, is really understand a wide variety of language. So it has been trained on a huge amount of tokens, so six trillion tokens, and so it understands really a lot of words that are quite out there, but it can't really do, like, complex tasks. But we will add that information using some fine-tuning. And also, if you want to try out a small language model, that one's really good because it has great instruction tutorials. Google even gives you hardware to run it on. But actually, it turns out, You don't even need that. You can just use your computer. I fine-tuned the model for this on the 100,000 pairs easily on this MacBook M3 Pro. So it's like 10 hours of training. That's overnight, totally doable. So we see here, it's not even that long. And the stack is also great. So I use the Hugging Face stack. So that just is like these three imports at the top. And then you can immediately start fine-tuning. And the tutorials are very good. So it's not that hard. What is a bit hard is if you want something very specific because the language model space is exploding so rapidly that you have these classes with huge different amounts of configuration. You don't even have to read it. You just see it's a lot, right? So you have to settle on something for your configuration. You have to pick all your number of epochs, how much you want to pack the data in the training, and you want to figure out what gradient accumulation and all these things you want to do. But once you have that, just go, and then you get really nice results really quite quickly. And yeah, so does it work? Yeah, totally. I can just now write down something like, give me a plot for a customer name, some information, and the word gimme has never been in the training data set. These are unknown customers and dates and data set UUADs, all that. And it just gives me a nice instruction that immediately works in my CLI. So our model has learned something there, which I find is really nice and demonstrates something that I find quite useful. Unfortunately, it doesn't always work. So you still have like, I mean, you have to be honest here, right? This is doing, it's working okay. It's like 85%. It really gets the correct command. But then sometimes you have missing flags or extra flags, stuff like that. Yeah. And I think the three categories here on the left can mostly be explained by this issue where you have, when your original data set information is missing and you're forcing the model to hallucinate a bit and that leads to instability and then gives you the wrong flag when you basically do inference. The two things here on the right, then this means that the output is totally broken. The UUID it only has to do a simple copying step and no mistakes there I'm pretty sure and not a command and it's just a garbage output. That can happen with an LLM or SLM even more so you need a better model there and I think that would be a good next step here. But actually I then noticed in my talk proposal, I had promised a second application. And here, I chose FFmpeg, because that's quite popular. I know I said bad CLIs, and FFmpeg is maybe not a bad CLI, it's just a very complex one. And I think that is evidenced by the huge amount of AI wrappers. You might have seen one of those in Hacker News or something like that. I really like the name of WTFFmpeg, which basically uses a consistent prompt or a custom prompt to wrap the FFmpeg CLI, and then you can just write in natural language what you want to do. But, of course, you have all the drawbacks that we already talked about. You need an API key. You need, like, upstream stuff. We don't want that. So I did the same thing for FFmpeg, and this is also, I put up some code, so you can have a look if you want, if you're interested how this really, really works. But it turned out to be a much harder target, so you need much more dataset generation code. The FFmpeg CLI is crazy complicated. I was a bit naive there, I must admit. And I also noticed that I don't know the FFmpeg that well, actually. There are a lot of different switches that you can have and that you have to account for in your data set generation. But at least because you have initial code that generates these pairs, you can really dig down and understand what's happening and what's missing. So I think that really is nice in that approach. And what I found out also, it's really, really tricky to generate nice, realistic file names because you need to teach the model to copy this information. Sorry, it's a bit hard to see from the back probably, but you need to teach the model to copy information, what the file name is, from your instruction verbatim into the output. And doing that in a way, in fine-tuning, it's not that straightforward, but I settled on something where I basically combine randomly drawn words with data from the tokenizer that the LLM or the SLM already has available. And does it work? Yeah, kind of. I mean, it's surprising that it's basically the same ratio. So in 85% of cases, you get the exact correct command that you need, and then sometimes you have flukes where we have an extra copy or something that you didn't expect, and often that doesn't really hurt, but it does not give you exactly what you need. So there's some opportunity for improvement. But of course, I also had to test the claim, is it actually faster? Like I promised fast inference on your device, right? So I converted it using an inference engine, I used the Lama CPP, and we see doing it like generating commands a thousand times. We actually managed to do that in most cases, like there's one outlier in less than two and a half seconds. So I think this is really quite feasible as a workflow. I can wait two and a half seconds if I don't have to remember what the weird CLI instructions are. Yeah, so if you want to check that out, you can find that under this QR code here. So you can just see what this Fmpeg fine-tuned project is, and maybe that encourages you to also start something using one of these small language models because I think there really are an opportunity to kind of take your domain expertise, your kind of specific domain problem and put that into some small language model. And that's kind of it. Any questions?

Speaker 2 [26:36]

Hello. Thank you. Yeah, there are a bunch of questions. And if anybody hasn't done it already, just add them to the app here. I think the simplest one I'll just do really first. Are the slides available somewhere?

Speaker 1 [26:51]

Yeah, they're in the schedule thing. I uploaded those. Cool.

Speaker 2 [26:55]

All right, yeah, um Let's see. I'm gonna try to get it in order. Is there a risk of de-skilling you in bash as of SSH into a server without an MLM

Speaker 1 [27:10]

I mean, the nice thing about LM is technically it's so small, it's only, what did I have, like 500 megabytes. You can just bake it into your program, right? You can just bake it into whatever you're shipping, and then you don't have to care. It will just be in there. Can't do that with LM.

Speaker 2 [27:29]

Did you compare how good Cloud Code or Codex is at translating the natural language to CLI commands?

Speaker 1 [27:38]

I mean, it's very good. I mean, they spent a lot of money on that. So I better hope it's good. And it's kind of the foundation of how you do the dataset generation. But the point is, we can just not need that. And yeah, I think it's possible to show that you don't need that.

Speaker 2 [27:55]

How did you teach copying?

Speaker 1 [27:58]

Yeah, that's a big issue, actually. So how do you do that, right? You need to copy some parts of the input, yeah? And initially, I was really, really naive. All my pairs, my instruction pairs, just contained this kind of string, yeah? Because that's what my coding agent had just put there. It didn't know what the locations are called, right? So that doesn't really work. And what I did is, after this mutation step, I just replaced all of these identifiers. You treat it as a template, just replace it with randomly drawn strings just something that is from like three characters to like 12 or whatever you think might work and That worked really well That really worked you have to account of course you your ideas have a special structure, so you need to back that in But but that really works doesn't work that well if it's not like random identifiers Which these locations where then you need to do the trick with file names where you just conquer those think about okay? What would a realistic file name look like?

Speaker 2 [28:53]

This is an old school question, wouldn't a sophisticated Red X also do the trick to extract the CLI arguments?

Speaker 1 [29:03]

I mean, from where? Sorry, I'm not sure I get the way you would do the regex. I mean, there are regexes in there. I mean, that's how I do replace these strings post-mutation. It's just a regex. Yeah, sure. But I don't know how you would get from, like, a natural language reliably to a command line argument.

Speaker 2 [29:23]

How did you fine-tune the model? Did you use a validation set?

Speaker 1 [29:29]

Yeah, sure. I mean, I just took out a few percent and I don't have the loss curve in here I think. No, but it just took out a few percent of the data set and then checked how many epochs I should train. That's like not that much. No, I don't have any slides on that. Sorry. Yeah, but there's not that much special sauce there. Just stop before you overfit.

Speaker 2 [29:52]

How is this related to tool calling or function calling in language models?

Speaker 1 [29:56]

I mean, I think an approach to tool calling is not that far off, but in tool calling, usually you have a lot of crud around, basically. So you really fill the context really quickly. And I like that CLIs are something language models already kind of understand. The Internet is full of CLI commands, and they're really condensed display of information. So you don't need to care about types and things like that.

Speaker 2 [30:23]

Would it be better to perhaps use fixed tokens for UUIDs, file names and such, and do string replacements on the language model's input and output?

Speaker 1 [30:34]

I mean, it is kind of a placeholder where you just replace that later with randomly drawn tokens. I don't know if maybe whoever asked the question, we can chat about this later.

Speaker 2 [30:46]

Yeah, he's very approachable, so don't be scared.

Speaker 1 [30:49]

I don't buy it, don't worry.

Speaker 2 [30:49]

I don't like it. All right, did you achieve anything with accuracy over 85% during your process?

Speaker 1 [31:00]

process I suppose no but I'll again also only four months on the weekends so

Speaker 2 [31:04]

And only €1 or €50 too.

Speaker 1 [31:07]

Yeah, only 50 euros. I'm sure it's possible. I'm sure you get 90% plus. It must be possible.

Speaker 2 [31:13]

How did the users of the CLI feel with this feature? Are they using it or are you the only user?

Speaker 1 [31:18]

only user. Currently I'm the only user, so I feel great.

Speaker 2 [31:24]

All right. This is high customer satisfaction currently. All right. We started early, so we have lots of time for questions. This is great. Could you potentially...