Beyond Basic Prompting: Supercharging Open Source LLMs with LMQL's Structured Generation

Speaker 1 [00:07]

This talk will really build on some of the ideas from the previous talks. So from the previous talk So it's going to be fundamentally about what's the type of value we can unlock from building on top of logits and what's Taking a more practical approach to see how can we use logits and make our generations work better for us? and this is going to use this open source tool called LMQL and I work for Deidu, a Dutch service design company. We generate user journeys for different customers. We fundamentally mine these types of insights, so we transform unstructured data into highly structured data, and that's where I use LLMs daily. I myself have been working there for almost two years. Before that I co-founded a sales tech company called Comptura and before that I worked as an NLP engineer at Helix working on rare disease drug discovery and biomedical NLP pipelines. I'm also interested in edtech so that's another thing to talk about after the talk. So let's start with just going through the basics of what can LMQL provide us with regards to generation and then we will look at a basic retrieval augmented generator and how we would evaluate the system and have a little look at some potential improvement opportunities. At the end there's a little demo application that I prepared so everybody who wants to can play with LMQL on their phones hopefully for five minutes and then finally we will close with some remarks on where is LMQL with regards to taking things to production and if you want to do something like this in production what's maybe a better way to do it so let's let's start by LMQL is fundamentally a domain specific language language that uses Python-like syntax and allows you to interweave prompts and code. And this is a pattern I quite like about it. It's nice that you, here you can see these string literals will be, these will be interpreted as a prompt, and these are the prompt variables in square brackets and you can put different constraints on them, and here the logic bias stuff starts to, this is how we actually inject this logic bias stuff. So here's the, this is a basic open source small LLM from Hugging Face, 1.7 billion parameters with 4-bit quantization. That's what we will be using throughout. So what's happening behind the scenes, as we've seen before, is we send these tokens actually to a LLAMA CPP server, and then we get a response. So that's how we could look at this as LLAMA CPP is kind of the underlying server that is sending all of this information and that's what we're at LMQL outlines and these other tools are often like a layer on top of this that are just manipulating let's say that basic protocol under there. So you can set your system prompt as well and then the model will slightly change what it does. you can also look at it in different ways. But you see with the generations, there's also a problem. So here, look, this is the answer. And with these open source LLMs, they often start to either repeat themselves or the generation boundary is not super clear in like what it's supposed to do. So it will start asking questions itself, then it will start answering the questions and then it will go on an endless loop kind of question and answers. And these for, in particular, if you're working with kind of more small models, this is quite a common problem. So this is a recurring challenge. So one thing you can do is obviously you can set like a stop token here. So I just stopped the generation when the questions start reappearing. That's the, let's say some simple approaches, but there's some nicer approaches as well with structure generation. So the other aspect that these tools really shine on is when we're working with software, we all need clear interfaces. We need to define nice classes. We need to work in a kind of validated structure. Even if you're generating JSON with your LLM, often you want to further validate that maybe with by dandy classes or something like that. So a nice feature LLMQL also provides is you can directly take these data classes, And then you can feed them into the model here. I'm talking about myself. And then I wanted to actually generate a structured object following this object. So I give this object to the prompt here, and then I generate it. And then you can see that it will generate some kind of result. In this case, notice it also doesn't actually follow the prompt text. So again, like with a smaller open source LLM, you can see it actually hallucinates all of these results. So we have a lot of these problems, and we're going to talk a little bit about how we can kind of measure these and keep them under the wraps. Finally, I think a pretty interesting scenario is using LLMs as a kind of poor man's classifier. So when you want to do maybe something a sentiment analysis and you already have access to an LLM and you don't want to necessarily train another model or something, you may want to just try and do something like this where you get a movie review. Here it's in a prompt, you can see it's kind of like an f-string, and then you do some kind of chain of thought reasoning with analysis and then finally you also return distribution between two between a specific number of tokens and this is the same approach where you focus on the the logit probabilities of these particular these particular generations so in this case the positive and negative so then you can see that we would get something like here I really enjoy Shrek 2 and you can see that the model under seems to get that and then the classification it classifies it as a positive sentiment review and you can see it with 90 you could interpret this as 98% probability on this side and and with much less probability on it being a negative negative review but one one One thing that I was kind of asking about the previous question as well is if you change the labels here that you're trying to do your classification on, whether you do it on here we were looking at positive or negative, and then if you look down and you go more and you just change the string to good or bad and you're still doing the same sentiment score, you will get a widely different probability distribution actually with different models. In this case we're getting quite a stable result, but in particular when you're working with smaller LLMs with very few parameters, you will get a very wide range of changes here because of your training data distribution and underlying kind of logit distribution. So that's just something to kind of think about. And then as a kind of sanity check, I thought it would be interesting to also look at how does this model do with sarcasm. So here is a sarcastic review where, you know, which is still using some positive words, So a more traditional sentiment model would be kind of confused with this TF-IDF score or something like that. And actually, what we see here is that the analysis shows that the reviewer is not a fan. It's kind of on point with what things are being criticized, and it identifies it as a bad model correctly and a good correctly. I also tested this on one of the smallest available open source LLMs, the small LLM 135 million, which is just a few hundreds of megabytes from Hugging Face. And with that model, for example, here, it would actually think that this is still a good review, actually. So when you're working with very, very small LLMs, again, this can be kind of a challenge that some of these capabilities that emerge with the bigger ones are not going to work well. So I think RAG is a great way to kind of look at this problem space. I guess a little show of hands who is working on RAG or working with a RAG application at the moment? Okay. Actually, I was expecting maybe more of the room, but I'd say 50% of the room at least. So let's build a very simple RAG model, RAG system here, really just to kind of demonstrate how we can use this tool. And instead of doing a vectorization, we will just do a kind of search interface that we will give to the LMQL, and we will just search through Wikipedia, and we will just take relevant context like that, and we will do that as part of our augmented generation workflow. So to do this, I also want to demonstrate how easy it is to do kind of tool usage here. So we define this very simple method, which is literally just visiting Wikipedia and putting in a keyword in this URL, and then we'll take the first 500 characters of that, whatever we get as a result, and we'll call it a day as our most basic rag option. So this is what this looks like as a LMQL program. So you can see here we've defined the question at the top, like, which countries did the Norse originate from, and then we define here, you can see we're interviewing some Python code here, so this is not in quotes, and we're just, this is where the model is called, and this term is a variable, this is a generation from the model that we define here with some Conditional requirements where it has to be whatever length that's a term and then we give it the final answer So this is going to be two generations one is generating the keyword for Wikipedia and to the final answer of what happened So when we actually fill in the prompt, this is how you can see orange is the first generation So this is the keywords LMQL generates actually the small open source element generates and then based on this filled model until here you generate the rest and again we're seeing the same similar problematic behavior like questions start reappearing. It's not not bad but the answer is actually not very on point either like it doesn't necessarily answer like where they're originating from so not necessarily the best answer either. So yeah this is just a recap of what you've seen, like you can see the action statement, the term variable, then we do the Wikipedia fetch and then we get the final answer generation. So let's kind of, I like this question of how was rhetoric taught in ancient Greece as what we're trying to go for here is also kind of a multi-hop reasoning. We're trying to frame this problem more as as like potentially what happens if you need to put together information from multiple Wikipedia articles and how can you kind of put something like that together so here this is the same the same prompt that we've seen before and it's here this is another kind of interesting rag failure scenario that we're seeing is that the context so here you're seeing the model searches for rhetoric it gets a general definition of rhetoric here and then based on its own knowledge it answers how was rhetoric taught in ancient Greece but this is not based on the retrieved information so in some domains this could be considered hallucination and potentially something that you definitely don't want to do and as we're we're not based on on it's not based on our result. So next, I wanted to show like we can also combine LMQL with some, we can do some control flow, some Python control flow, which is also another very nice property of these tools, these logic bias processors, is that with these control flows you can define workflows or agents on your own. Like for example, common agent loops are potentially just a while loop and then you allow this program to break out of some while true where it can reason about something so here we're building up towards you know more and more capabilities so let's look at four Wikipedia articles instead of just one Wikipedia article the mighty for loop is coming here and we try to answer the same question and we're bringing in some some other tools here so let's let's we add the frequency penalty here so that the model will be penalized when it keeps repeating itself. So with these smaller LLMs we often see that they will generate the same thing all over again but this is still not solving our problem. So let's kind of try another approach here. So this is kind of a continuation of us just continuing to keep looking at these we're we're gonna look look for more and what we're doing now is we're generating a list of keywords and then we will we will search for those list of keywords and there's a we will look at this a little bit later why but in this for loop actually these are generate these are two generations each time ancient Greece is being generated so the for loop is generating there again we're seeing this repetitive generation problem this is going to be a common common issue that we're going to try and fight around. So one solution here is that we generate the sequence fully so that way at least the items in part of the list are kind of conditioning things. So you can see here this is the first generation, this is the second generation, this is the third generation and then there's a separate step where each one of these results, each one of these results is then retrieved from Wikipedia using a Wikipedia search tool and then we will give the final answer. So this will be working much better than the previous approach. But let's take again a little bit of a detour around, so let's we've set up like kind of the worst possible rag system probably, but let's try and think about it like how do we can evaluate this and what does good and what does bad mean here. So we're gonna use these three rag guess matrix, factual correctness, context recall and faithfulness. Factual correctness is fundamentally how correct we are with our answers, so whether we got the actual truth from a reference answer. This one is very useful but in most cases out in the real world we don't always know what the real answer is, so it can be a bit difficult to use there. Then context recall is again a ground truth based property where we look at the claims in the ground truth and we look at the ones that in the ground truth that can be attributed to the context that we retrieved. So the idea here is that if the claim in the ground truth cannot be reasoned out from what you've taken out from the vectors, then you've got a retrieval problem fundamentally. But this is also a bit challenging because you you look at each sentence in the ground truth, you check can the sentence be attributed to the context using some natural language inference, if yes you give it one, if no you give it zero and then you do an average of these. So for example here's a context recall little scoreboard where let's say we've got three claims, one of them is present in the context and two of them are missing, in this case we'd get 33%. Faithfulness is probably my favorite Ragus metric because this one doesn't need ground truth. So what we look here is the number of claims from the generated answer and we look at whether these can be constructed from the context retrieved. So this is the hallucination factor, you could look at it like that. Like a good behavior is when faithfulness is very high, in particular if you're working in more sensitive domain or you want your model to work around your company's own data, then you really need faithfulness. Because if your model is not faithful, it may hallucinate in the worst case possible. Like you could see before, it wasn't faithful, it was hallucinating things even when there was no Wikipedia stuff retrieved. That's a bad behaviour if you're doing anything security related or anything with your company fundamentally. So again, what's happening here is we look at the generated answer, we look at the claims made in that, We try and tie it back to the vector, to the vectorized retrieved context. We look if it's supported, and then we calculate the faithfulness score, and this can be, here's a little scorecard where, again, let's say there's three claims. One of them is unsupported, but most of them are, and then let's say we've got 66% faithfulness score. Okay. Context, recall, and faithfulness are actually the same metric. You just use one with ground truth claims, the other one with generated answer claims. So it's the same idea with a little twist, but definitely faithfulness is my favorite metric because you don't need ground truth, which is a bit difficult sometimes. So let's move on to actually the evaluation. Sorry for this being a bit small in this display, maybe I can. So we're going to have a little detour to the decoders, which is the final piece of the kind of structure generation capabilities of LMQL, and we're going to look again at initially, I was trying to generate a list of things before. I was generating a list of keywords, and let's take this example where you've got a list of items here that you want to take with you to the beach. we just want to see what does the LLM do in this behavior and what we see again is this repetitive behavior so again this is just with the argmax greedy token so it's really like sunscreen it really wants to take sunscreen to the beach so it keeps it's like more sunscreen more sunscreen it's like myself you know with this pale skin you you definitely need all that sunscreen so and what what we can do one idea is we we increase the temperature to the maximum and we hope that you know the random gods are gonna say favor us and maybe they do maybe they don't in this case they don't so random is not helping us then another idea is maybe we look at this kind of beam search approach where we fundamentally we're looking at multiple paths along these generation routes and we're gonna look at looking at multiple of them and we also tweak temperature to the maximum just to hope that okay this time maybe we will generate some different items as well and now we also take a bounding suit with us in addition to a lot of sunscreen. But another approach that we can take again which is something I really like about LMQL is combining Python and actually using Python to control the state. So what happens here is I'm actually just I have a set variable here with all the choices and once I've chosen an item I just remove it from the set so this way the model can't choose the same item again anymore and this way we can manage to take everything with us if we want if we want behavior like that where we don't want to keep taking the same item so this type of control flow is very nice where we can combine Python logic with this logit generators. So this is a view of what's happening with these decoders. I think it's a good way to look at it is that you're trying to get through a maze. And the argmax decoder is your greedy search where you'll just go on the, let's say, easiest path through the maze. And if you choose beam search, then what you're going to do is you're going to go on let's say all the paths available to you, you will have a look at all of them, then you will choose the two or three or whatever number of beams you look at, your favorite ones, and then you will keep going through all of those. And so that way you have a bit more chance of not getting lost in kind of a local trap that the greedy search will fall into. So if it's been a while for you as well, since you've actually seen how kind of beam search works, here's kind of a quick reminder. So let's assume you start here, and then you look at the next probabilities for all of these tokens, then you select the top two, in case you're looking at two paths, and then you keep going, you keep going, and you will look at these top two elements. So for example, if you look at this sentence here, the top two candidates in the first node are arrived and D. And then after D is higher, you will look at green and which. And then you will look at both of these, which and arrived. And here, as you enter and these paths, you will have one which arrived, or you will have just arrived, or you will have the green which arrived. This one doesn't have an end token, so actually you won't get this path. So yeah, this is from the wonderful speech and language processing, it's a great reference. So let's kind of go back now to our really horrible rag application and try and use some of this control flow. So what happens next, like we were really struggling with our context retrieval before, Like the model was really struggling in actually finding anything on Wikipedia. That was partially because it was a very horrible way of finding things. So now I gave it a nicer search interface where it can actually search for articles. Then it will get a list of articles back. And then the model can choose which articles it wants to get 500 character dumps from. So what happens here? This is where we look at these page candidates. So there's one little method for that, one prompt. Then there's another prompt here where we're going to use this control flow of actually retrieving the summary from Wikipedia for the page articles that are of interest. And here we will keep track of which pages are being searched. And then finally we will get back the pages and so this is going to be our retriever. And then this is actually the prompt which is connecting all of these together. So here you can see that here we're invoking the page search, then here in this if statement we actually retrieve. we do the retrieval through the previous prompt, then we print all of this stuff out so the LLM can actually see it, and then we will do the final question answering based on the augmented generation. So let's see what happens. So this is a question of which companies are the main contributors to greenhouse gas emissions and their role in global warming according to the carbon mayor's database so this is an Amnesty International question answering data data set that we're looking at and you can see here that the final answer is is it's a list of the 200 largest companies who are responsible for most of the greenhouse gas emissions so far so good looks looks interesting and then then let's have a quick look at what are the pages actually we managed to retrieve so some is a list of countries by carbon dioxide emissions per capita bit of a miss list of locations and trees by greenhouse gas emissions okay this is looking promising and then climate change in China maybe climate of China I don't know it doesn't sound like it's it's so useful but looking at the actual retrieved context there's you don't need to read this there's actually nothing about the companies, the particular companies that are in there. So in this case we're still actually getting this is all by the model and this is quite likely behavior because all of the open source LLMs will be fine-tuned quite a bit on Wikipedia so this isn't a surprising kind of behavior that this data would be memorized somewhat but it's still interesting that again we're seeing actually very low faithfulness here and the model would not be answering this but let's so let's take this this approach now of finding the Wikipedia pages retrieving all the summaries and then answering the questions and we build a little some glue code on it and then we import all of the ragas wrappers we interestingly ragas uses langchain because under the hood all of these evals that I show you are actually using LLM as a judge patterns so all of those evals are using you know that natural inference stuff there it's actually not using dedicated models but prompts so it's using langchain under the hood and then we here are the metrics that we define context recall factual correctness faithfulness so just to reiterate factual correctness are we correct faithfulness is our answer actually based on what we retrieved and context recall did we have any chance of actually answering this based on the context we retrieved so next let's let's look at some of the some some of the actual answers that we we gave and just to just to get a bit of feeling of what happened so here's a question what are the global implications of the USA Supreme Court's ruling on abortion and the answer the model says it has global implications but then it frames it as a you know as a multiple choice question so not not an ideal solution okay I retrieved contacts are actually quite good the retrieval is is working but we're having quite a bit of challenges this was the greenhouse gas emissions Here, this one is what actions did Amnesty International urge its supporters to take in response to the killing of the Ogoni 9. And this is the reference answer. Here there's, you know, an appeal to the Nigerian authorities. That's the reference answer. And the model kind of ad-libs it more like it doesn't really know what's what's going on though it did figure out that maybe something is going on related to Nigeria but it is not so direct but this would be actually a kind of correct answer probably this would be scoring as a kind of correct answer in our in our rag set up here here there's a what are the recommendations made by Amnesty International to the Special Rapporteur on human rights defenders. The recommendations are focused on child and young HRDs in future work, raising awareness about differences and challenges they face. And here the model kind of just says yeah Amnesty International has made the recommendation, so it kind of taught tautological answers. So we're really seeing, I wanted to just show you like these are good examples of the model's reasoning capabilities are failing again here, like you're seeing these tautological processes, you're seeing the repeated questions. So these, all of these patterns are showing you that there's some kind of reasoning challenges and we keep having issues with reasoning. So What I would take away from this Next let's look at our actual model performance. So this is how we performed compared to just running the for loop of for Retrieval questions and how faithful we are in each case and how how many times are we are we factually correct? So in this this case According to this factual correctness metric and you could see how the results were not very good but according to Rangus, we're right 22% of the time. Do you think we were right 22% of the time? You know, I would say that's a bit of an overestimation of what was going on here. So, you know, factual correctness is, because of the way the prompt is working, it's actually a bit at the threat of these tautological answers. That's a Rang evaluation challenge. Like, if you have these tautological answers, maybe they're slipping in here. The other interesting behavior is we're hallucinating like here it was faithfulness at 25% and context recall at 5%. Our retriever just doesn't have the information and that's maybe not so surprising that still we're looking at multiple Wikipedia articles but just the first 500 characters of multiple Wikipedia articles maybe won't be the right way and there is a reason we would actually use vector search for this obviously. So we're really seeing these clear retrieval problems here. But the fact your correctness is also showing a very big issue with our model reasoning. So I really like these Rangas metrics because the faithfulness and context recall are giving you good warnings on your retrieval problems and some reasoning challenges and factual correctness is really showing us that this model is kind of too dumb fundamentally. I've redone this same process with GPT-4 and it performs about two to three times better than this open source model so there's still a kind of large gap here. I didn't actually rerun this with DeepSeq R1 I think it would perform actually quite similar to GPT-4 and actually be something useful in a kind of production environment but working with these very small models I think they're still only useful for more like casting things to JSON kind of structure extraction and that type of processes and when you need more reasoning at least in my experience the very small models are not very useful fundamentally and I think this illustrates that they just don't really have necessarily the right kind of reasoning capabilities so I wanted to also give you a chance to actually play with this so I made a demo with with GPT for available on this QR code actually let me one sec let me give you a chance to potentially scan this so let me also maybe zoom in a bit so if you scan this or if you visit this URL then okay I think everybody had had a chance to do that and I'm actually a bit slow. So the URL is kinavigator.usml.net It's a previous conference this has appeared at. So the username is navigator or lowercase and the password is ki navigator as well and then you can see a basic Wikipedia search prompt here and if you tap on the example prompt it will copy paste itself into your into your phone and then you can ask a particular questions of like for example here is one of the questions from the amnesty data set that we were looking at what are the implications of the USA Supreme Court's ruling on abortion so then if you hit run prompt it will it will give you some some results we'll see how how this handles people and then here's the result where you can see the the question at the top then the action like we're searching Wikipedia for the we got row versus weight as a article then we've got some failures there's some no results and then the final answer is the information available according to GPT-4 does not provide it doesn't doesn't give us an answer that we we want to look at but this I wanted to highlight this behavior here because this is something here was the open source element there was much less of so in this case this is faithfulness working so in the prompt I also ask it like I don't want it to actually answer if I don't want it to answer if the retrieved context doesn't actually give you any evidence to what you what you're saying so GPT-4 is much better here and with the open source LLMs often there's this kind of confidence confidently wrong answer more so but if you if you change this to whatever question I don't know some other other like more straightforward question then it should potentially work and you can also change the the prompt and run whatever little LMQL prompt you want so I don't know what is the origin of the Norse that was one of the examples maybe there's some slight demo challenges here well I'll just give it a Bit of time... I don't know, did anybody in the audience manage to run a prompt by any chance? No? Didn't run? I'm not entirely sure why it's not updating to running the next one but well this was meant to be a little bit of an opportunity to see this in action but maybe some slight technical challenges there hmm yeah it did actually okay okay it's really that's really so I I wanted to kind of go back like these this logic bias processing libraries or having some kind of structure generation interface I think the key advantage of being able to do this is that you can generate pydantic objects and validate them through whatever interface you do. The logits can be very useful to generate your types. So for example, if you constrain your logits to only be integers and a dot, then you can get floating points or you can get integers. So you can do kind of simple type generation based on constraining your logits. That's how LMQL does this type of behavior under the hood. And blending prompts and code is a really quick way to iterate. I think when you're taking this code to actual production, it's better to use something more modular and with kind of more cleaner interfaces. There's some challenges with potentially writing a unit test for something like this that you saw, like, or when the code and other things blend together. But I think it's a great prototyping interface. The LMQL itself is not currently maintained. It's been abandoned for a while. But all of these ideas of using logits are something that you can use via guidance and outlines. And these are also actually a lot of these tools, for example, you can do logit biases for up to 300 logits with OpenAI. You can do frequency penalty like on OpenAI, on Hugging Face models. So a lot of the underlying API ideas are something that you can just take away with you and you them hopefully with your daily work and as you could probably tell I'm a huge fan of ragas as well which is definitely a wonderful tool in in actually getting an idea of whether your context retrieval is having challenges or whether you have some issues somewhere else and yeah so if you'd like to connect on LinkedIn very happy to connect I will publish the slides on pre talks as well and they will be on my website later so open to some questions

Speaker 2 [39:48]

Awesome Chris. Thank you very much. I think that deserves a round of applause So there are quite a few questions, I'm not sure whether we will well get through all of them But again, I guess you'll be available during the coffee break, right? So and yeah, the QR code is there. So let's start

Speaker 1 [40:10]

Um...

Speaker 2 [40:12]

Can I use LMQL to query several PDF documents placed in a folder on a local machine?

Speaker 1 [40:19]

So I think you should probably use something like MCP or something like that with file read access But if you have a method in Python to read those you could write some an LMQL could actually Then interface with your Python method. So the answer is yes, but probably not the best tool for your use case

Speaker 2 [40:41]

Thanks. Next one. Can we use this tool on text-to-image models like stable diffusion? How would you do it?

Speaker 1 [40:48]

So this doesn't work on text image, but some alternatives I believe do have some of these alternative libraries do have actually text image support either guidance or outlines does have I think Text image support, but the idea I think works there as well. Probably you can limit Your generation in some way

Speaker 2 [41:10]

Then I like this one aren't LLMs notorious for being overconfident with their decision when asked to give some score What does that percentage true slash false actually represent?

Speaker 1 [41:22]

Yeah, I think this overconfidence is a really big problem, and that's why Ragas' factual correctness is such a kind of bit of a tricky metric there. Because you're seeing that even if you answer something like, yeah, he did that, like, what did he do? And you just say, yeah, he did that, like a kind of tautological answer, as you could see, it can still score as a factually correct answer. So LLMs are confidently wrong when answering the question and they're also confidently wrong when evaluating the answer

Speaker 2 [41:58]

How did you quantify the claims and the correctness of the response in comparison to the claims in the retrieved context?

Speaker 1 [42:05]

Could you repeat the question?

Speaker 2 [42:06]

How did you quantify the claims and the correctness of the response in comparison to the claims in the retrieved context?

Speaker 1 [42:14]

So the way it works is it generally I think it splits things into sentences into small sentences And so that you can check out the ragas prompt It built it splits your your ground truth or your generated answer into Little sentences and then it will look it has a little method where it looks like does the does your retrieved context imply this so does it imply each one of those little sentence chunks so you just iterate through those little sentences and you just do a natural language inference task on it but the interesting thing there is that you're not using a dedicated model you're just using an llm for it

Speaker 2 [42:58]

One more, I would have asked that one as well, so I'm glad somebody did that for me. What would you recommend as a successor of LMQL given it's not maintained anymore?

Speaker 1 [43:08]

Yeah, so I think guidance and outlines are the two main options that are quite popular So I would recommend either of these libraries and if you're you're starting something like this today Then use these libraries don't use an MQL. You're gonna save yourself some pain

Speaker 2 [43:24]

A question for me. So why didn't you present guidance or outlines instead then so I

Speaker 1 [43:28]

So I used to, when I co-founded my own business, I used to use LMQL to run a lot of prototypes myself. So I invested a lot of my own time in building rack systems with this, a bit more production rack systems. And I was just way more comfortable with this tool.

Speaker 2 [43:47]

All good. I'm also going to present a library which is no longer maintained. And final question, could you compare agents and LMQL? When would you use each one or even both at the same time?

Speaker 1 [43:59]

So, one way to look at it is LMQL, and these tools are actually the way to implement agents. So, I was hinting at this, but you could look at all agents are fundamentally programming control flows. Often an agent is either a while loop, or it's some way of maybe a while loop that also has some memory, that it can also send information to another while loop that has memory. And if you look at it this way, then these are building blocks for agents. That's one way to look at it. The second way to look at it is that I think agents in my work are not so useful. Workflows are more useful. We generally want to do things in a repeatable way where we can kind of have a measurement on different stages of the process and we know whether it's working or not. And an agent type system is a bit too random. There's a bit too much uncertainty potentially for my use cases. So I prefer workflows, and this can help you in setting up workflows. But it really depends what you mean by an agent. Is a kind of browser capability an agent? I don't think so. I think that's just a tool use. I think that's just for a workflow element. I think there's a lot of branding and marketing hype around agents. And I think, in general, workflows with a lot of tool access are kind of indistinguishable from agents.

Speaker 2 [45:24]

Awesome. That was it. Very efficient. Well done. You went through all the questions. Thank you. Thanks for listening.