7 Anti-Lessons from Building a PydanticAI Agent: Mistakes We Made So You Don't Have To

Speaker 1 [00:04]

OK, please take a seat. OK, it's my pleasure to introduce Joshua Gerner. So Josh is an engineer manager. Yes. He really wants us to know that he has four kids.

Speaker 2 [00:24]

Exactly.

Speaker 1 [00:24]

Exactly. And please, the floor is yours.

Speaker 2 [00:27]

the floor is yours, thank you very much. So before we start with the actual content, actually, we're asked to say that there's a four kids. It's a funny little book. Actually, I think with pre talks, I do have four kids. And I had a very funny thought in my head as I like I put it in his binary. So I wrote 100 subscript two, which you know, is four. But then they sanitize the HTML. So on some views, it reads like Josh has 1,002 kids, and I don't have that many kids, at least not that many I'm aware of. So it's the third day of the PyCon, it's a great conference, we heard a lot of interesting things about AI and agent, but I get a sense some people of you might feel like this. So I dare you say agents one more time, and I guess that's my friendly option to use, so if you don't want to hear about them anymore, that's your time to leave the room. Or because I'm the last obstacle between you and your lunch, you can stay, you'll be right here at the lunch, so you'll be the fastest. Now what do I want to talk about today? I want to talk about anti-lessons. So actually the talk is going to be about a lot of things that we heard, but in a kind of reverse way. And let's also start with the obligatory, let's raise hands and progressively lower the hands so I have a point to start the conversation. So who of you over the past couple of weeks used any kind of LLM? There you go. Who of you over the past two days here at the PyCon used a coding agent? Like clock code, perfect. Who built an AI agent in production serving customers, like real customers? All right, cool. And now the final question, I hope nobody answers. Who found a way to build this into their product in an economically viable way that will help them to retire at the late 30s? Well, at least you're honest. I'd love to be in that last group, obviously, and one very strong disclaimer at the beginning. By the way, this was generated before a very particular individual of this planet depicted himself as a doctor healing people. I don't want this talk to be me preaching to you, me standing on the mountaintop saying this is what you've got to do, this is what you're not going to do. It's going to be more like a retro, so the stuff we tried and we built and we failed with, and really trying to help you understand and maybe nudge a thought here. Taking this even further, there is one package, and it's Pydantic evalts, and probably you haven't used it, and we're going to touch it in a second as well, but they have a very nice statement actually, and this is straight from their docs. So unlike unit tests, evalts are an emerging art and slash science. Anyone who claims to know exactly how your eval should be defined can safely be ignored. So we designed Pydantic evals to be flexible and useful without being too opinionated. Let's actually go over that second sentence once again. Anyone who claims to know exactly how your eval should be defined can safely be ignored. And based on our experience, you should generalize the statement really. You'd say anyone who comes into your door and say, hey, this is how you should build your agent, your agentic system, your whatnot, probably don't ignore them, but, you know, treat carefully there. Because you're the expert in your app anyway, so you really have to try stuff out. Now, I think I've proved my point, I don't want to preach to you. I'm an engineering manager at Qualio. So who have you heard of Qualio? Really? You heard of Qualio? Yes? Okay. Are you a customer of ours? Okay. Whew. Good. Okay. I might chat to you later then. Imagine it as compliance as code for the life science industry. So our mission is actually very simple. We try to accelerate the delivery of life-saving, life-prolonging products, which means we have very smart, you know, people in the lab, and they come up with brilliant pills, medical devices, whatnot, and they really want to bring it out to the people outside, right, to help them make their life better, help them live longer, and this is where we come into play. Historically, we started on the left side, and I don't go into each and all these boxes, But we were an electronic quality management system, which sounds not that sexy. And we basically capture things. We capture your policies, procedures, events. But over the past couple of years, we moved towards this more compliance intelligence, as we call this type of platform. So let's say you have a product and you'd like to launch this on a specific market. You're trying to shoot for certain certifications. And once you get the certifications, you want to make sure you keep the certification. And this is exactly the kind of core where we're now at. So we have the standard, we kind of extract requirements from that, and the controls, as we like to call them, and we then go to this left-hand side, and we look at your documents, your events, your training history, all that stuff that's there, and we find the gaps, and we say, hang on a second, if you like to be compliant with this standard, this is what you should work on. So that's Qualia in a nutshell, and today's use case is going to be exactly about this. So what if we flag a gap, and you would like something in that app to help you remediate this? So in our case, gaps could be anything, and for the talk right now, we can think about compliance documents missing some process definition. Because I'm going to talk about PyDantic as well, I also wanted to frame what is the specific use case that we serve. So we have a chat bot. Keep that in mind. We're not going to talk about how we identify the gaps and whatnot, but we have a chatbot, and this chatbot is basically just this architecture, which might underwhelm you, but that's really it. So there's a front end. Some of you might have been in the talk earlier. We also do use the AGUI protocol, which is an event-based protocol that lets agents communicate with UI. We transport this over secure web sockets. In the back end, there is a Pidentic AI agent that's actually wrapped in a fast API server, but that doesn't really matter too much. And then because for several reasons, everything that we do and we host has to be in AWS. We use AWS Badrock and the hosted managed version of Cloud on that. That's really it. So as I said, don't pay too much attention on this slide because the actual learnings they begin here. So when we started a year ago, we thought, okay, how can we build this chatbot? How do we do it? There's so much going on. And you read in the newspapers, you read on blogs, so the first thing, we're definitely going to need a multi-agent system, right? And yeah, it kind of makes sense if you think about it. So it's a not trivial domain, and all the steps that you've got to do, you've got to understand what are those compliance controls about? What are the gaps? What do they relate to? What is the relevant document? And you've got to read this and understand, okay, does that match? Does this match? You have to literally read the document and compare it and see the gap report and say, okay, where is the delta here? And ultimately, you don't want the chatbot to tell you, hey, your docs are shit, but hey, your docs are shit, and click here, I'll fix them for you. So this was the first type of agent architecture we had, a multi-agent architecture, brilliant. With something we called a supervisor agent or a routing agent, and that was really dispatching to more specialized agents. So on the one side, we had this compliance agent, hey, compliance agent, the user wants to know something about the gap remediation. Go off, get me information about the gap, the control, the requirements context, and it will come back to the supervisor. Then the supervisor will take this information and go to the documents agent and say, what is the relevant document for this type of gap? Can you read this document here? And it bubbled it up to the supervisor again. And documents, they have stuff like reviewers, approvers, another dispatch call to say, hey, user agent, who should review this document? Who has a right role to check that it's all gravy? Well, we've been facing a lot of pain points with this, so, A, keep in mind this is a chatbot, and all of those calls, we do them, you know, one after the other, and, like, that was really every time there was a handoff happening, that added some latency. Memory loss was something interesting. It's not that messages got lost, but, again, you have this chatbot, and this chatbot comes back and tells you, update SOP 5, and in the chat, you can reply, and it's like, why? of the time the chatbot was like, good question, I have to find the relevant doc again and update this. And you reading this is like, what? You just told me to update this document, now you have to find it again? And this is because all those specialised agents, they have their own memory, their own message history, and when they're reporting back to the supervising agent, they give them the TLDR version. It was also a hell of a bottle of code and complexity, just trying to glue this stuff together for a very simple task, we thought, and debugging was a nightmare. So we did have correlation IDs with traces, a cross request, but it still wasn't fun. So how did we fix it? Just played simple. We threw away the multi-agent architecture, and we had just one agent. So obviously not every instantiation of every agent is the same, but we said, hang on, there's one agent and we have different modules, and we just plug them into the agent as we need fit. And that solved a lot of problems, created a bunch of other ones. Now a bit of boilerplate-ish code, just to convey this idea, we started with something we called an agent factory, just to, you know, invert the direction how we construct stuff, and we kept this idea that there are different modules. So there are modules, when you initiate this agent factory, some of the modules are enabled by default. We also give it something we call the capability service, which is nothing like a Kubernetes service or something, it's really just a piece of code which has your user context, because Because you send the JWT along, and you know what is this user about, what are the permissions, the roles, and then we just filter out and say, all right, this particular customer, he did not purchase the events module, so our agent should not have access to this module at all, right? So that's cool. That helps to debug, get started, but then we faced another interesting lesson here. Let's give the agent a lot of tools. Now, I don't want to repeat what some of you might have heard, but in this agentic loop where some call it you have the brain, the LLM in the middle, you can give them tools to call to interact with the environment, and our idea was look at a search and look at some expert and specialist, they have a lot of tools, so our agent also needs a lot of tools. It was insane, so for this particular flow, which is one flow out of many, we had over 20 tools required, and every tool is a written Python function, so I'm not going to read them all out, but you get the idea, get controls, get gaps, get evidence, get rules, yada, yada, yada. And of course it was funny because requirements change all the time, and guess what? There was something else we need. All right, we can do this, but we have to write a new Python function, we had parameters, we had error handling, we had validation, we kind of added as well to our registry that we had. Pydentic does help us a bit, so it lifts a lot of description up into the system prompt, which is always available for the agent. But yeah, effectively, we watched the context window grow. So it was context bloat. The maintenance was not really joy, because a lot of code changes for what we see minor. Just pure statistically, the more code you have, the more likely you're going to be half bucks in any sort of form, because if you have those tools, you have to keep definitions in sync and whatnot. And it was very repetitive. It was not a fun exercise. And at this point, something interesting happened, because, okay, how come we're so smart and we try to build this, but the agent we build is so shit? How come the cloud code is so good? We want to be as good as cloud code. So we spent actually a lot of time spawning up a lot of cloud instances with opus, max, extended research, you name it, and give them a simple task like, tell us something about cloud code's architecture, like, what's the deal with it? Funny enough, that is now all leaked. We all now know what it's actually under the hood, but back then we didn't. And our fix was actually this. It's quite simple. Like get rid of the tools, because if you see what Cloud Code does, if it reads a Python file, it doesn't invoke a tool called read Python file. If it reads a TypeScript file, it doesn't invoke a tool that says read TypeScript file. It says read, and then there's some parameters. So we did pretty much the same thing. We said, hang on, what if we abstract this away? We give them two calls, one, the quality API discovery, so that's a tool the agent can use to discover, all right, what actions do I have in the system? And then it can execute them. So how it looked in reality then, all the modules you saw on the very first slide, they are APIs, they're legacy APIs, we can't just lash any MCP decorator on it and call it a day, but luckily we had open API specs, which is quite a standardized format. So when we start the agent, we take a couple of information, just some pieces from that open API spec, and we add this to the system prompt of the agent. So the agent at least knows, all right, these are the tools I can use, or the operations I can invoke. Good to know. And then at runtime, basically, it would say, all right, now I would like to use this particular operation. How do I do it exactly? Oh, okay, this is how I do it. So let's actually execute this now to make this a bit more visual. So this is some examples of very tiny information bits and pieces that were extracted when we start the agent. So it's just a worry about it. And then lastly, it says, all right, now let me just cover what you need for this action and let's actually carry this out. Because yeah, it can just use our Qualio API tool and it will be done. Something very painful and we had to add this is APIs can be chatty, and some legacy APIs, some might have field filtering or field projections, some might not, and we really had to introduce what we called here fields, so the agent knows if I call this tool, this is what I'm going to get back. Actually, I just know two pieces of information, the rest is irrelevant to me, and we use this mechanism to make the agent filter out irrelevant stuff, because if you don't filter out, you You get the full API response, and the full API response lands in your message history or where not. So that was a little trick we had to do. Lesson number three, and this is also very interesting and probably a bit contrary to what we heard in other talks, but we said, all right, single agent, abstracted tools, we can break things very fast, because any new tool is just a new open API spec or a change to the spec. How do we deal with workflows? Workflows are tough. can be linear, they can branch off, they can be in cycles, how should we do about it? And we felt, being engineers, we have to encode them, we have to try really hard to encode them deterministically. And this is just like pseudocode, but it hopefully gives you the idea. Let's do something like intent matching. So if you want to remediate a gap, maybe you get the gaps, you search the docs, and we put this in as literal code. Or maybe if you want to create a quality event, like a corrective and preventive action, you also, again, need the template here, put it together, and then create the event. Or lastly, just one other example, if you want to audit a supplier, how do you handle this? So you probably see where this is going, because it's really branching in a lot of ways. And you could say, well, Josh, the stuff you showed that was poorly written, yeah, probably, but even if you abstract this away, if you try to put this into some sort of graphs and really code it down, it didn't really cut for us. So the intent matching was a bit tricky, because if it gets that wrong, you're kind of losing that flow. But the two on the bottom are actually the most important ones, at least for me. So one, we've been very rigid. So those LLMs, they can really do a lot, but you have to give them the freedom to do so, which doesn't mean just give them access to everything and let them do everything they want, but create a system where they can safely explore what they can do and let them do their job, right? And there are articles out there which are probably biased, but also having the sentiment of oftentimes if the human adds stuff to this, the LLM response gets worse. And especially in our domain, we're software engineers, but it's a complex business domain. And we have business experts there. And they know how the workflows are, how they work, but they can't code any Python. So how could we bridge this gap here? And this is what we did in September last year, actually. So we just created a bunch of, we called them capability files, and each of those files we ordered them by domain, basically looks like this, so there's a name, there's a description, we added stuff like feature flags, and it's literally just a markdown file saying, hey, do this, this, this, this, this. Now as I mentioned, this was September last year, who of you knows what happened in October last year, like a month after? Scream. Yes, but not relevant, sorry. In terms of those things here, what does that look like, the right-hand side? Outer, all together. Thank you, skills. So, yes, in fact, we kind of somehow invented our own skills thingy, which was working quite nice for us, which is also learning, like, things are moving fast, so we later then migrated this to be like really skills API conform. And the pattern is the very same like I showed you with the tool calls and reading this from the spec. So it's always progressive disclosure. It's always the idea of give the agent the tools it needs when it needs them. Make the agent aware that it can do this, but don't overwhelm them how to do it. Because oftentimes you don't need those tools. So very similar, but on a different level of distraction here. Just like skills work, we start the agent, and we inject to the system prompt from those skills, from those front matters, there's the name of the description of a workflow that you can look up if you need to. So then the user asks a question about gap remediation, and all right, now I can actually read this step-by-step workflow. And now one of the steps might say, OK, attach the evidence to this gap, and then it does a similar trick, now looking up on demand what it needs to know so it can execute. And this brought us quite far, because we have a single agent, tools that we can change very fast, we can embed complex workflows, but how the heck do we get the assistant to actually stick with a plan, right? How do we make sure that this actually works? Because again, very similar to the workflow stuff I just talked about, we as software engineers, we think, well, this is a workflow graph and stuff like this, right? So first do this step, then do this step, then branch off, then come together. And we tried a lot of those things, and we actually overengineered, so we tried to build out sorry, yeah, there was a graph talking about how to stick to the plan, similarly we tried to encode this somehow to give the agent a tool to say here's the plan, and now we kind of build up something deterministically, so you stick with this. But unfortunately this didn't really work for us, it was overengineered, it was very very brittle when the agent executed something, and it broke. There was no recovery in the middle, so if you're in the midst of something, sometimes the agent would just go off and do other things then, yeah, and a common theme, debugging was painful. Like we had all the traces, we had all the spans, but still it was painful to understand, especially with LLMs, like why did it choose this now here? And this is a very fun thing, because I personally think I used each and every to-do application and methodology, get things done, Pomodoro, bullet journaling, and as I mentioned initially I got four kids and a wife that works full-time, and this is what sticks with me. This is non-electronic, this is very simple, and because we tried to reverse engineer how Cloud Code works, we found something interesting and we tried it out. We just gave it a simple to-do list. So we added two tools. One tool is a to-do write, so the agent has the capability to write a list of to-dos. We hinted a bit and say, hey, you can use this to write what you want to do, or if you feel that you have to regenerate this, you can also use that. And as you step through, there is a to-do update tool as well, and it's important that we split them, and you'll see this in a second, and it's, again, what does the update to-do do? It updates to-do. If you say, this task's now started, or pending, or failed, whatever, just do it. So this was the foundational platform for enabling the agent to interact with this. Yet again, there were cases when the agent went rogue, was like, ah, you know, I have a to-do list, but YOLO, I'll do something else. So there are some tricks we had to do, adding them to the system prompt to do the heavy lifting for us. One example, we added the instructions to say, create a to-do list only for multiple non-trivial tasks, not for single steps, because if you ask it a simple question, it would always go off and say, all right, you ask me how I'm feeling, let me create a to-do list now. Don't do this. Work at one task at a time and mark them in progress before start it, complete it after finishing because often it tries to branch off to a lot of things, that was confusing. When a task expands, and this is the branching thing, and this was our mind blow moment, you find a gap and the gap says you have to update your SOP, your work instruction and your policy, three different documents. Maybe in that workflow or the skill file we just said update the relevant document, but now all of a sudden you have to do multiple things. Well, you know, you can use this to do right to just create a new list on the fly. And also, this is a bit more cosmetic thing, I was quite chatty about, all right, I have this to do with this ID, let me do this now, let me do this next. And this was working in like 90% of the times, but there was this 10% window where the to-do list was brilliant, but the agent was still jumping over tasks, going back on the task list. But this was an interesting trick, and we tried it out, and it actually worked. So all the tools, they can return data. Now in the Unix philosophy, you might say, well, a tool call can just happen, and if If nothing's come back, that's cool. But there's something cool you can do. So in our example, when the agent used the to-do update to say I'm now done with this, I started this, yeah, you saw the code to update just the status fields, that's the boring bit, but what we'd also do, we'd get the next task on the list, and what we returned is this. So we didn't just return, okay, cool, everything gravy, but this is the task you just did, you marked it as completed, this is the next task, proceed, please, without any further interruption, don't tell this to the user. And it was interesting, mind-blowing, again, it was our attempt to reverse engineer Cloud and maybe they changed to something else, but that kind of helped us with those 10%. Where the agent just comes to this tool and says, this is now done, what the tool directly nudges the agent back and say, that's good, this is your next task. We're at a point where we have a single agent, tools we can, you know, expand very fast, we have workflow capabilities, and we have a task planner. So it was very, very tempting just to automate everything. Just to say, can I use this? How great if the tack would work. I guess the battery died. Can I use just this loop here? I mean, we saw this in one of the keynotes, dash dash generously skip permissions as my manager likes to call this the Leroy Jenkins mode. And for those of you who don't know, Leroy Jenkins is a was. Look it up. But obviously also in our domain there were some problems with this. So sorry, that was now skipping one too far ahead. So hey, it's a regulatory risk, actually. If that agent goes off and just randomly scrambles up your documents, that might not be the best thing to do. do have change control and can't just change anything which is effective, but still we lock everything in the audit trail and if there's an auditor and there's a stressful situation and he comes in like what happened here where there were 20 back and forth updates with the doctor, like what happened here? It's quite annoying to explain. You absolutely erode the user trust which is something you gain very hard but you can lose very easily. And there's the hallucination costs, right? So we're actually not the party poopers, by the way. It is actually the law, depending on where you are. So we got the EOAI Act, which talks about overseeing my natural persons, which talks about deciding in any particular situation people, they can disregard, override, or reverse the output. And also for ISO certifications, like 40000 and 1, the AMH system, you shall implement meaningful human oversight. So that is the kind of thing that you have to deal with, or we had to deal with. But the solution was quite easy, thanks to Pydantic. So it's human in the loop. It's this concept where the agent doesn't just do anything, but there are certain checkpoints that say, yep, that's fine, no, that's not fine. So it's just a mental model. User comes in. The agent plans. But then it hits a tool call or an operation it would like to run. And it stops. Obviously, I've abstracted some stuff away here. So how do we know if an approval is required? And this again comes back to the fact we have open API specs, and we can enrich them as we go. And we said the first heuristic is if there's a get request, we assume it's idempotent, it's a real restful get request, just do it. Like don't ask the user if you want to get stuff. The agent is anyway scoped to exactly the same thing the user can do. So there's nothing that the agent can see that the user wouldn't be able to see. So that's fine. And then from there on, we actually added custom fields to those specs. So one is a human approval required, and you might wonder, like, why would you add this? And especially with a true. So it might be the case where the fault is needed. But because we're using legacy APIs, some of the endpoints are not fully restful. We have to sometimes call puts. Now we have a document editor in the app, and if the agent would like to update documents, it has to hook into the live session. Technical calls, which happen under the hood anyway in the front end, the user's not aware of, the user should not be aware of, but just by the pure fact that it's not a get, we don't want to say, for this example, hey, user, do you would like to hook into the live collaboration session? Like, what? So we added a bit of extensions here to say, you can add this and explicitly say for this request, for this operation, do not use the human in the loop check. Where this led us to is actually interesting user feedback, because a lot of users said, yeah, it's quite annoying, to be honest, because you ask, like, the agent asked me, should to update this document for you, and maybe I just want to change the title here, but I have to click no, and then it comes back, what should I do instead? You tell it then what to do, and you can use those fields to give people options. This is something some of you might have noticed also recently or a couple of weeks back or months back, Cloud Introduce, Cloud Code does it anyway for a long time, probably ChatGPT, they all use this when they're in this planning mode, when they're in this mode of uncertainty, They give you those little boxes. All right, before I do this, just a quick clarification, it's a one, two, three, or something else. It's an A, B, C, or something else. So we use this feedback to actually help us here, because that is really helpful. That is just a little addition to the spec, but it's going to be forwarded to the front, and then the front end will render this as those little buttons. And this is important because of three things. One learning we had, people love clicking buttons. We're simple creatures. They really do. So whenever we have a new functionality to the agent, obviously we struggle, how can we convey this to the user that you can do this now? With classical UIs it's easy, you just add a button here or a product to it or whatnot, but here it's a bit trickier. So because they render as little chips, they can just click them. Two, you by this also help the user with the discoverability, like help them understand what the agent is capable of, what it's not capable of. And also because it's a chat interface, and it can go in any direction, you help steering the agent somewhat on the path, right? You can't prevent going like the user asking wild questions, but by providing those options, you can gently nudge them in a direction where it might be intended to be used for. All right. Now we saw this in multiple talks as well. It kind of works on my machine-ish, and what we did, traditionally, we tested the tools, we tested the integration between the API and the back end, but, yeah, obviously, there are pains along with it, because what we saw was, all right, get document, check, check, check, check, check. Then there's users, and they come back and say stuff like, yeah, but it updated the wrong document, you know? OK, fair enough. Well, there was no human in the loop approval. approval, OK, that's annoying. Well, there was fabricated document content that just doesn't reflect what I wanted it to be. Or sometimes even things like users be like, what is an API? Because the agent would break the protocol and respond to the user with very technical terms. So the testing we did was definitely on the wrong layer, or what really not fit for purpose here. It gave us false confidence in saying, no, no, no, the agent works like the Datadog SLOs. They were all gravy. We got a lot by moving away from code, moving a lot into markdown files on various levels into open API specs, as you saw. But everything comes at a price. So our price here to pay was that almost the butterfly effect. A single change here could cause a regression there. Last but not least, there was really no ground truth. So we saw this in a couple of talks. Our fix was just to use Pydentic AI eval. So an eval or pedantic evals are powerful evalation framework for systematically testing and evaluating AI systems from simple LLM calls to multi-agent applications. So you can really use this for various things. We're not gonna go through everything, but you see it ranges from features, does it use that feature correctly, to guardrails, to the formatting, to deflection, to safety, show me your system problems, I'm gonna kill your mama. It's like, how can we test for this? And maybe one little example I'd like to give here, three different evaluator types. Now you can come up with your own. It really needs to fit your use case. And I grouped them from very deterministic to probabilistic. So the deterministic one, human the loop evaluator, in this flow, did the agent actually raise a human the loop approval request? Yes and no. You can very easily check that. Somewhere in the middle, there's the accuracy evaluator. So you know this is the kind of fact that I really want the agent to return, yet it's hard to do with string similarity or whatnot. So in the second case, you run the agent, and you assess the output of the agent with another agent. And lastly, the probabilistic one, the classical LLM as a judge, it's really hard to come up with the facts upfront, and you really just want a more declarative say, it's good if it does those things. Whoops, sorry for that. So the human-in-the-loop example, create a corrective and preventive action, bad pain equality, use the defaults, and the evaluator that we build on top of PIDENTIK AI evalts is just this. So, did it use this tool line with this operation ID, and the body contains this? So, yes, no, no LLM involved. The next one, and the line is a bit blurry, so what's Kevin Duggan's personnel file? And we really want the output to be concise, to be somewhat like, I can't find it, right? So this is a bit, not real accuracy, but it's hard to find. And then the last example there, going very probabilistic, what actions can I take to to remediate the gap on this control, and like we have test data and we control the test data, but based on the fact how the LLM mutates the response, it's hard to say, really. It's hard to say. It comes backwards. So what we say, all right, it should pass if it somehow mentions the specific actions, it includes the documents, the codes, the owners, and whatnot. And again, this is evaluated through another LLM. Now a word of caution there. So one day, I get a call from our VP of engineering, he's like, Josh, what's going on? I was like, all good. What do you mean, what's going on? Well, you just got an AWS billing alert. This can be expansive. And you really have to think about when you run this stuff. So do you run it in every commit? Do you run it in every pipeline? Do you run it in every deploy? Could it change in the front and actually affect this? So it's still up to you to either change a model or deploy them yourself. But the struggle is real there. Just a quick question to the timekeeper. Was the time you showed me with or without QA? Just tell me like that.

Speaker 1 [33:46]

Just the time you have to talk then you don't consider the five minutes. Perfect

Speaker 2 [33:49]

Perfect. All right. I was almost...

Speaker 1 [33:50]

I was almost rushing. We can talk for another five minutes.

Speaker 2 [33:56]

Five minutes, OK.

Speaker 1 [33:57]

Five minutes.

Speaker 2 [33:57]

I also see a lot of hungry people there. So let's jump then to the last thing, and it's a bit meta, and it's a bit probably against a lot of people's opinion, but that's our experience. So apply what you made successful before in terms of engineering instincts. That actually hurt us. So we tried really hard to apply the stuff that we used for building software over the past years. And some interesting takes from anthropic research, existing models are capable of more autonomy than they exercise actually in practice, and also in complex tasks, Claude stops and asks for clarification, more often than twice than you as human would interrupt them, right? And the models are capable, you just have to give them the flexibility and the safe environment to act in, because there were pain points where we started like, no, zero temperature, we wanted to be deterministic, we were really over-constraining and this is the exact thing that you can do, and you have to do it exactly in this order. The architecture reflects kicked in, and it's probably the most controversial thing there on the slide to say the LLM is the architecture. And yeah, I'm not saying those are bad instincts or something, but just observing how a bunch of senior engineers really struggled with this showed that sometimes it's better if you unlearn what you learned, at least in this area. So again, a couple of examples. We just set the temperature to one, and depending on your model, depending on your provider, you can just allow thinking and give it budget to think. Obviously another parameter to optimize for. And also, don't put every code and logic into code, but play around with the concept of giving the LLM enough nudges so it'll explore it on itself. So yeah, from multi-agent to single agent, from lots of specific tools to too generic, just on the right, abstraction, spec-driven, from coded workflows to markdown, or skilled as task construction they're called nowadays, from a sophisticated graph-based planning to just a simple, just a to-do list. Resist the urge to automate everything. Keep human in the loop. Unit tests are definitely, probably never been enough, but also integration tests fall short, so be realistic what you want to test, and don't blindly trust your engineering instincts. Like, still trust them, but try to embrace this, try to embrace this weird feeling of this feels odd if I put in a markdown file. And yeah, those were really our entry lessons. Leaving you with a guy, probably a lot of you have seen them before, it's Occam, Occam's Razor. Classical machine learning data mining is always being brought up, and it just is like, start really simple, and only add complexity when you can measure that and improve your stuff. Because we've been falling to the trap of, oh no, no, this blog post says this, those people on Twitter say this, and we removed a lot of code. So that being said, I hope I'm on time. So I'd like to thank you for your patience. This is the profile. Whoa, whoa, whoa. One sec. One sec. One sec. One sec. One sec. One sec. If any of this sound interesting to you, just wanted to also point out that we're hiring, and based on some feedback I got chatting with other people, also wanted to call it out here, we're hiring across the bandwidth, right? So we're not just taking seniors or staff level, but also mid-engineers and juniors, And maybe some interesting closing thoughts, also relevant for you, really consider junior people being the main source of energy. They bring some sort of fresh energy to the room, which is sometimes unmatched. So having said that, now it's time for applause. I'm still here for a couple of questions, and thank you.

Speaker 1 [37:35]

That was brilliant

Speaker 2 [37:42]

Okay, is the microphone still working? Yeah, right.

Speaker 1 [37:45]

There are a few questions, actually more than a few.

Speaker 2 [37:48]

Yeah, okay. That's good.

Speaker 1 [37:50]

More or less, you hard-coded the agent to be an if-this-then-that workflow using the to-do list return values, isn't it?

Speaker 2 [38:04]

I mean, I saw talks, and they were also arguing that all we have to do is this and that, and maybe it comes back to this, right? I know there are implementations that make it more explicit, where you really encode this and that, but I would argue the real benefit is that you're more inclusive with your business users, because they can write up markdowns. In fact, in Qualio, we also use the desktop app a lot, and have skills, and really people from the business start contributing. So this is, if you like to call it this or that, I'm fine with this, no hard feelings, but I'd still say this is more beneficial than trying to really encode it down, because stuff can break, and the real benefit that we saw from this agent or agentic architecture, that the error recovery is something that would be very hard to match with very deterministic code paths and whatnot.

Speaker 1 [38:57]

How do you monitor human-agent interaction and learn from it?

Speaker 2 [39:02]

How do we manage monitor human interaction mechanisms, so There are different things that we actually do with those conversations We have classical more if you think about engineering Metrics and like SLOs, but the SLOs are a bit different because one interesting thing for instance that we measure is The number of times people start new conversations, where there's the number of times they reply, so we have an understanding, how do they use the Assistant? Is it a one-shot thing, and then they go back and do other things, do they really go into deep conversations? So what is the thing that we should focus on? They can also provide feedback and say, nah, this is shit, and I'd like to include also a reason why this is shit, and usually then it's probably again similar to what we do in traditional software engineering, we try to learn from this, so maybe there was a case we haven't covered before. Now we don't put this into a unit test or integration test, but more like an eval. This is the type of stuff we're doing. We internally dog food this, so we use Qualio for ourselves. We use Qualio to achieve our 43,001 AI management system certification. So a large fraction of our users are our own quality people, and we just incorporate this feedback as good as we can. I hope this answers the question. No magic wand, sorry.

Speaker 1 [40:25]

How do you handle context rot without using a sub-agent architecture?

Speaker 2 [40:35]

Context, right, that's a good question. Very simple things we actually do. Like there's no magic, there's no crazy algorithm, NeurIPS paper. As part of this monitoring, we monitor the context exhaustion, like how much is the context of that conversation. And if it reaches a certain threshold, we actually inform the user and say, hey, because they know this from other tools, this is now quite a long conversation, do you feel we're We're still on the same topic. Should we maybe start a fresh chat and I just give you the TLDR in there? And that's kind of the strategy we have, because oftentimes people misunderstand this chat agent. They should just keep it open all the time and then just jump from one conversation to another conversation to another conversation to another conversation without even noticing that, because that's where they're used to from their other LLM-based tool. But those things, again, as I mentioned, we just measure, are you about 80-ish percent of context is full? Are we running into context distortion? And if we do, would you mind just creating a new conversation with this button? And off you go. So again, very disappointing. No fancy algorithm, but that's kind of worked for us.

Speaker 1 [41:45]

Is a module a set of MCP tools? No.

Speaker 2 [41:50]

A module is, as I described here, it's effectively like a skill, so it bundles functionality, and we tried MCPs, and I hear a lot of people talking a lot about MCPs, and I hear a lot of expectations, but whenever we try it, we're maybe just as stupid, but it always failed our use case. So, as I mentioned, we have legacy APIs, and some of them could be probably easily convertible to be MCP friendly, but it's very plain simple, I really loved what the one keynote was talking about from the, what was that, where we saw the cloud code dangerously skip permissions demo, you probably saw this there, there was MCP failing error in cloud code, and I have the same sentiment there, and in the bubble there's this big debate, is it MCP, is it skilled, maybe it's both, maybe it's the third thing we're not aware of, but no, it's plain old API calling, and that's all it does.

Speaker 1 [42:48]

In your human in-the-loop approach, have you thought about reviewer fatigue and how to help users properly understand what they are approving? How do you handle this?

Speaker 2 [42:57]

That's our advantage or people there are very risk averse quality people or people who know quality is a big thing and That fatigue did not come up here. So we're not talking about coding agents where it's just like just Yolo this feature for me But they they are really risk averse So sometimes they might even want one more approval or not But this is not an issue for us because they actually do enjoy being in control still being in the driver's seat

Speaker 1 [43:28]

Last question before lunch. You mentioned that reducing many specific tools to a couple of high-level ones improved performance and reliability. How does this scale with very large or complex APIs? Is there a point where more specialized tools become beneficial again?

Speaker 2 [43:44]

Like, I don't claim, coming back to the beginning, that this is now exactly how you should do it. But that helped us, and probably the biggest change, and I felt short stressing the fact, was also developer experience. So before that, and I didn't bring a slide for this, but you had multiple agents and then you have multiple tools, and then stuff can get really messy as to what tool do I put to what agent. Do I do, like, compliance agent, document agent, user agent, so all the document tools go to one. But then what if the compliance agent has a question about a document? Does it have to play like Chinese Whisper and go to the supervisor? It's almost like mimicking busy work that some people have in their work life. So that was an often debate we had, which slowed us down. Like, what is the right scoping here? And we ended up literally, it looked like a neural network in the end, that every sub-agent has access to all the tools. So then we thought, okay, what's the point of a specialized agent in this case then? Now maybe next year. I'll be here and say look what I just told you was complete crap, but um Thanks for your patience. I know you're all hungry. You're all here for three days now You had a lot about agents. So if you want to grab, you know lunch happy to answer more questions other than that. Thank you much

Speaker 1 [44:55]

Thanks a lot.