Building Reliable AI Agents for Publishing: A DSPy-Based Quality Assurance Framework

Speaker 1 [00:07]

Hello, thanks. So, yeah, the name is long. A lot of words, agents, publishing, DSPI, framework, assurance, and so on. So let's start about our journey. So I like the touch, the personal touch from the previous speaker. So I studied here in the electrical engineering department a couple of years ago. So I also did the PhD as you see but in engineering as well So I decided on the to stay on the practical part as far as the AI goes not the other things and yeah, and in the last seven months now we have a system in In a newsroom in Germany, which helps them with producing text in there for them Cutting editing videos doing some audio things as well and we'll want to share with you Our experiences, so basically also our journey, how to assure the quality so that they don't dump us as a customer and really remain and work with us. And we deliver the value they need for their work as well because, as you will see in a minute, they have quite high requirements for AI systems in their work so that we really satisfy that over a long term of time. so short overview i'll do a short introduction about what we do and what are the requirements from the publishing side regarding the ai systems then okay our focus is actually able to test basically evaluation driven development so how we how we think about evaluation of lms and different aspects in our work then i would introduce the spi which we use as one part of a bigger system and I think it should be used more broadly, so therefore I would like to share some examples with you so you can have a clear understanding of how you could apply it for yourselves, and then I would like to show a little bit also about some key aspects about the reward implementation in this newsroom and some general conclusion at the end. So who we are in TZXIO, so we founded the company four years ago during COVID time, and we started all of individual development and other services, but our main focus is on the right. So basically we have this co-pilot created for journalists and newsrooms, editorial teams to process the text how they want it, add the video to that, and also audio features as well. basically to allow smaller publishers to become a multimedia house coming from the print side where they are right now. So enough marketing, challenges of what are the features of LLMs and AI systems in general in newsrooms. So basically from our experience, the journalists, they have very high standards for consistency of public specific style and the tone. So when you read the Spiegel and cite, you always have some expectation as a reader how they would present this information. And they are very, very specific. So for example, when you ask someone who is not programming, is there a difference between different types of to write a program and so on, they will say no. But every program knows there are many, many aspects there. And this is the same for them. So when we see the text, it's nice. And they see hundreds of things which are wrong or correct there. So this is one aspect for them and it's also highly related to the business side as well Then the second point as well high stakes for errors. So When we make a hallucination adding wrong name or add the wrong fact it could cost them ten thousand one hundred thousand Euros for them. So it's also clear business side issue for them. So the errors we produce Can be expensive Then low latency evaluation requirements, so they are used to work very fast So we cannot just say okay, we will evaluate the quality for what an hour and then we'll come back This is not feasible for now And then in general LMS blur the boundaries between different steps, which they are used to do and have to a quality assurance is there Therefore we have to come up with some systems and approach us out to handle that and on the right or on the left for you You see an image I think it is two or three years ago when one of the bigger publishers said, yeah, we could maybe replace the journalists and so on. But from our experience, not a single journalist got replaced by AI until now because of these issues we see on the site. So how do we think about evaluation when we have these high requirements from the users? So one thing which is very relevant for them is the spectrum between objectivity and subjectivity. So, the text which they produce, there are aspects which maybe can be objectively evaluated and there are, of course, objective performance criteria from their side, their style, other issues as well. Then also there are different task types which we work with them. So one thing is for maybe a typical machine learning task is summarisation of the text, maybe another is classification and all these things together are often required to produce a single artifact for them, then of course the benchmarks, there are some benchmarks which are relevant for some specific attacks, and also metrics. But generally we have the spectrum where we have on the objective side some metrics, so the basic ones of course are a blue score or a recall, then we move more and more into subjective side when we have maybe some data sets which we have standardized we have the judges somewhere in the middle where we use some other lms other ai systems to evaluate certain aspects then of course we have av tests on the customer side and also manual human in the loop analysis we have our own journalists in house to do also that because this is basically also needed to scale other systems as well from our perspective Another thing, when you read news, is factuality is often pretty high on the requirements list as well, and what we apply are knowledge graphs. We try different aspects, but basically we apply LLM-assisted knowledge graph generation, so we generate bigger or smaller knowledge graphs to check the factuality of the final text or intermediate aspects as well. And one issue there with the knowledge graphs are that on one side they can help, of course. They are deterministic, so if there is something in the graph, it can actually evaluate and really 100% say, is it there or not? But the issue is generally with the graphs, because it's an older technology, already keeping that up to date and also populating them accordingly. And LLMs can help there, and especially if we combine it with some already defined ontology, so basic ontology of what aspects do we want to have in our graph, that can actually support quite well, and also help with explainability of the system to the journalists in this case, how to check for certain aspects or basically questions. For example, in journalism, they have , which they are used to check, and we can also separate them into smaller pieces, but then we can use this as a basis for knowledge graph generation. This is also very explainable for them, because the customer is used to work in a similar manner. Another aspect is the evaluation of the style of the text. So, for example, when you read Bildzeitung, it's a much different style of text and also the emotionality and many, many other sub-aspects, I would say, than maybe Financial Times or other texts as well. So there is the question of the style and different sub-aspects of that. and we use judges quite extensively for that which we train or manipulate based on the feedback and also the knowledge from our experts in the company but one thing of course to mention there that of course this is not a self-fulfilling prophecy that we have a LLM which checks another LLM and so on, so of course we have to make sure to take care of some biases like positioning learning, right, that the learner learns basically in which position is actually the same model, or self-affinity, that the same model thinks always likes its own answers, and also verbosity, so very long texts tend to be graded higher. And therefore what we do is generally we use multiple judges, and we also use the classification type of the judges, so basically we cut the output layer of the LLM which produces the tokens, right, so where it generates the distribution to select the next token and just define maybe four or six tokens for the classes to do a classifier of the LLM. So of course for that you need an open source LLM, you cannot use the API for that. Another thing is to go a bit further with the judges is role playing, so we use personas for that, so basically this is, to make it very simple, I think everyone has done it, that you just say you are for this and that, and what we use in the different aspects, these are two examples, but we use generally more to have some objective persona and subjective persona there, and there we define different criteria there, and this is basically, this This is basically already the output from the optimisation which we've done with dspi, but I will come back to that later. OK, so a lot of moving pieces, how LLMs can be evaluated, and I think all of them have their places for specific tasks. And now in the next steps of how we could implement some of these things with dspi. So what is dspi? It is an open source module from Stanford, an LP group. And their core idea is to move from direct prompting to programming of LLMs. And basically, the general concept is to create a text transformation graph. So they see the problem really as a multi-step process where a LLM transforms, in this case, the text. And it has the key features there, the signature. So this is basically where we define the inputs and outputs of our system. For example, from the publishing side, we want to define the length of the string. So we have a print publisher, which publishes every week a newspaper in print. And they want exactly 60 characters on that part. Not 65, not 67, but 60. And there we want to define the exact output of the string. Then we have the modules, for example, to abstract the prompting techniques, for example, chain of thought, program of thoughts, or multi-chain comparisons. Then there is a big part of DSPI is evaluation. So basically where we can define some metrics or use other DSPI programs to evaluate intermediate steps as well. And I think the most, the most famous is actually the optimization to tune the prompts there. So basically there they have several algorithms there to synthesize future examples or even optimize the optimization for the instructions which we do. They also offer fine tuning, but they haven't tested it, so I used, when I used the fine fine-tuning other libraries, but it is actually also there. So how could one implement that? So for example, we have a task, we want to do a summarisation, so a very basic task, a daily issue we have, and we start with a signature, so where we define inputs and outputs. So for example, there, we just decide, OK, we have the text, and then as output, we want to have the summary. This is a very basic example, and we can add some descriptions there as well. Then we create a class, a summariser module, which looks very familiar from the structure to PyTorch, if you are familiar, I think. And then we also define some reward functions, so basically a selection criteria based on what we want to select the texts which are outputted. And then, for example, in the end, we say we want a best-off of 10 examples. We select the one which most is the best regarding our criteria in this case. So this would be a very simple program where we basically generate 10 summaries and select the one which is the closest to our 300 word limits. But of course, you can define your own easily, defined on the report functions there. Then, of course, when you run this piece of code, you will find, OK, we are nowhere near 300 words, because the LMs cannot count the words. I can tell you that. OK, let's try it on the next step further. We heard, OK, the chain of thought is a good thing and works quite well, or is applied quite often. Let's implement that. And then for that, we can create a class for a chain of thought. And as you see, it uses the summarize, which we created before, for initialization there. And then we can use it to run it with the same reward function. So then this is how we can basically do a multi-step process. And this is for another example. We want to really have very defined multi-steps, so we can define each step of your chain of thought. This is basically also how we, in the end, progressed in our case, because the task was to process news, basically news articles, to shorten, to define length. So basically, we get any length from the text, and we need to provide a concise, correct summary in a predefined length range. So basically, the range was always plus minus 25 characters. And we need to process everything. And this was basically the idea. We have maybe, I think, six or seven steps which we defined, how we decompose the text, and how we add it together, and so on, because we need to basically to produce typically around 1,000, 2,000 words in a short period of time. So we need to have a fast way to summarize the text. So yeah, so basically the structure here is that we have the chain of thought, and then we have basically the two step process where we first derive, in this case, the key points, and then we generate the prediction. And then again, we select from 10 texts, we select the best one which fits our ideal length reward. And if you would run this text on your computer, we'll see that the result is much better for when you run it. Basically, we already see in very simple examples the improvements in this task, how DSPI can help structure exactly what the LAMP should do in this regard. One bonus feature of DSPI is tracing and logging. So basically, when we run all these classes and tags, all the outputs and also intermediate steps were automatically locked. So we can easily use it for the improvements or also some error handling there as well. I talked before a lot about the judges. So this is an example how a judge could be implemented. So essentially, in the beginning, as you see, we have, again, input and output field. So input field. And then we have the key points. So basically, this is the judge which will work with the previous example. We take the key points, we take the summary, and then we evaluate the quality of that summary with a judge. And as you remember before, I told you that the judges can be wonky, can have some biases, so maybe we want to expand the judge with our specific metric or some criteria. So this is an example where we have individual grades for factuality for key points. So did the summary really have all the key points? Is the structure good? Is the clarity good? Is the text neutral in this case? And this is basically a metric which can be applied, again, in the same judge to evaluate the text. So, and the last thing about DSPY is the optimisation, so basically what it can do is, can do for you the few-shot learning, where basically it can select most optimal from what you have, the examples for your prompt, where you select for the few-shot prompting, and it also can produce some instructions to optimize the outputs, and it uses different algorithms, but they are based on Bayesian optimization in the end. And this is just a quick example. If we would want to optimize just instructions using Mipro and v2, this is how you could implement it. So basically, we have a teleprompter class, which we can then put into the double step Summarizer which we created before. Okay, so for the end, so how does all of that relate or how was it implemented in the practice in the last seven months which we had? So basically we created like a quality gate, and it has a lot of things there. So basically we have many rules which were defined by the customers for the style expression, So everything which was clearly definable and deterministic, we have there. Then we have a lot of factual correctness of entities, graph-based methods to check the logic, and also the relationships between people, places, and so on, and then also the style and coherence evaluations. This is high-level quality gate architecture, which we have. So basically, we have input text, we have a task classifier, and then basically, it So to determine what things to activate or basically for the graph analysis for entities Judges and also the rules and then we have also a text processing post-processing step where we do some Basically, you have here a general text generation there We have some specific requirements for example, how the date should be written exactly. Where should the point be or the anything? So this is them Basically the styling of the text and then for monitoring we use a nofollow which integrates nicely with the SPI and for a server-side we use Grafana so standard things and Yeah, this is just a screenshot which I made from our dashboard. So basically we have around 13 Yeah, 13,000 requests in a month where we push out these texts and And we basically can see, so basically we computed here everything, all the requirements on the latency because when we have an issue, then some feedback loops start and so on, so we take more time. So already from there we see that we have some issues with our artifact there. And the goal where we aim is actually, I come also from mechanical engineering, as this sigma quality, so basically in mechanical engineering you want to have a certain defined deviation range and you want to remove it or really based on standard deviations. So right now we are somewhere between 3 sigma and 4 sigma, we try to get there, but it's already a general fact that over time we generally also come back, so we are running to stand still right now, because over time your actual process, your actual standard deviation or actual errors only appear. So basically we still have some ways to go, and still, yeah, every day we get at least one e-mail where it says your system is not working, even though we push hundreds or sometimes thousands of texts per day. And we are happy with the accuracy, but still we get very negative feedback every day. So still ways to go. Yeah. So what's the conclusion from our side? So we think for a really reliable system which over time delivers what it promises, we need to embrace the test, but basically evaluation-driven development of LLMs and their systems. What helped for us is to guide the process based on what kind of task we have, how subjective it is, how we can decompose it in subjective and some unobjective criteria. Yeah, we use different methods. We use the graphs. We have judges. We have also humans in the loop to actually to boost and to scale the evaluation process. And from our experience, the personas helped as well to basically align the judges with our experts so that these people don't need to review 1,000 texts per day, but maybe just 10 or 20. So thank you for your attention.

Speaker 2 [24:01]

So thank you for your talk, and we received about eight questions in the Slido, and I will just start with the topmost. Do you work with editorial teams to evaluate the text? Do you use their suggestions to reinforce the models?

Speaker 1 [24:19]

Yeah, so we have our internal people so they do the 90% of the work because customer wants the result But yes, we do we basically So we use the SPI for that so we get the negative examples And then we have automatic pipeline basically to re-optimize the prompt and then we run the whole pipeline And we see if it's still okay

Speaker 2 [24:45]

Okay, next one. How often does your generated content get published verbatim or with minimal human edits?

Speaker 1 [24:55]

Yeah, that's a good one. Basically what... So from... We generate, let's say, two types. We generate short artifacts like headlines and teasers, and then we generate generally three or five examples so that the people can select. Then one of them gets used, but basically they still have a lot of freedom which one to select. But yeah, that's an easy task, basically a five-shot task. And regarding the final text, I would say from general metrics, 70% have minimal things. Most of the time they remove some text because we have this text range and then we sometimes add some text which may be not that relevant for the text.

Speaker 2 [25:52]

Okay, next one. For productionized application, how do you ensure these evaluations are met in the deployment pipelines?

Speaker 1 [26:04]

Yeah, we run basically a canary deployments Where we are we have several servers and then we start with one the run the relations and then we scale the update basically then over time

Speaker 2 [26:21]

How do you ensure that all claims in the text will be checked for factual correctness?

Speaker 1 [26:28]

So the factual correctness we run with we have two two levels basically we have yeah these journalistic V-frame check so basically a system to check if does the content actually relate to that so Because then we have if for example we require in our text that we have a date or time and the input doesn't have it So it's very high risk that we will hallucinate that because we force it we want the date and it's not there so we basically do quality check of the inputs and then also we as I said we use the graph so basically we create entities graph for example in the relationships graph of the input text and then of the output and we check for the basically for the arrows which creep in and then correct it.

Speaker 2 [27:13]

What frameworks or libraries do you use for working with knowledge graphs?

Speaker 1 [27:19]

So we use Neo4j for for the graphs But we use but our graphs are small. So I think if you want to create rack with the graphs, this is Maybe not the best I would say because Neo4j is as a good marketing and decent product But I think maybe not for everyone our graphs are small because Texas has like 3,000 characters the graph is small So no issue there

Speaker 2 [27:44]

Okay, how do you balance the value of optimizing prompts versus fine-tuning or even pre-training transfer learning of new LLMs?

Speaker 1 [27:55]

So we have and are also using fine-tuning as well so the main thing is basically that I Think so what we saw the fine-tuning when we started for example We go to the publisher and we take their last three years of text and do a fine-tune for that and it gives a big bump But then okay, we receive some text every day, but that's not the big change than for them So the distribution is similar so therefore then we switched or to updated more So we can basically do a fine-tune every half a year or a year But prompt optimization when we have two negative examples then it's much much faster than what want them to fine-tune them

Speaker 2 [28:43]

Do you address the writing style and if yes, how?

Speaker 1 [28:47]

So the writing style is basically one part is, okay, the persona, so you should basically, am I, yeah, do I write with a lot of, so are my titles with a lot of question marks and so on, and I'm very emotional and so on or not. So basically, this is a persona there. So for generation, then as well, the same then for the judge to check if that works as well. And then we have some styled parts on the rules. So basically, these are just deterministic rules. This should be there, for example, keyboards for SEO parts and so on.

Speaker 2 [29:25]

Yeah, so now we are out of time. So let's thank the speaker again