Don’t call your LLM too often! How to build your dialog graph with confidence and sleep at night.
Large Language Model (LLM) integration in corporate environments often leads to excessive operational costs and system inefficiencies due to redundant API calls and complex, looping dialogue graphs. These issues frequently emerge when systems evolve from simple prototypes into production environments without rigorous observability, resulting in "death paths" or infinite loops where competing evaluation checks—such as faithfulness versus hallucination rates—force the model into repetitive regeneration cycles.
To mitigate these inefficiencies, a structured approach to dialogue graph optimization is employed. This involves implementing observability tools like Langfuse, Arize Phoenix, or MLflow to trace individual spans, track request inputs and outputs, and analyze cost breakdowns. By analyzing these traces, developers can identify redundant paths and restructure the dialogue graph. Optimization techniques include implementing a routing layer to bypass the retrieval process for simple queries (e.g., greetings or out-of-scope questions), disambiguating queries before retrieval to avoid irrelevant document searches, and summarizing conversational history to reduce token consumption.
The effectiveness of these optimizations is measured by comparing a "redundant graph" against a "clean graph" using a golden dataset. Success is evaluated through a trade-off between routing quality—measured by the number of LLM and database calls and the depth of the graph—and outcome quality, which includes metrics for groundedness, usefulness, and correctness. This methodology allows for the reduction of latency and cost while maintaining the integrity of the final response, ensuring that LLM calls are only executed when necessary for the specific intent of the user query.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Natural Language Processing & Audio (incl. Generative AI NLP) and was classified suitable for novice domain / novice python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
Building reliable dialog flows for LLM-based conversational systems remains difficult once interactions move beyond linear question–answer patterns. While early prototypes often rely on prompt chains, real-world systems quickly require branching, correction, clarification, and multi-step reasoning. At this stage, dialog logic implicitly turns into a graph, yet is still implemented and reasoned about as a sequence. This mismatch leads to structural problems that are hard to detect without explicit modeling and observability.
Complex document retrieval systems are not born out of theoretical itch. We’ll exemplify practical problems framing them around the following practical use case from the area of electricity/power production.
Use Case: Aladdin and the Case of the Almost-Exploding Power Plant
Rick and Morty are operations engineers at a large electrical power plant. Every single day, they face the same heroic challenge: too many documents, too little clarity.
The technical staff produces a constant stream of operational reports: free-text summaries describing the health and performance of steam generators. These reports are rich in knowledge, but poor in structure. Rick’s daily ritual is to read, compare, and summarize them, trying to predict which units will soon need maintenance. If he gets it right, the plant saves money by avoiding unnecessary service routines which are prescribed by regular maintenance guidelines. If he gets it wrong… well, let’s just say steam generators have a dramatic way of expressing dissatisfaction.
But unstructured reports are only one part of the story. Alongside them exists a well-behaved, structured world: databases containing results of regular, non-invasive ultrasonic inspections of pipelines, used to track corrosion development over time. Morty has built a quantitative model that predicts the probability (and timing out of this probability) of a pipeline rupture based on these corrosion measurements.
Naturally, Rick and Morty want everything. They want one system that can: 1) Understand messy human-written reports, 2) Reason over numerical corrosion models, and 2) Answer simple document questions without investing into unnecessary intelligence.
Thus, the system Aladdin is born.
Aladdin combines three very different subsystems:
- An agentic indexing component, which dynamically builds a search index for a GraphRAG over heterogeneous documents, given a pre-defined graph structure.
- An autonomous analytical agent, which evaluates pipeline failure probabilities using Morty’s quantitative corrosion model.
- A lightweight text-based RAG, backed by a vector index, for fast and simple document retrieval.
But what is the challenge? Once these components start talking to each other, the dialog graph becomes unpredictable. Execution paths depend heavily on what information is actually present in the documents. And this is something that cannot be fully reasoned about in advance. Loops appear, branches explode, and theoretically “clean” dialog designs fail in practice.
This use case illustrates why observability, tracing, and empirical optimization of dialog graphs are essential when building real-world document retrieval systems for industrial environments. Especially when Rick just wants a straight answer and Morty really doesn’t want another pipeline incident on his watch.
Given this use case we will exemplify several structural pathologic cases in the dialog graph which we observed in the practice and for which we found curative approaches.
Non-ending loops in the dialog graph A frequent failure mode is the emergence of endless circular dialog graphs. Typical examples include:
- correction loops (“Please rephrase your input” → user rephrases → validation fails again → same prompt),
- clarification cycles (“What do you mean by X?” → partial answer → same clarification),
- fallback loops where a generic catch-all path routes the conversation back to an earlier state without introducing new information.
Such cycles are rarely intentional; they arise from local fixes applied over time and are difficult to identify by prompt inspection alone. In production, they manifest as stalled conversations, increased latency, rising token costs, and user frustration.
Beyond circularity, several other structural pathologies commonly appear in document retrieval systems.
Dead subpaths after non-matching branching conditions
Dialog graphs often include branches guarded by semantic or data-dependent conditions, but changes in document structure, embeddings, or preprocessing can make these conditions unsatisfiable, creating dead subpaths that are never executed. These paths are dangerous because they give a false sense of coverage, increase maintenance and reasoning complexity, and in production often manifest as mysterious fallback behavior where the system always takes a default route instead of a specialized one.
Redundant validation and re-validation steps
Another common issue is redundant validation, where the same or equivalent checks are performed multiple times along a single dialog path. This often happens when validation logic is added defensively at multiple layers: once at input parsing, again before retrieval, and again before response generation. While each validation step may seem harmless in isolation, their combination leads to inflated dialog depth, unnecessary latency, and increased cognitive load when analyzing traces. Worse, slight inconsistencies between validation prompts can produce contradictory outcomes, for example, an input being accepted in one step and rejected in the next.
Overly generic catch-all branches
Catch-all branches are often introduced as a safety mechanism: a “default” path that handles unexpected input or retrieval failure. Over time, however, these branches tend to grow in scope and responsibility, eventually becoming overly generic handlers that do everything. Such branches blur the distinction between genuinely exceptional situations and routine cases. As more logic is added to the catch-all path, it becomes harder to reason about what the system is actually responding to. Specialized logic may be silently bypassed, while unrelated scenarios are forced through the same generic response strategy.
Linear sequences that should be collapsed
Many dialog graphs contain long linear chains of nodes with no branching, no state changes, and no observable side effects between steps. These sequences often originate from iterative prompt development, where small transformations are added one by one (“extract entities” → “normalize entities” → “rephrase query” → “check relevance”). While conceptually clean, such linear chains are rarely optimal. They increase token usage, latency, and the number of failure points, without adding expressive power. More importantly, they obscure the true logical structure of the system: what could be a single semantic transformation is spread across multiple opaque steps.
An additional aspect of an overcomplicated dialog graph - especially baked by an autonomous agent - are barely predictable costs. Autonomous parts of the system need a very tight observability net to stay under control and not to burst cost prediction by an order of magnitude.
Working within a specifically regulated environment of a power plant posts additional restrictions on the explainability of the results. Every fact must be trackable to the source of the information and model hallucinations must be recognized in the very early step.
All the above requirements result in a setup which is heavily based on an LLM Operating Platform like Langfuse.
When combined with dialog-oriented orchestration frameworks such as Langflow, experiment tracking extends from single calls to full conversational trajectories. Complete dialog traces expose path stability, node utilization, dead branches, fallback prevalence, and user-facing metrics such as turns to resolution or correction-loop repetition.
Over time, this empirical evidence replaces design-time assumptions. Dialog paths are merged or removed based on observed execution rather than theoretical intent, with unreachable branches, redundant validations, and unstable loops revealed directly through trace analysis. Dialog graph optimization thus becomes a continuous, reproducible process grounded in measured behavior.
This talk proposes an engineering-oriented approach that models conversational logic as explicit dialog graphs and treats execution traces as first-class data. Using Langfuse instrumentation, developers can analyze concrete execution paths—branch frequency, loop formation, latency hotspots—and compare alternative graph designs through aggregated metrics and A/B testing, enabling systematic optimization based on evidence rather than intuition.
To sum up: using concrete production-oriented examples, the talk shows how graph-based dialog design improves multi-step retrieval, explainability, and robustness across languages. Endless correction loops are detected and eliminated, dead branches are pruned, and overly generic catch-all paths are replaced with targeted recovery strategies. The overall message is that scalable conversational systems require not just better prompts or larger models, but explicit dialog graphs combined with rigorous tracing and data-driven optimization.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:00]
Today, we have a nice starting talk in this room, one from Andrea and a junior. They're both from ION and are AI engineers. They work mostly with textual data in the AI space. Before we start the talk, I want to remind you, we have a Q&A website, talks.piken.de, where you can ask questions and then I will simply later reply to questions after the talk is finished. So, then I really want to welcome on stage Andre and Eugenia with the topic Don't call your LLMs too often. How to build your dialogue graph with confidence. Sleep at night. And everybody of us wants to really have a nice sleep. There you go.
Speaker 2 [00:59]
Good morning everyone. Welcome here for the first talk of this day, third day. You are still fresh and energized. And you may know E.ON in these red colors as a company first selling electricity across Europe. It is the probably first idea you have in your mind or an old hardware company managing a lot of energy networks, replacing fuses, climbing pylons and doing a lot of hardware stuff. But actually, nowadays, E.ON is more represented in the cyberspace because we have to manage somehow this infrastructure. And the infrastructure got smarter with every day. We got smart meters. We can offer a lot of plans for our customers, cheaper, more expensive, but more stable. Reliability is a very interesting topic because we have multiple suppliers on the market and everybody can sell the electricity to the grid. You need to think about the congestion problems when at some point you have our productive state and you need more consumption somewhere else hundreds of kilometres away. And all these problems match actually very well on the topics we would like to present you today. The topics for today are how we can manage the load in this world, information load for the customers and for the employees of Eons and one of the very important topics here how to do it efficiently. The efficiency means you don't have to waste your resources. Eons are very cautious about not producing more and selling more but doing it environmentally friendly. That's why we don't want to call our LLMs too often and sleep well at night for these reasons. If we look at the last years, meanwhile, four years back, LLMs pioneered a very interesting class of applications. We do not take seriously the necessity of training a new classification model for customer feedback text, for example, or we do not provide our custom models for extraction of personal information from texts we get every day in the emails or in the transcribed voice messages. We do not try to write emails blocks for the customer support. We all delegated it to some cryptical model which lives somewhere in the cloud, and we aim for the newest one always through the information there hope for the best and get some result but the reality is uh in this shiny world uh very simple uh this gets very expensive and after four years we have a very clear statement the eye cannot burn more money in the real sense of this world than it saves uh in the end in the last four years we went through a very clear evolution starting in the 2023 with a very simple rack applications we took models at that time it was OpenAI GPT 3.5 family. We took a lot of code from Microsoft. Eon is a Microsoft job to the biggest extent. Our 90% of all our code is running in the Microsoft code. We used a lot of predefined path on the Asia cloud using, for example, document intelligence and AI search and not bothering with other vector databases. And we had a lot of chat GPT like functionality for internal users of us, not for the customers even does not offer any digital solutions of these kinds to the outside market, but we automate our internal work a lot. And then we went next year to the point where we battled a lot with external sources, not only using the sources provided in the PDF documents with very simple ingestion pipelines or the model knowledge, we tried to give the chance to the model to learn from our SharePoints, for example. In a big corporation, it is not easy to get all the permissions and combine all the sites of SharePoint into one big mesh. Another pain was definitely handling non-English input. We have Hungarian customers, we have Romanian customers, we have definitely people from Sweden and Italy, and it made our life much more complicated. And having learned that, Jenny is laughing, much more complicated was her work, working with Hungarian customers. the next year was actually the idea of doing not only a single application but combining it all into backends which can be called from anywhere and custom frontends were a very important step where we offered centralized services across even for people who would like to build their own information retrieval solution and for that we had our AI hub hosting very different LLMs behind it. The Genia architecture from that made us cautious that we do not need to only give access to the models but give all this stack and make some elements of this stack unavailable if you build new application. And I would like to point your attention to observability today, because it is a point which is mostly neglected. If you build a fresh POC, you look at your logs, you try to understand what is happening. But in the end, the system evolves. And today I and my colleague Jenny will show you mostly how this evolved system can look like, how bad things happened. to us. These systems were built overnight and then evolved over a month and then landed in a situation where they were not efficient anymore and how we circumvented a couple of problems. Talking about the architecture, retrieval and evaluation, we can look at the generalized architecture of a retrieval system at E.ON. It is not one specific project that is part of the architectural board overview where we can see that we tackle multiple sources in two phases the runtime where we get answers generated on the bottom of the slide and actually the whole magic happens previously when you extract data from different sources running it in batches for example on data bricks or in custom pipelines in our Kubernetes clusters and as usual looking at the bottom and the green bar you see the magic word lang fuse the mostly used solution for observability in our in-house solutions it came up not from a sudden here it was a learning that without a system for this functionality you cannot understand what happens to your system. Length use is not necessarily one solution and something bad happened to our video signal. I can entertain you in these two minutes, telling you jokes about how the cockpits are, but the signal is back. And Lengfusers is not only the one solution. I can show you a very quick commercial overview for that, starting with OPIC, which is a very widespread solution in our on the UK market EonNext and EonUK use this cloud provider for observability a lot we'll be talking about the functionality on the next slide very reliable partner for us if you are not concerned with storing your data definitely in the European region and you can outsource to the U.S., which is possible for our U.K. departments. The next very widespread solution is definitely Langfuse, an open-source solution with a big office in Berlin. You can talk to these people directly. They are approachable, and they went, actually did the exit a couple of months ago. very successfully and we are bought by click house after they reviewed all the architecture to the click house back end and the next one is the solution which is probably known to everybody in this room is ml flow if you have a databricks account or you have ml studio running somewhere and do not want to have a third dependency, again, go for extended functionality of MLflow. But those were commercials of the systems. Let's look at what is important actually for us as people who operate these systems. First of all, we do want to understand how the input and output of the LLM looks like. tracing and tracking of single spans and requests. We do want to annotate every span if we would like, if they are problematic to custom data sets and run experiments on them. Calculate scores. One of the examples for this course is on the left. And definitely you would like to compare after deployment the performance of your system based off the predefined golden data set uploaded to this tool you can manage your prompts mostly we do it on the code side and synchronize it to the ui of length views to enable our non-debt coding affiliated people to do the experimentation on the length use ui and a very important thing for the management is actually the cost breakdown because you don't understand your costs for the llm or specific path for optimization if you look only on the Asia coast panel. But let's look now at the things which can definitely go bad in the system. These two examples I will be showing you now are not created from scratch. I mean, nobody would go into this situation willingly. But if you do it in a team which evolves and new people come in, and you don't look at the behavior in the production, sometimes you can get a situation when you try to generate an answer and then you run obvious checks in parallel. In this depicted case, you have the check for faithfulness, how trustworthy your answer is, the correctness, if you can actually compare it to the golden truth in this case, and the hallucination rate. And understandably, if you would like to force your generation into a state And to give you more details, the hallucination rate rises up because the model gets creative. And if you force it to write more text, it writes it, but not necessarily based on the text. In this case, you may end up with an endless loop. The loop is definitely not endless because you have some retry count and you exit after five times calling your model but you end up with a situation where one check with one prompt forces to do the opposite as the other check in this case we have the trade-off between the hallucinations and faithfulness another example could be if you do not differentiate the error mitigation techniques here and in all the consequent checks of the generation you always exit if your check fails after some time of retries and your user and badly your logging system does not understand what at the point where you exited. How to fix this state? Fixing this state means that we definitely need to observe the system and see what happens on the trace level and then handle it. How we can handle it, Jenny will explain you in two minutes.
Speaker 3 [15:14]
Yeah, so as Andrei has already mentioned, we have various projects related to different drug systems. So in this graph, I show more or less the possible issues that Andrei has already discussed in details. And basically, to tackle it practically, we need to first formulate the issue. So basically, that the long dialogue graphs might accumulate various pathologists, such as death path or infinite loops, all that Andrei has already discussed. And what we want is actually to reduce those graphs by adjusting maybe some thresholds for evaluation or restructuring the node system. But before this, we need to ask us several questions. So first of all, if this graph should happen at all, if in this way, so if we need the retrieval at all. Because, for example, there are often some questions, not questions, but queries from the users like for example hello or some polite form or some malform questions like what is and this can actually be a problem because in our early systems when we always did a retrieval because well it's right we need to retrieve the documents exactly this what is was working very badly because we had did retrieve some documents and there was just some very weird answers it can also be a question that more complex case for example we need to disambiguate the question because, for example, if in a system we have information about several departments and the question is about, please tell us about the travelling guidelines, the question is okay generally, but exactly this system should first understand about which department it goes and so it should also ask the question, so please make it more precise. Or there can be also some questions that are certainly not about our documents and those Those would require for example web search it can be a question like what is the weather today? and it's obvious that we also don't need to do the retrieval and We have different systems some would contain web search some would not If it does contain it we can go in that direction if it doesn't that we should Just finish the graph and say well, sorry. I don't have this information another question is is how much history or conversational context we should use, because it is very talking consuming. If you have a long history, maybe we should use only last several sentences. Maybe we should summarize it. Maybe we can also check the query, and then we would know that it doesn't have to consider the history at all. For example, if it is, again, just thank you, OK, thank you. We don't need a history for this. Or we know that the new question doesn't have anything to do with the previous one, because some users working just in one chat and it is actually very confusing for them when the history is too much present in the chat because we often had also questions for example if I start a new chat I have this answer but if I somehow do it in the old chat it's completely different answer and of course yeah there is a question of how we do the quality check for the answers how high is the threshold for the evaluation, how good basically the answer should be and it can also be considered differently because for example correctness. We have an example answer but it actually can also be correct when the answer is quite different because the user who created the data set didn't think of a different answer based on the document base. And when we considered all of this we can start creating a new cleaner graph by tweaking all those thresholds as well as restructuring the nodes. Here, for example, we see that for certain questions you would just go directly to the answer using LLM and without executing all of the graph with the database calls that are also actually very time-consuming and expensive. And so our method would be keeping the redundant graph and compares certain metrics for the redundant graph and for different options for the clean graph so basically we did it automatically partially using something like auto research we had some skill files that would adjust some parameters of the graph and so when we have the answer first we need to evaluate like in any reg system because evaluation basically defines the success of our project how good the answer is if it is correct, if it is well grounded, if there are hallucinations and if it is actually useful because it can be perfectly based on the documents but not really on the point to the question. Then of course there is a question of latency, efficiency, depending on the use case the answer should be, it is crucial that the answer is fast and our users not getting annoyed. For our system it's very important that the answer is really thorough and thoughtful and then they can also wait for five minutes because before the answer is there but exactly for our use case it is also important to have a good routing quality basically that is something that we are going to evaluate here so how many data paths are there how deep the graph is how many LLM or database calls we had and basically so here we evaluate at each experiment step our outcome quality versus the routing quality. If some reductions in the graphs are affecting our results in a better or worse way because we can reduce the graph completely, it will be very fast and nice, but the answer quality will be horrible. And then I would like to show just a very small demo. Basically, it is exactly where we compare. Sorry, I don't see my mouse.
Speaker 2 [21:09]
Uh, yeah.
Speaker 3 [21:15]
No, it's here. I'm sorry. Yeah, so right now here. We just use just more or less generic Jupyter notebooks because for different projects we use different observability systems like LengthViews, Opic and ML flow as Andrey said and here basically we can see that when we run it we execute our graphs and Wait a second executed and I think my token has expired but we'll still be able to see something, wait a second, demos as usual, sorry, yeah I can show it here before the token has expired it was working, yeah I can show you the result then so So basically it builds our graphs and we get such tables. Here we see the query and the comparison of the clean and redundant graph. So basically we can see how many retrials there were, how many generations there were, and their LLM calls. For example, in many cases it's going to be the same because maybe the questions are straightforward too complex that is why here for example we have the same LLM calls and but in this case we see that we have won three LLM calls which can also already be a lot of time saved then we can also build the graphs that was actually doing the note that didn't work because my token has expired and yeah so here we see for example that for clean and for redundant graph it was the same because the question was very ambiguous but in this case for example we can see that actually the clean graph didn't do too many checks and didn't go into the loops because the question was answer was good enough and the redundant graph had built already a lot of looping trying to improve the answer that was already good enough and basically in this way I can continue and see with different parameters how it works and yes so in this case we continue developing our graphs and we actually so it sounds a bit theoretical but it is actually the approach that we use in various project related to our customer emails or some chatbots that we use for our customers with their internal systems or some internal documents that they have so thank you very much
Speaker 1 [24:12]
So super nicely done. Thank you for the nice talk. I think there is your LinkedIn.
Speaker 3 [24:17]
Sorry.
Speaker 1 [24:17]
Yes. Feel free to connect with the speakers or also feel free to pick up the speakers after that for like simply asking a question in personal. So we have some questions already there at talks.bicom.de. So we'll start now with the first question. And this is a side question. Which Langfuse version do you use? The self-hosted one? If yes, what is your experience regarding the cost of it?
Speaker 2 [24:48]
I can answer this question because I was a lot concerned with different installations exactly of length use. Currently, we are in the middle of a migration between the second and third generation. It is not very straightforward because of this big architectural change. I would say that it is affordable both in cloud and self-hosted scenario. We are happy with that. The only problem is that you need to dedicate a team who would look after this installation. That's why we definitely would like to centralize that. Not every team needs its own installation of landfuse. We have now three bigger of them, like the, for example, installation of EDG, E.ON Energie Deutschland GmbH and the centralized EDT installation for E.ON Digital Technology, which is the hub of technology for the whole E.ON, and definitely a couple of widespread installations. The centralization here is, in my opinion, the key.
Speaker 1 [26:07]
Super, thank you for the answer. The next quite upvoted question would be, do you try a GenTech architecture which overcomes the DAG approach completely? DirectX.
Speaker 2 [26:23]
The gigantic architecture, which all comes DHD.
Speaker 1 [26:27]
D-H-D-A-T.
Speaker 2 [26:32]
Ah, DAG, okay, direct acacyclic graft, okay. I mean, we have a lot of agendic components in the racks themselves. They decide on themselves which part of the system needs to be called, especially if you have tasks, in this case, conversational tasks, retrieval tasks, or actionable tasks as well. Yes. For information retrieval purely, where you have especially different types of documents, you can do it deterministically better, in our opinion. Not led to the services.
Speaker 3 [27:11]
It depends, it's a little bit on the project because we also did some demo, it's not in a real project but where we didn't use only this vector search but also built a graph based on the data so we combined the vector search and this graph since the document base had a lot of somehow interconnected documents and for building this graph we actually also used some agentic approach but usually it is yeah the agentic part is mostly in the part after the documents are already there and processed
Speaker 1 [27:51]
Thank you for your answer. The next question will be, will you share your slides?
Speaker 3 [27:57]
We will. Yes, sorry.
Speaker 1 [28:00]
And like pre-takes?
Speaker 2 [28:02]
Yeah, we've forgotten to upload them to pre-tax.
Speaker 1 [28:02]
Yeah.
Speaker 2 [28:05]
They will be available shortly after talk.
Speaker 1 [28:07]
I think you can also upload it to the Discord channel on the slide channel, so maybe it's also easy to spot them there. Thanks for uploading the slides. So the next question would be, how do you tell what is a redundant graph? Did I understand it correctly that you look at the intermediary generations to figure out when a response would have been good enough without any additional large language model calls?
Speaker 3 [28:35]
All about the redundant graph so usually we just observe it for example in certain project We're observing it in a language and we saw that there were a lot of calls Before we set a good threshold for the number of attempts. So for example, we saw this infinite rule in their graph or we saw that For example, we go through the whole graph and still the answers to general that is why it means that probably it was some generic question and we didn't have to go through it completely but yes I think the easiest way is to observe it and see if there were some loops that were taken too long or some conclusions that don't lead to anything
Speaker 1 [29:19]
I see, thank you. So another question, from your session name, you call it as don't call your lll too often. Could you explain what it means? Are you referring to building a simple graph versus an redundant graph? Like simple versus redundant graph.
Speaker 2 [29:38]
Yeah, it is the reference exactly to this case that we try to make the amount of calls as less as possible, trying to build the optimal shorter graph in this specific case, yes.
Speaker 1 [29:53]
Another question. How did you measure correctness of the answer in the chat? Maybe longer graphs have better answers in the end. Was there no jumps from the question, Oscar?
Speaker 3 [30:03]
It is exactly our problem with the correction as I said exactly this step is The trickiest one because the groundedness for example is clear if it is based on the documents or not Or usefulness is also more clear, but for the correctness We have some data sets where our users say what they expect but as I said often it is not correct actually what they expect because they don't know all hundreds of documents in the database and And, yeah, probably there are some documents that actually answer the questions better than they expect on the documents that they think of. That is why exactly these parameters are what we use for evaluation, but cautiously.
Speaker 1 [30:45]
The last question will be do you run any component level evaluation of a graph or are you evaluation or are your evaluations mostly targeting end-to-end quality?
Speaker 2 [31:00]
It is a very tricky question because we're currently trying to modularize these approaches and we have tests for rack components specifically. If additional dialog component is employed, then it will be an end-to-end test. On the architectural diagram we showed you in the middle slide, you probably saw the rack as an extendable component and would try to pre-test them, let's say, on the unit level. But generally, the acceptance test will be end-to-end, having the friendly user group, and then testing from the front-end to the retrieval back-end the 100% of the pipeline. So in the most cases, we are talking about the end-to-end testing.
Speaker 1 [31:56]
Thank you for your answers. This was Jenny and Andre with their talk and I hope you as notaries can now sleep better at night.
Speaker 2 [32:03]
Thank you. Thank you.
