Jeannie: An Agentic Field Worker Assistant

Speaker 1 [00:07]

Good morning, everybody. It is not morning anymore, but we see us for the first time. And I hope you are still energized to follow this talk in the late hour. Pepe and I, we both work for E.ON, and E.ON is not only the sponsor of this event, it is one of the largest companies across Europe. We provide stability and give you the energy for the daily tasks, electricity, heat energy, and the red color. And unfortunately, E.ON cannot operate only the cyberspace. We have a lot of tedious daily tasks. Our field workers go to prepare broken pylons, stolen cables, pipes which we are not functional anymore. And somebody needs to do it. We cannot replace people working in the fields, but we can make their lives easier due to things we develop in our digitalized world. And one of these examples is Gini. Before we come to this highly sophisticated agent, or actually swarm of agents, let's look at the GNI timeline like it was the last two and a half years, because it is crucial to understand where we stood and where we are standing currently. in the year 2022 the life was much easier we had our decoder and encoder models we used bird-based fine-tuned models to accomplish a lot of tax-paid tasks eon has a lot of strings flying around from customers to customer support within emails floor we have a lot of internal documentation everything needs to be understood summarized entities needs to be extracted we need to translate things but by the end of 2022 it actually became very apparent that the world is not there where we stood the last 50 years chachapiti appeared and everybody at eon was actually thinking what we can do with that and by march 2023 we've got first models of opening eye based on asia and started thinking about what can be done with that we have our colicria which was the originator of the many nlp based products and starting in march 2023 based on some blueprints first ideas on internal chat GPT-like products were developed we struggled a lot with the input size window because early models were very limited in that we developed very sophisticated chunking and history management techniques for that but it got very simple after the problematic small windows became history and two other things happened in 2023 at least for us we adopted in many product products langchain which was our framework of choice early adapters of this one and we started building custom front ends using streamlit and chainlit to rapidly develop prototypes for customers another highlight of that was that not everything can be done with textual racks we need to represent knowledge these experiments are still in doing by 2025 but graph rec was exactly one of these approaches time went apart and in the year 2023 when we had already a lot of running project based on simple texture rec we understood that not only Gemini or GPT for all can be added as an enhanced reasoning models to what we have. Very crucial idea was that we do not have to compete with the whole outside world. You see the logo of Microsoft Copilot and Copilot appeared and gained a lot of market for text processing tasks. If you would like to generate a presentation, if you would like to generate an email, you don't need a custom solution for that and you can not compete with Microsoft and all the developer fleet behind it that's why we try to focus on things we can do better and for that we understood that we need to apply agentic things we will be looking in a couple of minutes besides agents and the underlying graph then graph framework for that we definitely made a big step into custom formats and especially into processing of visual data with large language models. And by now when we have first agentic approaches it is clear that agents needs to cooperate with each other and for that we are looking deeply into structured output of large language models and to topics of coordinations between and among different agents. other big topics are not taking ready models not even as foundation models but train our own probably as continuous product incorporate knowledge into the model itself and not use direct pipelines and for that we are deeply looking in the evaluation mode now into deep seek and the whole llama family of models you've seen the state of technology and the state of products currently was very promising in 2023 we've started with the whole open AI stack on Asia and Asia I stack using the AI search document intelligence for extraction of textual and visual informations we used exclusively open AI models and our products were circling around conversation with the model itself or providing simple data transformation tasks it changed in 20 to 24 to usable products where we have corrective approaches to rack we can understand when the screen Now, where we can recognize hallucinations, we can correct unappropriate output of the model and we now can deliberately combine models from different worlds where each model shines and does it better. And current state of solutions is actually sophisticated chatting with a lot of documents with your custom document collections, focusing on one document, on on many documents combining reasoning across documents and combining custom data sources because most of our customers are interested actually not in reading text but for example tabular data but that's not the end currently we would like to learn how to automate the daily doing by agents which do not only tax extraction tasks but tasks which enable people outside in their daily tasks we are working on a centralized general architecture for custom solutions which sounds like a rocket sign it is not necessarily a rocket sign by its nature but it is very inspiring to be part of such a big initiative which will facilitate the lives of many, many, many employees. And one of these examples is actually our agentic genie. Pepe will be talking about that on the next slides.

Speaker 2 [08:34]

Thank you, Andrei. All right. So, E.ON is one of the biggest energy companies in Europe. So, that means thousands of millions of euros goes into infrastructure, thousands of field workers, and as a proper German company, a lot of processes. We've noticed that in the past few years, work incidents have risen, especially field-related incidents. So, we dove into incident reports to get to the root cause of what was causing them, but what we encountered was tables with dozens of columns, which came from questions and free text field answers, so it was a nightmare to analyze. In fact, the root cause analysis became on itself a full-time job. And that's when we realized that in order to fix the data, we also needed to fix the process. And this is where our journey begins. Oh, sorry, should have clicked through the animations. So that's what's going on. All right, so imagine you ask us to install a PV panel on your roof. So this request is translated into a task, which is assigned to someone, Marcus, who's our field worker for the day. And then Marcus needs to prepare for the task, perform it, and mark it as done. So simple, straightforward. But actually a lot of things can go wrong before he actually even gets started. Depending on the task, the preparation process can be long and tedious because he needs to find the right equipment. If he needs to wear a helmet, goggles, certain type of gloves, he needs to go through the security guidelines, through the instructions on performing the task. Also, if a thunderstorm is coming, it's probably not the best idea to install a PV panel, or if there was a recent flooding, then maybe the site is not reachable. So all these are risks that we have to account for, and if something slips through the cracks, that can lead to delays, costs, and in the worst case, an accident. So that means that we have a critical factor here, which is task preparation. That is the first problem that we're focusing on today. So what happens if, despite all of this, an incident still happens? Well, we have the incident reporting process. Imagine that Marcus, while installing the PP panel, slips, falls off the roof, and breaks his arm. Now, on top of being injured, Marcus now has to create an incident report. And as I mentioned before, that's dozens of questions, many of which are not even related to the accident that happened to him. And what we end up with is a frustrated worker, exhausted and stuck with a process that is highly inefficient and outdated. And this also has some risks because we end up with incomplete or inaccurate reports that make it harder to get to the root cause and preventing the future. So at this point, we have two main issues. the potential risk of a poor task preparation and the inefficient incident reporting process knowing the problems we now present our solution

Speaker 1 [11:49]

So, this solution is actually artificial, ingenious, green-eyed. It solves all the problems we were speaking about. Actually, the task preparation workflow, what I have to do to be prepared for the next field day, we have the task preparation agent for that. The worker is alerted about the whole history of incidents at the place where the next task has to be accomplished. And we gather or Gini gathers for the worker all the surrounding problematic points like weather condition, like time, like the urgency of this task. And we definitely streamline the positive or negative incident reporting in the end. Genie is based on a couple of agents and a workflow, and the name is coming actually from a very well-known German soap movie where Genie was an ingenious helper. We took this as an inspiration for that. The internals on Genie you will see on the next slides and in the presentation.

Speaker 2 [13:07]

Thanks again. So now that you've met Jeannie, let's take a closer look of what makes her special. So as Andrew mentioned, we have agents and we have workflows. Anthropic has a very nice article about those two definitions, but in a nutshell, think of a workflow as a recipe. So we define the steps in advance, and for the case of the task preparation, we want to get information of the user, the next task, then things like the weather report, the distance to the site, instruction, incident, etc. And we end up with a final PDF report. That is something that happens sequentially as we defined it. So what we're looking at here is something reliable and predictable, and that's good for some use cases. While agents, on the other hand, they have more flexibility. They have the liberty to choose and reason to select the right tool depending on the interactions. So instead of asking, give me this final thing, it will be more dynamic. Like if you tell the agent, I need to know my next task, it won't trigger a workflow. It will just simply choose to call the database to get your next task. So what we're looking at here is flexibility versus Predictability or reliability depends. So that's why we have the two things because we Jeannie blends both of these things into a single solution and that's why we call it a super agent Now for the case of the task preparation what we have is it's the same thing that I mentioned here to set some things are running parallel, but essentially we have database calls that we that we build with SQL alchemy to connect to to our databases we have external API is like to get the weather forecast we also have the retrievers of course to get the instructions related to the task or related incidents and finally we use playwright to generate a PDF report from a pre formatted HTML that is that is coming from the report that the LLM generated at this point okay so next we're gonna see a live demo it should work can't promise anything but I have a recording in case it doesn't all right so the screen is not changing so give me a second all right so we first log into our application as we said we had agents and we have workflows and one of these starters are pretty fine to go directly into into each of them so if we just simply click the task preparation workflow in the end it's a chat we'll just simply input the message that that we need and what we'll do behind the scenes it's as you see it's running the agent and it will call the different the different tools that the workflow has like getting the upcoming tasks which we already see here it's a PV panel installation at a fictitious address when it is due the distance from you and starting from your starting point the weather forecast has description guidelines etc etc and some references here and well the final step is remaining but maybe you don't want to PDF let's just say yes give give me a PDF typo there doesn't matter all right so it will now work on generating the PDF what it does here is it will take this report that we already generated it will pre it will actually ask an LLM to give it some specific fields to pre-fill an HTML file and then we play right it it will generate a PDF document, which we see here, nicely formatted. It's just a one pager with everything that you need to perform the task right to the point. So no need to take your time going through each of the steps. This is one of the capabilities that we've introduced, the workflow. But we also have the incident reporter. And because this is a smart agent, we can simply say in the same conversation, I want now to report an incident. This is transferring me to the incident reporting, just say, okay, sure, just say it's fine. And then, all right, we are now in the incident reporting agent. It's asking me, well, she's asking me if it's my incident or on behalf of a colleague. Let's just say it's mine. had it during my last task not true check again right okay so it looks in the database and it finds the latest test that I perform and so I will say and this is something that I we added some hour ago so this will be the first live test, but let's try to see if it works with audio. So while doing the pylon maintenance I slipped and fell and twisted my ankle. I went to the hospital and the doctor said that I needed to rest for two or three weeks. And let's just hit it done. Text-to-speech. Now, well not pilot, pylon, but I think it will work. And here we have it. No need for dozens of questions. Out of a single description we already have a pre-filled incident report we need to review it we need to we can add additional information but I think that this looks fine so I'll just say sure looks good and then it will simply insert it into the database we see it here I mean she said that she inserted it but show me the latest incidents let's see if we can see the table right so here it is it's the same ID right 41 yeah 41 41 yeah so we went from having a full manual process that took nothing short of 30 minutes to a short conversation where you can even record your voice and send it to Jeannie so it can do much much more things but we only have five minutes left so I'll just go back and finish the almost finish the slide so as we wrap up this presentation there is one thought that we want that we want to to live with you that this isn't just about AI it's about people it's about safety it's about purpose at Eon we're not just building AI for the sake of novelty we're building it because it can change lives and you make workers safe, make processes more optimal and more humane, and with Genie we see the potential on how agentic AI can bring clarity when there's complexity and support when there's risk. And this is just the beginning, this is just one use case, but in the end the real goal isn't just to replace people, it's to empower them with right information at the right time in the right way. So thank you very much.

Speaker 1 [20:34]

If you would like to be part of this mission, we are three days at our booth downstairs Come to us talk to us and probably somebody will want to work with us

Speaker 3 [20:45]

yeah we have some questions came up so the first question is does a graph rag really help increase accuracy if so do you have any idea how to improve retrieval accuracy compared to simpler chunking techniques

Speaker 1 [21:02]

It's a very wide question, but yes, GraphRack can improve the accuracy of the retrieval if you have the well-built knowledge graph. The question is how to provide the quality of the knowledge graph. We are actually striving for measurement on the gold collection for different custom projects. The definition of what is good and what is bad is actually the crucial thing here. you need to know your document collection if you have very short documents then your chunks are not overlapping probably or the kind of overlapping is too problematic for a retrieval engine since you have like 90 percent of the window here you need to think in details about that there is no silver bullet to solve this problem in general

Speaker 3 [22:00]

Okay, the next question is, can you say more about the hardware you are running with your solution on? Is it LLM self-hosted? If so, which inference engine do you use?

Speaker 1 [22:15]

Eon is a cloud native shop since three years. Most of the workloads were migrated from on-premise data centers to mostly Microsoft Azure, but we are normally cloud agnostic and we think in three big providers and even in a couple of more, let's say, not that America heavy. That's why the compute is obvious here. Those are either hosted services like container apps where the current solutions run, or those are more vanilla Kubernetes clusters, but managed on Azure. We do not have our hardware, probably except for some bare metal graphic cards, but it is a very, very narrow case.

Speaker 3 [23:12]

Okay, so we have a couple more questions. So one is how do you avoid workers abusing the LLM? Like write me a realistic report so I don't have to work for the next three weeks so how do you avoid workers abusing the LLM like write me a realistic report so I don't have to work for the next three weeks

Speaker 2 [23:41]

I mean, LLMs are a tool. They're not bad by nature. So I encourage everyone to use it. I mean, there are plenty of agentic solutions. And in the end, people are benefited by it. This is just an example of how you can use it to make someone's life easier. But in the end, same as with the whole programming paradigm, you need to have, in the end, a person reviewing that what you're producing is actually quality. Because in the end, no one is going to blame the LLM, is going to blame or thank you. So you are responsible. It's about responsibility.

Speaker 3 [24:18]

One last question, so which framework is the frontend build with?

Speaker 2 [24:22]

Chainlit it's Python based and it's it's only for chat applications, but it's very nice

Speaker 3 [24:27]

yeah thank you so much thank you everyone for the interactive questions and give them one more round of applause