From Pixel to Payouts: A Multi-Agent System for Real-Time Insurance Claims Processing
Insurance claims processing for vehicle damage is traditionally a slow manual process, often taking weeks or months due to the sequential nature of investigation, cost evaluation, and approval. Traditional deep learning approaches using Convolutional Neural Networks (CNNs) for this task are often limited by a lack of labeled datasets, a lack of adaptability to new pricing, and a "black box" nature that hinders explainability.
To address these inefficiencies, a multi-agent system was developed using a Python-based framework to maintain governance and stability without the constraints of external orchestration libraries. The system utilizes a ReAct (Reason, Action, Observation) loop, allowing agents to reason through tasks, execute functions, and observe results. The architecture consists of an orchestrator agent that manages a sequential pipeline of specialized agents: a vision agent powered by the OpenAI Vision API to identify damaged parts and classify severity (minor, moderate, or severe), and two cost agents using the Perplexity API to provide comparative repair estimates from web-based market data. A final shop finder agent identifies local repair facilities based on the user's location.
The system is deployed on Hugging Face Spaces using Gradio. In testing, the pipeline processes a claim in approximately 50 seconds. Key advantages over linear prompt flows or CNNs include modularity, the ability to perform end-to-end assessments (from image analysis to shop location), and transparency provided by the ReAct trace. While cost estimations remain approximations based on web searches rather than static databases, the system demonstrates how multi-agent collaboration can automate repetitive data review and accelerate the insurance payout lifecycle.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Autonomous Systems & AI Agents and was classified suitable for intermediate domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
Project Goal and Business Impact
Imagine filing an auto insurance claim. Instead of waiting days for a damage evaluation, photograph the car with your phone and, within minutes, receive a detailed assessment.
The primary objective of this project is to drastically improve the efficiency and objectivity of the initial auto insurance claim process. Current methods rely heavily on human adjusters and manual estimates, resulting in delays and potential cost inflation. By deploying a sophisticated Multi-Agent System, the aim is to provide a fastly, data-driven assessment that benefits both the insurer and the customer.
The Multi-Agent Architecture
At the heart of this solution, there is an orchestrated system of specialized AI agents, each with a distinct role. The architecture follows a sketch where an Orchestrator Agent works as the brain, creating execution plans, managing agent lifecycle, coordinating the execution, and aggregating results into coherent outputs.
The Vision Agent, powered by OpenAI GPT-5.2, acts as the system's eyes. It analyzes uploaded damage photos with technical precision, identifying specific damaged parts (bumpers, panels, headlights, etc.), classifying severity levels (minor, moderate, severe), categorizing damage types (collision, scratch, dent, paint damage), and generating detailed technical assessments.
Two specialized Cost Estimation Agents run, representing different repair philosophies. The OEM (Original Equipment Manufacturer) Agent focuses on premium repairs using manufacturer-certified parts from authorized dealers, while the Aftermarket Agent explores cost-effective alternatives using quality certified aftermarket parts from independent shops. Both agents are powered by Perplexity's sonar-pro model, which provides access to current market data and pricing information.
The Shop Finder Agent searches for repair facilities near the user's location, provides contact information, ratings, and availability, and adapts its search strategy based on the information retrieved.
Technical Highlights
The system is built in Python, leveraging several key technologies. The Gradio/Streamlit framework provides an intuitive web interface for image upload, location input, and real-time results display. OpenAI's GPT-5.2 handles computer vision tasks. Perplexity's sonar-pro model accesses current market data for repair costs and local business information.
A sophisticated state management system provides each agent with memory of past interactions, confidence scores to assess decision quality, performance tracking to optimize the system, and context-aware autonomous decision-making.
At the core of each agent's execution is the ReAct loop: a Reasoning, Action, Observation cycle. Each agent doesn't just call an API and return a result; it first records a thought explaining why it's taking an action, executes the action, and then logs its observations. This trace is accumulated across all agents and surfaced in the UI as a collapsible reasoning log, making every decision in the pipeline fully auditable and transparent.
Generative AI vs. Manual/Traditional Tools
While traditional automated tools rely on rigid, rule-based computer vision and static databases, this Multi-Agent System introduces a modular reasoning layer that bridges the gap between raw data and decision-making. According to the industry research from McKinsey (2025) the agentic workflows reduce claim cycle times from days to seconds with consistency in claim evaluations.
Traditional tools are often "black boxes" or monolithic scripts, instead this modular architecture give the opportunity to develop in the future every task as a swappable module for an hybrid framework where every single agent can be replaced by a non Generative AI tool, for flexible, custom and scalable solution.
The Future of Insurance Claims
This multi-agent architecture is a robust, scalable blueprint for automating complex decision-making business processes, such as insurance claims. It leverages the strengths of several large language models (LLMs) and specialized agents to deliver a fast, transparent, and comprehensive output that far exceeds the capabilities of a single model. The project demonstrates practical, real-world applications of multi-agent systems in production environments.
Links: Article Hugging Face App Repository
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:00]
Please do not ask questions verbally during or after the presentation. Simply enter your question as text or like interesting questions that have already been asked there. In addition, please mute your mobile devices so that they do not interfere with the conversation.
Speaker 2 [00:21]
We look forward to you
Speaker 1 [00:22]
to you sharing your experiences with PyCon and PyData Darmstadt 2026 on all your social media channels.
Speaker 2 [00:37]
to having me today here, nice people, nice location, I'm happy. And today I want to speak about the agentic AI in the claim processing and much more in the car damage evaluation. But before to start, I want to introduce myself. I work with numbers, with uncertainty and risk. I'm an actuary during the day. and I play with data in the AI and data science in the free time. Ok, look at the agenda, it's really rich. We start with the motivation, the use case, the introduction and we go also with the demo. So, the motivation is coming from many years ago, not many but maybe 6-10 years ago and about the whole idea to develop a deep learning card image classifier. The idea was simple, to use an app to upload the image with the binary classification or the multi-classification, depending on the damaged part, and also to annotation with the repair cost. What is the issue? sure that there wasn't the data set with the labels and also with the annotation cost. So I shoved the idea. But some months ago this idea came back to me in mind because I thought about the use of the iGentic AI. This is the state of art about some years ago when I developed the app with the deep learning classification. So I found this Streamlit app and they are based on binary classification, multiple classification or otherwise with some parts with scoring but something about guessing. So if you look about the claim processing, it is really slow because it starts with when you have a car accident, the customer gives the notification and some of the documentation, Then there is a claim handler for the claims to investigate about the fault, the evaluation of the cost. And there is also the approval about the claims and then there is about the communication and the reimbursement about the payment. So, the claim processing can start for 2-3 weeks for a simple claim or otherwise add more than one month, more than one year for litigation claims. So that here what can be, if you use an agent TKI you can really become your claim processing faster with the data review process. But before to start to talk about the Gen-TKI, the revolution is coming from the transformers, yes. Because years ago from the communication, the paper Attention is All You Need from Google in 2017, these transformers changed everything. The idea is brilliant because data are not processed sequence to sequence, but at once, with attention mechanisms that give the weight, give the much importance, what is the part of data that matters to each other. And this way you can take about the long range of dependencies between data. The model language models are using transformers, yes, different from the first used for translation, and are similar, for example, for the GPD for the text generation to create the responses, text responses. The revolution for the computer vision is coming also from transformers, because transformers is flexible and you can use transformers also for visual language models, just to use visual encoder that capture the images, create the visual features, this visual encoder is trained on large image or text data and image and create these visual features. Then there is a projector that is a bridge between this visual encoder and the transformer. This projector maps the visual features into the embedding language with image tokens. In this way the transformers have the same embedding with image tokens for images and text tokens for the text and can process both. So this is the idea for visual language models that can be now the state of art for the computer vision. But what happens if you give to your visual language models or language models the possibility, opportunity to act, to interact with the environment? became an agent. So an agent is a system where it is able to perform a task, a complex task or a simple task, given by reasoning, by the planning, by the action. So it starts with the perceiving, with the perception, retrieving the information from the external environment, and then a reason about the act to do, the decision to make do, and in this way is handled by the large language model or visual language models. And then with action, interact with the environment, just calling some tools such as APIs or other querying database or something else. At the layer above this there are the planning that the agent can plan breaking down the complex tasks in a sub-task and also is able to give the priority of each step. Also there is the memory and knowledge. With this memory and knowledge, they retain the context of the interaction and for some agents there is opportunity also to learn with some external knowledge. So, why they use the object agent? The use of the object agent is why because they are able to give some complex task or a repetitive task in an autonomous way without the human intervention. So they are also able with the tools to adapt to the context and in this way, given that they are able to give some repetitive task, they improve the augmented human knowledge for brainstorming, for problem solving, for automation and they can be used in for repetitive tasks to increase the productivity, leading people with into higher level activity, much more efficient, much more important activity, so improve the efficiency inside the company. And also for the customer is able with knowledge, with external knowledge can be also memorized some personalization, so give some personalization experience for the customer. So there are many opportunities to use agents. There are different types of agents, just two extremes. One is using a single agent, in this way you have one agent that gives the whole task, but it's easy to realize, but it's less scalable. But instead, on the other side, there is the agency. There with agency you have multiple agents that work, that collaborate. In this way they are processing much more information and they are really good in the company, but the problem is coming from the coordination. So what I realized? I realized in an app that the stack that I use is coming in with using the Python framework. I decided to don't use any external framework for one reason. Okay, you have high stack, you have long chain, long graph, you have a clear power outage, but you have some strict framework. So with Python, you have governance on everything. You are done, so you are free, and you are not linked with the external break change. So for the company, it could be good because you have a stability, and you are easy to implement, and also you can grow in your implementation. And the others are framework, the API, OpenAI and Perplexity. OpenAI for the vision agent and Perplexity for the cost agent and the final shop agent. The deployment happens on a hugging phase, so for this reason my first choice was Gradio. And OK for the backup, Streamlit. Before, to start with the demo, just an introduction. Okay, I started months ago just for a simple pipeline based on the function, the call API with open eye and perplexity for this type 3 type of agent. The visual agent, the cost agent, and the shop finder agent. But it was easy. So it didn't adapt to the contest. So I thought about something much more debatable, much more audible, and I thought about how to use the agent system. With the agent system, I thought in this way, creating the react loop, reason, action, and observation. With the reason, okay, you have the field start, react, thought, the agent state, there is what happens, that the agent in this way thinks about the decision to make, then there is action, So, with the reaction action, the agent decides which function to use. And then there is the action fn, that in this way it works, because in this way it calls the API and retrieves the data, the raw data, and with the observation, it sees the data, they are registered in the logs summary. So each agent has an internal state with a Python dictionary, and when you call, when you start your agent, you call with the create state agent, agent state with the type of agent and with the configuration. In this way, you have an agent that is explainable, you have a self-tracking with the drug trust, and also is stateful because you have the state. So this is the architecture of the app. You upload the picture with the location, you have this information go into the orchestrator agent. This orchestrator agent maps everything, so this plan, each step of each agent, each agent works sequentially. There is the vision agent, this is the first that analyzes the images and provides the detailed information with the damaged parts also with the classification of the severity mean or moderate or several this information does in into the context and arrive to the two cost agents in this way they have there is a dealer service and the independent shop why two cost agents because i don't have a data set so i use only the picture from kaggle and my what i've trust about the cost, the estimation of the cost. Using two cost agents I have the opportunity to have a comparison, because these cost and these estimations are coming from the web. And then there is the shop finder, it was looking about from three to five shop locations, shop repairing costs into the location, and then this information are grouped and then for the output in a stream little radio interface user friendly phrase okay now i start with the the app okay the demo live okay you can see this is the um grade your apple on your game face i upload the i written the api i upload an images and i okay put the the the location just you Push the button, okay, it takes just less than a minute, 50 seconds, so it's really quite fast, but because there is just one loop, and I think that it's a really good achievement for the customer, also for the worker, because it can be used both for the customer and both for the worker. So, first of all you have the damage analysis, in this way you have the description of the damage analysis, you have the affected part, in this way, how the bonnet, front grill, assembly, left hand map, assembly, right hand map, assembly, so on. And you have also the classification, in this way is several. After that, scrolling down, you have the two comparisons about the range of the cost. And for each cost there is also the breakdown of each component. After that, also, there is a shop finder. We have the description of each repair shop with the details, and there are address, phone number, website, maybe, okay, depends, and okay, in this way, also we have at the end the React trace, that is the nice things of this app, because, and we look about, for instance, we start with the orchestrator, the total we have, have received inputs with image, yes, and location, location, new, Germany, I need to determine which agent to activate in what order the action is a build execution plan so observation we have the execution plan will with four step vision cost primary cost primary cost alternative and shop finder thought i have a plan with four step i will execute a charging priority order vision first then cost estimator then shop finder passing results downstream and context action to execute agent plan is the function, observation, all four agents, task completed, vision, cost, primary, cost, alternative, shops. For the vision agent, we have the thought, I have received a car damage image, I need to call the vision model to identify damaged parts, assess the severity, and extract a structured description. The action is the call OpenAI Vision API, the observation is a vision API responds successfully, detecting severity, sever, confidence, 78%. cost agent we have the two cost agents with the tote i have damage information with severe server and i need to generate a detailed estimate for the primary repair philosophy in munich germany using euro i will query the perplex api with the structural requirements the action is a call of the api perplex api the observation is the cost api the primary responded successfully estimated the the range the same for the aftermarket and also the shop finder agent yes okay the tote i need to find out about the repair shop near Munich, Germany, it will require a perplexity which is a structural request for shop names, addresses, rating and the contact details. The action we have, they call the API and then the shop, the observation is the shop search API responded successfully for location Munich, Germany retrieved a shop listing three selection, three section found in response. So this is the app that I realized. We come back to the presentation and to the slides. So this part we have just watched about the demo, but just to have a comparison between the generative AI, the AI and tools using about convolutional neural network. So the transition, what happens? Okay, we think about the convolutional network, you have a general in the internal generalization, if you want to you change your task, you have to retrain your model if you have your goal is only to make a classification single classification binary or multi-classification single task so it's not adaptable with the adaptability with the new prices require new training and also about the explainability is a black box and also they are monolithic scales so the entire pipeline if you change something you need to retrain with the genetic AI, with this model, what happens? First of all, yes, you have the reason patterns, so you have something with a react loop, you have a trace of your reasoning, and also you have a description, you have a language description, so it's much more. Also for the goal, you have end-to-end assessment, much more tasks, so you have a reply to much more questions. The cost, the analysis, the cost, and also they find the repairing shops. Also, you have much more data update with the market because you have web search data from the market. It is considered transparent because you have the description of each action from each agent. and also from the modularity you can switch each step each specialized agent replaced with also with the convolutional network for instance or you can add other stuff other agent well the other ways there are some constraints for instance yes your estimation is approximation because i haven't a data set so it required for instance for your this this this cost estimation is coming from the web search so it's necessary the professional man so human in the loop then also depend of the quality of the image because if the image is blurry yes your estimation is bad and okay you don't have for instance your static you either don't i don't have a static database in this way what happens that the data is not guaranteed from this static database so it can be used for instance it can be implemented as a database for a comparison because now i'm using only market search and then for about the regularity the insurance world is much more regulated, yes there is, the app is auditable because you have the trace of each agent but maybe for the put in production requires other alignment. So what I have learned from this experience for building from scratch is important because you learn how it works the ecosystem and also it's possible to realize for instance multi-agent system. This app can be used as a starting point for the claim processing and maybe can be used as the initial process then building from scratch before to transition on external framework because at the moment the external framework are unstable because they are changing a lot. Also, what is the opportunity from the multi-agent system? The opportunity is that you can break down your complex tasks and change the claim processing into sub-tasks. And then you can create end-to-end pipeline. And then, okay, you can create your multi-agent system not only with the specialized agent but also using different brains, using different type of language models, visual language models. I used OpenAI and used Perplexity for different tasks. Okay, there are the references and that's it. Thank you.
Speaker 1 [21:35]
Thank you very much for an interesting talk, Claudio, and thank you for asking questions. If you have more questions, you can access here talks.pycon.de and ask them right now. We have a little bit of time, and the first question.
Speaker 2 [21:47]
Yes, you look
Speaker 1 [21:48]
From the audience is, how do your agents calculate confidence scores? For example, calculating confidence and uncertainty scores of a multi-classification task in the image detection is a statistical challenge that vanilla agents are not capable of.
Speaker 2 [22:05]
Yeah, exactly. This is an important question, a really good question, because I thought about the confidence. What happens? This confidence is given by the system, retrieved by the agent. So I'm not able to check about how is it calculated. This is a black box point. But it's important. I consider to use the confidence for one reason. because with the confidence you can understand how your result is good and for instance for in a next development to give a next step in the pipeline so yes that a good question because you building a deep learning model you have the framework that you can build your confidence here is external confidence
Speaker 1 [23:00]
Thank you. The next question. How do you quality assure the proposed multi-agent system?
Speaker 2 [23:07]
Okay, it's difficult because, good question because, okay, the problem exactly is that I don't have the data set, welcome if something want to propose a data set with the database with cost and we can have the quality assurance, quality evaluation about the multi-agent app. I created two cost agents, for instance, to have a comparison. But yes, a good quality is coming from a benchmark, and I have to create. So someone that is working with this type of data is welcome to give the help for a comparison.
Speaker 1 [24:00]
Thank you. The next question is, how do you leverage insurance conditions documents? For instance, how would you decide coverage determination using LLMs?
Speaker 2 [24:12]
Okay, here there are not about the documents, this app is just about evaluation of the picture and the location. So how it can work? It can work that you have the app, the customer, also the worker, upload the picture, it's just for one picture, but you can upload many pictures, it can be realised, and with the location you have an evaluation. It can be extended with for instance you think about the MSCP server you can create another app and you can assemble this app and you have for instance about the documentation you can retrieve a documentation from the customer and then you can build just a rich pipeline so this is a just a piece of the claim processing.
Speaker 1 [25:07]
Thank you. The next question is, how is such a system deployed in production? Which stack does it use?
Speaker 2 [25:14]
Okay, it's deployed on Hugging Face Space. You have the address, for instance, in the slides, also in the channel of the Pack-on-D, from the MyGitHub repository. And what happens? I deploy it on Hugging Face using Gradio, so it's a simple deployment, easy deployment, not a sophisticated deployment. So to put in production requires much more work. So it's just a prototype, for instance. But I think that it works. And this was, as I said, a Python framework, essentially.
Speaker 1 [25:58]
Thank you. The next question is, why do you consider the agents more transparent? How do you explain 78%, for example?
Speaker 2 [26:08]
Yeah, yeah, yeah, okay, exactly, the question is interesting, okay, what happens, what is the reason that I consider much more transparent, pick up a deep learning model, a convolutional neural network, okay, it's fine, it's the state of art of some years ago, it also can be used now, but you have a deep learning model that is a black box, the agent inside the LLM or visual LLM is a black box. What is the gain that you have between the previous system? You have the react loop. You have the summary of each action taken by the agent. The reason, thought, the action, so the calling from the API, and then the observation. So you have each step is just documented. So you have the movement of the agent. Yes, inside, I agree, it's a black box, but it's just an upgrade despite this convolutional neural network. What happens about the scoring at 78%? The threshold is 70%, I put. It's a general model with a computer vision from GPT-5.2. You can improve your accuracy with a fine-tuning model. just I think that is good with a generalization model you have it is used in a zero-shot learning way so it's good for my opinion is good but it can be improved with fine-tuning model fine-tuning yes
Speaker 1 [27:45]
Thank you, Claudio. The next question is, given the flow seems very linear, have you tried to compare the multi-agent architecture versus a prompt flow based architecture? If so, where do prompt flows fall short in your experience?
Speaker 2 [28:00]
Okay, just I created the pipeline at the beginning, just a linear flow, and okay, I haven't now to watch, but what happens is that you have just a description, You don't have the classification of the severity, and I have just the range of the cost. So the next step with this app is much more informative than the previous. It depends. You can build just with a flow chart, but what happens? That is not adaptable with the contest. If you use the computer, the visual agent, and the visual agent gives the information that is vague, With a simple flow chart, this information is passed to the cost agent, and the cost agent doesn't have the opportunity to improve the information. This vague information is spread into the downstream, falling into the cost estimation. Instead, with this multi-agent system, the opportunity is much more informative, gives the classification. So, if they're failing about the cost vision, for instance, give a GAV-VAG description, but give a classification, the cost agent is raising about which type of cost estimation to do. For instance, if it's a sever, decide to give a detailed cost estimation. If instead it's a moderate or minor, decide to have a light estimation. So, there is only one loop, but if you improve with much more loop, you have a reasoning. So, with just a flowchart, you have one shot, you have a result from the API, and if it's good, okay.
