Electify - Retrieval-Augmented Generation for Voter Information in the 2024 European Election

Speaker 1 [00:07]

We'll talk about Electify. It's an open-source tool that we built together with Anna Neiffer and Joshua Nowak and it's a tool for retrieval augmented generation for vault information. But first I would actually like to start with a question which is what is the source of this sentence? So for all the German speakers, now is your time. Any ideas? It's very long, I know. Okay, five minutes later, now unfortunately I have to speed it up. This is from Party Manifesto for the German federal election, which was just held in February. And as you can see, it's very long, right? It's not only long, it's also complicated, so the logic is like split and then right Right in the end you have the verb, of course, got to love German. And long sentences is actually not the only problem in these manifestos. So you have also things like this, . This is also nice. It's like a long compound word of both English and German words and whatever the hell this is. And on average, party manifestos compile like 73 pages for the last election. And this was actually short because usually they are longer. So I would say this is, well, first of all, you can even quantify kind of the readability of these documents. And here I steal data a bit from a really nice study from Hohenheim University, which you can access via the QR code. And basically what they do is they have an index which quantifies how readable a text is, right? This is called the Hohenheim Comprehensibility Index. It's defined on a scale from 0 to 20. And you can calculate it for different text sources, right? So BILD articles, that's the German BILD site, are actually really readable. They are like 17 or something. Parliament speeches are actually also not so bad, around 15. Wikipedia is like 11, and then the party manifestos of 2025, they are at like 7, right? So, not great, like substantially worse than Wikipedia, although they're still better than dissertations on political science. So, yeah, I would say this is kind of a problem, because party manifestos, in essence, are the promise of a party for the next four years, right? They tell you what they're going to do. And if I have to scan through 73 pages of text, which is harder to understand in Wikipedia, not that likely to do it, right? So, and this is just one party, of course. So at the same time, of course, we all know there are solutions to this problem, right? Like big one is journalism. Of course, you can read newspapers, right? And there are also other really nice tools which were invented to exactly simplify this process this process of informing yourself over the party positions. A really nice one is Valomat, and there are a couple of other ones that go in the same directions. So, in principle, how these tools work is you get a statement like, linting tools shall be banned. Code style is a personal journey and a form of self-expression. And the statement is then presented to all the different parties, right? So party X might say, yeah, wipe code away. Party Y is against it, and Party Z is like, what the hell is a linting tool? And you have several of these statements. And then as a user, you can go to Walmart's website, and you can also give your opinion on the different statements. And in the end, you can compare all your statements and what you chose with the parties, and for each party, you get kind of an alignment score. This is great, right? But I would say there's also like an intrinsic limitation of these tools, which is, if you think about the space of all the political topics, right, that are currently somehow relevant to this election, what these tools do is they try to sample this space evenly. And I would say even they do a great job at this, right, this is highly skilled political scientists who spend a lot of time to compile questions which are unbiased and like fairly sampled across all political topics. But the problem is, and I hope this shows now on this color scheme, what I'm interested in is a subset of this, right? I don't care about some topics. Some are not relevant to me. And on the other hand, maybe there are some really niche topics I'm interested in and they are not covered by Palomar. And I think exactly at this point, there's actually a big opportunity for LLMs because They work kind of in a reverse way, right? You can ask any question. First of all, you can have individual questions about exactly the topic you're interested in. And you can also choose how coarse they are, right? You can ask what to do about the economy, but you can also ask how should this specific fish be protected. It can be really detailed or broad. At the same time, we know asking for elements is not enough, right? And if you do that, then you have different problems, like LLMs usually don't have access to relevant data, even if you use some agent with a search tool, it's not guaranteed to find the right data. There's a training cut-off, right, so the training data is outdated. And in the end, after the response, you typically don't know why the LLM told you that. You don't have access to the actual sources. And of course, they sometimes just make stuff up. Okay. So one tool that I think everyone knows and is almost old by now, I feel, is Rack, right? And the idea is this was a tool kind of created to fight all these problems. And although I think probably most of you know what it is, I will quickly go over it just to not lose anyone. So also we start here with a user question. So this time first we retrieve relevant context from a vector database. So this is a database in which we store relevant documents for the typical queries in our system. In our case, this will be the party manifestos. And then we augment the prompt with this retrieved context, right? So the prompt now consists of some system prompt that tells the system kind of what to do and how to behave, then the user question, and then the retrieved context. And this way, the LLM has access to the relevant information needed to answer this question. Finally, you pass it to the LLM to generate the response. Okay, so let me now explain a bit how we implemented the system in our case. First, starting with the vector database. Yeah, so... Okay, some boxes are missing. Okay, so we start with loading the party manifestos. These are just PDFs in our case, and they are actually not very bad PDFs. There's no tables, no weird layouts, so it's relatively straightforward to parse this with standard PDF libraries. In our case, it was PyMuPDF. And then an important part is actually slicing. Because in the first step, what you get is essentially a huge string, and you have to chunk this in smaller slices so you can eventually put it into the vector database. Important parameters here is the chunk size. so in our case it's 1000 characters which is really large for a rack system but we found that in our case it works best because i guess political content is very nuanced and really depends on the context and at the same time also chunk overlap which shows very large this is basically if you have a sentence like this with no overlap it would be chunked into two slices right red and yellow and if you have an overlap then these orange words they would belong both to the red chunk and to the yellow chunk okay and then finally you create the embedding vectors right so the embedding is basically just you take some text and you use the embedding model to create a embedding vector which is a long list of numbers which kind of encodes the semantic meaning of the text okay and finally you install all this information in a vector database in our case chroma which now contains the text slices the embedding vectors and also some metadata most importantly metadata about where to find this particular slice or kind of which pdf and which page okay um and then comes the retrieval part right so question comes from the user again you create the embedding vector of this question and then you compare this question embedding to all the embeddings in your vector database and we retrieve the 10 slices which are most similar and then another step which we found to be really useful in our our case is maximum marginal relevance reduction won't go into much detail there but basically what it is is a re-ranking step which reduces redundancy in different slices and we just basically saw that this works better which is why we used it but the intuition is a bit if you have a broad question like again what to do about the economy then there will be a lot of slices which are kind of similar and this step kind of increases the diversity in the context that you retrieve Okay, and finally, you build the context by basically just concatenating all the slices you found. All right, so here I already told you ten slices, five slices, some chunk overlaps, like why, how do you choose these parameters, right? And I would say this is where Ruck becomes difficult, in my opinion, because it's really fast to do it, right, and it kind of works. But evaluation and hyperparameter tuning is, I think, not so straightforward. So I would like to explain a bit how we did it and what we learned, especially on the way, I think. So first of all, why is it hard? I think it's hard because compared to like a classification problem, you don't really have hard metrics like, I don't know, the precision or something like this. So typically what people resort to is these LLM as a judge metrics, and they can be quite unstable, irreproducible, and so on. So they are not so nice to work with, I have to say. But one thing you can use is that there are metrics which kind of depend on different parts of the system. So if you have the retrieval and the generation part, then the metrics like context relevancy, which basically tells you about the fraction of slices which are relevant to the question. They only depend on the retrieval part. So this is nice because then you can tune certain parameters which influence the retrieval and look at this metric to see how you do. And on the other hand there's faithfulness which is kind of tries to extract statements from your response and then rates how many of these statements are actually supported by the context. So this is a kind of a good measure for hallucinations. And there are also other the metrics which kind of depend on the whole system right so so you can use this fact that they depend on different parts of the pipeline so here's an example where we tuned the embedding model and basically the thing to look at here is the context relevancy for example okay so we did this for the embedding model LM the number of retrieved slices etc etc but also we did it for the instructions in the prompt and this next slice kind of pains me a bit, because I hate prompt engineering and the whole concept behind it. But the prompt is really crucial, it is really important. And to showcase why, let's go back to our example again. So linting tools shall be banned. If you just use a very naive rack prompt, out of the box kind of, you get this response. So the party is committed to a liberal culture of debate, blah, blah, blah, blah, blah, and Because linting restricts personal development and individual style, right? And maybe it doesn't come to surprise to you that in none of the party manifestos they talk about linting, okay? So obviously this is nonsense, right? The LLM just kind of tried to cook up an answer which kind of made sense, right? Because it tries to follow the instruction. The instruction was give me the party position. So it does, right? So first of all, the prompt is really crucial to manage irrelevant context. In this case, the context is not relevant to the question. So I would expect an answer like, I don't know, I cannot say, right? And the second thing is that metadata is also really important, because you can choose, for example, to put the party name in the prompt or not. And in our case, we saw actually the performance is worse if you do, because I guess the LLM of has preconceived knowledge about the political leaning of a party which might be outdated. And third, of course, you can tune the response style, right? Like most prominently, let's say, how concise do you want the answers to be and so on. Okay. But let me actually focus on this first point, the irrelevant context. And let's talk about refusal rate. So refusal rate actually means, if you have some evaluation dataset, the fraction of questions in your set that the system does not reply to, where it says, like, I don't know, basically. And we did an experiment where we looked at different prompts. And on the y-axis, actually, I plot the faithfulness score. So one basically being the optimum, you have no hallucinations at all. And on the x-axis, you have the refusal rate. And in this plot, each data point corresponds to a different prompt, okay? And the only part of the prompt that we changed was the one instructing the model to refuse an answer. So here on the left, basically, there is no instruction to refuse whatsoever. And indeed, the refusal rate is zero, right? And then here on the other hand, we have a really strict instruction kind of like do not answer unless you're certain, et cetera. And you actually get a high refusal rate. But the important part of what this plot is that you see that there's a tradeoff between refusal rate and faithfulness, right? So basically this means we can increase the faithfulness at the cost of a larger refusal rate which i think intuitively makes sense because especially in the cases where the context is not relevant you have a high risk for hallucinations right and for our system actually we chose this prompt which actually it's a bit painful because now it means our system does not reply in 15 of the cases right which is a lot and actually a big chunk of the user feedback we got but on the other hand we have 99% faithfulness right and I think for a system like ours that tries to inform voters it's really important to actually choose this one because the kind of the worst case that we can have is that we do misinformation right it's kind of the opposite goal of what we wanted to do okay and finally another valuation that we did which which is nice is we can benchmark our system against the Valomat, because the Valomat in some senses is a great data set, right? I already told you that they really take care to avoid bias and so on, and also the responses they come directly from the parties. So this is a really good ground truth. So what I did here is I took the Valomat statements, I pass it to our system to generate a response, And then I take another LLM to map this response onto agree, neutral, disagree, because these are the categories of the Valomat. And then you can compare it on to the actual party responses and compute metrics like the accuracy for example. And here I was really happy that this is 91%. And especially happy I was because all the misclassifications are only between disagree neutral or agree neutral. And of course, neutral is kind of a slightly ambiguous category, so it's not as bad as confusing disagree and agree, of course. And the second thing that you can see in this plot is that also here, the refusals really increase the accuracy, right? So with no refusal prompt, you have like 77 percent accuracy. And then with our system as we deployed it, you get 91 percent. So this is a really, really big difference. Okay. So now it's time for a quick demo. So we deployed this last year for the European election the first time. Had about 6,000 users who also asked questions. And then we did it again for the federal election now in February. Yeah, so this is a pretty simple web application, so based on Streamlit and then Google Cloud Run or Azure Container Apps, whatever gave us the most free credits at the moment. And yeah, so let me just quickly walk you through it. So this is how it looks. You have here the question you can type. For example, I can ask, how should artificial intelligence be regulated? And you can choose which parties you want summarized, right? Oh, wait, actually, let me switch to English. Okay, and if you start generating, then first you see a quick banner which tells you like okay this is ai generated be careful and then you see the party responses right so the order is is random we have a big small disclaimer for afd and then you see the the different responses for the parties right and if you click on these links here you are redirected to the pdf which is the party manifesto to the page which is relevant so here it's about digitalization and further down here they also talk about AI, the AI act. If you scroll down even further you have more information on what this stuff is based on, so you can click on the sources and you see the actual chunks that the model used. And finally you can give feedback. Okay, and the last feature I want to show is that you can actually also choose to hide the party names, and then the answers are presented like this. So ideally you should do this, of course, first, right? But the idea behind this is that if you look at such a system and you see the summaries, of course you have already your kind of favorite party in mind, right? So you have a pretty big confirmation bias that you kind of want to like the party that you want to vote for. So if you hide the parties, you can then first, like, read the summaries, like, free of judgment. And if you say, like, okay, maybe I agree to this, you can read the party. Yeah. And that's it. And there's also a small support button to help us with inference costs. Okay. Yeah. So with this, almost at the end, I would like to go to the conclusion. So the first one is let's build open source RAC applications. First of all, it's a lot of fun. It's surprisingly cheap, even if you have a lot of users. It really doesn't break the bank. You learn a lot. And also I think there's lots of public data which could benefit of things like this, right? Think about all the open government data, like parliamentary speeches, there's so much. So go wild. Also, the second take-home thing for me is let's embrace refused responses in LLM-powered applications. I think very often we tend to see this as kind of an error, like if you refuse, like it didn't work, but it can actually be an opportunity to avoid wrong responses, right? So handling refusals I think, yeah, is a nice handle to increase the accuracy of your systems. And finally, yeah, let's discuss. So looking forward to your questions and also I'll be around later. Let me know if you have ideas or if you think it's a terrible idea and irresponsible. I don't know. Let's talk about it. So with this, thank you very much. I have a quick link repo here where I link, for example, the study that I showed before and also some other open source projects that we stumbled upon on the way. So if you're interested also in our code, you can scan this QR code. And thank you very much. Looking forward to your questions.

Speaker 2 [21:47]

Yeah, thanks, Christian. There's a bunch of questions. The first important may be, whom do you work for? Who has financed your work?

Speaker 1 [21:57]

So we have financed our work actually. So it's a completely 100% private project. It's not no company. Nothing Yeah, I mean we had some support we are buying your coffee of roughly I think 70 euros something like this But it's it the cost of this project is really on this order. So Yeah, maybe this is also something I should highlight so I think overall we spent on the order of hundred euros and And we, I don't know, people ask like 15,000 questions. It's not expensive.

Speaker 2 [22:34]

Okay, next one. Have you thought about chunking the content based on the structure of the document by sections?

Speaker 1 [22:42]

Yeah, I think this would be an obvious improvement actually to go because these manifestos usually they have sections with like pretty obvious headings.

Speaker 2 [22:43]

Yeah.

Speaker 1 [22:51]

So we did not like mostly because we have all full time jobs and this was kind of a side project. But this would definitely be an obvious improvement.

Speaker 2 [22:59]

Okay, the next one. Besides mentioning the party name in the prompt, did you observe any issues with political bias baked into the LLM?

Speaker 1 [23:10]

Yeah, this is a very good question. So, of course, all the evaluations that we did, you can do for each party, right? And indeed, you see there are differences. However, it's a bit hard to say why they occur. So, I mean, one observation for sure is that parties that tend to have more polarizing content perform worse. I'm not entirely sure why. It could be bias of the LLM. It could also be that retrieval is worse because I think also one thing is if you create such a data set, it's not easy because you can phrase a question very differently, right? You can say, what the hell is wrong with whatever? This is a question. But you also can say, like, what is your opinion on X, right? And this makes a difference because the style is also part of the embedding, right? So yeah, short answer is it's hard to say.

Speaker 2 [24:17]

Okay, so, oops, that was a ring. Five minutes now for answering, so this is good. How much did running this cost for the last election? Sorry? How much did running this cost for the last election?

Speaker 1 [24:34]

Yeah, last election we made actually we're able to do it even cheaper because we also found free credits on ever open AI

Speaker 2 [24:34]

Yeah.

Speaker 1 [24:43]

So I think yeah, I think last election was pretty much we had to renew the domain like for some euros That was it

Speaker 2 [24:52]

Okay, then did you notice much difference in the results between the German and the English versions?

Speaker 1 [25:00]

Yeah, that's also a really good point. So, how we do the English version is actually really simplistic. So, basically, we just take the English prompt and do the retrieval. And then we just instruct the LM to answer in English. But, of course, the context is German, right? And surprisingly, this does not degrade the performance much. Like, it's a very small difference. Of course, you could improve it by translating the prompt, then doing the retrieval, but it doesn't have a big influence.

Speaker 2 [25:32]

Is extending support for smaller or newer parties like ValoMod does too challenging for LLMs?

Speaker 1 [25:46]

No, no, no, no, I don't think so. So of course you can, that usually there are like 30 parties or something that take part in an election, right? Here we chose to only take the ones that were currently in the Bundestag. Just I mean, also basically there are two reasons. The first one is a bit time. So yeah, as I said, it's like it's not none of our main projects, right? So and the other one is that actually in the previous version, we also used debate data. So data from parliamentary debates. And that, of course, you can only do if the party is already represented in the parliament. So basically, that's the second reason.

Speaker 2 [26:29]

Okay. Have you thought about using other information for the parties like interviews by leading figures or past votes in Bundestag? Yeah.

Speaker 1 [26:40]

Yeah, also a very good question, of course you can do that. It's more challenging, though, because typically party manifestos are at least in general a neutral style, but parliamentary debates have very challenging things like sarcasm and humor. So people might say something but actually mean the opposite, and this is really hard for LLMs. And the same with interviews, yeah, the content is much harder to understand for LLMs to pick

Speaker 2 [27:11]

Okay, then is co-junking at headings work?

Speaker 1 [27:16]

Yeah, I think we had this question. Oh, okay, that's good. Yeah, it works and probably is a bad idea. We just didn't do it yet.

Speaker 2 [27:24]

Okay, there are lots of questions that are incoming and I cannot sort them. But you can if you vote up some. For document parsing, have you considered dividing the content by headings and sections rather than using predefined chunk size?

Speaker 1 [27:40]

Yeah, again...

Speaker 2 [27:42]

okay good thanks sorry did you consider choosing developers with different product political views for example in designing your problems

Speaker 1 [27:57]

Yeah, also very good question. I have to say, gotcha. So, of course, we are friends, right? We tend to be kind of from the same political bubble. That's true. So, yeah, I think the best I can answer is that we evaluated against the Val-O-Mate questions, which are, I think, a pretty good benchmark.

Speaker 2 [28:21]

Okay. Let's see. Are you looking at data gathered from the prompts? What are people asking more about and so on?

Speaker 1 [28:31]

Yeah, actually, I'm really sad that we didn't collect it, because we were afraid of data privacy. Because we talked to some people, and especially for the last election, I think now people care less. But back then, a lot of people were really concerned that we would get a shitstorm kind of about this. And then we took great care that at least the data privacy side is really like we don't collect anything basically except just like usage in general, like visits and so on. But yeah, so we don't know.

Speaker 2 [29:08]

So the last question for today, for here, but there's also a coffee break maybe. What effort goes to explaining users how the system works?

Speaker 1 [29:19]

Yeah, I mean, I think from the feedback we gathered, most people didn't have problems navigating the site. I mean, we have a quick about section where we explain how it works, kind of. But that's it. Like, there's not any further instruction. It's literally just like putting in a question and clicking go, right?

Speaker 2 [29:42]

Yeah, thanks Christian, and yeah, I think you enjoyed this, so let's maybe give him an additional applause.