Tracking Knowledge Diversity in LLM-Generated Responses.

Speaker 1 [00:35]

Okay

Speaker 2 [00:37]

Let's start today's session. I'll be your chair and welcome. We have three very nice talks scheduled here, starting with Sara Masood, who will be telling us about tracking knowledge diversity in LLM generated responses. Please take it away. Let's welcome her with a round of applause, please.

Speaker 1 [01:02]

I hope you guys can hear me. So this work is actually based on a research that we've been conducting. So this is the first time we are trying to convey that research in a non-researchy manner. So I hope that I'm able to present some useful information out of this. Yeah, so we all have been there where you want to get some information from a large language model. You usually are dealing with a chart-based system, and you go there and you type, tell me about X, Y, Z, let's say tell me about democracy, and you begin reading the responses, and you start interacting with the system, and after almost 20 minutes, having read through multiple prompt variants, you feel like, I could have just Googled this, because now I've lost track of across these different prompts that I gave, what was the unique information, have I even been gaining any new information or not? So the question of, like, how diverse the responses that we get from these LLM-generated, like, answers is, and, like, is it better than Google? So that's basically the premise that we are working with. But the question becomes... And yeah, the goal is that, okay, in our research setting, we have around 200 topics. These are your general knowledge topics, so democracy, politics, things that you have a good Wikipedia page about, then you prompt for this particular topic, you prompt your language model with multiple prompt variants, so we take almost 200 writing prompt variations So give me an essay, give me a 500-word essay, give me a 1,000-word essay. We try to avoid things that include referencing. So we don't say give me a research article because we don't want it to hallucinate citations. We want it to be more of an essay type or a writing or a small paragraph. So it could be like explain this to me like a five-year-old. That also counts. And, of course, we want to compare model size. So small models versus large size models. models, we want to compare it across versions, so new releases that are happening over time. We want to compare different family of models, and, of course, we want to see if using some source of external information, say, as a RAG helps. And all of this is still happening in a prompt-based setup, so we are not training any new system here. We are trying to replicate as much closely as we can how humans generally take information from AI systems is just through multiple prompt variants. So that's the goal. Now, the question is how do you measure how well these systems are performing? Because how do you measure the concept of diversity? How do you determine whether this model is more knowledgeable about this topic than the other model? So a very simple assumption is, of course, this is a very simple assumption. What we have originally is a blob of text, and we convert this text into a list of sentences, and these sentences are then the basic input that we use to measure diversity. But let's begin with the assumption that you already have your model give you a list of information. So even if it's about democracy, it will give you information like it is a modern concept, it is a nice concept, and things like that. So it's a list of information. And a very naive approach would be to just count how big this list is. The bigger the list, the more diverse the model. But it comes with a caveat that the models can generate synonymous information. So saying democracy is good and democracy is nice is not giving you any new or extra set of information. So just by counting the number of items in your list is not a good measure of diversity. So what we can do, we can start clustering them. So again, you have these lists of sentences. You cluster them based on whether they convey similar information or not. And then again, the number of clusters become the number of unique information you have. So higher the number of clusters, better the diversity. Again, we face the same problem that some models can generate information that is only singleton, which is like it just gives you one information, but there is no coherency of that information in the rest of the list. Whereas if you think of reading a Wikipedia page, the sentences are coherent, so one piece of information is helping you understand the other piece of information, so still, even though you are clustering, just by looking at the number of clusters is not a good idea, because some models can generate random piece of information, and again, you face the same issue. Bigger lists or more number of clusters is not just equal to good diversity. clustering is still better than just using list. And what we have done here is creating these clusters not just based on semantic similarity but also entailment. So, within the cluster, you can think of sentences that give you information about a particular aspect of that topic and no more information. So, you can think of one cluster, say, that talks It's about Berlin Wall, so it will talk about when it was brought down, and all the sentences in that cluster will talk about that, so they will either be able to answer the date or they will be able to give you the year, but it will be a very specific information. So clustering for us is not just semantic similarity, it is also mutual entailment. Everything within that cluster gives you or is able to answer a very specific set of information. So once we have these clusters, we say that, okay, we have a better measure of diversity than just looking at the cluster through semantic matching. Still, these clusters still suffer from the issue that if you have large number of clusters, some of them may have just one or two items in the cluster, it is still giving you random information. So how do you reduce the impact of this random information? Yeah. So we have already covered that just looking at the number of items, whether in a list or a cluster, is not a good approach. Better approach, when we think of reducing randomness or assessing randomness, we go to measuring entropy. What we mean here is that if you have clusters where you have one cluster which is very big, it contains all the information that you need about the topic, but then you have multiple small clusters that are giving you random information, you want to penalise that setup. So entropy as a metric helps you capture that skew in information. So a more balanced cluster, a system that gives you more balanced set of information and coherent, here balance is coming from coherency of information, so if it gives you coherent information, the diversity will be higher. We measure this based on the number of clusters that we get for a given topic. Of course, we aggregate this across topics for the model as a whole, but this is the building block where for a given topic, let's say democracy, you prompt one model, you see how many different clusters of information you get from it, and then you use the cluster sizes as your base for a data mining entropy. So PI actually captures the size of that cluster within your system, and it is normalised by the total number of information that you have or the total number of sentences that you have in general versus the number of sentences that are captured within that cluster. It's a very good improvement from just measuring sizes or cluster sizes or even just measuring list items, but Entropy operates on a log scale, which is difficult to measure and compare. So we want something that is linear, which is easier to understand and compare, which is where another version of this entropy metric comes into play, which is called the Hill-Shannon diversity metric. This has multiple variants, but the easiest variant is to think of this as like, okay, we are in log scale. Let's do an exponent of the log, and we are back to linear scale. So this is where we end up. So we still use the concept of uniformity and diversity where we want balanced cluster sizes. But now by doing this exponent of log, you are back to linear scale. So you can now do comparisons which are easy to comprehend. So now we have this metric sorted. We go back to getting these list of sentences in the first place. And as I said, like, we do this comparison across, like, a huge set of, like, topics. We use search as our baseline. So we use the top 20 and top 40 search results for that topic from the Google research. And we set US as the location. Then we compare for that same topic, we compare the results coming from the models. So we have 27 different models. These are different family, different sizes, different versions, so you can track it across time. And as I said, we prompt each of this topic when we give it to the LLM using different writing prompts. So you have almost 200 variants of that. And then once we have the search text, we use the search text as a RAG input for these models. And we see does giving the search results as part of the prompt, does it help the model at all? And of course, we do clustering for each of the system, and we get a diversity score, which we discussed before. So this is a score that we are getting for each model topic pair, and then you can aggregate it to get it at a model level or at a system level. Now that you have all of these results, all of these outputs, now we start comparing. This is the figure that you will see again and again, and we'll try to cover as much information as we can. The first thing that we notice is that, yes, models are getting better with time. I don't know if I can, yeah. So if you can see, except for Quen, which seemed to be degrading in performance with time, for Lama, Gamma, and OpenAI models, we see that they are getting better with time. But if you look at the absolute difference between search and where the models are, there is still a very huge gap. So yes, models are getting better with time, but they're still not as good as search results as we would have hoped them to be. Another interesting thing to notice here is that the size of the model is not a very good measure of, will the responses be diverse? We do know from our anecdotal experiences that yes, a bigger model will give you a good writing. It will give you sentences that look good, it has that coherency to it, but is it actually giving you diverse set of information compared to a smaller model? Not necessary. So if you're looking at just the diversity, if your aim is to get a diverse response, it doesn't, even if you are okay with less grammatically coherent sentences, then using a smaller model is still good enough. So yeah, models are getting better with time, but whether they are getting better with time and size is not something we observe in our study at least. And as I said, you can still see that for majority of these systems, you're still better off with search. So if you are looking to gain diversity of information, if you want to get unique piece of information, you should still go back to your usual search. What we do see is that the OpenAI GPT-5 variant is the one that comes closest to matching the quality or diversity in terms of search. But even, like, of course, this is top 20 search results. You can make this variation with top 10, top 20, top 40, top 100, however, like, you want to compare. But, yeah, we see that still, like, the first few pages of your Google search still giving you good enough information. And we also see that because these models, which are usually trained on your internet information, we see that a lot of that is still not getting captured in these systems. And yes, then the comparison with RAG. So as I said, we use the search results themselves as an input for the RAG. So you have the search output. We give certain sentences that are comparable for this topic as an input for prompting the model, and we see that this RAG-based setup is helping the models improve their diversity. So you can again see in all of these cases the dashed lines are the ones which are with RAG, and in most of these cases they perform better than just simply prompting the model for the same topic. So it is just using the search output as an input for the models and it is helping them give more diverse information about that topic. So yes, search is helping, but again, it is not helping all models uniformly. What we actually observe is that if it is a smaller model, then it benefits more from a RAG or a search infusion rather than a larger model. So, again, larger models may not necessarily be better at adapting to external knowledge sources. So, again, this is not based on any fine-tuning, so just off-the-shelf rag infusion, you are, again, better off with smaller models. So the three things that we learn is that, yes, models are getting better, but they are not as good as we would expect them to be compared to a rudimentary search result. And yes, RAG helps, but it helps models of certain sizes, and that utility is not uniform across model families and across model sizes. So we are heading towards the end, and I'll try to give two main set of takeaways. The first is comparing LLM with search. So what we also see is that the open weight models are more coherent if you're comparing them to each other, so say compared to a GPT model. So you should, if you are looking to, like, compare the diversity of information in terms of consistency, you should stick to using open weight models. Because the LLMs are not still, like, coherent in terms of giving the information, similar information among each other, you should, if you are looking at getting multiple point interviews, you should use multiple models, don't trust the output of a single model, in general don't trust one knowledge source. And yes, because search is not equal to LLMs, at least as of now, you can use one to determine the blind spots in the other, so you can use your search to get a basic level of information, and then see what additional information can LLMs give you, or vice versa. So you can use these as two diverse sources of information. Think of it as a system like an AI education system where you're trying to teach students about a certain topic, then you would be better off combining multiple sources of information instead of just relying on a chat-based interface. The other takeaway is, of course, in terms of RAG, As I said, smaller models are giving a better performance with RAG systems. But what we also see is that the information source of RAG is actually impacting the output a lot. So as I said in the beginning, we use like almost 188 topics. And some of these topics are specific to certain countries. So we go to the Wikipedia page of, say, India, and we look at a very specific concept. Let's say we look at Diwali or we look at a political concept or a political entity and we use that topic then as an input. So for each of the topics, we have a mapping of which country it is mostly associated with. And what we see is that when we start looking at the measure of diversity, if I can break down this table in terms of topics that are country-specific and keeping the setup of prompting without RAG versus with RAG same, then we see that the impact of RAG is not same across different context. Here we are using country as a parameter, but you can think of this as you can change it to language, you can change it to knowledge source, you can change it to expertise. So we see that how your RAG is grounded changes how it impacts the output. So localization helps. Going back, the RAG is based on search, And the search that we did was on Google with US as the location. So that then becomes the orientation through which rest of the prompting is impacted. So you can see that certain countries that might be closer to US in terms of the news or engagement that they are producing, which gives you the top search about that, is influencing how the RAG is impacting. So localization of RAC then becomes a question that you need to account for, not just use a general knowledge source, but use a specific knowledge source for the context in which you are operating. And of course, so far we have assumed that these search results are not impacted by LLM generated content themselves, but we know that this is now becoming a vicious cycle where you have a knowledge base. you know that this knowledge base is not complete. It has loopholes, and you're trying to cover those loopholes of information by using an LLM to generate extra information. And what will happen eventually is that your knowledge base, which is supposed to give you extra information, will get information which is same as that of a language model, and the two will not end up helping each other. So keeping your knowledge bases secure and as LLM-free as possible is a good strategy. Trying to make sure that the information that they capture is diverse. You have a metric of diversity. You can use the information that you have in your knowledge base, give it a label, a metric of diversity, compare that to the LLMs that you're using and see, is my knowledge base still diverse? Do I need to improve its diversity? If I do, then going back to the concept of localization, like how can I improve the knowledge in my knowledge base in a localized, in a contextual manner? So that's the major takeaway. Yeah, and these are some useful links. The code and all the data that we have generated through all of this prompting is available online. You can go and read our paper. It has a lot more interesting results. We do comparisons with Wikipedia. media, we do comparisons, as I said, with different topics, and of course we also explain in detail in the paper how we go from getting the text blob that you get from a language model, converting it into a list and then forward. So, yeah, do read and do CITAS. Yeah. That's about it.

Speaker 2 [21:48]

Thank you. Wonderful. There's a lot of time for questions. So I'm going to go into that. Let's start with the popular one. When I use LLM for coding assistance, i.e. cloud codecs, diversity of output is good or bad in your opinion?

Speaker 1 [22:07]

Okay. That's a slightly tangential question. The diversity here will depend on, like, is it able to solve your specific problem? Because in terms of, like, the way we did diversity here measures a difference of opinion in terms of, like, coding. It is less about the difference of opinion and more about correctness of the output. So there your diversity metric itself will change. So I I don't know if it will directly answer your question, because the metric changes when you're looking at coding outputs.

Speaker 2 [22:44]

Okay. Next one. I think you talked about this, but this was asked earlier. So maybe you have an additional comment. Does the diversity vary by question if we are something different than democracy?

Speaker 1 [22:56]

Yeah, yeah. So the output is not shown for democracy. This is for models for across all the topics that we have. As I said, we have around 200 topics. This is the aggregated result. But at topic level also, you see similar patterns emerge. So yes, it changes with topic. But this is like overall, this is not topic specific result.

Speaker 2 [23:21]

Does the cluster granularity affect the findings?

Speaker 1 [23:26]

Yes, it does. So one thing that I didn't show, but if I can quickly show what we do in terms of clustering, is that once we have these clusters, we also check for singleton clusters.

Speaker 2 [23:27]

Yes.

Speaker 1 [23:45]

So we penalize systems that have a lot of singleton and what you can say, clusters that have only two sizes. And we also see if the clustering is done purely through semantic matching and we get different cluster sizes, then how does it impact diversity? So yes, it does impact and we have reported that.

Speaker 2 [24:08]

Right. There's another popular one. Is always diversity the way to good? For example, when asking for homosexuality, we do not want diversity and being an ally one time and homophobic the next. And also it is good that every time we use the LLM gives a different point of view. I think there's a little bit of English.

Speaker 1 [24:34]

Yeah, I understand the question where it's headed. So, yes, it is not like an always measure. Sometimes you would want certain aspects to be brought up more. As I said, when we were dealing with these 200 prompt variants, we didn't ask it to give us a positive or a negative opinion. So we didn't explicitly give that nudging by itself just so we can get whatever information it generates by itself. The second part of that question that aligns with the work is that when you want certain aspects to be prominent or not prominent, you can actually think of them as, hey, I want to give this cluster more weightage than the other cluster. So right now, the clusters are not manually mapped, but think of it as, hey, I have these clusters, I can associate a sentiment or I can associate a hatefulness label to it and give the clusters that have higher negativity, you know, weigh them down. So you can modify this metric to your specific need.

Speaker 2 [25:48]

All right, let's do some more. Is diverse, okay, I think things are moving a lot, so give me a bit.

Speaker 1 [25:56]

I think there was a question. Yeah.

Speaker 2 [26:00]

Sure, I'm going to give you my mic.

Speaker 1 [26:03]

so thank you for the presentation I'll definitely take a closer look at the paper but maybe I missed it how do you label the measure coherency within the cluster is it human involved or is it like done automatically using an equation yeah so it's both we do have a human evaluation so what we again use entailment model but this is not a language large language based model this is a bird based entailment setup that we have and we measure the entailment both ways so a entails b and b entails a if this is true then they get added to the cluster and once we have these clusters we do a random human evaluation of this and then we check okay this is like the entailment that we have done is good so yeah

Speaker 2 [26:57]

Okay, give me one second, there's just so many questions, but if your question doesn't get answered, please feel free to talk to Sarah later, and you can also post them on Discord, she'll answer.

Speaker 1 [27:12]

answer I am not you're

Speaker 2 [27:14]

not on this card so you can

Speaker 1 [27:15]

But feel free to make fights.

Speaker 2 [27:15]

Yeah, you can talk to her. Let's see. Is diverse output generated because of different data set of pre-training or different model architecture?

Speaker 1 [27:30]

It seems to be a combination of both, but largely it seems to be a combination of training, because as you can see here, the model size is not impacting, which means that, and we know that most of the time, whether it's a bigger model or a smaller model, they're usually trained on the same set of the pre-training data. So it seems like the pre-training data, if it is same within the model family, then it doesn't impact the size. So I think the training is more important than the data set as of this analysis.

Speaker 2 [28:04]

Okay. And with that, I think we can conclude the session. Let's thank Sarah again for a very wonderful talk.