Building a HybridRAG Document Question-Answering System

Speaker 1 [00:07]

My talk is about how to build a hybrid RAC document question answering system and I wanted to start off with a quote of Andrew Ng who said that knowledge graphs can improve the context you pass to the LLM and the performance of your RAC applications. He was saying that when introducing their new course about knowledge graph and graph rock but also kind of to set up the stage and understanding of the of the topic where we'll be talking today as to I like the library analogy to compare vector rock or like traditional rock if you want and a graph rock so imagine that if you're trying to search something in a library vector rock would be or traditional rock would be something like having a search index to to search across books and you just search for similar books but maybe you don't want to do this you want to have some smart and experienced librarian who knows you know some hidden relationships and and what's tidying up different books together so having such a smart person experienced person is like having graph rock so and today we will be talking about first we will talk about what is rock then we will talk about what is graph rock and how to how to marry them together how to merge them together we will try to figure out what is hybrid track and what are the challenges if you want to build your hybrid track system and how you can approach the building process and also this talk is more like a platform agnostic it's more about the process and understanding of the concept behind behind the topic and then we will move to key takeaways and Q&A so my name is Daria as I was introduced I will just skip this part let's let's talk about rock I guess many of you already familiar with rock and have some experience just quick show show of hands okay yeah perfect perfect so I will keep it short and simple when we because large language model has a context size limit limit and we cannot just quiz long documents into the context we need to chunk documents like split documents into chunks we need to convert them into embeddings store in a vector database and then we need to do a search using semantic similarity so we just embed user query and we are searching for the most similar chunks of the most similar pieces of data inside our vector store and we then return it to the model and the model has some information to reply with so it is using like you know like external knowledge so yeah this is basically yeah this is this is basically what I said yeah the query is being embedded we're searching for the similar items and the model has food for sorts to generate the answer but in reality when we are trying to return top or n top 10 to top 20 it depends chunks of the documents it is very difficult to understand up forward what is the most suitable number of chunks or documents is and also when we need to combine and understand some relationship between inside your documents RAC and LLM they could be not great at certain types of such such certain types of questions for example I will be sticking to legal documents use case but you feel free to to adapt it to your use case this question is which documents include a termination clause and specify penalties in euros so essentially you can break it down into two questions where we have termination clause and where we have mentions of penalties in euro and imagine that we have some documents and they can be lengthy they can be worded in different terms we have a new document one some mentioning about termination we also have some mention about late fee in euro and we have a second document also about termination and about some penalties right that that would be nice if you could just abstract away of all this you know uh linguistic garbage and uh to see the relationship between different terms so uh and and i have like a high level overview so we have document number one and we have termination clause inside it we have document number two and we also have a termination clause inside it and in one document we have late fees and another we have some penalties and both are errors so this is what brings us closer to graphs and first I wanted to talk about what is graph itself so graph is constructed from explicitly linked entities and relationship from the data and it's stored can be stored in a graph database or some kind of triplet storage and they like the base basic element of your graph is a graph triplet it consists of subject predicate an object so it's it's like a like a sentence like PyCon can be subject located in it can be predicate an object Darmstadt so it's simply as that and if you move to a graph rock like pure graph rock this is a rock methodology that incorporates knowledge graph as a part of its retrieval process so it leverage uses the graph to return results based on the entities that can be found inside graph and their relationships so the main difference is in the retrieval part so imagine that we have okay we don't have such a simple graph but this is a part of our graph and this is what being retrieved and we can have this triplets we have doc1 contains termination clause and we have like multiple triplets that are extracted and then they are passed as a context to a large language model and it can formulate an answer based on on the information so we have two documents with termination clauses and penalties in error but to create a knowledge graph yeah i just assumed we had a knowledge graph but you need to create it first and one possible approach is to use llms uh for the creation because you need to have some entities you need to have relationship between entities this this was mentioned uh this creation step where i mentioned in a paper uh speaking about hybrid truck yeah and they used a large language model first to do a cleaning of your text of your raw unstructured data and then they had like a second step using this more more clean data to some extent to extract these triplets and then to store them in the knowledge graph and i was talking about like legal um legal domain but there are a lot of possible use cases so if you can imagine a vast amount of unstructured data that can be can be linked in somehow together like company policies internal documentation legal documents or medical documents here you you see the example guys where we're trying to link together medical diseases have some logic behind the diagnosis or like even market analysis you can have a lot of articles what's going on which company acquires which company and what is the outcome so all this like a vast amount of unstructured data are good for doing graph constructions but there are certainly there are a lot of challenges because how to build the graph it's maybe the main challenge of it because it's not just you know have an LLM and say go and build me a knowledge graph you need to have examples you need to think up forward what kind of relationship what kind of entities are the most important for your company for your domain and to have these examples and to also think up forward for the retrieval part how would you deal with it how to turn the user request to query to a knowledge graph so so so so model can can query the graph and have the results back and also like how to how to make sure that this all thing is scalable integrated back into your systems so we were talking about traditional rock and a graph rock but can we marry them together can we merge them together can we combine them yes we can because you probably notice that the return of the graph it's it's pretty dry you know you have only triplets and you basically have like very small output for the large language model which can confuse your model and possibly cause some hallucination so what if you try to concatenate the information from a rock like traditional rock and a graph rock and have this like both as a context for LLM so one paper called this approach a Hebrew truck that's why I'm referring this as a Hebrew truck and they were talking about that this approach can balance high quality answers with comprehensive context retrieval but not only like paper I noticed this I noticed that some companies started to explore this approach and the company called Letria I'm not affiliated with it I just found a very interesting blog post about it they try to you know run a benchmark and compare our vector rock like traditional rock system and then don't be misled by graph rock here is essentially the combination of graph and a vector rock so it's hybrid rock approach and you can see that if we talking about correct answers it's like 57 versus 81 and if we if we combine together correct and acceptable answers it's it's like even more so we have if we have like 90 percent for a hybrid rock and this was like a human humanly evaluated benchmark across law industry finance data set but how do you combine them together we mentioned that we need to do some kind of calls to our vector store to have the chunks returned and we need to have calls to our knowledge graph and have the entity in relationship and extract it and then to merge them together and pass to the model So, if we think about legal documents, for example, if the user, for example, is interested in what's going on into the documents, for example, they may be wondering what are the payment terms for the company A or company B. And, okay, we have filters. We can filter out all by metadata all the documents related to the company A. And then we have call to a vector store and we have chunks with payment or payment terms, something related to payment. And we have call to our graph and we have triplets with agreement has payment terms of 70 days and amendment which alters the agreement has payment terms of 45 days. But these two calls can be done in parallel and then fed back, returned back to LLM and LLM can use it to have additional insights and have the understanding that there were agreement and there were amendment and now actual payment terms is 45 days, which can depend on your company, whether your company is happy or not with that. And yeah, there are a lot of challenges in building such type of systems First of all, as I already mentioned when I was talking about GraphRack Is how to build a high quality knowledge graph So the first step you want to do with your unstructured data is to extract the knowledge You want to focus on entities, recognize these entities, recognize what are relationships between these entities, and do co-reference resolutions. It means that if in one sentence companies refer to the company, and in another sentence it can be referred to as supplier, and you need to understand that it is actually about the same company. and then when this step is done we have like a prototype of our graph but still we can do better we can improve our knowledge graph by do complexion and fusion maybe there are some missing notes or missing facts or maybe there are like some like some triplets that are actually saying about the same thing or some facts they are controversial to each other so we need to pay attention to it and examine it also this is this is pretty heavy domain specific as you because as a company you would like to focus on your terminology of what is essential to your business and as a data scientist who built this graph you may you need to drill down into one domain whether it's legal clauses whether it's finance health care etc etc but um but uh what if you work as a consultant and you need to switch domains or you're working at the outsource company and you need to understand each client domain uh so you it's it may be the thing that it is not just like lift and shift once i worked with legal i i can pretty understand the finance and use the same like same setup to to create a knowledge graph now maybe it's not not the case and also there could be a case where it's a mixture of domains you need to work on legal documents but for oil and gas yeah and that adds another like layer of possible terminology that should be included in your knowledge graph also because we have essentially two systems two calls it's one query transforms to two calls and this is computationally this is computational overhead and of course like management overhead because you need to keep in mind both both systems and also this is this is more like from a terminology and research perspective there is different there are different ways to define Hebrew track itself because you saw this picture already and this is like a plain setup okay we are having some vector rock we're having some graph we're calling we're like passing them together and we have we have like a hybrid rock now but there are another possibilities to approach the hybrid rock or and some companies still call it graph rock because it contains it contains graph inside for example this is this is example of automatically built automatically built graph from raw documentation by AWS so and you can see I hope you can see that inside this nodes there are not not like terminology not a word itself inside but it's a chunk of a document so chunk of a document serve as a node and then you have an entity detected inside this node so in the middle you have entity referring to for example a city or some term and you have multiple chunks of documents that contain this entity and yeah this was created automatically and this is the way also to approach the Hebrew truck because they have combination of vector rock so they are doing cosine similarity to to find this entry point to the graph and they're doing afterwards crop traversal to have different nodes and relationship combined so there's all this there's all this information yes there are different there are different ways to approach the building of Hebrew truck and your setup you can do it like do it yourself approach you have a raw data you need to create a vector store you need to create knowledge graph and you need to take care of the retrieval part one one benefit here is that you have a complete control what's going on or you may have complete control of what's going on on each stage but drawback you need to do it all by yourself and yes it's definitely time-consuming and also you need to decide yeah where where do you store your where do you have your vector storage and where do you store your graph is it like you can use some open source um open source databases to store your graph but it's also a question of your company setup uh also recently aws introduced uh like automated solution no code quick way with bedrock agent to have graph rock i i decided to try it um you know the benefit here is you don't have to do any code you can do it quickly you just you just create a knowledge base and you connect it to bedrock agent and you can ask your questions so just like several clicks set up and you have your uh hybrid rock or should i say graph rock i don't know um yeah you don't have you have very little control how the graph is created because it's being done automatically and you are tied it to code provider it of course depends on your company and also cost you need to if you want to use this thing you need to keep amazon neptune analytics graph uh set up which is not cheap and yeah basically you you can explore how the graph is being constructed but you have no control you you don't know you don't even have the possibility to tweak the parameters how the graph is traversed so it's just like a question what is going on it's just like black box but also recently AWS announced the semi semi-automated solution with AWS GraphRack Toolkit. It's an open source package to facilitate creation of graphs. One benefit, you have more control, you have an open wrapper supported by AWS, you can open issue, you can reach out to them, but also this involves having Neptune Analytics as your graph storage. You're again they tied it to a cloud provider uh yeah you can see the cost it's like uh almost half a dollar per hour for the cheapest setup and it is still a new and still a bit of raw i hope the team will manage to do it more mature but so far it is like this so we are moving to key takeaways So, in my talk, I was trying to show you the combination of how the hybrid rack can combine different size of rack, like vector rack and a graph rack, and can provide better answers quality. it's better to use foreign structured data in complex domains it can bring a lot another level of insights from your data and it's so far it's an open question how do you approach the build because it is highly it will definitely vary depending on your company depending on your domain depending on the providers you're working with so pick up what works best for your use case thank you

Speaker 2 [23:53]

Now it's time for Q&A. So the first question is, what approach for weighting between traditional vector similarity scores and graph-based relevant signals during retrieval? Learn weights or heuristic rules.

Speaker 1 [24:07]

Sorry, I didn't get the second part. Yeah, so I would say the way to approach this question, it will depend on how you will approach the building, because if you are doing the do-it-yourself approach, you can be pretty flexible to define these rules and to see I would say that yeah it depends on your data and your domain and also on what type of relationship what type of relationship and insights you are expecting from your data but if you're using some kind of automated solution maybe you don't have you you don't even have such a question because you don't have much control of how the traversal and retrieval is being done

Speaker 2 [25:16]

The next question is, how does hybrid RACS knowledge graph integration differ from fast graph RACS page rank based approach for multi-hop queries?

Speaker 1 [25:29]

yeah it's a great question because i i also attended the talk uh yeah yeah yeah yeah it was interesting insight how they um you know decided to to bring this all together and uh yeah to be honest i i haven't tried fast uh graph rock yet and i i would be happy to try it and see but i definitely see the the difference in the mindset how they approach their like communities and just in my my first impression they have a better way to not to miss essential information from the graph not just doing like separate traversal and that's it and i think they would would have much better results than all this your like do-it-yourself approaches but i haven't tried it yet thank you

Speaker 2 [26:25]

The next question is, how does the graph rack identify the most similar triplet to the query from the whole graph?

Speaker 1 [26:34]

Can you see?

Speaker 2 [26:35]

How does the graph rack identify the most similar query to the query from the whole graph?

Speaker 1 [26:41]

Yeah, so this is, if you're building a retrieval part yourself, so the one approach that I saw in a paper that introduced hybrid truck, they were trying to convert user query or user question to, for example, cipher query. so like I have an user question what are what are the documents with payment terms like example question right and this could be transformed to open cipher language query so and this query transformation is being done by large language model yes the question is how accurate the query and how accurate the information extracted from the knowledge graph and also the big question is the big question is when how do you deal with the situation when your LLM constructed query doesn't return anything from the graph so should you go back and forth and try to replace your or replace regenerate your query and try to do it like once again but but all these are open questions

Speaker 2 [28:10]

What's the simplest way to create a knowledge graph using Azure OpenAI API with minimum custom coding?

Speaker 1 [28:19]

I would say Microsoft. They have Microsoft GraphRack. I haven't tried it yet so far. I was playing only with AWS-based solution. But if they claim that they have their own GraphRack, I guess it would be the easiest way to go for Microsoft-based solution if you're using Azure.

Speaker 2 [28:41]

Yeah. How do you deal with the case when the connection between documents is spread over a very big context, bigger than the LLM model context creating the graph?

Speaker 1 [28:55]

So the query spread over all the context to better understand the question Okay, so yeah this this is also a great question because Depending on the domain you can have like very big documents one approach is to you know just yeah maybe the simplest approach is to try to split your documents and try to feed it in batches and just hope that all this entities would be related in a crew in a final knowledge graph another way is to use kind of summarization and trying to shrink the document and you know just filter out all unnecessary information and just to summarize and bring it like spin it to a smaller smaller size so it fits the context size

Speaker 2 [30:04]

So I think we ran out of time I will post all the remaining questions in discord To the speaker and the big thank you again to Daria

Speaker 1 [30:13]

Thanks, everyone.