Using LLMs to Create Knowledge Graphs From a Large Corpus of Parliamentary Debates

Speaker 1 [00:06]

So, indeed, today we're going to be talking about analyzing a somewhat sensitive data set using language models. What can go wrong? So, about myself, I work as a machine learning engineer for Exibia Data, based in the Netherlands. It's a kind of data and machine learning consultancy. So, we work on a wide range of projects with different clients and with different use cases. and based on how the data landscape is changing, most recently those projects have a generative AI component. Prior to that, I worked for several years as a data scientist before having an identity crisis and becoming an engineer and now I tend to work as a kind of hybrid of both those roles in my projects. Within this vast world, I'm particularly interested in graphs and networks. I think they're quite interesting from a mathematical perspective and they also throw up lots of very interesting engineering problems. And most importantly, they can solve some problems. So, we're going to discuss how most companies are trying to use language models at the moment and some of the limitations as a result of that, which then leads nicely into how the concept of knowledge graphs can be useful. To make this more tangible, I'll show you how I applied this to a specific data set and the results that I saw from that and some of the learnings around applying this particular methodology. So I don't need to discuss too much here. If you didn't know much before, I'm sure throughout the course of today you've heard lots that has got you up to speed on this. Although what's perhaps more interesting to note is that it wasn't necessarily the model architecture itself that was perhaps the biggest innovation of language models, but more so the data used to train the models. I think for GPT-3, which was when this explosion happened, the actual model was publicly available for 11 months prior to its release. So it's really the internal data of the model that was the big innovation. and the internal data seems to be most of the internet. So very quickly, the power of these models became obvious and companies began to ask themselves, well, this is very nice, but how can I use this? And more importantly, how can I incorporate my own data into these systems? That data can be databases, a lot of text perhaps in things like Confluence, and, of course, lots of data in our favourite tool, Excel. Typically, when we have a pre-trained model and we want to update it with our own data, we would use a process like transfer learning, so just tweaking the parameters. That's a little bit more complicated with language models. Firstly, the data that we're bringing to the table is quite small compared to the data used to train the model, highlighted by this tiny orange square here. It's also difficult to know up front whether this tweaking will have any meaningful benefit. It's not that the output is a classification or regression which is very easy to evaluate. It's a more flimsy experience evaluating language models. And of course there's also a lot of complexity behind it and so companies may not necessarily have the right engineering team set up to do things like retraining their own models. Typical use of a language model is quite simple. Input a question, output an answer, and then the solution by which we can incorporate our own data became known as this, which is now a very popular approach, of which there have been some very interesting talks today. But to quickly summarise, our private data set is split out into these chunks called documents which is then stored inside either a normal database or a specialist database like a vector database. And we have this concept of a retriever whose job it is to take a question and then to return some subset of relevant data which is then passed into a model and then we can get a output based on that subset which is relevant to the question. One of the challenges with this is when the retriever extracts this subset, it's extracting a subset defined by some length, say n. It's difficult to know up front what is a suitable n. It's also challenging in the sense that for some questions, maybe you need a larger n, some need a smaller n. It's not so easy to know. And as we'll see in an example later, for certain types of questions, language models don't perform so well. Types of questions where there's a single answer instead of many, many different answers. What do I mean by that? Well, if I was to say, can you summarize the following piece of text, there's numerous ways to do this. So lots of different correct answers. But for some kind of questions, more factual type questions, when there's something more quantifiable in the data, we are looking for an exact answer. So consider this question. How many companies have former OpenAI employees started? We can break this down into two sub-questions. Who are the former employees of OpenAI and how many companies they have started? And in this fictitious example, you can see how in these documents on the right-hand side, information that is relevant for the question is sparsely stored within each one. in one document you may have information that pertains to a particular set of people who've worked at this company and then in a separate document information about companies that have started. So even this really small example concerning two people you can see how this information can be spread out over four documents. So if you think about a much larger data set what would N be in this case? Perhaps it's going to be everything in which case choosing N and choosing this vanilla RAG approach may not be suitable. Of course in lots of cases it may be that all the information related to a specific entity in your data is stored inside a single document or within some dense documents related to each other but often it's quite sparse and then this becomes quite challenging. So how can we overcome this problem? What we have here is just a textual representation of some knowledge. In this particular example, the knowledge that we care about is the people who have worked at a company and which companies they have subsequently started. But it's not necessarily an efficient representation because there's lots of noise within there, a lot of other information that we don't need. The question is, how can we represent this information in a different way? when you have a situation in which your data contains entities and you have relationships between those entities a very natural conclusion to that is you can represent your data as a knowledge graph so you can see here on the right hand side in blue we have individuals and in green we have companies and the two relationships between them are either worked at or founded and created. And so this now offers a much denser, cleaner representation of the knowledge that is also stored in this text on the left. And the nice thing with the graph representation is let's say we have more and more documents coming through, being generated, then to update this graph we simply need to update existing attributes of these nodes and relationships. so we don't need to add more data it's a data efficient way effectively whereas if we just store them as documents you have to store an ever growing number of documents so it's a very dense way of representing the information stepping back a little bit and just talking about knowledge graphs as a whole whenever you have a data set in which there are many different types of entities and those entities can be related to each other in many different types of ways and a lot of many-to-many relationships in those entities, then a knowledge graph becomes a natural way to represent that data. And in doing so, certain downstream and subsequent analysis becomes more feasible because there are lots of nice, efficient algorithms to process this type of data. The first challenge is, okay, this kind of makes sense from a conceptual perspective, but how do we store this type of data? Technically, you can store graph data in standard SQL databases, which are relational, but then subsequent analysis becomes inefficient. Those databases store things either as a row or a column perspective, but specialist graph databases like Neo4j, Memgraph, ArangoDB store things from a node perspective. and the reason this is important is because we want our subsequent analysis to be efficient so to understand why this is important let's look at this very simple example here we have three types of nodes a b c when we store this on disk we store it using this kind of node-centric view so when we store the representation of A we store alongside it references to B and C because it's connected to B and C likewise with B it's connected to A so we store a reference to A this means when doing certain algorithms like say some pathfinding or some traversal along your graph you simply need to have a starting point and then because you already contain the references to the other nodes you can quite quickly traverse along those. When I say references, under the hood what's really happening is it's just a memory pointer to the address at which the other node is stored. Why this is important is because in a normal database if we had these three entities we would have to perform a join on three tables. Joins are quite inefficient. So even in this very small example with three nodes you'd have to perform two joins. So if the path is getting longer and longer you can imagine how many joins you need to do. So whilst graph databases are slow at writing data because you store a lot more up front when it comes to querying them you can retrieve information faster because you don't need to explicitly define the joins. So our aim is to start off with some textual data extract from there some entities and relationships which we can then use to create a knowledge graph which is a structured source of information. And then on top of that we can try to gather insights more easily from there. So coming on to the actual data set who's this handsome fellow you may ask? This is Thomas Kirsten Hansard most famous for starting to publish parliamentary debates in the early 1800s, 1804, 1805, and since then every debate that has taken place in Parliament in the UK has been transcribed and then published. The data from 1800s up until 2005 is quite easily available from this API. From 2005 up until now it's also kind of available, but using more complicated JavaScript which makes it harder to scrape so we're going to be focusing on pre-2005 but you can navigate through the centuries, the decades within a specific day you have a whole list of debates that took place and within each of those you have this clear structure of a person the area they represent and what they said about a certain topic. And this represents a very complex interaction of people talking about ideas and policies and all sorts of strange language rules and customs. Sounds like Harry Potter sometimes. So I thought it would be a fun task to analyze this data set using language models. So why is this interesting? Well, it's also a very useful kind of point of use for journalists, whether they're trying to write about recent topics and they want to refer to what was said, what happened, which decisions were made, or writing about a period of history further back, whether that's for some research or for a book. And the problem is, if you're trying to read this information, to gather at least a high-level overview of what the views were, who disagreed with whom, and what was decided. It's quite tedious to have to read through all that text just to get a high-level intuition. And for full disclosure, I've also read some of the stuff in there as well. Some fun guy. So how are we going to do this? How do we start from these text files and create a knowledge graph? well earlier I mentioned that language models had limitations in specific scenarios but they also have lots of strengths of course and one of their strengths is entity recognition they can not only extract entities from text but also extract the relationships of those entities as well and we're going to leverage that for this pipeline The end-to-end process looks a little bit like this our input to the model is a set of instructions and then the context then becomes the specific transcriptions of that debate. Input into a model which outputs two components, the entities, i.e. the nodes and then the relationships between those nodes Once you have data in this format, which is kind of like maybe a CSV format or something similar, it's very trivial to load it into a graph database. Then once we're here, we have a structured source of information which represents the knowledge from those texts. The development of this really highlighted that the entire effectiveness of the quality of results was very much dependent on how high quality your prompt was, which is a rather strange experience. Typically, our data pipelines are dependent on, you know, good software principles, clean, maintainable code, and now it's just about how well you've written a set of instructions in natural language, which is a slightly odd experience. I don't need to read this full now but I wanted to just give a bit of a feeling for how long the end-to-end instructions were, so not crazily long, I hope you can see in the beginning we just defined some very high level instructions this is a debate between people in the House of Commons the task is to extract the identities of those politicians, extract the policies that they're discussing, whether they disagree with them, agree with them, and so on, and providing a few simple examples of what an input looks like in terms of the individuals and then the text, and then what we expect as an output. So the entities, like organizations and politicians that may be mentioned in the debate, policies, and then the relationships between those policies, between those entities. But as I mentioned before, it was a rather strange experience in doing a lot of this prompting, not something which I particularly enjoyed, if I'll be perfectly honest. Like I said, because such a big, large and important component of your entire system becomes this opaque API and model, and you don't truly know if the same model is being run for different invocations, and running the same thing twice and getting different results is quite frustrating and therefore becomes very difficult to test and maintain your system. To give a bit of a fun story around this, I ended one day with some ideas about what I wanted to do for the following day, so within this prompt that I had I left myself some to-do notes in there, you know, maybe simplify this or add this, and then the following day I just ran everything to get an active state, and I noticed the results were, like, better, and I thought, I haven't done anything. And then I checked what was in there, and it was all my to-do comments. And somehow that just gave better results. So the initial output is something like this, just a list of politicians, organisations, and so on, along with the policies that were discussed, and then the relationships between all those aforementioned entities. There was some post-processing that was required to do this. For example, here you see it says the United States administration, but then also the Bush administration, both extracted as political entities. And I guess you could argue there's maybe a hierarchical relationship between these, but it really just came down to the fact that when people talk about things, they often talk about the same thing but in different ways and so we needed some kind of way of grouping these things together or else you end up with a lot of clutter in your nodes and then that can become a bit annoying so again we leveraged the power of language models to do this and so a second post-processing step this involved trying to clean up some of this stuff to make it simpler there was also an issue about not constraining the output enough So, of course, you can use a few different text files as a validation, but when trying to run this on everything, you start to see some odd things coming up. And one of the things that made me laugh was two entities were extracted, one for rural areas and the other one broadband, which had the relationship not covered by. So quite a damning indictment that 22 years later, the same problems still exist. so in the end our cleaned output is like this it's just two data frames one with the entities and another with the relationships between those entities and at this point it's simple to just export these as CSVs and upload them into a database and so this is a representation of just a single debate so not even a day but just one debate within that day and just using a standard graph database UI you already have a slightly more intuitive way of looking at this. You can see which nodes have been extracted, what relationships they have and if you start clicking into these relationships you can already get a feel for who's agreeing with who, who's disagreeing with who and which policies a politician supports. Although in these two cases, it's a bit simpler to see what's going on. But now you can see here, I see this individual supports five different policies, but I have to click into each of them to actually see the text, which is not so intuitive. And so the next question becomes, well, we have some knowledge now stored within this. How do we extract it out of there? And especially thinking back to our journalist, what is the most natural way for them to interact with this stuff? So everything we spoke about now was using a language model to load data into a knowledge graph. The second part of the equation is how do we extract information from there? How do we get some insights? So a typical SQL query will not work. Usually graph databases have specialist querying languages called Cypher. And they look a little bit like this, the top here. and as I mentioned earlier you don't need to perform lots of joins on a graph database so these two queries are equivalent to each other, the bottom one is in more ordinary SQL and the top one in Cypher, so you can see how efficient the query is in Cypher simply because we don't need to be explicit about the joins but of course we don't want our journalist to write SQL or Cypher so we implement something quite similar to a text-to-SQL system although this time it's a text-to-cypher the process at a very high level goes we ask a question, the question is then converted into a cipher query that query is then executed against the database which then returns some results and those results are then used as context along with the original question as an input to the model so that the model can reason about the structured data in your graph database. So earlier you saw we had this situation here where we couldn't quite see which policies this individual was supporting. Well, now I can ask a question like, can you summarize the policies so and so supports? And then you can see here it generates this cipher query, which is this green line here. it returns these results from that query and then using that it provides this as the answer which I've just copied out here so a much more natural way of understanding that part of the data I was quite interested by this last question here it says here additionally he supports resolution 1284 so as a follow up I decided to ask can you explain the details of resolution 1284 and of course it tries to generate this query but it says no, I'm sorry I don't have specific details of resolution 1284 I thought this was interesting because it was very specifically named resolution and I thought well this happened over 20 years ago this information must exist somewhere in the model so I decided to ask outside the context of this whole system the exact same question and indeed the same model has a pretty good idea of what resolution 1284 was just thought it was an interesting way in which it can use its internal knowledge but it doesn't, which can be a feature in some cases actually, so it wasn't just trying to invent something, which is good I guess So to conclude I hope we've been able to see how we can use language models to load data into a knowledge graph and why that may be useful where the standard RAG approach might be a limitation and also how we can then extract data from there as well on this fund data set. In general what we see is the second part of that solution, i.e. extracting data from the knowledge graph, that's something companies are a little bit more interested in. The first part, performance, is at least for mission-critical or sensitive applications are just not there. You need to have a lot of humans in the loop to make sure that the information that's being extracted is actually quite in accordance with the source data. Whereas the second part is a lot easier. What happens is, usually the graph databases have two the schema is too complex and so we start by simplifying the schema building system and then slowly building it out and trying to make sure the performance stays in line there. That's all for now, so if there are any questions, I will happily take them.

Speaker 2 [25:47]

Thank you, Usman. We have quite a lot of questions, actually. So I read them out in the order of how many people actually upvoted them. So the first one is, when building knowledge graphs from data, how can one deal with conflicting information coming from different documents?

Speaker 1 [26:13]

One of the interesting outputs of this was there's also some conflicting information because, as it may surprise you, politicians are not very consistent people. What helped there was by having a temporal component to there and then some resolution on top of that to account for these inconsistencies. But it depends a lot on the use case. I'd be curious to hear your specific example. maybe the consistency is kind of by design or not I think that's also important so I'm going to give an annoying answer and say it depends

Speaker 2 [26:53]

What is your debugging strategy in terms of identifying and correcting wrong relations?

Speaker 1 [27:03]

Yeah, good question. So one of the things that I used was I just tried to keep up to date with what was the best practices out there in terms of getting good results. So how to structure imports, how to structure schemas. Also, for certain things, there are exact outputs you require. For example, in this case, there's two types of main entities we're extracting. The politicians that were taking part in the debate, which is a fixed number, so that's unit testable, so you can choose several different files over a longer time range and then run your system and validate it's extracted, the correct ones each time. What was harder to do was validate the policies being discussed because they're being discussed in abstract ways. It's not the case that someone says, I support policy X and the other person says, I disagree with policy X. They talk about it in a much more abstract way with varying language. So for specific exact answer outputs, you can just write unit tests to validate that. When you're iterating, it doesn't break anything. But for the policies, it was more difficult. That still requires more of a human in the loop.

Speaker 2 [28:15]

So, how can graph database model change in relationship, how does graph database model change in relationships over time? Does longitudinal data pose an issue for graph representation?

Speaker 1 [28:35]

Uh, no.

Speaker 2 [28:40]

Long question, but short answer. Where do you see the advantages of knowledge graphs versus RDBs for representing your data?

Speaker 1 [28:53]

As I mentioned, with a standard database, you have a very efficient write time because you store the minimum amount of data you need, but then it means when you query data, it's usually slower if you're performing a lot of joins. So if your data can be represented as a knowledge graph and for whatever reason you have a long kind of pathfinding algorithm you need to use, then it will be quite slow on a traditional database. So, you need to have different types of entities in your data which exhibit many-to-many relationships and a valid use case for which you need to use them. So it's always good to think about, yeah, if you think about a social network, for example, that really lends itself to this type of stuff because you often need to do this like path finding and trying to see how things are connected to each other. that's kind of place you'd want to use this

Speaker 2 [29:56]

Okay, so there's a lot of questions, you know, they're constantly jumping. Do you use simply triples RDF or property knowledge graphs? And how do you handle relation properties?

Speaker 1 [30:16]

Yeah, so the data that was extracted here was simple triples, and that's what the system expects to load in there. So you have a distinct set of entities and then entity relationship, entity as triples. So it does expect that.

Speaker 2 [30:38]

How do you handle relation properties?

Speaker 1 [30:41]

So it depends on your database system, but usually you can define attributes for your relationships and attributes for your nodes. And so you can upload that as a separate file. So here we had a data frame of nodes with some features, and we only did the relationships without any attributes. You would then have a third data frame, which has a distinct set of relationships and then a set of attributes as well. But that was not included.

Speaker 2 [31:07]

When extracting data, do you limit token generation to match ontology for higher quality of extracted data? I'm a bit confused. When extracting data, do you limit token generation to match ontology for higher quality in extracting data?

Speaker 1 [31:30]

I'm not sure I fully understand you.

Speaker 2 [31:32]

Okay, I didn't have time to moderate the questions, but neither I am an expert in this field, so really tough task. You may even pick the questions yourself.

Speaker 1 [31:54]

Yeah, so this question is quite similar to the previous one. How do you validate them? So, yeah, as I said before, some entities you're extracting are quite specific, and then you know that's what it should extract, whereas some, like, policies are kind of more abstract and fuzzy. So you can validate the specifics when you expect them, but not so much the other stuff. I'm still trying to read this other one and see if I can understand it, but, yeah, it has a lot of upvote, so maybe other people understand it, but I'm happy to discuss it afterwards. But no, I didn't do any token limit generation here. Well, so one of the things that has come across in the last couple of months is language models with increasing context length, but what remains to be seen is just because they can take a lot more context doesn't mean they will actually consider everything in that context. That's something I have noticed. So you can import everything if you want to go, but it may not do a good job in actually extracting all those things. So, shall we do?

Speaker 2 [33:03]

One more and then hit the coffee break and then prepare for the lightning toast. Is that okay? Was there a time when you had to change whether something was a node versus edge, such as something said edge was what was actually said, a.k.a. conversation? Changing the nodes into relationships.

Speaker 1 [33:32]

No, not necessarily. But one interesting thing that may be worth pointing out is the way in which we named the nodes, the way in which we named the edges, was having a huge impact on the kind of questions that would be asked. So there was two relationships that I extracted regarding entities, one which was debate with, so two people having a discussion, and another agrees with. And then when you ask a question about which politicians are having discussions with each other, it would confuse whether to connect them to the agrees with or the debates with. And I think this was really just down to maybe a slightly ambiguous naming convention between two relationships that were too similar in concept and maybe had some overlapping features.

Speaker 2 [34:25]

Okay, thanks for being such a great audience, and let's thank Osman once again.