Conquering PDFs: document understanding beyond plain text

Speaker 1 [00:08]

Yeah, hi. I think it's been a very successful first day so far, and, yeah, I'll be giving a talk that's maybe a bit different from my other talks because this really is about something I built more or less recently and, you know, an implementation and more of a really practical talk. And a bit before we start, a bit about me. Some of you might know me from my work on spaCy. SpaCy is an open source library for what we call industrial strength natural language processing, so it was really designed from day one to get stuff done and to be used in real applications, and we also always put a lot of work into all the boring open source maintenance stuff, not breaking people's code, ensuring backwards compatibility and so on, and this actually had a really fun side effect more recently, which is that ChatGPT and other large language models, code models, are really good at writing spacey code and also writing It's basic code that works, so I definitely recommend checking that out because it's a really cool meta use case. And we also developed Prodigy, which is a modern annotation tool for creating training and evaluation data for machine learning models, and it was really designed for developers and is entirely scriptable in Python and really lets you implement these workflows where you use a larger model to create data for a smaller one. And actually also some of the workflows I'll show you in this talk are also things, yeah, we've implemented, you can implement and try out with Prodigy as well. So yeah, all of this type of work would be really easy if all our data came in as nice, clean, plain text or as nice spreadsheets, but of course it doesn't. A lot of it is stored away in paper or more recently now in PDFs, Word documents and so so on. And yeah, the PDF format actually has been around since the 90s. Briefly, Adobe, who developed it, considered cancelling development on it because it didn't catch on. People needed specialized programs and so on, but they did continue. It became an open standard in 2008, and ever since then, it's really ubiquitous in our life and our work and business use cases. And I think there is one kind of obvious reason for it, and that's paper. Businesses want electronic copies that really map one-to-one to paper documents. And PDFs are really just the best way of doing that. And that means we kind of, as developers, we have to deal with that. And ever since then, we've had, I would say, a pretty complicated relationship with this particular file format. And I think one reason for that is that as developers, we really like having a source of truth or something we can develop against that doesn't change, that has the same consistent format. And PDFs really don't give you that. If you think about it, saying I have the data in a PDF is about as meaningful as saying I have it on my computer. It could be text, it could be a scan, it could be an image, it could be a combination of all of it. The layout that's embedded could be super relevant or it could be completely arbitrary and just decorative, we don't know any of that, and that makes PDFs pretty difficult to work with. And with more powerful vision language models, we now do have the ability to process these tasks end-to-end and actually ask a lot of questions and complete a lot of tasks on PDF documents directly. But in these workflows, we still use PDFs as the source of truth, and they are actually some lessons that we have learned from other tasks and other use cases in industry and in practice where using very large models, even though they can solve the task end-to-end in a monolithic way, is actually very counterproductive and makes our tasks unnecessarily complicated. And so a lot of the work in applied context has been around how can we decompose and modularize our workflows again so that we can actually debug them individually and have systems that run faster, because if we think about it, a lot of NLP tasks are actually pretty straightforward classifications. They just become a lot more complex if you shove it all into one, and especially if you're using a monolithic approach, that becomes a lot harder. I've even seen people convert their plain text into PDFs so they can use it with their PDF tool of their choice, so please don't do that. I think there are better ways of solving that. If you want to classify PDFs, you don't need to have a model that does everything at once. You can break the task down, and ultimately what it becomes is text classification, and that is something that we've really solved for a long time. That was really the place where we were trying to get, and that's something we thought about a lot because I've always wanted to build a solution for this, and eventually what I came across was a library called Docling. You might have worked with it before. It's developed by a team at IBM, and it's an open source library and also includes their own models for document processing. And what initially really attracted me to Docling was that it uses one unified structured format, the Docling document. And I think one reason I like this as well was that it really reminded me of something we had built for spaCy, which is spaCy stock objects. So if you maybe have worked with spaCy before, you might be familiar with this, but this This is really one structured format, and so a source of truth that holds all the information, and that's obviously really relevant. So I ended up experimenting with Docling and built on top of that, and what I put together was basically a wrapper around Docling that integrates with spaCy and gives us NLP capabilities on PDF documents. So what we can do is we can load in an NLP object, and we can process a PDF with it just like we would process a text, and we get access to the text-based contents of the document, and we also get access to the document layout, and that includes the page layout, but also all the layout sections. And all of that is included in these layout spans that we can also process and iterate over. have the sections of the document, we have the content, the character offsets, the token offsets, it all is integrated into a single text-based document, we have the types of sections that we get back from Dockling, like for example, is it a table, is it a heading, and we also of course get the bounding box, which is more on the image layer, so we get the coordinates so we can map every section back to where it is in the original document. So yeah, that's really cool. And what's even cooler and more interesting, if we think back to how can we break down these larger problems and make our systems more modular, is that we can actually in spaCy not just pass in a text to process, we can pass in an already processed doc. So we can actually take this doc object that we get back from spaCy layout that uses docling under the hood to extract the contents from the PDF, and then process it with another pipeline that we already have. So if we've done NLP before, the fact that the data comes from a PDF really becomes secondary. So if we have a system that works on text, we can also run that on the PDF document. So in this case, here in this example, I'm using the transformer-based pipeline for English. We process the PDF into a doc object that goes through the NLP pipeline, and we basically get an annotated doc object with whichever predictions, either something out of the box or something entirely custom. And for example, name entities, part of speech tags, dependencies, text categories, and so on in the PDF. And there's a lot of cool stuff we can do with it. I have some examples here. This is how it would look. This is what we get back from the doc link model. Because everything is spans and everything maps back into the original document, we can, For example, iterate over all of these layout spans, and we can find one span that contains another. This is all basic Python, and this becomes basic maths, and, you know, it's completely separate from all the other logic of bounding boxes, for example, and, you know, we can really just do basic computing with it. And so in this case, for example, given an entity, in this case Apple, we can find the bounding box it's in by just computing. And similar logic is used in the heading detection. So we know if we have a section or some content somewhere, well, what's the closest heading? Well, we just go backwards in the document and just look, okay, what's the first heading that we come across if we go backwards? Well, that's the heading of that particular section. So yeah, we can also do that and find the section heading and really iterate through the document that way. So I think all of that can be really useful. And one of the main questions, of course, I get is, well, what about tabular data? Because tabular data is kind of special, because on the one hand, it is something we like. It's structured data. But on the other hand, it's this completely different format that pretty much always needs more context to interpret. And so DocLink actually has its own models for table detection and table processing. And you can read more about that in the table format paper. That's actually what's implemented in the library. And it goes based off standardized images of the different tables that are detected in the document. And well, what's a good format for a table? Of course, Pandas DataFrame is the most logical data format in Python to use. So you could actually access the bounding boxes for any tables in the document. And if tables are found, you can access them as a Pandas data frame and, well, do whatever else you want to do with it. The fact that this data came from a PDF is pretty secondary, and you can just do maths with it or apply any other logic you already have that you would apply to a normal data frame. And another cool feature I think that could be quite interesting also for use cases like RAG is that, well, in a normal document, how do you represent a table? There's not really a good text-based representation of it that you can apply other NLP methods on, so you can define a function that basically creates something, like a summary of a table. So you can treat that table like you would a normal paragraph that just states the numbers. So in this example, really just prints the columns, but you could use a generative model to generate a summary of that table, and then treat that table like a normal paragraph. And I think there's still some experiments we want to run with this if that is maybe a nice opportunity to combine the same methods and also include tabular data. But maybe you can even think of other use cases for this functionality. But I thought that was interesting to include. And what we can see is modularity, just like in normal software development, is actually really powerful here. And the fact that our data came from a PDF shouldn't really matter as much as it currently does. And I think that's why it's crucial to really get the data out of the PDF, not treat it as the source of truth, and instead treat it simply as a different input source that we can also apply more traditional natural language processing methods on. Because these are things that are really tried and tested over a long period of time, that we have used for a long period of time. And yeah, we don't need to train a new model that needs to do everything at once. We can really rely on what we can already do and focus on making the layout analysis more powerful and creating a source of truth that we can actually rely on. So in terms of practical workflow that we can use, and also if we're thinking about, well, how can we improve these models and workflows going forward? Obviously, as we've seen, we can start off with the processing. We create a structured data object that holds all the information and that we can work off. And then we apply an NLP pipeline that we've already trained or one out of the box that works on text. And we create a structured doc object that holds all the annotations that we're interested in. And now, of course, depending on what we get out, we also might want to improve the whole system And especially improve the NLP pipeline, because that is what really gives us the predictions we're after. We're usually not after the bounding box. The bounding box is often just an intermediate step. What we're after is a prediction of, like, what is mentioned in the text in the PDF? What category does this PDF belong to? And so on. And that is something that the NLP pipeline is really giving us. So what we can do is we can take that doc object, we can annotate it, we can correct it, correct predictions that the model makes. And we can feed that back into the model with transfer learning. So with only very few examples, we can create a model that's hopefully even better and more specific to our text type. And, of course, since we're working with PDFs, there is the whole question of layout and whether it's relevant. And instinctively, I think, because design and layout is something that as humans we use to communicate information, instinctively we do think, hey, this is like super relevant and the layout always matters. But what we've seen from actually looking at a lot of documents and use cases is that it helps to take a step back and actually think, how much does this layout really matter? And how much can you abstract away? Because your model will always generalize better if you don't include layout features. For example, whether something is a headline or what type of text it is, how it's laid out, where it is on the page. Is that really relevant? Is that not relevant? If you could take it out, it can generalize better, but maybe if you have a lot of similar documents that you're processing, all your documents essentially look the same. There's also a big opportunity in including these layout features in your model and in your process because it means that you can recognize common patterns, treat these documents the same, and learn from one document things that also apply to other documents. So, this is something I think that's always really important to examine at every new project, and it might be different for each project. And to give you a quick example of, well, how could an annotation workflow look in practice, I've actually implemented this in our annotation tool Prodigy as well, but I think you can also apply the same ideas in other tools and contexts. So here, for example, the task was, well, how can we handle a PDF layout where maybe the layout is relevant or not and annotate spans in it? So we have a recipe, which is a Python function that orchestrates the whole workflow. And all of this is actually open source. So the underlying logic you can check out, and that's on GitHub. And yeah, we have input data. It's just a directory of PDFs, and we also can, in this case, specify some sections we want to focus on, because often you might not want to focus, you know, you want to include graphics, you might want to exclude graphics, exclude tables, you might want to even exclude headlines if they're not so relevant, and really just feature the text and focus on it, you know, to also reduce the cognitive load and only really work with the data you're interested in. And in the UI, it could look something like this. You can see the original position in the document, and you can very quickly see the predictions for different spans of text, different entity types, whatever is relevant for the use case, make corrections, but also get a sense for the original document. Or if you've determined that, hey, layout features are completely irrelevant, you can even leave that side out and not even consider what it looks like on the page if the page just for no good reason at all, uses two columns, and yeah, that's like super unimportant. So it kind of also encourages you to find a workflow where you really only focus on what's relevant. But yeah, we've definitely been playing a lot with different use cases here, and in the end, the data is saved to a data set in the text-based format, which means you can train from it pretty easily. And that's where it's, you know, that workflow really comes together, because, you know, you have output directory here, you have the data set that you've created and corrected, and here this holds back an evaluation percentage. This is really something, I mean, actually it's kind of a bad example to have on a slide, because it's really not a great practice. Ideally, you want to have an evaluation set that doesn't change. If you always randomly hold back some percentage, your evaluation never perfectly compares, so this is really just for simplicity. copy this part too much, but it gives you a really quick experiment, and it gives you an NLP model that you can run on any text or on your PDFs. So it really factors out completely the file format that, as I've shown, we have a very complicated relationship with, and that's often really just a container for text and optional layout features, especially if we want to apply NLP methods. So at runtime, we can have the model, we can have the PDF, and apply that separately, and apply the model to the doc, which actually holds the text extracted from our PDF. And what we ship and deploy is then a pipeline that can run on any file format, and separately, the layout extraction component that we can also test and debug separately, and then use to process documents at scale, kind of similar with the pipe method like we're used to from different libraries, also including spaCy, and I think a lot of workflows implement methods where you pass in a sequence of documents and that takes care of batching and so on. So that's kind of the full end-to-end process. Again, this is still quite early work. It's something that I've actually only worked on quite recently. But I think there's some interesting takeaways from our work that we can use when we are trying to conquer new file formats, including PDF, which is really everywhere. And I think one of them is that PDFs really are a bad source of truth. And we want to work from something that's more unified, that we can really rely on. And that's the same between all processes, whether the data comes from a PDF, whether it comes from a Word document, whether it comes from a scan, or whether it comes in as really nice clean text and how we like it, and that also means that we really want to get our data out of the PDF as early as we can and not treat it as the input for an end-to-end process. That might be nice for a prototype, but actually I think these kind of workflows are also one of the reasons why a lot of projects do get stuck in the prototype phase and never make it into production because you have this monolithic system that you can't easily debug and take apart if you really need to and if your process requires it. So I think instead of looking at building better methods for working with PDFs, we should focus on getting stuff out of PDFs, and we already have the methods if we get our data out of the PDFs. And I think also modularity is something we follow in our normal software development and I think this really gives us a huge advantage here because it lets us apply trial and tested methods and components that we already know work on our documents and it means we can totally develop these independently, improve them independently, and so, yeah, I think We have a lot of experience with NLP. We should use it, and we shouldn't have to reinvent the wheel just for every file format that we're dealing with. And finally, I think we're still really only at the beginning, especially when we're looking at these models that turn a layout format into a structured format. Dockling is definitely worth checking out, and they're constantly improving their models. But of course, if you're looking at, oh, the most common problems users reported, they're definitely around, hey, the model predicted something that's slightly weird. And again, we're still at a level of modularity. We can go even deeper, break it apart even further. But I think this is just something we have to accept when we're working with machine learning. But on the plus side, these models are definitely steadily getting better, faster, and smaller. And I think we'll definitely continue working on this workflow and I think we'll see a lot better document processing capabilities that we can use as one component in those systems. And if you're interested in a lot more background on this, I wrote a whole blog post on the development process and some of the motivation behind the code that we built and the integration and how a modern workflow for PDFs could look. So you can check that out. It's also on our website. And yeah, if you try it out, definitely let us know how you go and what you build. Thank you.

Speaker 2 [22:29]

Thank you, Ines, for a tremendous talk. And we have a couple of questions. So, first of all, how does you do a multimodal reading of data from the PDFs? For example, text-to-image tables happens with spicy layout or spicy. And how do we do RAG on top of this?

Speaker 1 [22:52]

Okay, so what was the first part of it?

Speaker 2 [22:55]

How does it with a multi-modal reading of data from PDFs, so text to image and so on?

Speaker 1 [23:00]

and so on. Ah, okay, so basically a lot of these things are specific to the doc link side, so it processes the document and represents everything that's in that document. So if you have an image, so if you have text in an image, it will be OCR'd. If you have an image that is a diagram or, I don't know, a photo, it will be preserved as a photo. That's kind of something we sort of exclude for spaCy because we're focused on everything you can do NLP with, and you have the tables that we can convert to a data frame. So this all happens on the model side. You have that document format that holds everything, and we focus, basically, in spaCy layout, we focus on the stuff that you can do NLP with, and that's the text. And then I think what was the other... There was another part of that question.

Speaker 2 [23:49]

Yeah, tables happens with space layout and how to drag on top of this.

Speaker 1 [23:53]

on top of this yeah so basically I mean if you dad different approaches yes you can do the rag approach where you use the PDF as the source of truth but I guess the question is oh how do you do it differently well in that case you have text and you have you can do rag on top of it kind of in a similar way as you would do rag on top of text that nicely comes in in a database so in a way you kind of already taking over part of the job that normally your model would have to do in Rack, and the thing about Rack is you do pay a big price for all the flexibility you get, where you can basically do anything, but if you have a use case in industry especially, where you kind of know, you might even know what types of questions can be asked, you know exactly what type of documents you have, because a lot of use cases are very specific, then you can take advantage of that and already do a big part, so your model has to do less, so you do the text extraction, maybe you actually do NLP already on it and put it in the database. So then you have a database and all your system has to do is basically translate natural language questions into queries. And so you have a lot more control over what the system can give back and maybe what it shouldn't do. So it depends a bit on the use case. But I think there are a lot of opportunities there as well.

Speaker 2 [25:07]

Yeah, that sounds good. So, also, the next one would be, do you have an opinion about the emerging OCR by the LLM? So, for example, Mistral OCR.

Speaker 1 [25:20]

Oh, I haven't really evaluated that in detail, so I do think it's definitely quite promising that we are hopefully getting better at OCR, it also makes sense because if we are training larger and larger models on more and more samples, and there are only so many letters and so many options in OCR and so many fonts, it makes sense that we should be better at it and I think if we view it as a component in the system and really if we have one model that really only does OCR and it doesn't have to do anything else and it's not a massive multimodal model that can generate you cat pictures, it's really just like this thing does OCR. I think there's a huge advantage in that and I think actually Dockling lets you configure which OCR solution to use and that means if you could configure it in Dockling you can configure it in spaCy layout as well. So I think if we have more and better OCR systems, because I think that's still a bottleneck we can plug that in and just get better performance throughout the whole pipeline because we have components and not just like one model

Speaker 2 [26:25]

Great, thank you. So I also want to point out that a lot of guys appreciated the design.

Speaker 1 [26:32]

Oh, thanks. Of your slides. Oh, okay. Now, speaking of design, I mean, sorry, it's a tangent, but, A, I have stickers. Also, if you haven't been there yet, on Level 3, Room 305, all throughout the conference, we have our feminist AI LAN party, and we'll also be making a zine about PyCon, so that does tie back into design, so really collage, digital collage, so if you are into design and making things look cool, you can come and contribute. We have workshops, all kinds of things. And yeah, stickers, naturally.

Speaker 2 [27:08]

Great. The next one would be regarding the best practices. What would be your best practices approach for PDFs that are composed of flat images with text or figures, for example, an image embedded inside the PDF?

Speaker 1 [27:25]

Yeah, so an image, if you have an image embedded in a PDF, and you're lucky and your PDF is not completely a mess, again, you know, we have a model here, model makes prediction, so it's kind of, it really depends what the results are, but it should kind of work out of the box. So the nice thing is about the doc link models, they do handle PDF, docx formats, and so on, even images, and will OCR if there is text that is part of an image. So if your PDF allows it, you might be lucky and can actually use this end-to-end workflow, so you load this in, even into spaCy layout, and what you get out is text. Of course, now if, I don't know, if you're very unlucky and the model does not like the way your PDF looks, then you will have to do a separate OCR step, but in theory it should work out of the box. So you have your PDF, and what you get out is text with the bounding box related to the embedded image, and also figures and tables extracted.

Speaker 2 [28:27]

Great, thank you. And the last question here is regarding the Docklin. So Docklin is slower than, for example, PDF Plumber. Yeah. How good does your spaCy layout works scale for multiple thousand PDFs in an NLP pipeline?

Speaker 1 [28:44]

Yeah, so I mean it does make sense like if you have, you know, a vision language model or like a larger model that is always slower, that is really kind of the trade-off.

Speaker 2 [28:44]

Yeah, yes.

Speaker 1 [28:53]

So it's actually something we looked at as well. I think DocLink provided kind of a good mix of trade-offs in terms of like accuracy, speed, and also, you know, use of more recent larger models. But, of course, if you, you know, go for more traditional solution, you're still much faster, just like how, you know, using a CNN model, you know, you'll never beat that in terms of speed compared to, you know, a transformer model. But yeah, sorry for the tangent, Dockling, what was the second part of the question?

Speaker 2 [29:26]

PDF Plumber.

Speaker 1 [29:27]

Yeah, so that's a much more, you know, established traditional solution, unsurprising that, yeah,

Speaker 2 [29:28]

Ah.

Speaker 1 [29:35]

that is faster. Spacey layout itself adds, like, pretty, it's very negligent, like, the overhead it adds, so it all comes down to the model, basically. That is the part that makes it, so, you know, the docking speed is about what you get with Spacey layout. And you can also configure that. That really is a trade-off. Maybe there is room for an extension that really uses a much more traditional approach that's faster, but then, of course, you have lower accuracy. You might have to implement your own OCR. That is always the trade-off.

Speaker 2 [30:15]

Okay, thank you very much. Let's once again thank our Ines for the great talk.

Speaker 1 [30:23]

Thank you.

Speaker 2 [30:23]

Thank you.