Simplifying RAG Document Pipelines with Multimodal Embeddings

Speaker 1 [00:00]

Good afternoon. It's my great pleasure to introduce our next speaker, Arne Grohbrücke, and he will talk about how to integrate text from PDF files into an LMM by using natural language processing and other techniques. If you have questions, and I'm sure you will have, please ask him to.

Speaker 2 [00:21]

All right, thank you everyone for joining. So the topic of my talk today will be simplifying rack pipelines with multimodal embeddings. So first, I'm a machine learning engineer at Sioneers. We're a small consulting company, and in that role, I do all sorts of projects in the gen AI or data science. Lately, a lot building rack system or agentic systems in general. So first, while I'm sure most of you already know what rack or retrieval augmented generation is, just to get everyone on the same page. Retrieval Augmented Generation is the concept where you enhance language models, large language models, by integrating external knowledge to reduce hallucination and incorporate up-to-date information without the need of retraining. So you basically see it here. You start with your input query to the language model, and before passing it to the language model, you query your vector database, which stores all your external information, and you retrieve the most relevant documents. And then you basically pass the input query and augment it by these relevant documents and pass both of them to the language model, which then generates the output. So the idea is basically to take away, so to not rely on the language model to actually have the information, but to give it all the information and then to mainly use it as summarization tool in the end. The main question now is, how do you get your data into the vector database? For this you use the so-called ingestion pipeline, which is the integral part of any RAC system. So this is the processing layer that turns raw documents into embeddings, which again can be stored in your vector database. The core part now is that most embedding models, especially originally, only accept textual input, which requires you to convert all these documents into text streams. games. So you basically see this here. You start out with any raw document format, which can be a PDF, which can be an Excel, which can be images. For some of them, it might be easier to convert them to text. For others, it might be more difficult. That is basically handled by your processing type pipeline, which takes all of these documents as input and output text chunks, which are ideally equally sized, which you can then run through your embedding model to generate the embedding. So the question is now, why is Rack still challenging? If you build Rack system, you're definitely confronted with the document format PDF, because it's still the dominant document format in enterprise settings. The challenge is that PDFs are display-oriented, meaning they are designed to look good if you print them out or read them on the screen. They're not designed to make the data easy to extract. So converting a PDF into text to then pass it to your embedding model is a lossy transformation, and visual information gets lost or requires separate processing. There are obviously some cases where it's easier to extract the textual information. For example, for text-heavy reports, which mostly contain linear text from left to right, from top to bottom, with maybe some headers or footers. It already starts to get more challenging if you now have multi-column layout, for example in scientific papers where you need to think about reading order. It gets even more difficult if you now have financial tables where you need to align rows and columns, you need to handle merge rows, and OCR solution can easily misalign these. It also gets more difficult if you think about slide decks, for example, PowerPoint presentations which are converted to PDF because most of the information is actually encoded visually, and there might only be sparse text. And it's also getting more difficult for scanned documents, which requires you to use an OCR solution in the first place because the text is not machine-readable. And it gets really difficult if you now want to handle charts or infographics because the meaning is purely visual, and only extracting the data labels of your charts might not get you anywhere. So how has this been originally handled? So the classical approach is you start with a PDF and you apply a layout detection module which basically segments the PDF or a page of the PDF into text, figures, tables and so forth. Then you run your OCR solution or text extraction module to basically extract the text from all of these segments. And afterwards you recover the structure of the document, so you basically want to recover the reading order, you want to process tables, and you want to merge all of the texts. And this then gives you a structured textual format of your complete PDF document, which you can then chunk into equally sized chunks, embed, and store in your vector database. This works well for text-heavy reports, as I already mentioned, where, for example, contracts, technical documentation, and so on. Another advantage is that there's a major ecosystem with open source options for PDF processing, such as Paimu PDF, or even complete OCR solutions such as Marker or Docling. Also, you have fine-grained control over chunks, overlap, and other hyperparameters. However, it becomes more challenging if you think about tables where OCR solution might misalign rows or columns, as I already mentioned, which would produce wrong data assumptions. It also becomes more challenging if you want to reconstruct the reading order. So, for example, multi-column documents might create garbled chunks if the OCR solution misses that there are two columns and just reads from left to right, which we've seen actually in production quite many times, even with paid OCR solutions. It's also difficult to capture charts because they are mostly not handled by the OCR solution, so you need a separate processing layer which handles your figures and charts. And you also lose to complete layout information. So the question I want to ask in this presentation is what if we stopped deconstructing documents into text and instead embed them the way humans read them, as images? So this will simplify the pipeline actually a lot, and this is possible due to the rise of multimodal embedding models. So you basically only take your PDF, you split it into different pages. From each page, or even multiple pages, you create an image, and then you embed only the image of the page, capturing all of the information that is present. So to start off, I want to give you a short overview of the landscapes of models that are available. Just one short thing to mention is I'm focusing in this talk about B-encoder models, which output single vectors. There's also another family of models called latent interaction models, which are mostly included in academic benchmarks. However, they're not really used in production settings because they require you to use much more storage because you need to save thousands of vectors for each page then. The first model I want to mention here is Cohere Embed 4. This handles text and images, and both of them interleaved. It's a proprietary RP, and it's mainly mentionable due to its large context window and because it's fine-tuned for finance, healthcare, and manufacturing. A new addition to this list is Gemini Embedding 2, which is special because it not only handles text and images, but also video and audio natively. It's also a proprietary API. It's still in public preview and not generally available yet. Then there is Voyage Multimodal 3.5, which advertises itself with an excellent Matryoshka representative learning compression. For those of you who don't know what that is. I put that in brackets for dimensions. It's basically a learning scheme where the vectors are trained in a way that most of the information is encoded in earlier dimensions, which allows you to cut off the vector at any point and only, for example, retain the first 250 psychs dimensions. And while you lose quality, you are guaranteed to use as little quality as possible. So, for example, a Voyage multimodal 3.5, they guarantee that you only lose 1% of quality if you only retain one quarter of the dimensions, which can be quite nice in a production setting to save some costs. Then there's also an open source model called Nomic Embed Multimodal 7b, which is built upon QEN 2.5 vision language model, and it's fully open so it weights the data as well as the code. And there's JINA Embedding V4, which is also an open source model, however, in a self-hosted variant it can only use for non-commercial settings, otherwise you need to go via the API, but they also report excellent retrieval quality and they have lower-rank adaption adapters for certain areas. Alright, one thing I haven't talked so far about is the costs. So this was actually the motivation why we in our company first evaluated this topic, because OCR solutions can get quite expensive, as you see here. So for example, the Amazon text drag solution with the table and forms model costs 65 euros per 1,000 pages. And also, cheaper options are still quite expensive. For example, Azure Document Intelligence with the prebuilt layout version or Mistral OCR 3. So they make up basically most of the costs of your complete processing pipeline, while the embedding model only takes like a marginal, not even mentionable aspect of the cost one consent in the case of text embedding three small, which is an OpenAI embedding model. So this is a big advantage of this multimodal pipeline, because you only need an embedding model, and while the embeddings are a bit more expensive, you need a few more tokens to embed these images, the price is still much lower compared to a complete OCR pipeline. So the takeaway so far is multimodal embedding models make the pipeline much simpler, and they can reduce costs tremendously. So the final question is, are they any good? Can they match the retrieval quality of an OCR solution? And this is what I want to look into next, and for this I designed a benchmark. Just a note, internally in our company we ran this on real customer data first, but for the sake of this presentation, because then I couldn't have shared any real information, I reconstructed this evaluation with a public data set, so it's easier for you to follow and you could even reconstruct the experiments yourself. So the data set I used is the PDF association data set, which is available on Hugging Face and which is the filtered version of the safe docs corpus, which are just documents scraped from the internet, so it's real world PDFs. And I only used a tiny portion of the data set because I didn't have unlimited costs, so I used 1,300, roughly 1,200 PDFs, and I I ended up with roughly 14,000 pages, where the PDFs had eight pages on average. So most PDFs had between two and 20 pages, with some having much more. I chose this data set because it's diverse across many sectors. So there's documents from governance. There's documents from legal, education, science, and health care. These are real-world PDF data set presenting all the challenges which you might also encounter in your customer data. And they are quality filtered already. so they are filtered to be only English and there's no corrupted files, so it was easy for me to set up. So basically what I did is now I embedded all these roughly 14,000 pages into a vector database, and the next thing I did, I created myself a ground truth data set, and I used a fairly simple algorithm for this. So I just randomly selected PDFs, then I randomly selected a page from this PDF, and I selected the page before and after, and passed these three pages to a vision language model, in this case GPT 5.4, and let the language model generate a question which can be answered based on these pages. So this then generated me one ground truth evaluation entry. I repeated this 300 times, ending up with 256 evaluation entries. Why the gap? Because I filtered out pages with no substantive content, for example a bibliography or a table of contents because that led to trivial questions such as how many sources did this paper cite, which can be obviously answered by a lot of documents. So the evaluation protocol was then rather simple. So using these generated evaluation entries, I could simply run them again against the complete corpus of documents and basically reproduce this needle in a haystack search. It's important to mention that I only wanted to measure the retrieval quality, so I did not generate an answer using an LLM afterwards and evaluated the answer as well. This would have been a different benchmark. I just wanted to know, if I ask this question, is the retrieved page among the top K pages? So these are also my metrics. First I report the recall at K, so it's the correct results among the top 1, 3, 5, or 10 results. I also added a second metric, which is a bit less restrictive, and it counts as a hit if the retrieved page is within plus minus two of the correct page, which I would have expected. So this is the case where actually a question might be answerable by many places in the PDF. And I report the mean rank. If we now look at the results, which were actually quite surprising to me when I ran this first, is that multimodal embedding models do not only perform on par with OCR pipelines, but they actually outperform them in the native settings. So the evaluation is led by Cohere Embed version 4, but it's already tightly followed by Gina versus 4 and also Gemini Embedding. So, I mean, I got to be fair here. You can always spend more time with your OCR pipelines. So I evaluated this in a rather plain setting. I just used Azure Document intelligence with their best model and use text embedding three small. You can obviously spend much more time improving this OCR solution, as I said earlier, having separate processing modules for your figures, and likely you can get this 59% up a bit. But on the other hand, I also spend just as much time on the multimodal embedding, so basically I think it's kind of fair if you compare it with the time I spent to set each of this up. So now the complete results are basically coherent. So all multimodal models actually outperform the OCR pipeline in its plain setting at lower costs. I mean, you've got to see that, for example, the recall at 10, so looking at if the correct result is among the top 10 results, all of these basically pipelines or models perform quite well above 94 or 95%. So it's not like the OCR pipeline fails completely, but it's still a bit worse than the multimodal embedding models. All right. So basically what can we take away now? What are the tradeoffs to consider? So first of all, what you gain, what you pay, sorry, what you pay is page level granularity. So you retrieve complete pages and not chunks. I mean, you can control the chunk size by, for example, embedding multiple PDF pages into one image or something, but mostly you just need to take the pages as they are. And the second, and I think this is maybe the crucial point is that you need to use a multi-model LLM during generation time because all you have during generation in the vector database is a link to your image. So you don't have the OCR solution which extracted the text, but you only have the image, so during generation you need to pass all these page images to the LLM as well, and this basically requires the LLM to do the OCR on the fly, which also likely requires you to use a rather strong LLMs, so you might not be able to do this with very small LLMs because, again, they need to perform the OCR on the fly. However, on the other side, what you gain is a much simpler pipeline. You don't need to worry about OCR, no layout detection, no chunking, so it's very fast to set up. You preserve all the visual information by default, so all tables, all charts, and the complete layout is preserved the way human systems, and usually the documents are at least currently still designed for humans. This is all at much lower costs, and you usually have much higher throughput. So usually your processing pipeline is bottlenecked by, for example, separate processing modules for your images where you need LLMs again, and then if it uploads 1,000 documents, you need to worry about rate limits and everything, so that is much better here, assuming your embedding model has a quite high throughput, which is usually the case. And this whole thing is also formic agnostic, so it doesn't only work for PDFs, you can also use it for photos directly, you can use it for Word if you convert it to PDF, you can use it for slides or whiteboard photos. You simply render them as an image and embed them. Alright, so I want to finish this talk up with some practical guidelines. So first of all, I've always said that you only have the image during generation time, but obviously nothing prevents you during your ingestion pipeline to simply use an open source tool to extract the text anyways, because most PDFs are machine-readable, and a simple open source solution such as Pymo PDF can extract the text and then you can save it alongside the page image, which then still enables you to pass the text to the LLM, so it basically sees all the information it needs directly and can still look up the image for like visual information or layout. And this also enables you to use hybrid search if you would like to for higher recall. The second point I want to make is how you choose your embedding models. So as I already said, there's RP-based ones, such as Cohere, Gemini, or Voyage, which are fastest if you just want to get started in production and which don't require you to set up an infrastructure. Then there's self-hosted one. If you need full data control, you can deploy them on your own GPUs. And the thing I already mentioned is the Matryoshka embeddings, which most embedding models support, which basically allow you to truncate the dimensions, which in my projects I've done quite frequently just to reduce costs. And the final point is, as I've did in these benchmarks, it's always a good idea to monitor the retrieval quality and not just the generation quality, because the wrong answer of the LLM is mostly traced back to actually retrieving the wrong page. So having these pipelines where you have a ground truth data set and measure the retrieval quality is always a good idea and usually costs you nothing because you just need to to generate a retrieval data set once. And running the pipelines only accords a few embedding costs, which are usually not very high. All right, with this, I want to thank you for listening, and I'm open to any questions.

Speaker 1 [19:29]

Yes, thank you very much. It was very, very interesting. We have lots of questions, so please answer them not so extensively. The first question is, what about models like co-Pally? If I understood correctly, you did not consider late interaction models.

Speaker 2 [19:44]

models. Exactly. So these models are late interaction models, and they usually work in a way that they output many 100 or 1,000 of vectors. And obviously they work a bit better, but they also require much more storage costs. And this is why at least I have not seen them used in production, but they're always included in academia benchmarks, and they might be ready for production at some point. But this is why I did not include them.

Speaker 1 [20:09]

Next question. Did you intentionally leave QN3 VL embedding out or is it just worse than the others?

Speaker 2 [20:20]

I've actually looked this up and I think QEN 3 VL itself, I'm not even sure, so you could probably use one of the later layers as an embedding model, but as far as I understood it, it's not natively, so it's not natively fine-tuned for this embedding task, but rather for like real Vision QA tasks. So this is why I left it out.

Speaker 1 [20:40]

Next question. Did you do an ablation study comparing multimodal embeddings versus unimodal, e.g. only text versus text plus image?

Speaker 2 [20:52]

I'm not fully sure I understand the question. Maybe just read it again one more time.

Speaker 1 [20:58]

one more time then i should do an ablation study comparing multi-model embeddings versus unimodel

Speaker 2 [21:05]

Maybe just come to me after the talk and then maybe you can explain it.

Speaker 1 [21:09]

Okay, next question. Would you recommend Gemini or Coherent for German data sets?

Speaker 2 [21:10]

Okay, next question.

Speaker 1 [21:14]

Also, I'm curious if having the images during generation time would delay generation. Does the image consume more tokens?

Speaker 2 [21:23]

So yes, the image consumes more tokens, so it might delay the answer time a bit. I did not get the first part of the question. This was, would you recommend Gemini?

Speaker 1 [21:31]

or cohere for German data sets.

Speaker 2 [21:34]

German data set. Ah, yeah, so they have very strong multilingual performance. I actually by accident tested this because first I didn't restrict the retrieval data set to English retrieval queries and it performed equally well. Yes, so definitely.

Speaker 1 [21:48]

We have time for maybe three or three more questions. Do you account somehow for information overlapping two pages?

Speaker 2 [21:57]

I mean, yes, this is a challenge if you, for example, have a table and it overlaps two pages because, I don't know, someone just converted an Excel file to a PDF file. This is always a challenge. In this setup, I didn't account for this. In production settings, you could be creative. This is actually an issue you have frequently. So as I said, you're not restricted to only always embed one page. You could consider embedding multiple pages in some kind of grid layout. However, I have not tested it. I mean it's even more challenging for the embedding model so you might see some decrease in retrieval performance, but it's at least an idea.

Speaker 1 [22:34]

Maybe two more questions Are the overall costs really that lower if you need a multi-model LLM for generation and have to store bigger embeddings?

Speaker 2 [22:45]

I mean, the embeddings are itself not necessarily bigger. I mean, I've shown the dimensions in the earlier talk. So, for example, if you use Voyage 3.5 with only 256 dimensions, I mean, that's not necessarily bigger. You don't need to store the images themselves in the vector database. You can just store a link to a blob storage or something, or to a storage container. Yes, it depends. If you currently use a very small model during generation, then it might rise the cost because now you need to use a larger language model. However, if you're already using GPT 5.1 or 5.4, then I wouldn't say it's an issue because they already are very strong at these capabilities.

Speaker 1 [23:26]

Okay, thank you and more questions coming. When you generated benchmark questions from a random page over AI, how did you make sure that these questions make sense and AI was not hallucinating?

Speaker 2 [23:43]

I mean, I've looked through them because it was just 256 examples. I'm not, don't push it, so I cannot repeat all of them now, but I've just scrolled over them and then kind of verified that they looked sensible, but I have not obviously verified each question. I mean, yes, this is always something if you create synthetic data sets, there might be some questions which are not that good, but I accepted this as noise and just hoped that my data set was large enough to cover this.

Speaker 1 [24:10]

One more last question, because this sounds very practical. When using your rack system in production, what embedding model would you use to query the user's request against the vector database?

Speaker 2 [24:24]

So you need to use the same embedding model as you used to create the embedding for the images. And as I said, then you can just choose from the table. I think all of these models, they perform really good. I guess it's a design choice then, each of them having their advantages.

Speaker 1 [24:41]

Okay, great. Anna, thank you very much for this very valuable presentation. Thank you.

Speaker 2 [24:45]

Thank you.

Speaker 1 [24:46]

Thank you.

Speaker 2 [24:46]

Thank you.