Lessons learned in bringing a RAG chatbot with access to 50k+ diverse documents to production

Speaker 1 [00:07]

And yeah, we want to talk about our lessons learned in bringing a Rack chatbot with access to more than 50,000 documents to production and the challenges we faced. And as a disclaimer, this talk is going to be quite technical and requires basic knowledge about how a retrieval augmented generation, or in short, Rack, works. Alright, and I want to start with a small code example. So, I still remember developing my first Rack prototype. And I was amazed how easy this was. So, with early versions of Langchain, back in the time, you could develop a Rack chatbot with three lines of code that can basically answer questions over PDF documents located in the directory. So, I was immediately hyped, but then, and we faced this challenge also at Merck, you realize that actually the road to production is much more challenging. Because for a production-grade Rack system, you're going to need many more components. So for example, you can see on the very left side, you need a data ingestion pipeline that runs periodically, loads the data from the source systems, and ingests them into VectorStore. You want to have a chatbot UI, and you're also going to have a chatbot workflow that becomes more and more complex that can basically, you know, from the user question, derives the final answer. And you also want to have, like, continuous integration and deployment pipelines, a test suite, and also some LLM tracing and debugging capabilities. And so this brings me to the agenda of this talk. So we're going to first introduce the chatbot use case, which has the idea of providing seamless access to R&D knowledge at Merck Electronics. And then we're going to do a technical deep dive into three components. First the data ingestion pipeline, then the chatbot workflow, and finally the evaluation framework that we developed. And in the end we're going to conclude with some key takeaways. So let's start with the first point here, the chatbot use case. And for this, I want to give you a quick introduction into what R&D at Merck Electronics does. So basically, imagine like the smartphones in your pocket to, you know, the latest AI supercomputers. None of those would be possible without modern microchips and display technologies. And that's where R&D at Merck Electronics comes in. Their global team, which consists of more than 1,000 scientists and engineers, develop new materials, like, for example, pure chemicals, specialty gases, and also liquid crystals that are then being used by chip makers and display manufacturers to build these ever smaller and faster devices. And these materials are really used in everything from, like, the tiny transistors in your phone CPU to, like, the very crisp pixels modules in your OLED TV. And so, within this R&D organization, there's a lot of knowledge that is spread throughout various systems, ranging from various SharePoints to, for example, experiment write-up databases. And it used to be, for these R&D scientists and engineers, quite difficult to find relevant data easy and quickly. And as a result, historic data would sometimes not be leveraged for decision-making, and You would also have the challenge that knowledge would become lost or poorly accessible when people leave the company. And then finally, it's also challenging to onboard new employees and to deal with company acquisitions, because every time we acquire a new company, it comes with their own data and knowledge stored in various systems, and it makes it ever harder for people to browse these various systems to find the relevant information. And so this is where the chatbot that we developed steps in with the idea of providing this unified interface to the knowledge stored in all of those systems where you can really answer a question and then it browses all of these documents and comes back with an answer. And the chatbot currently has more than 20 sharepoints and also other data sources connected. We have ingested more than 80,000 documents and we have at the moment onboarded more than 150 users across Merck Electronics R&D. And I'm now going to hand over to Nico who will do the deep dive on the data ingestion pipeline. Thank you.

Speaker 2 [04:24]

Thanks Bernhard. Yeah, let's have a look again. So what Bernhard mentioned earlier is the easy tutorial that you usually see when you look it up how to do RAG. But this is really often just the the tip of the iceberg, right? So the reality below the surface is usually far more complex and when building our system we faced some major hidden challenges. For example scale, like how do you reliably process 50,000 documents or more across 20 share points without timeouts or failures or anything? or how do you handle permissions in enterprise context security isn't optional so it's important that users can really only access the data that they can also access in the source system or quality like how do you ensure consistent accurate responses throughout various technical domains or updates your documents change all the time this requires some robust synchronization mechanism to really keep everything updated and these challenges they consumed a lot of our development effort, but they're rarely discussed in the tutorials. So we want to show you how we solve them, starting with the data pipeline. And for that, let's have a quick look at the architecture overview first. So on the left, we connect to SharePoints, for example, using a combination of the Microsoft Graph API and SharePoint APIs to extract the documents and permissions. And we've also integrated this experimental write-up tool for R&D content, which is not stored in SharePoint and the core of our pipeline really runs on Palantir Foundry, where we've built modular data pipelines, which use a combination of PySpark for heavy processing and Polars using in containers for more lightweight transforms. And these pipelines, they handle the entire workflow. So from document extraction to chunking, metadata, embeddings, vector handling, and so on. And importantly, we dynamically replicate all of these pipelines for each data source to allow independent scheduling and monitoring of all the data pipelines. And on the right we leverage our platform teams managed AWS services like the NLP API for accessing embedding models or the various LLMs. And we also have a managed quadrant service for vector storage. We use this for the vector database or tools like web hosting where we can just easily deploy the chatbot application. And also everything as you can see in the top right runs across three environments, so development, QA, and production, which enables true DevOps so we can safely test new features and validate them and do quality tests before deploying them to production. All right, let's have a detailed walkthrough of the different steps throughout this pipeline. So what we start with, and I mentioned this already earlier, is that we first list all the files in the SharePoints recursively. And for this, we have an Azure app registration with which we can have graph and SharePoint API access to list all the files with valid extensions that we can handle and this also includes videos via transcript and then we also extract user permissions really recursively throughout all of the SharePoint folders on a file level so individual users, Azure AD groups that have access to files and we can resolve all of these and then a big component is also the pipeline scheduling so all of this runs weekly with incremental logic that we've implemented so we always compare the previous pipelines output with what is currently in the SharePoint to see which documents have been modified which documents have been added which documents have been deleted or had their permissions changed and then we can handle these changes accordingly throughout the whole pipeline and yeah for the file download process I mean this is pretty straightforward for for SharePoint we just download the files right but for the experimental write-up tool just has some different APIs which we need to use and combine to get the raw content pretty much. The next components are text extraction and chunking. And it's important to mention that we built the pipeline in a way that it's source system agnostic from this point on, so it really doesn't matter where the data is coming from. We treat everything the same. And then we use PDF processing, which we prioritize over a naive text extraction because in our More experience, this usually better preserves the layout of tables or stuff like enumerations as compared to a Word document, for example. And also this has the advantage that it can handle legacy file formats like doc or PPT, which other parsers can't handle usually. And then we just convert everything to Markdown, because one of our lessons learned is that Markdown is really the best language for interacting with large language models. And you can see this on the screenshot on the right. I mean, you don't have to look at the details, but the main takeaway is that any kind of text that you have, including headings, subheadings, texts, captions, tables, you can just represent all of this with Markdown and the language models, they are trained with similar data, so they will be able to understand even and you can even answer questions with like simple tables like this. You can feed it to a large language model. It will be able to give you an answer and understand the rough structure of this. Yeah, we've tested various extraction approaches. We selected PMU-PDF for LLM for the Markdown extraction. Also, we used LibreOffice for just converting everything to PDF first. And a special shout-out here to libraries like Dockling or Markdown. Those are also great libraries for going from any kind of document to Markdown. So feel free to check it out. But PMU-PDF for LLM is what worked best for us. Also, it's important to mention here that we have some error handling in place. So whenever either a download or a text extraction fails, we just notice that. We write it down. And for the next pipeline run, we just retry. So we make sure that always all documents from the source systems are reflected in the vector storage. And for chunking, we have a custom strategy with just large fixed-size chunks to capture all the complicated content that we have in these R&D documents. Sometimes you need large context there. And we also preserve page numbers and other page-related metadata because this is important so that the chatbot can later on actually link to the source documents and the specific pages that were referenced to generate an answer. Now the next components are, first of all, embeddings. So we concatenate, first of all, the text chunks that we create with document-level metadata so that we always, I mean, the pages themselves, they don't speak for themselves, We need to add some metadata on the document level, for large documents especially. And then we use a combination of dense and sparse embeddings. So dense embeddings is the usual thing you get when you use OpenAI for generating embeddings on your text chunks. But we also use sparse embeddings, we use BM42 for that, which works really well with keywords. And you will see that later in the next chapter, there's going to be some examples on that. And the next thing we do is, yeah, we have the embeddings, so we want to get the vector points and upload them to the database. So, one of our lessons learned here is that we are trying to disable the vector database indexing when uploading points. We have around 80,000 documents now, hundreds of thousands of points that we need to upload to the vector database, and if you start uploading the points already, Quadrant will start building the index and this is going to take lots of compute resources, so your upload will be slower, you will have performance issues, so just try to disable indexing, upload all of your points, preferably in batches, and then enable indexing again and also try to keep the metadata fields in quadrant or in the vector database lean this is designed to be a vector database tool foremost so our experience is that adding lots of metadata fields with lots of content can really slow things down and hurt performance and finally for quality insurance we just replicate the collection again just to be able to compare it with the with anything we uploaded so we can really ensure everything is as is or as expected in the vector database. So, we can just map this. Okay. Key takeaways for the data pipeline is that really building and maintaining your own pipeline is a substantial data engineering effort. So, and to emphasize this, it really took us months to have a scalable data pipeline ready that works all the time that you can add thousands of more SharePoint documents without things crashing. And you really need to question yourself on whether make is the right choice for you or your own use case. But for us, it clearly was the only option because there's no out-of-the-box solutions that fit our all of our needs for example the permission management other tools also have limits on the amount of documents you can add like these azure out-of-the-box assistants they have a limit of 10 000 documents or the hybrid search that we implemented so for us it's really um yeah there was no other choice other than make now let's get to the chatbot which is i guess the most interesting part for the front end we still use streamlit and we have it deployed on aws fargate as a container and yeah let's walk through the step-by-step workflow of what happens when you query the chatbot so the first thing we do is we start the llm tracing and we use lang fuse for that you can see it on the right here and we can really recommend this tool it's open source as well and this is really just a verbose interactive llm focused logging which you can easily add into your application and we use this to debug and analyze any conversations between users and the chatbot and it also serves as a great source for future test cases which we will pick up on later again now the next thing we do is we call it PII classification our chatbot has access to lots of sensible and confidential data and we don't want users to figure out for example which employee writes the best documents so this is first of all for the users to feel safe about using the tool they don't want to add their data and use it if they think people can abuse it for generating performance summaries about employees. So this is a really important thing for us to add here. We rather want people to use it for meaningful things like figuring out which projects are ongoing in R&D right now. So for this, we just use a prompt that explains the task that I just explained to you and some few short examples. And yeah, we also use a small pidentic model for just in this case holding the classification Boolean, and we use this combination of Pydantic plus LLMs using structured outputs with libraries like Instructor, which you might have heard of, in many places throughout our workflow. And with Pydantic, we have data validation, we have type safety, and this is always preferable over something like parsing a string yes in an LLM response or something like this. So we can just recommend that. Now the next part, which is probably the largest component of our workflow, is the retrieval itself. So what we usually start with is a query rephrasing, which is quite standard in RAC systems. And for this, let's have a look at this conversation. So a user asks, what is OLED? Then our chatbot explains something about OLED. And then they ask a follow-up questions like, give me tech details. And there really is a challenge when users follow up with brief context-dependent queries like give me tech details so these queries elect the context needed for vector retrieval so what we do is we use an llm that automatically just transforms the conversation history and the latest query into a nice standalone query that you can really use for for vector embedding so yeah give me tech details becomes tech details on oled which we can use for retrieval and then for answer generation and this really significantly improves the retrieval accuracy Yeah, so we always ensure that the vector search has the complete context needed for finding the right documents. Now for the next step, and I mentioned this earlier, is we have a hybrid search in place. So we need to create two kinds of embeddings. First the dense semantic embeddings, but also the keyword-based sparse embeddings. And yeah, as illustrated here, different query types, they require different retrieval approaches. So let's have a look at the first example. A user asks, I need information about product MG4524. And then with dense embeddings from OpenAI, for example, you will get a really high similarity match for a document that contains MG4524, but you will also have a super high similarity for a document that contains MG455. So both of these are really similar, just a small change in terms of the product identifier. And this is where the sparse keyword-based embeddings really come into play. You can see here in the yellow bar that the exact product match really has a super high similarity match. But the other document, which does not mention MG4524, has a really low match score. So this really helps you in finding the right documents for queries like that. And the opposite example is a question about which product involves hydrogen peroxide. Now, the keyword-based approach will not understand that H2O2 is the same as hydrogen peroxide. But these dense embedding models from OpenAI, they're trained on tons of data. They will understand that H2O2 is the same as hydrogen peroxide. So you will get a super high similarity match for a document that contains H2O2 or that contains hydrogen peroxide with the dense embeddings. But with the sparse embeddings, only the hydrogen peroxide will really match. So by combining both of these approaches, our hybrid search really delivers the best of both worlds, precision of keyword matching, but also the semantic understanding of vector search for conceptual queries. And, now, this is not just relevant for our use case, but for anything that any data that an LLM was not trained on, for example, like your company data. Now, the next step is the similarity search with the embeddings. As you can see in this example, each chunk contains not just the vector embeddings, but also metadata about, for example, user permissions. So when user A01 submits a query, then our system automatically applies a metadata filter that matches their ID against the user's field in each chunk's metadata. So basically you can imagine the vector database collection as just a normal table with one column being the vector and other columns being any other metadata fields like the users in this example, which can be filtered similar to SQL. So really this ensures that we only retrieve documents that the user is authorized to access, which you can see here with the green dots. And this allows us to have a single application for everyone where authentication is implemented on a query level. Finally, we add some more metadata to the retrieved chunks. And then we do re-ranking. And we use re-ranking because semantic similarity or, like, keyword-based similarity does not guarantee that the document chunks are really helpful to answer your questions. This just means that it's similar to the user query. But we instruct our chatbot to answer only given the provided context. So we need to make sure that the context we provide is relevant for answering the question and not just any fluff that fills up the context of the LLM. Now the next step is just the normal rag response generation. We just combine system prompt, chat history, retrieved chunks, special instruction, and the query. And then we truncate the chunks if necessary. We generate the answer and we instruct the model to really cite the sources it's using for the answer generation, which we can then replace with just hyperlinks to the documents in the source system. Now the next step is yet another retrieval. The same thing again, but this time we just invert this access filter. So we retrieve documents that are inaccessible to the user. But we don't use them for answer generation. We don't want to leak information. But we really want to present to the people there could be other documents that might be relevant for what you're searching for, but you don't have access to it yet. So we provide a link to a, in this case, SharePoint access request form where people can just find documents that might be relevant, and then they can just request the access. And we really do this to promote collaboration across the company and to break down information silos across the organization as much as possible. Finally, we let the users leave optional feedback. So they can just rate the responses and provide free text comments, which we automatically tag the corresponding LangFuse traces with with the feedback metadata. So with this we have a valuable feedback loop. We can automatically go to LangFuse filter for traces that have feedback to identify patterns in successful or problematic chatbot interactions. And then we can really generate targeted data sets for testing and improving our chatbot. Okay main takeaways from the chatbot workflow are like first of all the simple retrieval is usually not sufficient for any use case that is out of the domain that the LLM was trained on so we recommend to combine dense with sparse embeddings especially for keyword heavy documents or anything that is really your company data that the LLM can't know about the next thing is balance security with discovery so permission-aware rack really shows both accessible and inaccessible documents you can implement this on a query level if the data pipeline that you built can handle it. And now lastly, implement end-to-end observability. Use tools like LangFuse, they are super easy to integrate into your chatbot applications and this enables really true LLM ops. Now with this I hand over to Bernhard again for telling us more about how we evaluate the whole thing that we just talked about.

Speaker 1 [21:55]

Yeah, thanks a lot. All right, yeah, I now want to talk about our evaluation framework. And as you can see, both the data ingestion pipeline on top and the chatbot workflow that Nico talked about consists of a lot of components. And as a result, we really have a large number of hyperparameters that we want to optimize. And the biggest challenge that we found is that most of these hyperparameters that you can see here and also the corresponding components, they interact with each other. So we cannot really optimize them in isolation. Instead, we need to kind of evaluate them end-to-end. And so let me give you one example for that. For example, every time we add a new SharePoint, this is, we're going to, we can only really evaluate this by looking at the final chatbot answers. So this means, you know, a change at the very beginning of the data ingestion pipeline here can only be evaluated at the very end of the chatbot workflow, so by really looking at the final answers of the chatbot. And, therefore, before we add a new SharePoint, we basically create a new replica of the Productive Vector Store, then load this new SharePoint, and then run our tests here to really evaluate does this chatbot, does this new Vector Store improve the overall chatbot answers? Because it can really happen that, for example, you add a new SharePoint, and then suddenly in the retrieval component, you have, you know, you had a couple of questions that were answered correctly, but now you have new documents that somehow are semantically similar to the query to queries that you had before, and they basically push out more relevant documents that came from other SharePoints. So yeah, I think really having an evaluation framework that you can run here is key to kind of improve all of this hyperparameters. So the question is, how did we come up with a test suite, and how do we then run this test to assess the performance of the chatbot? And all of this starts with actual user feedback. And here I'm showing two pseudonymized examples. So on the left, you can see the first feedback that we received, where the user asked, what is the scope of Project Zebra? And then an early version of the chatbot answered, Project Zebra is a wildlife preservation project, and so on, which is, of course, kind of hallucinated. And then the user provided the feedback that the answer does not mention the Merck internal Zebra project related to ALG ligands. And then on the right side, you have a second example where the user asked, what is a BC37000? And then the LLM answered BC37000 is the standard for high performance computing. While this is factually correct, it's actually not really relevant. And so the user provided the feedback that it should actually mention that BC37000 is a container used for chemical delivery. And I now want to show you how we turn such feedback into test cases. And for this, we implemented a test framework based on PyTest. I want to basically show you a rather simplified version of the actual code that we developed. So first we define here a PIDENTIC class QA test data that consists of a question and an optional expected and forbidden key phrases. And then finally an optional test pass criteria that we define in plain text. And then for the first example here, what is the scope of project Zebra, we would define that as expected key phrases, the answer should always contain the term ALD, and it should contain any of the terms precursor ligand, but it should not contain any of the terms wildlife, animal, and zoo. And we refer to such type of test cases as key phrase-based tests, and they are very simply to evaluate, right, we just have to get the actual answer of the chat bot and then do some substring matching to check, okay, does it actually contain the expected key phrases and not the forbidden ones. And then we also have like these more advanced LLM as a touch test, and you can see also an example here. So for the second question that I showed before, what is a BC37000, we can actually just define as pass criteria a slightly reverted version of the user feedback that we received. So here stating answer should mention that BC37000 is a specialized container or cylinder And the other synonym used for chemical delivery. And as mentioned, we evaluate this using LLM as a judge with this custom pass criteria. And I will now provide more details on how this works. So first, you saw the test cases before. We would use the PyTest parameterized feature to turn each of these QA test data instances into a test case. Then for each test case, we get the actual answer of the chat bot. You can see this here in line three. And then in line four, we would validate the key phrases. I will not show you the implementation here because, as mentioned, this is straightforward. But I want to show you a bit more details about how this LLM as a judge conceptually works. So, if we have this pass criteria defined, then you would execute, you know, line six and following. And here we create first like a judge prompt that we would send to this LLM as a judge. And the prompt template, as you see here, like a simplified version of our prompt template consists of a user question, the chatbot answer, and then this test pass criteria that we defined. And then in the end we add some instruction. Like here, very simple, kept very simple, you could just ask, does the chatbot answer meet the test pass criteria? And then, as mentioned before, we use quite a lot of these open AI structured outputs with PyIdentic. And here we would define as a response model like basically just two fields that we want to get back from the LLM, which is first this test passed flag that denotes whether the test has been passed. And if the test hasn't passed, then we instruct the LLM to return an actual failed reason. And then we can just assert that the test has passed. And if it hasn't, if it has failed, it would basically return the failure reason together with the chatbot answer. And so, overall, we really find that this LLM-as-a-judge with custom pass criteria works quite well in our use case, and it's a great way to, you know, close the loop and directly incorporate user feedback into your tests. And one additional thing that I wanted to mention that we also stumbled on is that, I mean, as mentioned here, I think in our use cases, we have a bit more complex problem, but it's still relatively simple, and this works quite well. But I would also encourage you to check out G-Eval, which is basically an LLM as a judge metric that can be found in popular libraries such as DeepEval. And it's based on a research paper, and it's conceptually quite similar to what we are doing, so it's also LLM as a judge with custom criteria that you can define. But it's a bit more advanced because it also has some sort of chain of thought reasoning capabilities, so it does a bit more reasoning before it comes back with the actual answer. And yeah, this was in a nutshell how we implement test cases with using PyTest and LLM as a judge. And I now want to show you how we run the test suite and how this then fits into our entire development workflow. So at the moment, our test suite consists of around 130 tests, and this gets really run as part of our CI pipeline. And I think the most important takeaway here is that this test pass rate that we have is really the key metric that we optimize again. So this is really what we look at to kind of get a feeling, you know, what is the current performance of the chatbot. And regarding what we test, the two examples that I showed you before were tests that assess more like the question-answering capabilities of the chatbot, but we also test other things such as the PI classification module, which can be valued standalone. And then also retrieval, conversational capabilities, so, you know, how well does the chatbot, you know, do with, like, longer conversations. And for example, is the chatbot able, like, basic sanity checks, can it access knowledge from the system prompt? And you know, regarding the retrieval, I want to mention that one lesson learned for us really is that it's very difficult in our use case to evaluate retrieval standalone because it's usually infeasible to annotate the ground truth documents that you would expect for a given query. And this is not just difficult because of the very large number of documents, but also because the documents are constantly changing, right? If you add a new SharePoint, then this could add, like, the ground truth documents that you would consider for a given query. And also existing SharePoint documents get modified, deleted, and so on. So that's just one learning that I want to show you. And then, finally, I want to mention that the pull request that you see here that implements hybrid search is really something that was motivated by user feedback and some of the failing tests that we come up with. And I now want to basically put all of this together and show you what our development workflow looks like and how this resulted in the pull request that you can see here. So this is roughly what our development workflow looks like. And as mentioned at the beginning, it starts with user feedback. And you know, following the example that I showed you before, in our case, we got back a couple of feedback around the chatbot can answer what is XYZ questions. And XYZ, as mentioned, could be like typically Merck internal concepts like a product name, project name, or anything other that is Merck specific. And you know, when we got this feedback, we then co-created some test cases together with the domain experts, like from this user feedback, and these tests would fail initially. Then we did some error analysis, and I mean, Nico highlighted this already, but at this point we really found that these similar spellings, so for example, BK37000 versus BC37000, they have high similarity in the embeddings and that's kind of an issue because at the time we only had these embeddings. And so this results in irrelevant documents appearing in the retrieved documents. And then we started ideation, okay, maybe quick fix, can we just reduce the chunk size so that it maybe captures, like puts more weight on individual terms or should we maybe implement hybrid search? And so we thought, okay, let's start with, like, the easiest thing, let's just reduce the chunk size, see if this works, like, this improves our test pass rate, and then we can still go for hybrid search. And then when we evaluated this, we indeed found that smaller chunk size had quite some negative side effects. And this is something that is often very hard to anticipate up front, and that's something where you need to test, like, this test feed font, because otherwise you would introduce regressions regularly where, for example, some questions that the shepherds were able to answer before, you know, you play around with some hyper-formulators sometimes and then it doesn't work anymore and, yeah, and this is something that you have to catch here. So what we found is that, yeah, reducing the chunk size is not helpful, but really hybrid search substantially increases test pass rate in particular on these new questions. So what we then did is we merged this hybrid search pull request that you saw before to the dev branch, which was out in a deployment to our QA chatbot. We would then start the beta testing phase, and there the beta users approved this hybrid search. And then finally we would merge this pull request to the main branch, which would automatically deploy the app to the production system. And I really think, like, a key success criteria in LLM projects is how fast it can iterate through this process. And this requires not just a comprehensive test suite, but it also requires quite some infrastructure and automation in place, right, like the CI and CD pipelines and so on. And so, yeah, I'm now getting to the key takeaways. So first of all, we would say that systematically improving LLM apps with a development workflow based on user feedback is very crucial. And this also forces you to focus on what actually matters. Instead of maybe, you know, blindly chasing, as shown here in this example, the cool new ancient framework that you read about on HackerRank, HackerRank, Hacker News, sorry. And then second, I think our experience is that for most applications, you really need this domain-specific evaluations. And we really found that these generic metrics that you would find in libraries, such as for example like answer relevance or something like this, it's not really kind of a key metric that you can optimize against. So we found them to be rather unstable in our use case. And then third, yes, I have to say maintaining a test suite is time-consuming and it often also requires the involvement of domain experts. So it's also something that you have to think about when you plan your project. You would probably also need domain experts experts, because oftentimes I have no idea what a question, you know, what R&D specific content actually refers to. But I think it's really like a mandatory component when you really want to, you know, develop a productive app, so to say. And then finally I want to say that, you know, I think like creating a test suite doesn't have to be such a daunting task, because normally like when you start, you know, with a new new use case, you would do user interviews. And your first version would basically naturally evolve out of this, because you ask the users, OK, what type of questions do you want the chatbot to answer? Then you figure out what kind of documents do I actually need to answer this? And then you would also discuss with the user what would maybe an acceptable answer looks like. And then you would also maybe start defining different user personas and so on. And if you did your homework there, it's very easy to, I I think it's rather straightforward to create the first version of a test suite before you really start writing your first line of code. Of course, you then need, like, this old infrastructure in place to then run this test suite, but I think, like, coming up with that is not as difficult as it sometimes appears to be. All right. And then, finally, regarding evaluation, I also want to encourage you to check out a blog post by Hamel Hussein. I'm not sure if I pronounced this correctly. For example, the blog post, your AI product needs evals. Because what we found is that it's very hard to find high-quality content on the Internet on LLM evaluation. There's so much noise there. And yeah, everything that this guy writes really stands out with very pragmatic and detailed advice, and it was also inspired us really to develop this test suite. All right. And this brings me to the overall takeaways. So as mentioned, first of all, regarding the data ingestion pipeline, building such a component to self, really scalable maintainable pipeline is a substantial data engineering effort that you should not underestimate. And something that you need, you know, resources for. So you know, if you start a project, then maybe do some expectation management up front that this is going to be quite an investment. And then second, regarding the chatbot workflow, I think it's really important to, and I think this is kind of applies to all kind of data science projects, complexify step by step and only if needed. And I would also, we would also encourage you to add tracing right from the start because with LengthViews it's so easy. It's just like you add this decorator, observe, and then you have, you already get these traces and this really is kind of a prerequisite to doing error analysis where you can really see on a prompt level what you sent to the LLM, what you get back, and where things are going wrong. And then when it comes to the evaluation framework, we would encourage you, yeah, create a test suite together with the main expert based on user feedback, and then integrate this into your CI pipeline and into your development workflow that you really have a systematic approach to improve your application. And then at the very last point, I want to briefly touch on adoption. And I think adoption would actually warrant a dedicated talk because it's actually not is straightforward. So, you should not expect that you just develop your Rack Chatbot and then you hand it out to the users and they would understand it, you know, what it's capable of and so on, and then it would just, you know, adoption would just solve itself. Adoption really needs a systematic approach, so you really have to think about providing onboarding training and various other mechanisms such as providing regular office hours and And also establishing user feedback channels such as directly through the app or maybe other possible channels. All right. Yeah. That's our key takeaways. Hope you took away something from it and, yeah, we're happy to take questions.

Speaker 3 [39:26]

Thank you a lot for this amazing talk and we do have a lot of lot of questions I will try to address them as much as we can the first question is about comparison that the system Libra chat docker and deep seek Can you comment a little bit on that and how you make sure that after rock your response still relevant? Do you have any verification steps?

Speaker 2 [39:50]

I'm sure if I got the first question right whether that's about comparing models

Speaker 3 [39:55]

Yes, someone mentioned using a similar approach with the LibreChat, Docker and DeepSync. So I guess the point is if you have tried the same pipeline or...

Speaker 1 [40:08]

I have no experience with LibreChat, and I think in the end, really, this test pass rate is the only metric that we have. And, I mean, we also tried coming up with other type of metrics, but in the end, we find that putting all of our emphasis on, like, this, having this really key metric that we optimize against is actually sufficient for us.

Speaker 2 [40:28]

And it's also maybe important to mention that this chatbot doesn't aim to solve a very specific problem, but it's more like an entry point to the huge R&D knowledge that has been created over the past tens of years for hundreds of thousands of employees. So it's kind of like a tool that needs to do everything. So it's really hard to define, like as a developer, it's hard to define test cases on what to do. So we really need to interact with the users, find out what they want from the chatbot, what kind of questions they ask like we we're not R&D people we don't know what kind of questions they're gonna ask so it's really interactive as Bernhard mentioned and we have to start from there create our test pipeline and then as he said this is the only metric that we have to optimize against

Speaker 3 [41:11]

Thank you. And the next question is about how do you combine different scores from the sparse similarity and semantic similarity?

Speaker 2 [41:20]

Yeah, for this we use Rank Fusion, which is, I mean, there's two options, I think, DBSF and Rank Fusion. We use Rank Fusion. Works well for us. And again, this is just one of the two hyperparameters that we can evaluate with the test pipeline. So we just switch it to DBSF, run the test pipeline, switch it to Rank Fusion, run the test pipeline and see which one performs better. But it also doesn't matter too much. We just retrieve quite a few documents and then we re-rank anyways. so we add metadata to the retrieved chunks using the similarity scores and then we do yet another re-ranking to just filter out which documents are relevant for answer generation and which ones aren't, so that's how we do it.

Speaker 3 [42:02]

And it actually comes to the next question about chunking. Have you tried semantic chunking by paragraph, sentence, header level?

Speaker 2 [42:10]

Yeah, so, I mean, we have around 80,000 documents with hundreds of pages on some of these documents. So any sort of like smart chunking is really costly and takes a lot of time to run. So we didn't do that. And also the documents are super, they are super heterogeneous. So they are from, some of them are 30 years old, some of them are recent. And they are written by hundreds of different employees. they all have their different like the documents have different qualities right some some are with more time spent on the documents some have less time spent on the documents so the documents are really they can be everything from high quality to low quality so there's really no we didn't find a good structured approach to take when chunking so in the end we just resorted to having large chunks with around a thousand tokens because the r d documents are context dependent they are complicated so we just have these fixed size large chunks for now

Speaker 3 [43:12]

Thank you, and how long does it take you to launch the first version given to the users and how many people were working on that?

Speaker 1 [43:21]

Yeah, so I think, like, it took, and, you know, it's always easy to ship the first prototype, and I think we did this relatively quickly, but I would say it took us, I don't know, maybe four to six months to really get to a point where you feel like we have a first, like, so to say, solid version, and I think we had always between, like, one and three developers, but none of them, but they were usually also involved in other projects, so it's a bit hard to give some definite numbers on that so I would say like one to two resources that are really working on it

Speaker 3 [43:57]

Thank you. And the next question is about comparing your system with Microsoft CopyPilot. Have you compared it? Because you said that it missed some features. What was it missing?

Speaker 2 [44:14]

Yeah, there are some limitations with it. I think the Azure OpenAI Assistant feature, they have a limit to 10,000 documents which you can just dump there. So this was pretty much out of the scope for us. We have way too much for that and there are many plans to add even more data to it. And this was basically the main limitation for doing it and also you're limited in flexibility as well, right? If it's a clear purpose use case that you have and you know the document quality, you can really without looking at all of these thousands of documents then maybe this is a good solution but for us due to the amount of documents and the heterogeneity and the document quality and layouts yeah we just had to build our own thing

Speaker 1 [44:54]

Yeah, and I also want to highlight that we don't just connect to SharePoints, but also these experiment write-up databases And there's also ongoing plans to integrate other data sources, and I think this is something that Copilot will not be able to do

Speaker 3 [45:08]

Thank you a lot for your answers. Unfortunately, we're out of the time, but I'll save all the questions so that you can address them later in Discord. Thank you, everyone. Thank you for making time.