Securing RAG Pipelines with Fine Grained Authorization

Speaker 1 [00:07]

Thanks for having me and thanks for those watching on stream as well. My name is Sohan. I'm based in Amsterdam. I've been in the industry more than 15 years including for a long time at Amazon. Currently work at a company called OTTZ. It's a startup and I'll be talking a little bit about authorization and about racked pipeline so hopefully you learn something new. I'm going to start with a very like silly question which I think I know the answer to but who here hasn't heard of LLMs? No one? Yeah well of course right like my grandma was asking me about it recently. And I think it's in the industry. It's pretty big. But I think we spent 2024 trying LLMs, trying GenAI, integrating it with our applications. But in this year, I think there is some thinking around, hey, we need to think about the ops and the security of LLMs. I know of many people here have heard of this industry body called OWASP, which is Open Worldwide Application Security Project. So it's an industry body. And they put out reports about things that could affect your apps. And they came up with one for LLMs specifically late last year. And I want to highlight three of them, which is how LLMs could potentially disclose sensitive information. It sometimes can have excessive agency, where it does things that it's not authorized to do, because we have just rushed ahead to implement it. And people could also intentionally add malicious code to a vector database for it to do something. And all of these, three out of these 10 problems, so the ones I've highlighted, can be solved with proper authorization. So that's why I think it's important to talk about. And specifically in the case of using AI in things like health care and finance, you sort of want security and guardrails around what your LLMs can actually access. So that's what I'm here to talk to you about. And just from, I think, last year too now, we're already seeing so many different ways people are adding generative AI in production. When I was making this deck a few weeks ago, I was like, sure, apps, agentic AI, rack pipelines, LLMs. And now this new thing called model context protocol has come up. I'm like, I can't even keep up. But it's moving so fast. I think that's something we all can agree on, because it is exciting. And today, I'm specifically going to focus on RAG. A few of you all might be familiar with it, but let me set context. It stands for Retrieval Augmented Generation. And the idea is, essentially, say you have a large corpus of documents and data, something that is internal to your university or your company. An LLM typically will not know that data, or won't be trained on that data. And it's too expensive and time consuming to build your own inferencing model just on that data. So you can use an existing LLM to sort of work with the data that you have to build something like a chatbot or some Q&A around those documents. And that architectural approach is called retrieval augmented generation. At a very high level, it has two phases. Your first phase is an ingestion phase where you start off with the knowledge base, so all the way on your left, you preprocess all the information where you clean that data up, you clean up the formatting, you break the documents down to smaller pieces that is more efficient, and you basically set it ready for your embedding model. Essentially, embedding is a way to convert some data, like words or images, into numbers. Because guess what? Computers like numbers. So using an embedding model, you can convert the word king into a matrix of numbers. And the word queen will be a matrix of numbers very similar to the word king. But the word banana might not be very similar at all. So it's a way of representation. And those embeddings are stored in a vector database. So this is what you're doing with your documents. And now comes the second part of a racked pipeline, where you have your query, your retrieval, and your response. So the user, all the way on the left, has a query. like, oh, how do I do this particular thing? And that query also goes through the embedding model. You get an embedded query, which goes into your vector database. And the data from your knowledge base and the user's query is then fed to the LLM. So this is a very, very standard rack pipeline architecture. If you look closely at this, I feel there's something missing, which is there is absolutely no mention of security or authorization. Because what if that user is not authorized to view a particular document in that knowledge base? Right now, they will be able to access it. Like, hey, how much does my CEO earn? You don't want my user finding that out, right? But without proper authorization, that's what can happen. So just to get it into the understanding, your LLMs really shouldn't access unauthorized data. Again, especially in enterprise, health care, finance, and so on. My specialty is working in authorization, in the space of authorization. So I'll give you a very small, brief history lesson of authorization and what the latest and greatest is. And then we'll talk about that in the context of RAG. So this is a very quick primer on auth. So first things first, I think there is a difference between authentication and authorization. It's common to conflate the two because sometimes they happen together. Authentication is identity, who you are. So when you click on Microsoft LDAP or sign in with Google or whatever, that is identity. The permissions you have after you're verified, that is authorization. So authorization comes after identity. In the past, very, very early on in computing, people used access control lists, which was one giant list of everyone who can access a thing. So for instance, I can say all the people in this room can access this stock, but not very scalable. So things change to something called role-based access control. You might be familiar with this at work where, say, you are tagged as an engineer, and all engineers can check in code. Someone's tagged to sales, so all sales can check quarterly earning report. And we do that in our university and in our workplaces. It's not very flexible, though. It's called coarse-grained, where it's very broad. What if a salesperson wants to check in code? Well, if you create a new role, it's super confusing. That changed a little bit to something called attribute-based access control, which is quite popular now, where everything is an attribute. So you can say name, time, designation, room, which building they're checking in from. All of those are attributes. And when you have to check for permissions, All these attributes are taken into the decision. And if everything's OK, you say yes. Very popular even now. But at scale, it can be very complex and very resource intensive. Sometimes you're checking for like 300 attributes, and that is computationally expensive. So things changed in specifically 2019 because Google released a white paper called Zanzibar. Now, for those of you all unfamiliar, Zanzibar is also confusingly the name of Google's internal authorization system. Has anyone sent an email on Gmail with a link to a Google Docs, and there's a pop-up that says, does this user have permission to read this document? Some of y'all, yeah? So that is possible because Google centralized their authorization into something called Zanzibar, and they detailed all of it in this white paper. Side note, I have a few copies that I've printed out off the white paper, so when you're going, feel free as long as talks last. But you can read this paper. It's actually a pretty great paper because it's written very conversationally. And they spoke about how, at the scale of Google, they centralized authorization across Maps, YouTube, Gmail, Sheets, et cetera. And they got to the scale of 10 million queries per second. And this was back in 2019. So now it's probably a lot more. And they did this using something called ReBAC, or Relationship-Based Access Control. And this is what all the cool kids are doing. So I think here's where you pay attention. So they didn't do role-based or attribute-based. They did something called Relationship-Based Access Control, where everything is a relationship between one object and another. So to give you an example, say you have a user named Gru. Gru, there's a folder named Evil Plans. Gru's relationship to that Evil Plans is Gru is a viewer of that folder. There's a relationship. Maybe that folder has a document called Steal the Moon. There's a relationship between the folder and the document, in the sense that the document is in the folder. So everything in how Google looks at things is represented as relationships. Specifically, you can have a user, like a user ID. You can have folders. You can have documents. And relationships can be anything. can be a member of a group relationship, editor of a document relationship, and so on. So what this paper basically said was we'll represent everything as relationships. And those can be represented this way. So I want you to start at the hash sign and start what's on the right of the hash sign. This is Google's way of doing things. Speak to them if it's confusing. So basically, you say the owner is user number three of the document one, two, three. So this is called a relation tuple. And this gives Google a lot of flexibility, because you can also have one object relate to another object, or one object relate to another object's relations. I'm getting a little heavy. For instance, you can say the group security is a member of group engineering. At work, you've seen suddenly there's a reorganization. things change, all permissions again change in your internal system. But with Reback, you can change these things around quickly because it's just a change in this relation tuple. So all of this sounds confusing. What are we really doing here? All of this is essentially represented in the form of a directed acyclic graph. So again, say you have a document on the left, and you can have a reader and an owner of the document. And you have two people who are readers, Fred and Sean. You have Jill, who is an owner of the document. You will notice a dotted arrow between reader and owner. That's because all owners can also be readers. Again, similar to how you use Google Sheets. If you have edit access, that means you can also read the document. You can add complexity. So say you have an organization, and you have an organization named The Org. You have an admin of that organization. Anyone who's an admin is an owner, and therefore anyone who's an owner is also a reader. So say you have to make a permission check, say a page is loading, and you have to check if Jill can read some document. How you determine that is basically seeing if there is a path from some document to Jill. And in this case, there is. There is a directed path from some document to reader, to owner, to Jill, and therefore there is a relationship between Jill and some document, and therefore they can read it. It's a very different way of thinking of permissions, but at scale, this started to work for Google, because this is what they are implementing. Now this happens to be very, very good for securing things like rack pipelines, because Because with Reback and with a Zanzibar-type system, you get very fine-grained access control. You can specify what exactly someone can access. You can specify when they can access it. And instead of trying to fit them into a role or fit them into attributes, you can simply just define relationships between users and the resources. And any time something changes, which happens all the time, you can adjust these permissions to do it. So that's what we're going to do today. Now, 2019 was that white paper, and it made waves in the security and authorization space because everyone was like, whoa, this is pretty cool. Companies like Airbnb and Carta implemented versions of this internally, and they still use it. There are a bunch of open source products that actually made open source implementations for this. And one of them is SpiceDB, which I'm going to show the demo with. Essentially, it's an open source implementation of Zanzibar, quite a few stars on GitHub, because that's the only metric that really matters, right? But also contributors from Adobe, Red Hat, Google, et cetera. And what SpiceDB actually is, and this is the technical part about SpiceDB. So it is a highly parallel graph database. It's not general purpose. So you can't use it for SQL or RDBMS queries. It's meant only for authorization queries. And SpiceDB will typically store all these relationships in a different data store. So you'll have like Postgres or some database storing that. And as a developer, for us, we can write GRPC or HTTP APIs to do a few things. The first one, this is the most important, is you read and write schema. Now what schema essentially is is way to model your real-life use case into this permission system. So if you were modeling Google Docs, you would start with having a document, you would start with having a user, and then the relationship between the user and the document. If you were doing YouTube, you'd have a video, you'd have a user, you'd have channel admins, and the relationships between them. So that's what a schema is. Once you have a schema, you can write and read and update and delete relationships between the things in your system, and, of course, you can do permission checks as well, like does Jill have access to this document? You can actually also do reverse lookups, which is tell me all the documents Jill has access to, or give me all the readers of some document one, and it gives you a reverse lookup. And lastly, of course, because it is a dev-facing product, you have things like the dev tools and the testing around it. So let's talk about techniques now, right? So we have a rack pipeline. We have an open source product called SpiceDB, which can help with securing this rack pipeline. How do you actually do it? There are typically two approaches that you can do. One is filtering pre, and one is filtering post. So let's talk about the post-filter approach. Again, very similar to the rack pipeline architecture I showed you. You have a user who makes a query. Query goes through an embedded model, you get an embedded query, which is then stored in a vector database, and you get relevant embeddings. We are doing the post-filter approach here. So essentially, after you get this, you add a Zanzibar-like system, or like SpiceDB in in this case, to check whether it has access to the documents. And how you do that is in the query at the beginning, you can add metadata into the embedded query itself. So you can say the origin of this document is document one, two, three. So you're adding metadata into the vector database from where the document came. And then you make a check to see if, OK, does this user have access to this particular document? The good thing is, you can actually specify that I need five pieces of additional context before I make a call to the LLM. Because you've already processed everything you wanted and you're doing it post, so you can actually specify how many pieces of context you want before it goes to the LLM. So this is enabling fine-grained authorization on your embeddings. But if you have a low positive hit rate from your vector database, then this might not be very efficient. Because essentially, say you have about 1,000 embeds in the vector database, then you're filtering all of them and you're maybe just getting like five, which might not be very efficient. But if you have a high positive hit rate, this method works really well. It really depends on your use case. The pre-filter, on the other hand, works well if you expect a low hit rate. So the pre-filter auth works in the same way. It's just that the filtration's done before. Again, the user makes a query through the embedded model. And now you do a lookup request to say, hey, give me all the documents that the user has authorization to access. So you're doing that beforehand. And once you get a response, you feed that to your vector database as an embedded query with the filter attached. So you're filtering out what documents the user can access before, and then you feed that into a vector database. And of course, the rest is the same. You get the relevant embeddings, and that goes through the LLM, and you get a response back. So this is the demo I'm actually going to show you, the pre-filter authorization. I will use, again, SpiceDB, which is open source. We will use Pinecone as our vector database and the OpenAI API as our LLM. Honestly, you can swap this out for pretty much any LLM like Cloud or DeepSeq or whatever. And we'll also use to construct the demo. A question I get asked quite a bit is, how do you choose a pre versus post filter approach? If you have a high positive hit rate, then you would do a post filter approach. If you don't have that high positive hit rate on that, then I would say go for the pre-filter approach. So yeah, let's go for the demo. The demo is, again, on GitHub in a very self-guided way. So you can actually do it step by step and follow it all the way through. There's a Jupyter notebook there, so you can actually access it. So I'll pause if you want to take a photo. And I'm going to attempt and make a quick prayer to the demo gods, because you know how the demo gods can be. And I will attempt. The only thing is I have to look at the screen here and do the demo, but that should be fine. Now, first things first, I have an instance of SpiceDB. Let me just escape this. I have an instance of SpiceDB running locally on my machine here. I have the SpiceDB binary, and it's running here. So any checks, you will actually see it updating here. Let me just open my Jupyter Notebook. All right. So first things first, we'll skip the pip and all of that. You know all of that already. We are going to start with this part, which is what I told you about schema. And folks at the back, are you able to read the text? I can increase it a little bit, perhaps. Yeah, all good? Folks at the back? Yeah, okay, excellent. So as I mentioned, our use case is very simple. The idea behind schema is to model your use case. Our use case is simple, you have a user, you have an article, and let me back up and explain what I'm going to do in the demo, my bad. Essentially we have an LLM and we're going to feed it a document with a fictional piece of data, something that doesn't exist, and then we'll ask a question based on that fictional piece of data. Based on whether the user has authorization or not, you will get an answer from the LLM. If it's not authorized, the LLM says, I don't know. If it is authorized, it will give you that fictional piece of data. Hope that's clear. So my real-life use case is simple. I have a user. I have an article. The user is a viewer of the article, and all viewers have permission to view the article. Very straightforward, nothing too complicated. Can this user view this article or not? So I've run this code already. And now I have to start writing my relationship, where I create a user and I create an article. So this is using the SpiceDB SDK in Python, of course. And here's where I'm creating a relationship. You can see a user named Tim is related to an article with the ID 123. It's just boilerplate code to create a relationship. And we say, hey, Tim, you get one more article you're related to. User Tim has a relationship of viewer to this article 456. So essentially, Tim is authorized to view these two articles. That would be part of the rack pipeline. In this code here, I'm just using code to create a pinecone index. For those of you all not familiar, Pinecone is a vector database. It's quite an easy-to-use tool, and I've already created it here. So you can see I can click on it, and you will see that in a bit. And I'm going to add some data into this document. And this is the data I'm adding into this document. I've come up with something very weird, which is Bill Gates won the 2025 Oscar for best football movie. Now, there's no such thing as best football movie in the Oscars. And if there was, I don't think Bill Gates would win it. But that's why I'm just adding it into that document, right? It's a fictitious piece of information that is only part of our document base. So I'm adding it to Article 1, 2, 3 that Tim can access. And now is where I create a langchain embedding object here in this namespace for Pinecone. And I've up-certed that data into my pine cone vector database. Now what do I do? I'm just going back to my code to show you where I'm at. We have done till here so far. So now let's look up what data Tim can actually access. Going back here, and I'm going to run what's called a lookup resource request for user and Tim. I'm just running that code. And you will see, oh, I have to write these. I didn't write this. Yeah, I wrote that. And I can actually open my SpiceDB. And you can see 323, it's written a relationship. It's speaking to my local instance of SpiceDB. And let me actually look up what articles Tim can access. So Tim right now is authorized to view these two articles. Again, you can see the call that's made here, which is a lookup resources call. So rack pipeline, Tim can access these two, and that's it. And now, I'm defining an ask function, which speaks to the OpenAI API. I'm creating a langchain object, which is an embedding, which I spoke about. And I'm initializing a retriever. And what the retriever does is it creates a prompt first. Sorry, first it adds a filter of the articles that they are authorized to view. It creates a prompt with a context and question, which is the typical LLM prompt. And I'm creating a runnable parallel, which includes the question and the context, the prompt, the LLM, and, of course, your past output. And the question is, who won the 2025 Oscar for best football movie? So let's run that. And you get the answer, Bill Gates, because Tim is authorized to view that document in which this piece of information is. So it's spoken to the OpenAI API, and it's done this. Now I'm going to delete the relationship between Tim and that article which has that, right? So that's in 1, 2, 3. So I'm going to delete that relation right here, and that's done. And let's ask SpiceDB again what are the documents Tim can view. So Tim can view only 456, which is right now empty. Again you will see at the bottom these calls are happening right here. Let me just, in case this wasn't done. And now, if you run this, I run the ask again, who won the 2025 Oscar for best movie? You get, like, no response, because there's nothing relevant from the LLM, because it doesn't make any sense to the LLM, right? Because they're not authorized to view that one document. So in a way, I've used this authorization so that Tim, who can't view that document, can't view anything related to that document, even though they're speaking to an LLM. So that was the demo. Again, it's all on GitHub, of course. I put the link up. And yeah, congrats. You secured your rack pipeline. There are a couple of upsides to this, which is, one, you get audit logging, of course, through the entire process. But you can also optimize your query response times because you're doing filtration either pre or post only on the documents that they're allowed. If you don't have that, you're actually not filtering. So your query response times can be inefficient if it's a very large corpus of documents. And of course, at scale, you're building very efficient rag pipelines. For next steps, you can play around with it. If you want to play around with schema, you can try play.Outset.com. There's a Python client library and different resources you can learn from. And there's also the GitHub link that I shared earlier. We have, I think, four more minutes for questions. So if anyone has any questions, I'm happy to answer.

Speaker 2 [27:32]

Thank you for this informative talk. We indeed have some questions. Is there an in-process version of SPICE-EDB available or planned?

Speaker 1 [27:42]

Can you repeat that, Ana?

Speaker 2 [27:44]

Is there an in-process version of SpiceDB available or planned?

Speaker 1 [27:49]

In I didn't get the

Speaker 2 [27:50]

get the in process okay maybe

Speaker 1 [27:54]

I mean, I can maybe give you some comment, but SpiceDB is completely open source. Yeah, it's on GitHub. It has a very community-driven process, so if you want to contribute, you can. There is also an enterprise version of SpiceDB that a couple of large enterprises use. That's, of course, a paid version. So if you're interested in, you know, with, like, support and stuff, that's available, too.

Speaker 2 [28:16]

What do you think about open search efficient filtering, basically dynamically pre- or post-filtering?

Speaker 1 [28:23]

I've personally not used OpenSearch in the context of rack pipelines. So I'm not sure what the performance implications for a rack pipeline is. But OpenSearch is a pretty industry standard tool for, you know, yeah, filtering. So I've not used it for rack pipelines, so I don't know.

Speaker 2 [28:45]

the next question is is there any option to the filtering during the vector search itself not pre or post

Speaker 1 [28:52]

Ooh, interesting. So as far as I can tell, I don't know if there is. I don't think there is. Because in the way the rack pipeline is constructed, the way I see it is that the filtering will have to happen either pre or post. I think the way things stand right now in terms of how vector databases and Zanzibar systems work, I think the filtration will have to happen either pre or post.

Speaker 2 [29:22]

Would you let LLM know that there is information which the user is not authorized to view in order to improve the answer quality?

Speaker 1 [29:33]

I mean, so in most of these cases, LLMs don't, where someone takes an LLM and puts it on like an enterprise, that feedback loop isn't necessarily there. So it's one of those things where the user only gets information about what they have access to, right? And there is no feedback loop back to the LLM that you're not supposed to access this because there's no self-awareness there. It's not going to access it anyway.

Speaker 2 [30:04]

We have another question, how would that work with a directory service like AD, LDAP or ENTRA?

Speaker 1 [30:13]

Yeah, that's a good question. So that, like LDAP and other directory services, are authentication. So authorization is the step that happens after. And any Zanzibar system is independent of authentication. So you can have people log in through LDAP or Okta or whatever and still have it follow this entire process. So there is no dependency there.

Speaker 2 [30:37]

and the next question how well does the spicy db approach scale with number of users and documents

Speaker 1 [30:43]

Yeah, so SpiceDB, I mean, look, Zanzibar in 2019 had, like, 10 million queries per second. SpiceDB also is built for scale. Honestly, if you add SpiceDB, it is very operationally difficult to add it, you know, into, like, a workflow because you have to write relationships and permissions. So it really starts making sense at scale. And that depends on things like how you're deploying it on cloud, what your data store is and so on. But this model was essentially built for planet-level scale like in Google. So there are a bunch of enterprises like Reddit and large companies who have planet scale that use SpiceDB in production.

Speaker 2 [31:25]

And would this shown authorization concepts for RAC also work for any other authorization concepts like RBAC?

Speaker 1 [31:32]

Yes, great question. So Reback, you can consider relationship-based access control as a superset that includes role-based access control and attribute-based access control. So if you're like, hey, I want to do something that looks like attribute or role-based access control, you can do that using Reback, because it's kind of a superset. That's the flexibility Reback gives you.

Speaker 2 [31:57]

Is the pre-filter approach not fairly inefficient if I have a lot of filter hits? Are there ways to decide this dynamically?

Speaker 1 [32:05]

Typically, that's a hard one. And how customers we have seen doing it is they sort of know the hit rate, and then they choose either pre or post before doing so. So you know how your rack pipeline looks, and you make a call or you do some tests based on hit rates and then go for the method that works best for you.

Speaker 2 [32:28]

and maybe the last the very last question is when using a system with own permission system for your rack for example conference would you have to duplicate permissions by cdb as well or is there any other better approach

Speaker 1 [32:43]

Yeah. So typically if you're writing to SpiceDB, there is a dual write that happens because you're writing data into one database and you have to write data to SpiceDB. This of course opens up other technical challenges like the dual write problem, which is an entire talk in itself. But yes, if you're using something like Confluence and SpiceDB, you will be writing data to two places.

Speaker 2 [33:04]

Thank you, Lord Sohang.

Speaker 1 [33:06]

Thank you.