Securing AI Agentic Systems: Enforcing Safety Constraints in AI Agent

Speaker 1 [00:00]

All right, welcome to this next session, and as always, please, if you have any questions for the Q&A, post them in the talks, and then I'm happy to introduce you to John Robert, and he's going to give a talk about security AI agentic systems, enforcing safety constraints in AI agent. So please give him a warm welcome.

Speaker 2 [00:30]

Yeah, thanks so much. I'm sure that... Are you enjoying the conference so far? Yeah, I changed a bit of my slides because I know that it's the last day and you guys have listened to a lot of talks and now too much of information overload. So this talk is more conversational because I know a lot of you use AI already, wants to use AI, don't like AI, and etc. So it's going to be more conversational. And then can anyone just say what they want to learn at the end of this talk? Because I like to know what the audience wants to learn so that I can suit my talk to what they want to learn. Anyone? Why did you come here for this talk? What do you think you want to learn or gain at the end of the talk? Can anyone say something? Yeah. When I build an agentic system for clients, it's up, up, up. Okay, just say something. When I build an agentic system or chatbot system for my clients, There must be some constraints regarding safety. They want to get some insight. Okay. Someone else? Yeah, so what you said, you want to know the constraints you should put in place when you want to build AI agents in your systems. Someone else, please? Yeah, to prevent retargeting the AI to overdo some constraints. Okay. Yes, also it said the constraints you need to put in place to prevent the AI from doing something. Yeah, so at the end of this talk, you are going to learn how, what you know already in software engineering, in software engineering, in data engineering, in all the engineering we have, what you know already, and how to apply that in AI agents. So, this is in 2018, before chat tripity. Elon said, mark my words, AI is far more dangerous than nukes we all know what is happening now with the nuclear weapons and all those kind of things and in 2018 Ellen already said AI is more dangerous than nukes that means we need to keep them properly so that they don't explode and how many of us use open cloud or have tried open cloud very few people So wow, interesting. That means we are all scared of AI, in a way. So Cisco wrote, personal AI agents, like Open Cloud, are a security nightmare. This video is a 30-second video. I want you to look close to the video, and then relate it to what happens in AI. No, no, no, no, no, no. In that video, the monkey is the AI. And I explain what I mean by the monkey is the AI. Soma is one of the top AI security engineers at Meta, and see what she wrote, she's using OpenClaw here, she said, trash everything in my inbox older than February 15. And then the open cloud said, OK, I'm going to do it. And instead, it started deleting everything in an email. And then we have also something similar from Microsoft 360. Microsoft is always sometimes having a lot of security issues. So what Soma did is she gave the monkey, the AI, an email and then told the monkey, don't shoot. But in software engineering, we know what we call role-based access control, right? Anyone knows that? So do I need to explain that, or everybody knows that already? So everyone seems to know that. Why is it that when we build software, a new intern just comes in, a new intern on the first day, And we give the intern a production code and access to everything. And then we tell the intern, please don't push. Please don't do this. Please don't do that. We don't do that in software engineering. We don't do that in how we build our software code. But we do that in AI. We write, don't do this, don't do that. So it's like you've given the intern all the access to everything, and then you tell it, don't do this. So like what it did to the monkey, you gave the monkey the gun, a real gun, not a toy gun, not a gun with bullets, and you tell it, don't do that, why not in the other way like you have in access control, you give it possibly a gun without bullets, or possibly, maybe not a gun, something different, and then you see what it does before giving it access to everything. So now, if you relate what we have in software engineering now to AI agents, you don't give, like Soma did, you don't give your AI agent all the access. You give it access to maybe write email, never access to delete emails. Or if you give it access to delete emails, only, I don't know, already you have the access control that says only something after this. Because agents always, or its net, always, they are all probability system. And in probability, there's nothing that's 100% right. It can never, never be 100% correct. There are times it will always go wrong. So because you cannot predict when it's going to go wrong, you don't give it admin access. Instead, create a new profile for your AI agent and then give it only access to mostly write, read, never delete. And same thing if you want to do things with, if you have an open cloud and you need to delete some files or something like that, possibly you give it access to only a folder, not the whole computer because then it can delete everything. So this is one lesson from software engineering. We already know that you can already apply to AI agents. But I don't know why we are doing it the opposite way. We give the agent everything, and then we say, don't do that, don't do this, don't do that. And the other problem is it's easier to write. So previously in my slide, I had this allowed list and denied list. It's easier to write a lot of denied list and a little allowed list. That's what most of us do. but it's better we have only a loud list and mostly don't just say don't do that so that's the first concept here I just gave an example of if you're an administrator you have an AI agent, you give it a specific role, in the role it gives only read only cannot read PPI assets and then also only do summarization, don't do export things like that, you have like guardrails So that's the way we need to think about building and using AI agent systems. So, what do you pay most attention to when you are building or using AI? Can someone say something? The models, Opus 4.6, GPT 5.4 now. What do you pay attention to? I want it to be a bit more interactive. Can someone say something, please? what do you pay attention to your prompt what prompt someone else the response and response okay not the model tools tools hmm okay actually no one got the right answer the right answer is actually tokens because your prompts and your response tokens are the most in the most I think token is more expensive than oil now because that's what you pay most for every developer is measured by how much tokens they use and your token is mostly your prompt and your response. Yeah, has anyone heard about tokenage? Anyone? No one? No one? Wow. Okay. Does anyone know this guy? Have you seen his name before? for okay that is interesting okay does it make sense so technique is talking grenade I know there's a lot of things happening in the world now and I hope we don't get anything like that around here by pre everything gets off very soon but talking it is talking grenade and what does the grenade do it's small and it's expert can this emoji be three million tokens so that's what it's called token grenade you take something mostly emojis you encrypt it so the guy I showed you before yeah this guy is one of the sometimes 100 top 100 ai personality is the one that tests most of the new models and he has been able to um jailbreak most of them using this concept called token grenade so what it does is he encrypts it an emoji or a world just a single word and that single word is 30 million tokens so once you put it in your input like we all do it explode and if it's 30 million tokens, the AI reads it as 30 million tokens. Likely, if you spend 30 million in just one token, obviously your costs go up, it breaks your system, it does a lot of things. So that's what they use mostly for AI jailbreak. But we have the software engineering concept already. Does anyone know like input and output validation or verification? Everyone knows that? Can I see hands up so that I know if I need to explain? Just 50%. So verification is if you give me an input, I check if the input is what I want. So that's what everything is all about. So you give me an input, I check if this input is what I want. In AI, we have the input you get from your user when you're building AI systems. You get some input from the user, which is mostly called prompt. We don't do evaluation on that. We don't verify if the prompt is a grenade. We just pass it directly into the model, get some summarization, whatever, and then we get a response. And then once we get a response, the same thing you pass, you do sometimes people do a bit of verification and things with the response, but most times people don't do, they just pass the response. Again, the only thing everyone is concerned about is does the prompt template, you have a template, You get the input from the user, put it in your template, send the template to the model. The model gives a response. You send a response to the user. No one does the verification. So to avoid token grenade, for instance, you can do something like this. I don't want to bore you with so many codes. You have seen a lot already in this conference. So you can count the number of tokens. Immediately you get a test, a prompt from a user, count the number of tokens before you pass it to the model. So that way, you have constraints and you don't get this issue with token greening. So that's some of the ways you can use to avoid it. And again, because tokens is dollars or euros like we live in Europe, you can also have a budget for each section for a user. So a user passing a token, you count the number of tokens, then you know how much this token is when you convert it to, I don't know, OpenAI models and things like that. Because in OpenAI or Entropic, whatever, they all measure how much you use your token based on the number of tokens you have. So you can already know the budget for each user and then use the number of tokens to get a limit to avoid things like token grenade. I had a lot more to talk about, but I want to answer more questions. with these two software engineering concepts already we can solve some of the major AI agents security issues so there is something we call this OAAPS or OWASP they have the top 10 security issues in AI agents, AI systems and things like that with just these two software concepts you can solve some issues with token misuse by not giving it all the permissions. You can also see like privileged misuse, you have memory and context poison and then you also have rogue agents which is when an agent does what's not supposed to do. So that's all I have for you today. I hope I was able to to change your mindset on how you build and how you use AI agent and AI in general. So I want to have more time for questions so that I can get some feedback and also explain more concepts I have. But I don't want to bore you with a lot of information. Yeah, so this, I guess, linked to my LinkedIn. And now I want to answer more questions.

Speaker 1 [15:58]

All right. Thank you, John. So, yeah, so go ahead. Please submit your questions in the talks so that I can read them out loud and that also our remote attendees can hear them and follow the questions. So far, I'm seeing no questions. So maybe get out your phones and think about them.

Speaker 2 [16:23]

Does that mean everybody understands what I said?

Speaker 1 [16:26]

We just like

Speaker 2 [16:28]

Or no one is facing any security issues with AI yet. Or no one is using AI yet, maybe. Okay, somebody raised up his hand. Maybe go and give him the microphone, please.

Speaker 1 [16:44]

I guess we're going to do that now.

Speaker 2 [16:48]

sorry I was just gonna ask if you have any practical recommendations into like let's say we are working in a productive system and we want to quickly start an environment in which we can prove that our agent is secure for example right like what would be the first things you would do into preparing this sandbox or ideal scenario where you can test if you can break it. So are you asking how to create the environment itself or how to test the environment? I mean let's go for like how you could approach breaking an agent in a productive system. So one of the approach is try to do something like this token aid so there I didn't put a link to it but the name of the guy you saw he has a github repo where he has list of things he does when he wants to jailbreak so you can go to the repo I think I can try to get I know the repo is called lm whatever that's some very funny name you can use he has created some prompts you can already test to see And another thing people do, which you can try, is if you're doing expedited things with image, you can get an image like this, something like this, write something on this image, and change the background to white. So no one sees it. And then you send that into your AI system and see how it does once you do that. But the major thing also is creating a new profile for your AI agent, ensuring that this new profile doesn't have delete mostly except you have a sandbox already that can delete and you don't care mostly read write always good enough I think that's the first that's what I do most of the time I don't I create a sandbox and in the sandbox I avoid complete system deletes or if it does some delete is fine if it's in the sandbox most of the time is what I do do and if I'm doing testing I test with a lot of prompts like yeah I have I just checked that is Github and from Github there are a lot of prompts where you put some signs in your prompt and it does something different so I use that to do a lot of tests and then also try to give some prompts to read system prompt and things like that so it's really the beginning of getting your agent secured or your ai system secured

Speaker 1 [19:35]

All right, so we have one more here, but we're also getting questions and talks about which we should come to, but then, yeah, one more here.

Speaker 2 [19:44]

Thank you a lot for your good presentation. I wanted to ask, what are the current best methods against prompt injection, context poisoning? Like, is it, as you said, don't do, and then write all the malicious intents list? Is it enough, or are there more robust things? So, most of the time, which is what I said at the beginning, we always say don't do, and we never say do this. The problem with don't do is, like, don't do is, you can't exhaust the list of don't do. if i tell you if i give you a gun i can tell you like what we have presently in the u.s you gotta give a gun i can tell you don't do this but humans are mostly they don't like laws so they always do something else so if you do don't do this mostly it's good to also do do this mostly do this then you can add a bit of don't do this because don't do is always a long list and the context after maybe one million token you lose a lot of information about what it should not do so it's best to also have do this not just don't do this now most of us just give the gun to the monkey and say don't do this instead of us giving it a fake gun or a gun without a bullet and you see what it can do before you start giving it more instructions on what you should not do or what

Speaker 1 [21:02]

Okay, so then in the talks we have a question. For internal agentic systems, I often hear we don't need special precautions because it is internal use only. Nobody outside of the company can reach it. What is your take on that?

Speaker 2 [21:22]

So the problem with AI in general is that it's something that explores a lot of things. So you give it an instruction, and you get some results you are not even thinking of. It's not software engineering where you have all the rules. In a genetic system, in AI, even if you give it just one prompt or something to do, it's most times we do something more. Because it has the ability to do that. it's going to do something more so if you're building internal agents you may not need to do everything like you are going to do if it's outer and something from outside but you also need to put us some add some guardrails because agents like i showed you here this is someone from um like she she wrote here this is something internal this is open core for personal use don't do this but obviously it does more than you accidentally do so if you are building something internal you can you still need some galleries maybe not as much as if it's not if it's something that's going to be exposed

Speaker 1 [22:29]

Okay, so people in the audience are wondering, can you tell a few sample cases how to enforce the security limitation beyond general role-based access control?

Speaker 2 [22:41]

it depends on what constraints you want so the most important thing is in agenting system there are two things are important is your prompt and then the tools so row base is mostly for the tools and then this other one is mostly for the tokens and tokens is more on verification and input output you need to do a lot on that because it's a lot of things can happen with your token so in that case apart from rule base I would say write codes or write constraints on what the tokens can be tokens in this case prompts or context or whatever just write information or write code to verify once you get the prompt or once you get the response input and output write constraints to avoid maybe you want so ppi is something that is very important if you send if someone sends an input you want to check if that's ppi bias is also something that is very important if someone sends an input you want to check if there's some bias in it or if there's an output like what happened to microsoft or google previously when the model gave a response that was by us in this case so if you get a response from an ai don't trust it ensure you do some verification to ensure that it's done properly.

Speaker 1 [24:08]

What would you say, is it more effective to tell an agent what to do rather than what not to do?

Speaker 2 [24:17]

What to do is mostly more important than what you should not do. What you should not do, you should also do that because, as I said, in any law in the world, you have what you can do and what you cannot do. What you can do is always exhaustive. What you cannot do is always a long thing you can do. So you can do both, but add both of them. What you can do is mostly better, but also include what you cannot do. Because if you just tell me I can do this, then I don't know what I cannot do. So it's important to have a balance of both.

Speaker 1 [24:55]

Okay, and then is disabling delete access for an agent really more effective than disabling write since files can just be truncated?

Speaker 2 [25:08]

trying to get what I mean so delete it's more because if you delete something you can get it back if you truncate also you are kind of just deleting a part of it so it depends on our what you wanted to do but I think most every time the issues we have is more on delete on truncate is based on what you want to do it's that I'm not sure it depends on your use case

Speaker 1 [25:34]

Another question in the talks. Can permissions protection against agent malicious prompt injection? How effective is it in this case?

Speaker 2 [25:57]

So, prompt injection and role-based, they are two different things. So, that's why I showed you two software engineering concepts. For tool misuse, you have the role-based, which is focused on the tool itself, how you use the tool, what the tools can do. For prompting, that's when you talk about the token itself. So, they are two separate parts of the agentic workflow. So, for tools, you do more role-based for anything with your prompts, response, and tokens. you do more on verification and evaluation of your tokens or your contacts or your response.

Speaker 1 [26:34]

All right, so someone was checking the Harmless Liberation Token 8 prompt on the GitHub repo and it's only around 400k chars. How can that be 30 million tokens? Do you have a source explanation for that? So someone was checking the Harmless Liberation Token 8 prompt on the GitHub. Yeah, and it's it's only around 400k chars. How can that be so many tokens? Do you have a source explanation?

Speaker 2 [27:03]

Sorry for, yeah. If you check the GitHub, it has several examples. So you can write the code to do the encryption. So what I gave you example is if I want to do something, you can give it 10 million, you can give whatever. It's just like you're writing code. You can change this into 3,000, 10,000. On GitHub, the guy gave an example how you can do the conversion. So it's not really about the numbers itself. It's about how you can convert something small into something big also like in grenade some grenades are small some grenade can do whatever so it depends on the size and what you do what kind of things you encrypt in that particular emoji

Speaker 1 [27:46]

Okay, so the mythos model is rumored to have broken out of its sandbox. How do you secure an agent like that? Which one? The mythos.

Speaker 2 [27:56]

Ah, the Mithos. So for the Mithos, I've been reading a lot about Mithos for a long time. The problem with Mithos is mostly because it can evaluate security issues itself. So it's not really about sandbox. Mithos was not tested on a sandbox. Instead, it was used to evaluate open source codes, and then it already can figure out some of the security You choose it as, and it can export that. So it's not because it was tested on a sandbox. That's a different thing.

Speaker 1 [28:30]

Okay, what would be more examples on tokens beyond the length check?

Speaker 2 [28:38]

So for tokens, you have PPIs, and in Europe, you can do things on data privacy, you can do things on PPIs, you can do things on, which other one do I do? It depends on the company you're working and on the use case, because I do a lot of things on the context I get from the prompt based on what I want to do. For instance, when I work in real estate, I give an example. If you, in German words, there is this double quote up and double quote down in German. If you put that into some old models, especially models from Microsoft, it tells you it's a private, it sends you an error. Because it sees that as a, I don't know, change it to something different. So it depends on what you're doing. You need to, you can test a lot of things. just like in software if you have an impute i can't tell you everything you need to evaluate when you have an impute depending on the kind of software you're building and how you're building and what's the use case but there's a lot you can do just like you do in input and output in software

Speaker 1 [29:49]

So that was our from the talks questions. So we've got those covered. We still have a bit of time so if you have more questions you can still post them.

Speaker 2 [30:06]

And if not, thank you so much for attending my talks. I focus a lot on AI security. I think this is in every PyCon DE. I spoke in PyCon DE in Berlin, also on vulnerabilities. That was not on agentic. That was just on normal AI. And now we are now into agents. And maybe, I don't know what's next in AI. Maybe something after agents and tools use, possibly. But my focus has always been on trying to avoid misuse of AI and AI agents and AI workflows. Thank you so much for the talk.

Speaker 1 [30:47]

Yes, thank you so much, John.