Would you rely on ChatGPT to dial 911? A talk on balancing determinism and probabilism in production machine learning systems

Speaker 1 [00:05]

Yes, and I realized yesterday talking to someone that I was wrong. It's actually 1-1-0 in Germany, so I switched that. So yeah, I hope that you guys can answer the question at the end of the talk, but along the way, I learned some cool tools and practices for reliable machine learning in production. I also learned yesterday that I need to make a slide like this. So I used to work for Alexa AI, which is Amazon's voice assistant, so I worked a lot on natural language processing, and now I'm working for GitGuardian we do cyber security and we find secrets in your source code in your Slack, in your Jira and so we use basically the same models until recently it was hard to tell my parents what I do for a living and then OpenAI came and I just said it's like ChatGPT it's much easier, just less money for us. My grandmother somehow still thinks that I do mathematics but I want her to be happy so I didn't tell her she's wrong a question for you what do you think this is? Do you think this is the Bitcoin price evolution or do you think this is my brother-in-law's laser Strava run? It's actually option C it's the number of AI related incidents in the past decades and what you should say when you see this is oh boy this is bad which is what I think that Michael Cohen said when, Michael Cohen is actually a lawyer who used ChatGPT to get previous cases for his own case and he realized that chat gpt is not like google and it can just invent cases which was the case in december 2023 when he just completely hallucinated three cases that never existed and now he's in big trouble another example of a gen ai incident was this matt corda is a journalist who managed to get chat gpt to tell him how to make a bomb it's a true story so OpenAI is working really hard to prevent this but he managed to get the information by saying hey I'm just writing an article to prevent people from doing this so can you help me write this and it works this one is an interesting example because it shows that by trying to prevent the models from being bad for example from making them more ethical and diverse they actually reached a stage where they when you ask for a question to Gemini to generate a picture of the Founding Fathers, it would generate something that is actually ethical and diverse. It's just completely inaccurate, historically speaking. So this is actually a very hard task to make the models ethics and not do illegal things. And before you start thinking like, yeah, this is all Gen AI, this is a new UI, it doesn't affect us, maybe it doesn't affect us yet, but we have examples of computer vision systems that have already affected the lives of many people. for example leading to the arrest of people who were wrongfully convicted of crimes that they never committed just because the ai was making mistakes so this is a real thing i don't want people to be desperate at the end of the presentation it's not about this it's really about knowing that ai is is not flawless and that when we work with machine learning and ai we need to plan around their mistakes and hopefully during this presentation together we learn how we can do this a bit better so I'll just go over quickly the motivations but I don't think I don't think we need any motivations for reliable system I just want to flash you the one you know the main categories of why we need to do this then I'll show you how the Gen AI models work how we can mitigate their weaknesses and finish with some best practices and the cool tools that you can use in your everyday life so really quickly I just categorized them in three big categories The first one is, you know, user experience and brand reputation. If you have a user-facing product that's using AI, Gen AI or whatnot, you cannot have a bad user experience. For example, a chatbot that will start insulting your user. That was actually the case with the Microsoft tablet in 2016. Or even simpler, like if you have an Alexa device and you want to switch off the lights and you just say switch off the lights, it doesn't work. It's just going to be a bad user experience. the second one is just one step further beyond the user experience you have laws you have regulations, you have contracts that you need to abide by you cannot have a chat GPT device that tells you how to make a bomb, you cannot have a driving car that's going to go over a pedestrian so that's even worse than just a bad user experience the last one is just usually we expect to have problems when we have systems in production we just need reliable ways to fix them, to understand what went wrong, and to deploy fixes, everything, you know, hot fixes, reproducibility, root cause analysis, is just harder with machine learning. For example, reproducibility is harder because some of the systems are just not reproducible. They change their outputs every time. As far as the root cause analysis is concerned, many models nowadays, we don't understand why they make the decisions that they make. So, you know, everything is more complex nowadays with those tools, and that's not done. The last point is the cost. I just wanted to mention it really quickly because I won't talk about this in this talk, but you don't need machine learning or complex machine learning to do anything. Cost is a big factor. The rule-based engines are cheaper, faster, so if you don't need them, you don't use machine learning for this. But I won't talk about the cost much more in the presentation. so how do the models work um i take the examples of alexa because i've worked quite a lot on this alexa is you know allows to connect you to many applications whether it's the shopping play some music get the weather or make some jokes and so for example if you ask alexa to give you jokes she'll say you why can't you trust atoms because they make up everything and that's pretty much the state of the art of the jokes that we have at alexa and the question is how does this work so one approach in generative ai is you know very schematically speaking you have an input to a model the input has the user request and the words that have been generated until now by the device and it's going to try to find the next word and so it basically has a vocabulary of all the words that it can generate from and it's going to give them scores between zero and one for example and the one that has the highest scores are going be candidates for the next word generation and so for example when you start a joke usually it starts with what and so that's why it has a higher score and so you're going to pick randomly one of them and then you're going to continue and after what maybe the next most likely word is is and so you're going to pick this word and continue until the model terminates in reality it's more complex than this i'd be more than happy to talk with you about this uh in the in the following of the talk but that's the base idea of how it works. All right, so that's when the model is trained. Now, how do you train the model? Very complex. I'll just say in general how machine learning works. You assume you have a lot of data with what should be predicted and given what you have, so inputs and outputs, and you have an optimization algorithm, and you show all those examples to the models and usually just make random guesses, and then you adjust its parameters so that it can better predict and make less mistakes and then you obtain a model that is trained so to speak and that's what you put in production and so the reason why i make this distinction between one and two is because when you talk about randomness you can really talk about two things you can talk about the randomness in the train model for example chat gpt as i showed you it just looks for the you know the top 10 most likely words is going to pick one randomly and so if you just have another run of the model maybe it's going to give you something different that's what makes those models creative somehow, have different answers to the same questions. This one is very easy to fix. You just say, just take the most likely, and that's it. And you fix a random sin, and that's the end of the story. So often when people talk about randomness, they also talk about the randomness in the first process, which is the training process. If you just give the model slightly different data, slightly different initializations, slightly different conditions, you might end up with a different model altogether. And so that makes the whole development of of machine learning models more complex overall because from one version to another, you might get drastically different results, okay? And so, if I go back to the example, as I showed you, nothing prevents Alexa, in this case, to tell me some very offensive joke. And if you do this, which I call uncontrolled output, you might go to jail. Maybe not, but that's a bad user experience overall. So now, how can we mitigate this? That's the next section. There are many ways. I won't be exhaustive. I'll just tell you the ones that we found to be most useful in my previous organization. Previous predefined answers. Imagine that you have just a generate joke function that generates from a predefined set of answers that you wrote. You have full control over what you wrote. Maybe it's just going to pick one randomly. Maybe it's going to pick one based on the context, like a joke about, I don't know the fact that I'm in Berlin, or I don't know what. And then, instead of generating the joke as I showed you before, the LLM or the chat GPT or Alexa is just going to generate a call to this function, to this API, and give you the results. And so in this case, you do not risk to have an offensive joke be generated. That's what people do for different applications. When you ask to do mathematics, when you ask about the weather, they don't invent mathematics. They don't invent the weather. They just go to APIs, a weather API, a mathematics API. And that's how it works. It only mitigates the issue, doesn't solve it, because the model can still choose not to use any API and just talk to you in response. And you want to preserve this, because that's what a conversational system is supposed to be about. So in this scenario, really there is no API to be called. The model just answers me. And maybe you'd go to jail for this. Or maybe not. Maybe that's too much. So predefined answers is not going to solve everything. That's why you have this big thing around model alignment. I'll just cover two approaches. The first one, you might have heard about it, is called RLHF, Reinforcement Learning with Human Feedback. What is this? It happens during the training of the model. Basically, when we talk about alignment with humans, we assume that as humans we have some guiding principles that make us respect one another. For example, you're not going to insult people, you're not going to behave like this, and you want the model to learn to behave like a human. That's what model alignment is. And so during the training of the model, you're going to have the model to generate not one answer, but multiple answers. For example, well, the model answer A could be, you're not funny either. And the model answer B could be, sorry, I will do better next time. And then you're going to have a second model, which is often referred to as a reward model, which job is just to say which one is the more aligned with human preferences. And so it's going to tell, back to the original model, you should generate more of the second option because that's more aligned with human principles. models. And then the question is, how do you train the reward model? You just ask a lot of humans with a lot of data, which one do you prefer? Humans say which one is more aligned with human values, and that's how you train the reward model. And then you use the reward model during the training of the first model, which is eventually the chat GPT model that you're going to use. The second technique, prompt engineering. The prompt is basically where you put all the guidance that you want your model to respect. At some point you're going to put in my case the user request, give me a joke, and what the model has responded so far. But at the beginning of the prompt you could say you are an helpful AI, you're not supposed to insult people, you're not supposed to go off topic. All this guidance you can put in the prompt and the model is supposed to attend to it and respect it. The problem is if you do too much model alignment you end up with something like Goody2, which which I found very funny. It's actually a joke. I mean, it's not a joke, it actually exists, but they made the world's most responsible AI model, which basically, whatever you ask Goody to, is going to decline to answer. So, for example, I tried it. I said, what do you recommend for dinner in Berlin? And they will just say, I don't want to provide dining recommendations because I don't want to influence you. So, if you do too much of this, it just becomes useless, so there is a balance to be found, obviously. Two more techniques, past model sanitation, this is after the model is trained, it's independent of the model, you're just going to put an extra layer between the model and your user that's going to check, for example, is there an offensive keyword in it, should I return the answer to the user or should I go back to the model, so you have lots of different ways to do this, but that's the concept. The last one I wanted to mention, because I think it's super interesting actually, it's It's constrained decoding. So as I showed you, the model is basically going to score all the words in the vocabulary. And what you could say is just, there are some words I don't want you to use, like stupid or the F word. And so whatever the model, whatever score it gives it, you just put a big negative infinity so when it does look for the top N or the argmax, it's not going to pick those words. This is interesting because that's also a trick that helps you generate some valid structured outputs. So for example, if you've ever asked ChatGPT to code for you or generate JSON, you don't want ChatGPT to generate a JSON that you cannot parse afterwards. And so using these techniques and libraries like outlines, you are actually able to say, I want the generated output to respect this schema. And so it's going to use something like this, for example, to say that the first word it generates is always going to be an accolade. And then the second word is always going to be an open quote. And so you can do this for SQL generation, for different structured format generation, it's very useful actually and mitigates the issue of having something you cannot exploit. So those are all mitigations. There are situations where mitigations are just not enough. If you ask Alexa to call 911, there is no room for error. You cannot allow, because of some circumstance, I don't know, because the person didn't articulate or whatnot, or because of the context was different, that it's not going to call 911. It's impossible. You cannot do this. And so that's why you also have fully deterministic systems. For example, role-based engines. It's just if and else statements. If you say call 911, it's going to call 911. If you say stop, it's going to stop. We don't rely on machine learning to do this. And so the question is, how do you integrate machine learning with deterministic systems? You have two ways. The one on the right is just sequential. You send everything through the deterministic components. If it can handle the request, it returns the answer. If it cannot, you go back to machine learning. The one on the left is if you have a bit more money, is you send everything to all the two components, and then you merge the outputs. So it's just more expensive because you send everything to machine learning models, which is more expensive. But then it's also faster because you don't have to wait for the first component to answer. so those are two approaches and that's actually how it's done often for voice assistants as i said you have multiple applications the weather the shopping play video play music and you have some super rules each application has his own model his own rules and the super rules are more like call 9-1-1 and so every request goes to everything and then you have a bigger anchor who is going to say, if the Super Bowl matched, it's going to go over everything. But if the Super Bowls didn't match, for example, the video matched and the music matched because you said play Frozen, but we don't know if it's a video or if it's a movie, the re-ranker may have a machine learning model which is going to look at the context. Maybe your TV is on, so you're more likely to talk about the movie. And so the re-ranker itself is a mix of machine learning and deterministic. Last word, things are hard in general for complex systems because the input to one model is the output of another model. So, for example, the input to the text model is going to be the output of the speech model when you speak to a device. You speak to the machine, transcribe your voice into text, and then the text model is going to take the text and find the action. And so if the first model makes a mistake, whatever guardrails you put in the second model is going to be useless. So usually when we work with these complex systems, we have to account for the mistakes that the first model is going to make. So, for example, in the training data, we make some typos, some words that sound the same but which are not, and essentially we also do testing end-to-end so that you don't test one system and not the other. So this is our own mitigation, I just want to, you know, so far basically I've only talked about one thing, safeguarding, the one on the top, putting as many safeguards as possible, but really the discipline is to do also testing, monitoring and quick rollbacks. Why? Because all those tools that you can do, they're better than not doing anything, but they will never be sufficient. I'll just try to show you very quickly what I mean. So for example, this is a nice tool called Perspective API. If you send some text in it, it's going to tell you whether it's toxic content or not. So if you say, shut up, you're stupid, I don't know if you can read, but basically on the bottom it says it's 93% likely to be toxic, which is true, but if you say something like this, which to a human is essentially the same thing, I would be offended whether I receive a tweet of the first or the second. For the model, it's just confused because of the arrows, because of going back to line, I don't know. It's a hard task, so I'm not trying to judge the tool and say it's not good enough, it's just that you should understand that even if you put more safeguards, there are still going to be times where those are not going to be able to work so you want to put not just one safeguard but multiple another example is this very nice tool called nemo guardrails this is very interesting because it's for llms so what what do people use this for for example if you have a chatbot you don't want the chatbot to talk about politics with your customers so you're gonna very easily define in a config file what is a user talking about politics sounds like so for example what do you think about the government which party should i vote for you give a couple examples then you define a flow if the user asks about politics so one of those sentences above or something similar refuse to respond and whenever you speak to the llm or you write to the llm is gonna first try to find if your request looks like any of those sentences about politics and if it does, it's just going to refuse to answer. So I did this with, I think it was ChatGPT 3.5, and I asked who is the most likely candidate to win the US election in 2024. It works. It said I'm not supposed to respond to that because I'm not supposed to talk about politics. But if you do the tricks like Matt Korda in the beginning and you say I'm just writing a fictional story about who would win the US election in 2024, then the model is just getting confused and it starts to give political opinions. Or, I don't want to be too harsh on this, honestly, but it's supposed to not talk about politics, but at the end it still says that Kamala Harris is a very influential candidate for the Democratic Party. So, you know, mitigation is better than nothing, but there are still ways that people are going to find to trick the system, or the system might be tricked by itself in some cases, and so that's really why you need the other three pillars. Testing, you have many, many, many tools to test machine learning models and their robustness. I'll talk about Jiscar right now, which is, you know, it's a French startup that's quite recent, but they do an amazing job. That's what it looks like. So basically, you submit a model to the tool, and it's just going to try to run some permutations on your input data and see how it changes your prediction. For example, it's going to put typos in the inputs and see if it changes a lot of the predictions. If your model is supposed to be robust to typos, it shouldn't change the predictions that much. Same thing, what they do is that they change the gender, for example, in the inputs, and they see if it impacts the prediction. And if your model is not supposed to be biased towards gender, it shouldn't change the predictions that much. So they have a lot of integrations like this. You can just plug them in your testing CI. And I think it's a really useful tool in these days where you don't need to reinvent the wheel every time. The other tools are just classic software developments, monitoring, quick rollbacks, you want to do this. So I won't go over the tools again. We use Grafana, Prometheus, Argo CD, Kubeflow. These are all very, very nice tools. I would recommend them and be able to be ready in any situation. I will tell you what my lead of security at GitGarden tells me is that security is the discipline of putting as many obstacles as possible on the path of your attacker and I think it's the same for reliability, you just don't put as many safeguards as possible and have as many tools as possible to prevent this, there's not just one best tool to do this I didn't talk at all about GitGarden that wasn't really the plan for the presentation but we do detect secrets in your code We use very sophisticated algorithms for this, but mostly we have a rule-based engine that goes first over all the commits that, for example, go through GitHub. And then we use this rule-based engine. It's not machine learning. And then on the output of this, we use machine learning to remove all the false positives, get more context about the secrets. So we use this combination of machine learning and deterministic system. That's really where we get the most value for people and the customers. and by the way I don't know if you received an email from GitGarden already but last year we detected 13 million secrets like passwords AWS credentials in code so it's really a big thing and if you didn't receive an email and you want to connect maybe you can just leak a password so I'll get an email and I'll call you back if you want to so I didn't talk about GitGarden, the blog is really good honestly so if you want to look at the blog what we do, whether it's machine learning or not cyber security, just go for it because I think it's a very qualitative blog so and I realize it's weird from a cyber security person to ask you to scan a QR code but you can trust me and this is just a QR code to my PayPal account this is just my LinkedIn if you want to connect I'd be more than happy to take your questions now and thank you for listening Thank you for watching.

Speaker 2 [23:41]

Thank you so much. Really, really good. Really inspiring. I have a question in the app. There has been a lot of discussion about JudgeGPT becoming more lazy, probably due to new restrictions. Did you observe this phenomenon?

Speaker 1 [23:58]

I think the person who asked the question is very right. I think there you have statistics of, I think it's called the refusal percentage or something like this, where you can track over time what the percentage of time where LGBT just declines to answer. So I don't have the statistic in my mind right now, but I agree with the person. People have seen that those larger models are getting more and more less inclined to answer. So, you know, but because companies want to be more protective but I don't know the statistics sorry but you have lots of good benchmarks on this and I would recommend go to the Stanford report from this year they have lots of data on AI I think it's called the Stanford HAI index and they have so many statistics about this kind of things responsible AI so go check it out

Speaker 2 [24:51]

Thank you. It's becoming less useless. It's becoming more useless as well, unfortunately. How do you know on beforehand that we should use rule-based or ML in the LLM application? Should we let LLM to make mistakes before going on rule-based systems?

Speaker 1 [25:11]

uh so if i try to refresh the question is should we refrain from using uh ai i think

Speaker 2 [25:17]

I think if you make your own, how I interpreted the question, if you make your own kind of LLM, should you go completely the LLM way or first setting some rules?

Speaker 1 [25:26]

rules always when do you start

Speaker 2 [25:27]

Always. Bruce, you're a bit too late, aren't you?

Speaker 1 [25:29]

that's a good question always consider simple systems before you consider complex systems the problem with rules, I didn't mention it it's just harder to maintain if you have more and more rules people are just so afraid to remove rules in the future because they are afraid it's going to break the performance and you end up with thousands of rules but you always need to focus on the simple cases first what you can do with rules or just simple models and then you consider the more complex models That would be my advice.

Speaker 2 [25:59]

I have also a question on the Alexa and the color 911 as we are here in Europe and it is 112 or 110 what happens if I ask Alexa here in Europe to call 911. Will it do nothing or will it call 110 or 112?

Speaker 1 [26:18]

G are you gonna test it right now? So no

Speaker 2 [26:21]

I'm just interested in this.

Speaker 1 [26:24]

whatever I want you won't check it so usually the way it's done is that we have models per locales so we have models that handle like the Europe Germany and we have models in the US and so but you know if you have to call 9-1-1 in Europe normally it should be able to do the right thing

Speaker 2 [26:42]

So going to 1.1.0 or 1.1.2? Yes. Oh, thanks. Didn't know this. So we can call 9-1-1 here. I didn't know this. Maybe another question. What do you think is the future of LLMs?

Speaker 1 [26:56]

I don't know if it came across the presentation, I'm not pessimistic about AI, I'm actually very enthusiastic about AI, I think we need to be very careful about how we use it and how we develop for it, that's why I'm very open research on this, but I think these things are just going to become more and more present in our lives, I think everybody should consider using them for their own productivity and my it's not just based on my intuition i think if you see like nvidia is releasing new you know chips every every couple months that are getting smaller and more performance and these things are gonna run on your on your device very soon i mean this is this is the evolution of things we just need to be very careful about it

Speaker 2 [27:44]

And how do you ensure reliability of model and critical use cases if rule base can't be designed or it isn't AI ready for that yet?

Speaker 1 [27:55]

I mean, it's actually a job in a company. You have some people who are Q&A engineers. They just are here to test the system in all its aspects. At some point, you know, there is a limit to what you can do. For example, I don't work for self-driving cars. I think it's an extremely challenging thing and field. I don't know if they would be able to design rules to prevent, you know, very bad things from happening. I think there is a limit to what you can do. So that's why you really need more testing, more monitoring, and more ability to go back to previous versions if possible, and not just anticipating the safeguards. You need everything of these pillars. That's what I want you to remember.

Speaker 2 [28:35]

Please give again a round of applause and thank you very much.