Better safe than sorry: Threat Modeling for Python Developers

Speaker 1 [00:05]

Thank you, Thomas. Yeah, hi everybody. Very cool to have such a nice audience in this fantastic venue. Thanks a lot for coming and also hello at home. Yeah, perfect. Let's start right through. I was already introduced, so very shortly. My name is Clemens. I work with InnoVex as a software security engineer. That means I help our development teams to build and operate their software and their systems in a secure way. I'm normally living and working in Munich. And InnoVEX, you might know, we are quite present at this conference. We are an IT project center, eight different offices in Germany, and we do a very broad range of different projects, from development of mobile apps, embedded, front-end, back-end, also a lot of machine learning, data engineering, and, of course, Gen AI a lot in the last years. And that also means I have a very broad variety of teams I'm working with, technologies I'm working with from the security perspective. So that's everything from the marketing side back to the projects I'm working with. I said very broad range, but our general approach to security at InnoVEX is always quite similar. We think that security is something you need to do during the entire development process. It's not something you can do in the very end, some security tests, run them, perfect, everything is secure, but we think it's something you need to do really for the whole process. No matter if you develop a web application or some data endpoints to scrape data from or maybe a chatbot, a rack, whatever. So in all projects, we think that security has to be a continuous process that needs to be done in all steps. And when I'm working with developers, sometimes the question arises, and it is brought to me, is my software secure? We have built something, is it secure? And that's a very good question because it shows that you care about the security and the quality of your project and of your software. But it's not so easy to answer. So we see this in the media a lot of data being breached and leaked. We see web apps or mobile apps that were hacked, that were leaking data, that were allowing to access services you're not paying for. You maybe know this Tide chatbot from Microsoft, another system that got exploited in a way it was not intended to, or the breach of DNA data of 23andMe, where DNA data, so very sensible data of users, were leaked. And all these news, all these things are, of course, only the tip of the iceberg. That's the part that got public. That's the part we know about. But we can be sure there's a lot more underneath incidents that are not known, maybe not even known to the people who run the systems, actually. And these hidden problems, data leaks, other vulnerabilities in your software, they pose a lot of risks to you, to your system, to your organization, because they might lead to negative press, they might lead to fines, or they might lead to your business model being exploited because users can access your software without paying for it. or your whole data is used for different software. So all these security incidents happen and sometimes people tend to be, oh, it's hard to cover for all that. It's too much security. It is too hard. What should I do? And therefore, I think the question, is my software secure, is not exactly the right question. The question should be, is my software secure enough? So there are threats that you need to think about, that you need to cover against, that you have to take countermeasures, but they need to be in line with your software, with your business model, and with the stuff you are actually running. And then you are able to do security in a way that it actually helps you to mitigate the threats that are relevant for you. Because we saw in the talk before, and also if you're a bit into security in general, there are a lot of different security measures, and there are a lot of different things we can do, integrating tools, libraries, doing second-factor authentication, data validation, data sanitization, A lot of different security measures, but sometimes it feels like it's a big pile of tools and activities and it's hard to find the right things for you. And that's where threat modeling comes to place, and that's the method I want to bring to you today as an approach to systematically identify what threats are you actually facing and how to secure against these threats in a targeted way. So, thread modeling is actually nothing new, not at all, it's a very established method originating in systems engineering, but it's applicable to every kind of software that you build. If it's a web application with a front and a back end, if it's some ETL pipelines, if If it's a rack, whatever, everything, every system we are building can be investigated with this method, everything that has some value for you that you want to protect. Threat modeling is a structured procedure. We will see this in a minute, how this structures. And this also helps you as a developer to get some kind of peace of mind. So in this way, you do not need to make some gut-based decisions on what security activities to do and what not, but you can really have some assurance that you thought about the different parts that are relevant for you in a structured way. Also in some domains, it's a legal requirement to do threat modeling. If it would, then you would know what it is, then maybe you would not be here, but this might also change in the future. Threat modelling is something that is seen as a very powerful process in different tools, so this might also affect you in the future if you work in regulatory environments. But last but not least, threat modelling is a preventive measure, and that's something that is really, really important. We are thinking about security beforehand, not in the end when everything is built and we try to test if it's secure or not. But we try to be more proactive to do the right things beforehand where it is much easier to do so and also where we can maximize the security budget we have in time and also money maybe to get the best results in means of security for our whole software system. But that's enough for motivation. I think thread modeling is a very powerful tool and you should definitely know it. Let's have a look what thread modeling means. Thread modeling is, first of all, just a term for the systematic approach. There are different ways to do thread modeling. One approach that I really like is the for-question framework from Adam Shostak that is a framework for structuring the threat modeling process into four simple questions. And we will have a look at them now. So the first question is, what are we working on? So that's the first thing that you should think about when you are doing a threat modeling. And so that means get a common understanding. What is the system doing that you are now threat modeling? What are you operating? What parts of the software does it have? And then we want to break it down into these components, see how they interact with each other and what external dependencies we have that we have to look into. And a very good method for that is a data flow diagram. That's a very simple way to visualize the data that is inside of a software system because that's often where security problems arise if data is accessed, processed, stored somewhere. So that's a very good way to see which parts are actually communicating with what data. So this is a very simple example here of a simple web shop. We have different parts of the web shop interacting with each other. Data is flowing between them. We have some external dependencies, like payment APIs and stuff, which are not part of our system, but we somehow communicate with. And also what's very important is that we think about trust boundaries. That is this dotted line I added here. So some logical structures of things that are somehow meant to be in a trusted zone together where we have a bigger trust in the communication between them, for example, and everything that is outside, for example, the frontend that is running the browser, stand outside of this trust boundary. And these interfaces that are between the different boundaries, they are very interesting to see what happens there because that's often where security problems arise. So that's the first step. thinking about what are we actually building and using some visualization. Dataflow diagrams are a good way to do so. You can also start with an architectural diagram you might already have. Maybe that's a good time to see if it's still valid and start with that. That's also fine. You just want something that breaks your system down into different components. And then we think about the actual threads. So what can go wrong in our system? And we have these different parts now, and we want to think for every part what can happen here, what threats can arise. And at this point, it's very important to look into security threats. So some people tend to talk about general quality flaws in this place. So that's also a good idea, and you can do this in a similar way, but when we're talking about security, we're always talking about human attackers, someone with a malicious intent that is attacking our system, not some hardware error in our data center or some spilled coffee or something that's not in scope for threat modeling, but we're thinking about threats of a human attacker. And not all attackers are the same, and it's a very good idea to think about this first. So what attacker do I really want to face or what attacker is relevant for my application? So often people tend to think about some federal intelligence agencies like the NSA and we have to secure against them and it's too hard or some Russian hackers or whatever. But there might be some systems where this is a relevant attacker for you, and you might consider this. For many systems, there are other attackers that are maybe more relevant to look into. Common criminals, for example. There are people who want to make money out of something. Or what we call script kiddies. Some people who just learned about a vulnerability SQL injection. Let's try this out in the next application I get. Or also researchers, ethical hackers who are more into research motivations or personal fame who want to attack your application. And then if you know your attackers, you think about what threats could these attackers do to my system. And that's where you can take your data flow diagram and you look at every component. And one method I like a lot is stride. It's a framework for breaking down the threads into six different classes of threads. So each letter of stride stands for one class of thread. So we have spoofing, where the attacker tries to spoof an identity, claims he's someone else in some way. We have tampering, where data is tampered in a way that, yeah, it is different than it was before. We have repudiation, difficult word for me. So, yeah, that the attacker might do something and later claims, no, I did not do this, or something that is relevant there. Information disclosure, I think, is self-explanatory. Some data is leaked to somewhere. where denial of service are attacks where the availability of your system is down, your web shop is not selling any more things, threats that come from that. And E for elevation of privilege, so the attacker is able to gain more rights than he used to do things that he's normally not allowed to do. And you can take these six classes and go to your data flow diagram, And for each component, you can have a look and think about, okay, what can happen here to these different classes? So how can an attacker spoof an identity in my frontend? How can he claim he's a user he is not? Or how can the data in my ETL pipeline can be tampered with? So where are the different paths to do so? And then that's the most interesting part and also the most difficult part, thinking like an attacker and thinking what the attacker might do to reach some goals with threats of this kind. And then the third step after we have these threats, we think about, okay, we have these threats now, how can we mitigate them? What are measures against them? And that's where you can come back to your pile of tools, but now you know exactly what threats you are facing, and you can think about, okay, for this particular threat, do I have something that mitigates this threat? And for every threat, there is security control that helps you to mitigate it. So if you think, oh, the attacker might tamper with some data sending over the Internet here, then one solution would be to use TLS on this connection, then it's integrity-controlled, and by that you can go through the different threads and find matching security controls there. And often in these sessions you think about, hmm, I think we have something in place against this thread, let me check this later, and that's perfectly fine. So then you know, okay, I will check this later, write it down. That's the most common thing you discover in threat modeling sessions. And the last step of threat modeling is then did we do a good job? So looking back in the spirit of inspect and adapt, ask yourself was this session helpful? Did we improve the security of our system in some way? Did we gain some new insights? What can we do different when we do thread modeling the next time for our system? All right. We have some minutes left. So I want to share with you seven common pitfalls I see a lot when doing thread modeling and especially when teams start thread modeling. First common mistake is thread modeling. That's a type I see a lot. And if you do not want the people to come and think you are talking about knitting in this workshop, then make sure to write it correct. Okay. Jokes aside, first real mistake I often see in threat modeling is digging too deep. So people go very deep in one particular threat and go into one special case for hours. Don't do this in the beginning. Have a look at the most interesting things first. Have a look at different parts, but don't dig into one particular threat too deep. And also, dare to set things out of scope. For example, some physical attacks on the Azure data center might not be your most relevant problem in the beginning. So, focus on your parts and don't dig too deep. A third mistake I often see is trying to use sophisticated tools. So, there are great tools that might help you with threat modeling. But for the beginning, just start with a whiteboard or the remote equivalent of that and just draw your data diagram there. Put the stuff you find in the document or in your ticket tool you already use. If you later want to scale up your thread modeling, you might want to have a look at tools. But for the beginning, I would not recommend that. Keep it as simple and accessible as possible. Fourth, forgetting about human targets. So I already said in security we think about human attackers, but don't forget to also think about human victims. So as developers, we often tend to see the technical things, but humans might also be the victims, so social engineering, phishing might also be a thing you should consider in the threads when you think about how an attacker can target you. Okay, fifth mistake is think too early about mitigations. So in the thread modeling, it's really important that you start with the thread view on your perspective. Don't start too early with the countermeasures and the different tools you can use for that, but really focus on what can happen to my system and then think about what would be a tailored response to these threats. So it's also perfectly fine if you do not find a technical measure against something. You can also do it in another way with some processes or it might also be that you say, okay, this is a threat, I do not have a technical solution, it's just not feasible, we accept this risk. That's also something that's fine. But it's really important that you have a look at the threat perspective first. Next mistake I often see is that you work too isolated. So threat modeling should be a team activity. It's not something you should do alone, but bring together a team. Also include your PO, maybe some non-technical stakeholders to bring in their perspective. It's also a very good possibility to share knowledge across the team, especially in the first part. What are we actually building? That's also very interesting, but also to get a common understanding about the security activities you have. That's a perfect thing there. and also think about the teams next door or the external things you have. Maybe integrate them, validate the assumptions about who is taking care of which thread. Maybe you discover that you thought they are doing authentication, they thought you do it. That's something that's really important because at these interfaces, problems often occur. And the last and the most relevant mistake I often see is not doing thread modeling at all. And that's really something that I would recommend you. Try to start little, but try to start with it. Find a way to do it in your own development flow, in your work, and in a way that fits with your team. But just start with it. All right. I also have some further links for you. The QR code you can scan. The other links are in the slides I published, so don't try to click on them, please, here. I just published a blog post about threat modeling for AI applications. So if that's a domain you're interested in, then you might get some more insights into the threat landscape of AI there. Also I added two games I like. If you discover, oh, I don't know what threats exactly I should consider. These are quite nice games where you can get some hints about actual attack paths that might be relevant for you. To sum it up, I hope you learned what threat modeling is and how it can help you to start with security in a systematic way, how you can analyze your attackers, what the four question framework is to structure your threat modeling, and the most important takeaway I hope is start small, start simple, but start with it. And by that, thank you very much, and I hope we have time for some questions.

Speaker 2 [23:50]

Of course, we have time for some questions because we are just in time. So, thank you for this wonderful talk. And just to start with the first question. So, how do you actually address security all along the development cycle? For example, in a scrum setting, tech debt and security quickly gets depriorized. And I know it myself, I know a colleague. So, security costs. So, how do you reflect this in development?

Speaker 1 [24:14]

That's a very good question. Yeah, it's hard to place security in any company, I would say. In agile teams, especially because you often tend to focus on features there and non-functional requirements might be forgotten there. But that's also something where threat modeling can help you because you really can name the actual threats that might happen to your application and to the assets that are in the application. So how can the business model, for example, be targeted with a threat you are currently not working on? And that might help to get some investment to fix this.

Speaker 2 [25:00]

Cool. And when we are talking about scrum and teams, how do you interact with the various teams at InnoVec? So are you a standalone security team or embedded security expert or something? So how do you live security?

Speaker 1 [25:15]

Good question. Yeah, so we think security is done best when integrated into the development teams. So I'm not coming to a team and having a look at their code and saying, that's wrong, that's wrong, that's wrong. Fix it, goodbye. But I'm in the team. I try to develop with them. So most of the time in the back end then. I am with the team the whole development process. I talk with them about tickets. We can have a chat about security issues when refining tickets as well and I also have a look at security tests so I'm really integrated in the teams there and try to work with them and not against them.

Speaker 2 [25:54]

And how often do you work together? It's just in the beginning of a project or is it on a regular basis?

Speaker 1 [26:00]

So, the ideal thing is that I'm really a fixed member of the development team, and sometimes I code more, sometimes I do more security, depending on the phase of the process. Sometimes I also come in projects in the beginning, and then maybe later, so it varies a bit from the different projects.

Speaker 2 [26:23]

And I know you get a figure about your trust boundary. And one of your listeners asked, why is your sysadmin outside of your trust boundary? Well, I understand it because I wouldn't even trust not myself. But just answer it.

Speaker 1 [26:38]

Yeah, so the idea of the trust boundary is here, but I think there's not a completely right way to do it, and a wrong way is that trust boundaries are something of some technical concept. So people, or all external entities, and people are always external entities if you're not living inside of the server rack, okay, maybe then we're back to the sysadmin, no, all external entities are not part of some technical boundaries of a system, so they are always normally outside of this trust boundary. Of course, the trust boundary or the connection between the sysadmin and the log analysis should be tighter than the attacker and the log analysis, so this is exactly one thing where you should ask about how are we doing authentication here, so So how can an attacker spoof that he's the sysadmin and do the things the sysadmin would normally do?

Speaker 2 [27:41]

So you already said on your last slide that Gen AI is getting more and more important. And we got already one question. Do you consider Gen AI modal hallucination, for example, a new form of threat? Do we need a new ladder and stride metaphor for it? So perhaps you can do some spoiler on the content you just referred to.

Speaker 1 [28:01]

So there's a lot of things with security and then AI. So the blog post is more focusing on how to develop secure AI and how to develop what threats we are facing when we develop secure, when we develop AI. I think the question is more about how AI might also be doing harm to our software because hackers might also use AI. So there was, I think just last week, this example, a study from the USA where they taught JetGPT to just use this CVE, common vulnerability enumeration, so the explanation of a vulnerability and develop a perfect exploit out of it in no time. So I think Gen AI is also helping the bad side a lot, but it can also help us to develop more secure code if done correctly. I'm a bit skeptical when looking on Stack Overflow about secure code. I'm not so confident that Gen AI might produce perfect secure code, but that's something that's very interesting. I could talk about this for ages. Great.

Speaker 2 [29:22]

because we are also ending our time

Speaker 1 [29:23]

Thank you.

Speaker 2 [29:24]

so I just want to pass this one uncommented to you my favorite talk so far thanks a lot and I think we should also applause now again for Clemens

Speaker 1 [29:32]

Thank you.

Speaker 2 [29:33]

thank you

Speaker 1 [29:33]

Thank you.