Architecture Under Constraints: Designing Systems That Still Evolve

Speaker 1 [00:00]

Hello and welcome to our next talk, Architecture under Constraints, Designing Systems That Still Evolve, by our lovely Edward Tam. Please have him a warm welcome.

Speaker 2 [00:19]

Thank you, everyone. Hi. So, as was already read, architecture under constraints, designing systems that still evolve. I'm Edward. You're going to listen to me for a bit if you want to, and I'm going to talk about architecture in regulated environments. If this is your environment, amazing. If this isn't your environment, welcome to it. It is challenging. It is sometimes interesting. And most of the times, just really, really fun to be in. If your brain is wired for it. Before I get going, I'm Edward, and I work at OrthoP Health. I'm the DevSecOps lead in that company. We create software as a medical device. What this is, is it is an app for your phone, but it is certified as a medical device. So we go through all the regulations and all the certifications that you do for medical devices. What I do there is a variability of things. I make things possible that were not so before. That's like my main idea in that company, to enable things. And when I'm outside of work, I try to stay away from screens as much as possible, just for me and my mental well-being. I go bouldering, I go hiking, I try to be outdoors as much as I can. So there are not going to be a lot of specifics in this talk about a language or a framework. That means that you can adapt the concepts that you have, whether you're running Django as a monolithic thing or you're running some form of service-oriented or independent systems or microservice or however you choose to name your distributed system thingy. If you build it in FastAPI or whatever you use. Why do I see the need for a talk like this in the first place? Well, most architecture advice, books, talks out there will start from a clean slate. They will start from, I have all the freedom. They will start from, oh, the best tool or pattern that I can pick in this situation will win the game, whatever winning the game in this sense means. And they assume that technical trade-offs dominate your design space. Now, I do understand why this is for the most part. but I feel that it is not the reality most of us especially in regulated environments face every day right what we have what we face is we don't have this freedom right our architecture doesn't start with what we want to build or how we select our patterns it starts with what we're not free to ignore I think the things were not free to ignore are things like regulation the audit ability of our systems. The legacy in software, in hardware, in processes that we face and have to work with. The organizational boundaries which shape how our systems can be developed, shaped, and solved. And earlier long-lived decisions, maybe even our own long-lived decisions from before that we might not be too happy with right now. I see some people nodding their heads. Yeah, that. Also, there's sometimes procurement timelines which might be uncomfortable and have an influence on how you can build and shape your systems. This is the actual problem space I see many of my colleagues and many senior engineering people face. I have some definitions for the purposes of this talk for the next few minutes. Please bear with them, right? I know that you might not like them and we can discuss for hours, usually very heatedly about them exactly. Slides, by the way, are on pre-talks and will be available online afterwards. Still, if you want to take pictures, go ahead. So for me, for this talk, design is how we shape a solution to meet the needs today. And architecture is the design decisions that shape what a system can become, how safely it can be changed, who must coordinate to change it, and how we assert and properties of the system. Let's go with this definition for now, accepting that reasonable people will disagree. If you read clean architecture, for example, they assert that there is no relevant difference between those two things. And if you follow those definitions, you will also see that all architecture is design, but not all architecture. Okay, sorry. I'm arguing that constraints are not noise around your architecture. They define your design space. That is, if it changes what can be built, changed, operated, or defended in audits, then it's an architecturally relevant input to your decisions. So now, if you're in a regulated environment, what are things that shape your design space. Not necessarily this exact list. This is not complete. This is not a complete list of lists. But I find it useful to think in three broad categories about these types of things, in terms of regulatory and audit, where I often think about traceability, access control, evidence, retention, in terms of legacy and operational concerns, where I think about existing systems, cost of migration, burden of support, and reliability. In terms of organizational and political boundaries, teams, team structures, contracts, geography, procurement. And when deciding architecture stuff, then walking through those categories briefly to have an understanding of, okay, what's my impact going to be in these categories is something I find helpful. There is a common mistake that I see, and that I have also made around this, and that is treating these types of constraints as outside your architecture. We're going to create an architecture, and then we're going to treat all these things as downstream implementation friction. Amazing, friction afterwards. Yeah, that's going to cause a lot of pain usually. So if architecture is not just your tech stack, not just your services, your databases, your classical definition of, okay, what is my system made of and how do these things interact? Then it is also your capability boundaries. It's also how things can and should change and evolve, your control points, your operating model, your evidence generation, and your accountability. The last two get very, very important very quickly if you're in a regulated environment. so it is not just how the system is shaped it is what the organization is able to do or not to do through and with this system with all of these theses in mind let's revisit the definition from before so a more practical definition of architecture would be the set of decisions that shape what a system can become who can change it, how safely it can change, and how well you can justify those decisions over time. I'm from the medical field. And in the medical field, safety has a very specific meaning. It means that if something is safe, it is assured to bring minimal harm to the people that use it. So ideally, you don't hurt yourself or die when you use this. this is a very pointed definition and it is not what's in the textbook so the summary of this would be a decision is architectural when it meaningfully changes future possibilities if you follow this definition you're going to see that regulation is not downstream because it shapes your data boundaries, your access model, your workflows, your change control, and your evidence path. What do I mean by change control when I talk about this? Well, change control in this context has two meanings. The first meaning being, how does the system change, and what rules do I have to follow when the system changes? This is from code is generated, typed, text exists in some way, to there is something that runs in production over here, and in between, stuff happens. That's change control on the one side. You can design for this. You can architect for this. And then there is change control within your system. Let me give you an example of this. In our applications, we have people that have injured their knee. So they do or do not have surgery, but in the beginning, they have restrictions on how much load they can put on this knee and how far they can move it. So, for example, I'm not allowed to put any weight on my knee and I'm not allowed to bend it more than 30 degrees. Over time, as my rehab progresses, this will ideally improve. I'll be allowed to put a little bit of weight on it and I might be allowed to bend it to 60 degrees. Now, this is important information for us because then we can, when you, as a patient, tell us this, give you better exercises that will progress you quicker. However, these things need to be changed by the patient. We might not just assume that this is now, because they have trained for six weeks, a thing that is happening. So we need to be able to show the patient changed this data on this date from these values to these values, and those are allowed. That is a proof path that we have to build into our system. So that's also change control. The point being here that if your business must prove something in an audit, in a court of law, wherever they must prove it, then that proof path is part of your system design. If it is not, you will have a lot of pain downstream to then get this into your system or build something around it that might or might not hold up. So, auditability is a system property. Think about your own systems. If you're in a regulated environment, you maybe have already done this exercise. Think of your own system now and tell me, for any change to your system, can you tell me who changed what exactly within this deployment? Can you tell me at time x what was running in production and which configuration it was running under? If I point at any line of code and ask you why is this line of code written in this way, can you tell me the reasoning behind that and why this decision was made? And if I ask you at time x, for patient y, was control z in place and effective? Can you tell me yes or no for that? So for patient x, yesterday at 12 o'clock, was the control for going from treatment parameter set 1 to treatment parameter set 2 in effect? And did it do what it is supposed to do? Those are the types of questions you will eventually get asked in highly regulated environments and that you will have to build into how you build your systems. Now I've talked a lot about what your system should be able to do, but let me give you an example of how this specific set of requirements could be solved. And there are many ways to do this, but this one is one that you could pick. assume that you have a product and you want to deploy this product in two regulatory scopes. For example, let's say within Germany and within France. Random examples, right? You will have different regulation in Germany than in France. And if you have certified products, that is, if you change something you need to go through a lengthy progress process to then get this product to something that is okay to put into the market as per your regulator, then putting this into to two separate modules, a module for Germany and a module for France, might be beneficial because you might be able to go to the French regulator and say, oh, yes, we changed code. Yes, there is a new artifact that we got put into production. But look here. None of the code paths that are relevant to France have been touched. From the perspective of your regulation, the product has not changed. Same for Germany. Now, add another dimension. Go into, you have a software artifact that contains more than one business product. So if you apply the same type of split, you split it into modules, and I'm saying modules here. You can substitute things like service or microservice or whatever your phrasing of the day is today. you split this into modules and you have a module for product A and a module for product B then if you change something on product A you know from the code bath you can show on the diff nothing in product B changed now some of you will now be very skeptical with me because but then I have to copy all this code around and duplication and try and I don't want this um yeah that is a price you pay and you can have something like a common core let's call it this for now but be aware that if you have something like that then everything that's in there if you change it and yes there's an asterisk there you can get around this for a little tiny bit but everything time you change this everything you change in there will trigger recertification for everything that uses this. What this means is you need to be very, very careful what you put into this. Basically, you should treat this like an operating system. It's going to have to be very stable, basically calcified, and that means you're going to have to provide escape hatches for everyone using it so that you balance two concerns. You balance the concern of having an interface that is useful, that is not configurable in such a way that the abstraction is basically useless, but also allows the products to adjust what is happening to an extent that makes it useful for them. That is difficult to do. Welcome to your job. It's also incredibly rewarding and fun if you get it right, and sometimes you do. This brings me to the following point. Not every constraint on your system is equally real. There are some true constraints that you have to deal with. Regulation is a constraint. Contracts are a constraint. Actual technical limits are a constraint. The speed of light and vacuum is not going to get any faster anytime soon. Validated legacy and vendor dependencies. What I mean by validated is that you have actually checked if you still depend on it and if you have to. not just that it's uncomfortable or that you really don't want to do this now because, ah, no, you've checked and it's truly not feasible for you and you understand why it is not feasible for you at this point in time those are real constraints if you look at self-imposed constraints you will often find that they are based either in fear or in habit or you just copied this pattern from left, right the team next to you, the person before you, whatever it may be. Or someone imagined that the auditor will eventually ask for this, so let's just do it right now. I've had this discussion multiple times in my career where I found a very complex system that people told me, but we're going to need this for audit. And then I actually talked to compliance people and they said, no, no, no, we just need you to sign this paper that everything's fine. That's a good win. Let me give you another example here, because this is a point that is, in my opinion, very important to distinguish between. I have come across a system in my career where the true constraint was data at rest must be encrypted and the key lifecycle must be documented. The implementation was that for every microservice, there was a hardware security module that this microservice talked to. Separate one. Individually hosted. Okay. For those that have no context around this, that's very hard, very difficult, very complex, and also very expensive to do, when all you actually need is a thing where you say, okay, encrypt this address. Usually this can be done. And you need a piece of paper that says, oh, this is when and how we generate keys. This is when we rotate them. This is what we do with them once they're rotated. And this is when and how we delete them. So many expensive decisions are answers for constraints that nobody validated. We assume that is not what an architecture decision is. It's not what leads you to good decisions. And yes, sometimes you have to make assumptions. I would urge you that now where I've shed light on all the things that I find, oh, we should act on that in some way, to go with me into the space of, okay, what could we do to address some of this in some way? And my assumption would be, okay, let's start when talking about architecture to talk about capabilities and stop talking about components first. So ask, what must we be able to guarantee? What must we be able to prove? And what must we be able to change safely? Change safely in this context? Correctness, reliability, compliance, security. Depending on which regulated context you're in, this list will be different for you. Examples of capabilities. Controlled change, as before. Traceability. That's the question of what was where when. Secure identity boundaries. Policy enforcement. Recoverability of X. Controlled integration with APIs, vendors, external dependencies, whatever. Those are capabilities of your system. Which of those must you have? Which of those must you prove what for? Lists can be extended, adjusted, depending on your context, as always. There are some traps in this whole capabilities game that I have fallen into myself, that I see other people fall into. And I just want to point out three that I have found to be fairly common. this is one that's fairly classical in in my area compliance theater right let's add another approval let's fragment this process about across three departments to ensure that there is separation of concern and there is no conflict of interest for this let's add a bit more progress process here this will make it better oh and maybe we could do this slightly more complex so that it's harder to explain so that we can spend more time talking about this that would be good. This does not necessarily make for more or better controls. I would like to remind you at this point that complexity is the enemy of security. And I would encourage you to make things only as complex as they need to be to solve the problem you've actually found to have. Especially when talking to people that are well-versed in compliance, I found that they often don't need ceremony. What they want is evidence, a clear and simple path to show what is written here is actually also done. So control, traceability, and proof of actual existence. What I also found is that often people tend to push things into a platform. If you don't like the term platform, you can substitute infrastructure. And remember that infrastructure is more than just servers. And I often found that this then generates something that I would like to call codified centralized confusion. So this usually manifests in having abstractions that no one asked for, potentially only on one example. Having standardization without thinking of this standard piece of thing that comes out at the other end as a product that people will actually have to use. And automation of unclear processes, because let's automate all the things right now. If you have a process that is not mature, if you have a process that is still changing a lot, automating this often generates confusion. Point being, a platform is a product. If you don't have a clear idea who's going to use it, use it right now, or also potentially use it in the future. If you don't have an idea of what it promises to do, and if you don't have an idea of who's going to maintain and operate this, then it's very likely going to be centralized confusion. And I would urge you to think very hard before creating this. The third thing I've seen a lot is microservices as an escape hatch. I'm not arguing against microservices here. I'm arguing about the reflex of, oh, we have any problem over here, let's take microservices, this will surely solve it. What microservices do is they solve some problems. Yes, absolutely. They solve some real issues people are facing. However, they do this at a cost. Increased audit surface. All of this has to go through the audit. Operational burden. Evidence fragmentation. Now you have your proof chains distributed over five different services that talk through three different protocols on 10 different cloud vendors. And you have interfaces between all of this, so you need to govern those in some way, shape, or form. Distributed systems do not dissolve constraints. They multiply to places where you need to manage those constraints. Once you leave the comfort of your own process, things get very messy very quickly. So with those traps in mind, what can be a guiding principle that might help us avoid some of those sometimes? I would argue that the deliberate preservation of options is a guiding principle that we can apply to avoid some of the traps some of the time. Not every option, the right options. the bottom text down there that's hard to read is those ones that are expensive to recover later that's for those following the slides on their phone what the right options are that's difficult to tell in advance sometimes impossible to tell in advance this is where judgment comes in this is where experience comes in and there is no compression algorithm for experience let me give you an example of a trade-off we made that was fairly expensive for us and we don't yet know if it paid off we have an off-the-shelf solution for identity and access management and when we selected this we had to select how to integrate this with all our other systems the first option that is fairly obvious is right you plug it into everything directly there are libraries and there are integration points and this is all well defined and there's a path for this and you can just do that. And the other option is you build an abstraction in between this. You build some layer that does something. This comes with a lot of cost. There's burden of proof that this works. This is a highly sensitive field. It's identity. So you need to be very, very sure that it works. There are things like, okay, and now I have to teach this to everyone that works with this in some way, shape, or form, right? It's not standardized, documented somewhere. You have to operate this, right? You have to respond to this. You have to monitor this. The whole slew of things that comes with having this abstraction layer is extremely costly. We still paid that cost. Why? Because what we saw on the horizon was that we need to integrate with some, let's call them, non-standard identity providers down the line. We also saw that there might be some things that this off-the-shelf solution does in a way that doesn't really fit to our specific security requirements. So we need a way to change and adapt this. We also knew that the underlying tech stack of this off-the-shelf solution was not something we were extremely comfortable in at the time. And all of these things prompted us to make the decision to add an abstraction layer in between. We also documented all those things and said, if we find out that this integration with other parties doesn't turn out to be as complicated as we assumed, or that we don't have any problem with those paths that we assume to be problematic, we should look at this decision again and maybe reverse it. Yes, we're going to throw away some time, energy, and effectively money, But let's revisit this decision when those triggers happen. What's usually worth preserving are things like bounded change, stable edges, replaceability of dependencies, and reversibility of decisions. Structural change, structural options, those preserve your technical freedoms, right? Interfaces, change boundaries, modules, however you want to call this within your systems. For dependencies, think about dependencies not only as the thing that is in your requirements file. Think of this as well in terms of where your stuff lives. Vendor dependencies are something that you have. The term vendor login gets thrown around. Yes, this is a thing that can bind you to stuff. Yes, this is a price you can sometimes pay, sometimes want to pay. That's okay. Just make this decision intentionally, is what I'm trying to say. And explicit assumptions and trade-offs will preserve your freedom to revisit decisions in the future. We're going to go to this one slide now, and if you want to take a picture of one slide and one slide only, that's the slide. Again, the slides are on the Internet. A simple decision framework that I have found helpful when thinking through these types of decisions for me is, okay, I have a decision, I have a proposal on what I would want to do. What real constraint does this address? What capabilities of my system do I create or protect by taking this decision? What options for my system in the future do I preserve, destroy, or create with this? What burdens do I introduce on my team, on my systems, with these types of decisions or with this concrete decision? How will I prove that this thing works and is controlled as per the regulation I am bound to uphold? And when do I look at this again? To either learn that, ooh, I made a right call, great, let's do more of those. Or, ah, the world changed around me, this isn't useful anymore, let's get rid of it. A good framework does not hide trade-offs. It makes them visible. If you have a decision framework that you're already using, I encourage you to think through it and think if it does this for you. with any decision you make if you're in a highly regulated context i'm going to argue that it is not done until it survives three conversations the conversation you have with engineering where you talk about boundaries and dependencies and failure modes and operability a lot of us will be very used to these types of discussions the discussions with leadership where you talk about risk and cost and reversibility of decisions and speed of implementation. Maybe not only implementation, but just generally how fast. End the discussion with auditors. What's the control intent of this? What's the traceability of this? What's the consistency? What's the evidence? You're going to need documentation that survives scrutiny. Specifically, these three types of scrutiny. So a useful decision record, for me, contains at least context, the real constraints that were addressed, the options that you considered, the trade-offs you selected, the evidence paths that are inherent to this, and the triggers for review. To sum this up, if you have these things, it is likely that your decision can survive for as long as it is useful for it to survive. And that's what I suggest you aim for. A good architecture decision record, ADR for short, is not a monument. It's a lot more like a receipt. It documents why you've made a choice, what you traded in, and when you look at it again. We're going to go back to definitions just one more time. With all of that knowledge that I've now tried to put out there, good architecture under constraints for me means treating constraints as a design input, distinguishing real from assumed constraints, designing for capabilities, not just system shapes, preserving the ideally right options, whatever those are, and making the trade-offs you made explainable to people. To put this into one sentence, architecture under constraints is the disciplined preservation of future options while satisfying today's real obligations and with this i would like to thank you for your time thank you for attending for listening to me if you have any questions please put them into the app and if you have questions that you're uncomfortable with putting into the app i'll be available after the talk outside this room to discuss

Speaker 1 [33:22]

Edward, thank you for your talk. We already have a few questions. The first one is, you mentioned revising architectural decisions if the need arises. How are you keeping track of these review triggers?

Speaker 2 [33:40]

That's a very good and also one of the prepared questions that I had. How this is done effectively for me is that we basically have a set of architecture decision records which we keep in our knowledge base tool, and we have a standardized section in there. We cite these sections into another page, and we go through this page regularly. Yes, this is manual. Yes, this doesn't scale to large organizations. but how I would evolve this is that I would then categorize this and go through the categories on a schedule. However, what specifically works in your organization will depend on the habits of your organization and you might find that your organization is terrible at looking at old stuff. How to solve for this is, I would argue, an open problem. Sorry.

Speaker 1 [34:40]

What tools do you use to track your ADR?

Speaker 2 [34:45]

The one that my company uses for all other knowledge management No, it's conference someone whispered Jira for those that can't hear it on the live stream

Speaker 1 [34:58]

Wouldn't it be possible to have a microservice audited independently and integrate this into multiple systems? Wouldn't that globally reduce the complexity of auditing instead of increasing it?

Speaker 2 [35:14]

Yes, this can be a useful approach. It is a useful approach given a certain set of circumstances, specifically if you can encapsulate one compliance concern fully within the boundaries of this service.

Speaker 1 [35:32]

How do you deal with regulatory requirements in the development environment? Are you able to separate production from development to have some tech freedom of DevTools?

Speaker 2 [35:45]

Okay, I'm going to try to interpret this question in two different ways, because the way it was read to me, I hear two different concerns. One is development environments where I deploy stuff for integration with other developers, and the other is my environment where I want my tooling to be as free as possible so that I can experiment and have choice there. for the environment that I deploy to as dev environment depending on your regulation yes you can write your internal procedures in such a way that you have a lot of freedom there and put your quality and compliance gates basically on the towards production side of this environment and I would strongly advise that you do this if you value any sort of but can I quickly deploy this to check how it works type of workflow? Which I would assume most of us do. And for the freedom in tooling and your own personal, be it truly local, like on your own metal, or be it in some form of cloud development environment, again, this is going to highly depend on the reputation you're under. but one way to solve this is to separate the part where you do the real development for the regulated product and where you play around with how you could do development for the regulated product maybe eventually. So you could, for example, have two text editors on your machine, where one is configured exactly as the regulated environment requires, and the other is slightly more freeform. I'm not saying that you should, or this is a solution for you, I'm saying that this might be a possibility to do things like this.

Speaker 1 [37:46]

How to deal with evidence fragmentation with microservices? Proper observability tools are enough or there are more requirements?

Speaker 2 [37:59]

that very much depends on how specifically your evidence needs to be presented. I'm sorry there is no generalized answer for this that I have. If you have a concrete use case in mind, do come talk to me afterwards and we can discuss this. I'm very interested in this problem space. What I can generally say is that in my experience, just regular observability tooling that's out there will most of the time not cut it.

Speaker 1 [38:31]

We have just two more questions, but about 10 minutes. So if you have any more questions, just put them into talks.pycon.de. The next question is, where does a boundary between future-proofing and over-engineering lie when designing an extensible system?

Speaker 2 [38:51]

Might I recommend a complete architecture library books? For me personally, especially when I think in regulated contexts, what I find to be useful is, is this something that I know to a certain certainty that I'm going to need? The example I talked about with the identity and access management system, we were like 99.99% sure that we're going to have to integrate with national identity infrastructures. So that was a future-proofing that we decided we wanted to do right now. If we would have been less certain, this would have probably played out differently. In other words, I would like to rephrase the question as, how do I know if my scrying ball is any good? I have no answer to this, I'm sorry. At least no general one.

Speaker 1 [39:57]

Am I allowed to modify code where nobody in my company knows who changed what or why it is this way?

Speaker 2 [40:09]

that depends very much on your product and your company. And if you have to ask this in this talk, the answer is very likely no. Do come talk to me afterwards about specifics, and if I can help, I will, but the very good chance is that I won't, because someone in your company will have to tell you that, and I don't know who.

Speaker 1 [40:35]

Do you see fitness functions based on quality or non-functional requirements or constraints as a viable means for keeping track of the fit of architecture and problem space? Yes.

Speaker 2 [40:49]

I guess you want me to elaborate on this, but yes. So for those that are new to architecture fitness functions, think of them as, quote-unquote, unit tests for system properties in the architecture space. Yes, yes, yes, it's something different, I know. Depending on who you talk to, it's also something very different. But think of it as something that guarantees that the system will exhibit a specific property. and that means that if you have a fitness function that shows you a specific property then yes this is a good means to ensure that your system will have this as for revisiting triggers maybe we would have to play around with this find the idea intriguing

Speaker 1 [41:43]

Do you recommend any training to learn software architecture that considers what you presented?

Speaker 2 [41:51]

I don't know of any. But then again, I don't know of any really good architecture trainings out there. At least I don't know how people learn architecture in a way that is structured, repeatable, and will yield predictable results. If you do that, if you know this, please come talk to me afterwards and tell me. But that is what I'm basically trying to say with there is no compression algorithm for experience. The only way I know is to apply yourself to decisions that relevantly change future options of systems and learn by the mistakes you make with the knowledge that your feedback cycles are measured in years, not minutes, hours, or days.

Speaker 1 [42:44]

The next question is a bit longer, so please bear with me. You spoke about decisions needing to be discussed with engineers, leadership and auditors. What are the different communication styles you need to use to get your point across to each group? And which has been the hardest group to communicate with in your experience?

Speaker 2 [43:08]

Okay, how do you do your job and how do you do it well? Okay, sorry, I'm not trying to be glib, but this is what this translates to in my mind. A lot of the job is not only thinking deeply about consequences of actions, a lot of the job is also talking to people and having an exploratory discussion about their points of view on things. I don't have a general answer in how to convince people of stuff and I would argue that sometimes you don't want to do this there is a lot of good literature on how to have good conversations with a diverse set of people that you might want to talk to I'm going to try to give you a more satisfactory answer now if you have to talk to a group find out what they care about and communicate your decision in terms of how it influences what they care about. How do you figure out what someone cares about? First step, ask them about it. Second step, check if their actions match their words. If their actions do not match their words, follow their actions, not their words. And the end part was... The most difficult group to talk to. different per decision made.

Speaker 1 [44:37]

Are frameworks like ARC 42 a good starting point?

Speaker 2 [44:43]

Any framework can be a good starting point if you understand that you are going to need to adapt it to your specific situation. That is, take a framework as a set of ideas, try those ideas, and be willing to drop them if they don't work in your context. So, yes.