Robust Configuration Management with Pydantic's Data Validation

Speaker 1 [00:06]

Yeah, hello, my name is Philip, and as he said before, I'm going to talk about robust configuration management with Pydantic's data validation, and that sounds like a bit of a dry topic, but I'm going to try to convince you that it has some interesting depth to it. I work for a company called Meteor, our main product is MDBrain, and we do computer analyses of MRI brain scan data, so we take the brain scans and segment them, do some analyses on and generate some nice reports for the doctors to do some diagnoses. And for privacy reasons, we deploy this application on-premises at the doctors, so we need to tailor it very specifically to every network environment and their specific style of doing these analyses. So we have a pretty complex microservice architecture, a lot of components reading these configurations, and a configuration format that grew over the years to adapt to all of these different situations. And this is kind of how I came to this talk here. So what am I talking about when I'm saying configuration? This could be either like the programmer defining some constants in some module that is being imported everywhere. You can kind of configure the application like that. You can also configure it using a config file that is more of the administrator's realm. And then the user could also express some user preferences during runtime by, for example, setting a language in a web browser or something like that. I'm only going to be talking about the orange part here, the configuration of the system, basically by the system administrator using the config options that the programmers provide. Kind of we started out with a simple dictionary based approach, so you kind of define your configuration as a dictionary and you can even nest the dictionaries to get namespacing, so as I said we have many different microservices and these all have their own configuration section and then you also have some overarching ones which take care of options for multiple components. And this is a Python dictionary, but if you want to store the configuration on disk, you need to choose a serialization format. You could either choose, like, say, JSON, or we're using YAML because it can give you cross-referencing and object loading and all that kind of stuff, but it can be unsafe and complicated. I think the Python ecosystem is more moving towards TAML in 11.3.11, we will get Toml lib in the standard library, the PyProject Toml uses Toml, so I think maybe now I would use this, but in the end, my talk is not concerned with any serialization format, anything that will unpack into a standard nested dictionary will do fine, so all of these are fine, we can use any kind of back end for our system here. So how would we go using like a dictionary-based configuration object like this? Well, the standard library gives us the get method on our dictionary class to supply a default value and a key, and if the key is not found within the dictionary, we get the default value, so that would kind of like look like this, I get the key, and I supply a default, so if we're looking at our configuration here, maybe we would want to be interested if the sender is enabled, and by default it should be disabled, so we're going to supply But we're going to have to basically supply this default value any time we access any of the key values, and we also need to supply the key. So this gets a little bit importable if we use it across many different components and we want to change the default value down the line, we would have to change it in every single place. So you would probably want to extract this into some sort of constant and then use the constant, but then you also have to take care of importing this everywhere. And yeah, defaults can be a little bit of a problem. Because if we are now interested in specing it out even more, supplying a default value for, say, the sender port, then the whole boilerplate gets a little bit more complicated already. We need to supply an empty dictionary as default value for the namespace variable. And then on that one, again, use the get and supply the default there. here. And maybe because we're doing all of this parsing of our object, we also want to log it so we know what we parsed in the end. And the next problem that we would encounter is probably fallbacks. I'm going to say it up front. I think going through all of this journey, fallbacks are not a good idea. If you're designing a configuration file format, I would strictly recommend not doing fallbacks. But we ended up with a format that had them, So I had to implement them. And this would kind of work like this, where if you're saying we want to use the same port for the receiver as we do configure in the sender, then we kind of get the receiver namespace, supply empty dig there, then get the port out of that namespace. If that's not defined, then we're going to get the sender namespace, supply empty default there, and then get the port. And you can see the boiler blade is growing and growing and growing. And this also gets us to some philosophical question. is the supposed value here of the last default that we supply? Right now, I put in the default receiver port. So, the receiver port has a different default value than the sender. But if the sender is supplied and the receiver is not, then the receiver port default is used. We could also say just use the sender default. I don't know. Fallbacks are just not going to get you into a nice, well-defined place. You're going to get all of these ambiguities. I'm not even talking about circular dependencies in your configuration format, you definitely don't want to deal with that kind of stuff. But my main motivation to overhaul this whole system was the problem of validation. If you're having a system like we just designed and you're only using the values when you extract them from the dictionary, you might end up getting some runtime errors because you're not doing the validation well enough. In this case, for example, I've created little bit more boilerplate, say we have this port variable, and maybe our deserialization doesn't give us strongly typed objects, so maybe we need to check if it's a string that has been supplied, we need to parse it ourselves, and if it's not a valid integer, then maybe we erase an error, and then we use it later on to open some port with our socket server, and our program crashes because the user supplied a negative number or something like that. There's a lot of different validations you could apply, and we actually had a bunch of these things happen where it would crash some time down the line when some very rarely used feature was being used and ill-configured. And if you don't have a very clear specification of your format, you're not going to be able to catch these errors up front. You're always only going to catch them at runtime. So it would be very nice to validate the config statically before even running our program. And the last problem that we encountered was the documentation aspect of things so these default values you define them as the constants and then you also have a lot of boilerplate code that you use for input transformation so in this case I'm extracting the default value into some other variable because I need to normalize it and supply the default value and everything and log it and then you also need to access these keys during every access so if you want to change a key name in your format you have to change in all of these little places or use yet another constant. Maybe you even want to supply default values for keyword arguments, so there's more places that you need to take this care of. You probably want to mention the configuration variables in your doc strings and maybe also even supply the default value there if you're not using type annotations. And another thing that we do is supply a default config file that has all of the default options in one file. So with our installer, so if the support team sets up a new installation, they can just copy the file over, change some values and be good with it, but you also need to keep that in sync, and then we also have rendered documentation for the support team that doesn't want to read the code but looks up some wiki page, so all of these different places we're getting horribly out of sync, and I would like to just have a system where I supply the value one time and I generate all of the documentation dynamically from it. So I was looking at different options for configuration management systems, and there There is a bunch of really fully fledged systems, for example, Facebook research has Hydra, but I think that's more geared towards running a batch of machine learning experiments with different configurations. There's Lip Electra, which is like this industrial strength tool with database support and automatic cute GUI generation, and that was just way too complicated for us. Dynaconf is very simple, but has very little validation capabilities. There's a bunch of other ones that I looked at. So then I decided I want to do my own hand roll system that is just the right amount of complexity, and I want validation to be front and center, so I was looking at different validation libraries. Most of them somehow relate to JSON schema. There's JSON schema, schema data classes, JSON, Marshmallow, Cerberus, a bunch of these different libraries, but the star of the show is Pydantic. You maybe know it from fast API. It's usually used for serializing user data that is coming in, but we can try to convince it to manage our configuration as well to our benefit. So the idea is that you would define your format as a Python class hierarchy and use type hints to define the types of the different variables. And it's kind of like an object relational mapper where you define your model as code and then load into it. And that would give you the ability to construct these objects and get validation errors if you actually can't. So that is very neat. How do we actually do this? As a little caveat, I'm going to be talking mostly about pedantic v1 because I started this system one and a half years ago when v2 was not out yet and we are kind of forced by backwards compatibility to stick with it for a while. So if you guys have any hints on how to improve this with v2, please let me know during the Q&A. not super familiar with Pydentic V2, but it's the whole overall of the system, gives you much more flexibility. So Pydentic models, as I said, are basically like ORM objects. You import the base model class and then just basically create model classes for every namespace. So in our case here, I get the sender config and I supply the host and port variable. And here the host has no default value and is mandatory, so you cannot construct this object without supplying a host variable. But the port does have a default value, so if I don't give the port, it will automatically assume it to be 80. And then we also have our baseline config object that then can include these namespaces by using these custom model classes as types. So the sender namespace here, as you can see, has the sender config as a type, so it will require us to pass a sender config. The receiver namespace, I marked it as optional, so it can take a value of none, and that means by default it's just not going to be present, and if you omit it, it's going to be fine. So if we now want to construct this, we basically just pass our data, our object, our dictionary as keyboard arguments into this class and get a fully typed object back. So if you, for example, take external data like this, I supplied some values, I left out some other ones that have defined defaults here, I would then end up with this nice little object and as you can see, even though I'm not supplying the sender port, the sender port is already filled out as default 80 in this config object and this is the nice thing with the dictionary that is not typed, I don't know what's in there that could be any sort of keys and any sort of values and some might be present, some might be not. I always need to supply the default when accessing it. Whereas here, the whole model is fully expanded now and I can trust that there's always going to be a sender.port and it's always going to be an integer. That gives me a lot more flexibility while coding. I don't need to do a lot of error checking because all of these are always going to be present. How can we make this model even better? By adding more constraints to use the data validation really to enforce the correctness of our configuration format. So for example, we're using these literal type arrays as kind of enums but get IDE support by the type checker for it. But you can also use type aliases to add semantic meaning to your configuration. So for example, DICOM is a medical file format and we have a tag type that I'm just aliasing to string. So I'm not really getting anything more out of it. But now I can construct dependent types that feel much more semantic. In this case, a filter is a map from a DICOM tag to a string that is much more expressive than dict string to string. And maybe in the future we could go down and actually go to the link of this documentation and extract all of the valid keywords and make it more strict, but for now this is even just improving our developer experience, and, yeah, you can actually continue this even more and build more and more dependent types. But Pydantic also comes with a good library of a lot of very interesting types with interesting validation built in. So, for example, you could have this file path type which will guarantee you that the the file is already present on disk or you can say I want this to be a future date and it can be validated up front that it's not in the past. Email address validation is famously difficult and there's really good strong support for that and the library is much bigger. There's a lot of interesting types for UUIDs and all that kind of stuff and that already gets serialized into say for example Python daytime objects so you also don't need to take care of that And you can even take it a step further and use some constraint type primitives that PyDantic supplies to make your own constraint type. So this is something that I created for a time of the day, basically like a cron type system. And you can constrain the string with some regex. And if somebody does a typo or maybe thinks, oh, I supply these with a period instead of a colon, then this would actually only crash when we're trying to parse this doing like the usage of it. But now we can just supply this up front. And the port example that I had earlier, it's pretty easy to do here. You say, okay, port is positive number between 1 and 65,000. So with this one line, I can get rid of all of that boilerplate code that I showed you earlier. Pydentic also even has built-in support for configuration management using the base settings class. And we can construct it like the following. You can only use base settings on the base model class, all the nested ones are still base models, so our previous configuration we have the same system as before, but now for my base system I'm choosing base settings, and I can also use the config class within our class that also works for base model to configure the Pydantic model even further. You can, for example, supply allowing or forbidding extra values or changing aliases or something like that. But here we're just setting the nested delimiter and what is really cool about this base settings class is that it can source and merge different configuration origins with a well-defined precedence manner. So in this case we can, for example, override some of our specification with the environment variables. export these all caps environments, PyIdentic will automatically recognize them, and you can see I can even access nested namespaces by supplying this nf-nested delimiter variable. So this is very neat. You can have the precedence of the initializer, and then the environment. There's built-in .env support. There's built-in support for secrets directories mounted in slash var slash run, like Docker secrets, for example. and the default values, it's all going to be merged together very practically. There's another thing that I'm using here, the license key. I set that type to secret string, and that is another pretty neat feature of masking values in log output. So if I construct my model like this and supply some string value for license key, I can print the whole config object, and it will automatically be masked for me, even if I access individual values directly, it will be masked, and if I actually am sure I want to get the secret value, I can use some custom method on it, which can be pretty nice if you're logging your configuration but don't want to expose some API keys or something. And what I have been also pretty interested in is using validators, custom validators for unorthodox things. So, I'm calling these input transformations, which will be clear in a second, but validators are basically decorated functions on your class that can perform custom validation on types if you want to do anything more specific than you're doing with the type system. And in this example here, I'm using it to emit deprecation warnings for attributes that I deem to be deprecated, but they can still be used. We can also do something a little bit more complicated, which is maybe a little bit further from actual validation, for example, supplying dynamic values. Number of parallel processes might be an interesting thing to configure, and a lot of times the users want to configure it to use the maximum number of threads available on a system depending on the number of cores on your CPU. So I'm marking this as optional, positive integer. We can supply null as a value, and if we do, I want this to expand to the actual number of cores on the system, and I can do this by supplying this validator that will check if it's none, and if it is, instead it's going to return this CPU count value. So the nice thing about this is as a user of this configuration object, I don't need to care at all about all of these defaults or complicated expansions. I get the object, and it has some integer in this attribute, and I don't care if it has been dynamically generated or is the default value or has been supplied by the user. I can just use it, which is very nice. And as I said, my favorite topic, fallbacks can actually also be implemented using this validator method. And this is even more powerful in PyIdentic v2 because there you can actually control any number of validators and the order of them more directly. In PyIdentic v1, you can only do validation once. So that makes it a little bit more complicated to be sure you still have valid object after expanding all of these validators. But, yeah, I think this is a pretty neat way of using these validators to do expansion, even though they're maybe not meant specifically for that. Around these core functionality that Pydantic supplies us with, I've also built a little bit of tooling around it to make it more usable within our system, and one of them is G-series latency, so we also want to supply or save this configuration file to disk, and I created basically like a wrapper class for that that inherits from our base configuration class and that can basically take care of all of this boilerplate of saving, like getting all of the file paths and loading it into a dictionary. And since we're using multiple products with different configuration files on disk, but I want to merge them all together in one big namespace configuration object so I can do validation across all of the different products and maybe access different values across products. So I'm also injecting basically the objects from different files into one big object here. And then I just load it with validation and log the content. So we have one core microservice that basically does all of the orchestration and that is the one that logs it and everybody else doesn't log it so you don't pollute your log, but you still get a very clear picture of what has actually been parsed. As I said before, We also dump a default configuration file that can be very useful for an easier initial setup, and basically the way I'm doing it is again getting some modified file paths and getting the configuration objects for all of our different products, and then going through all of the products and dumping the object into a dictionary, and the important thing is that we cannot construct objects valid objects if we're not supplying all of the required values but some of the required values like license keys you don't want to end up in your defaults file so basically what I do is I set these values to some token value that is valid but I obviously don't want to put these fake information into the file because I don't want the application to start up with an invalid file, so now I render this into a dictionary and then I go back and supply null values again, and when I write it to disk, then it will not be a valid file, but it will include all of the defaults. So here I'm using just the Ammeldump to write it down. And one big aspect of it was also permissive loading, so in an ideal world you would always have a perfect spec, and roll it out immediately, and everybody uses it in peace, love, and harmony, but we have a very complicated product, and I was just not sure that our spec would be perfect for the first try, so I didn't want to have a situation where some customer configured the application in a valid way, but my spec wouldn't be valid, and the application wouldn't start up, so instead, we're doing a scheme where if we opt out of validation, function, emit a warning, validate the model with this custom, or with this bespoke validate model function that gives us an expanded dictionary that has all of these input transformations applied and expanded, but gives me a list of errors instead of raising them, and then I can construct the model from this object in a more relaxed way. This is unfortunately not supported by PyDank out of the box, I'm giving you this here for the slides, you can copy it if you want, this constructs the model permissively and uses dicts where the objects are invalid and does so recursively. Talking about the documentation rendering, we're using Sphinx for that so we can supply all of the metadata and documentation just in the doc strings and then render these out. There's actually support for Pydantic with this Autodoc Pydantic module so you can basically You just go ahead and use auto summary and auto module to completely auto discover all of the different configuration classes that you have and you never need to touch anything but your model specification to automatically update the default file, the render documentation and everything. So that is pretty neat. Just for your reference and interest, this is the configuration that we use to get everything nice. Then we use also CI automation to automatically update this to GitLab pages. So on every merge we automatically get a fully rendered updated documentation for our format and you can see it's all interlinked. You could click on these links to jump to the different sections. You get the constraints very clearly. You can see the default values very clearly and it's always up to date and nobody can forget to update it. And then we're also using some add-on to render it to Confluence because our support team users conference for managing their information, and you can see we can even do beautiful rendered code blocks and info boxes and all that kind of stuff. One important aspect is also incremental adoption. We want to roll it out step by step, and it's pretty nice because you can still basically break down these objects back into dictionaries and use that to roll it out step by step. redefining our configuration now and then we can basically get rid of all of the, sorry, then we can just use the loading but call dot dict back on this object to get a dictionary back so we're having exactly the same system of accessing the values but we do get the validation and then in the next step we could actually get rid of the defaults and supply The values form our already validated object. But again as a dictionary. And then in the next step we could go back and actually pass the objects. But then maybe we are not sure where we always need to update it. So we can use the type checker now because everything is statically typed to find the place where we still need to supply, need to accept the object instead of the dictionary. So that's been super useful. So in summary, you do want a strict specification for your config format to be able to validate it statically before running the application. You can do so with Pydantic settings and use their input validation and transformations to really spec out your system, and then there's a lot of nice tooling that you can build around making your developer experience nice and comfortable. Thank you very much for listening to me here. As I said, my name is Philip. You can find me on GitHub. I'm going to try to put some of the code on my blog if I don't forget. I work for Midiere. As I said, we are always looking for new talents. You can check out this website there if you're interested in working with brains and computers. And, yeah, please do check out Pydantic. It's an amazing library. The dude just founded a company geared towards developer tooling. And, yeah, just check that out, and maybe you can find a better use for that as well.

Speaker 2 [26:19]

Thank you very much for a very insightful presentation and quite a lot of practical examples. Let's go through the questions that we have. The first question is actually coming from the nice presentation you have.

Speaker 1 [26:32]

Which tool did you use for it? This is Apple Keynote, and you can do all of these typing things with the type animations. It's basically all just a lot of very manual labor, to be honest. I have a lot of these manual animation steps and stuff. Yeah, you just need to sit down and actually go through it.

Speaker 2 [26:56]

Okay, thank you. So regarding your talk, can you define conditional attributes, like, for example, change an attribute's value depending on another argument?

Speaker 1 [27:11]

Yeah, that's actually something that I didn't have time for, but we're also building this to enable us to do migrations. So if you actually spec out your config, you definitely want to add a version number, and then you can go ahead and define migrations between different formats. But if you don't have a specification, you don't have a format, you definitely cannot migrate so easily. So this also helps with that, cleaning up your format and stuff.

Speaker 2 [27:38]

and to use the base model for config and then to use other models. So this was not...

Speaker 1 [27:43]

Sorry, can you repeat that?

Speaker 2 [27:44]

Sorry. When you use the base model for config and when to use other models?

Speaker 1 [27:49]

models. So the base model is basically just the pure data validation, and then the base settings class has all of this support for loading from environment and secret support and stuff. And you have to use the base settings class for your core model, and then all of the nested models use the base model class. But there's no other models, Pydentic just calls them base because you always inherit from it.

Speaker 2 [28:13]

And is there any UI for Pydandics?

Speaker 1 [28:14]

Thank you. I have not checked that, but I'm pretty sure you could write a system that automatically generates some nice config. We just use Unix files and our support config is using a text editor. But because of the nice annotations and type system, I'm pretty sure you could build something nicely if it doesn't exist.

Speaker 2 [28:35]

One last question, do you have an opinion on using validators versus the field models default factories in terms of maintainability and also the code readability?

Speaker 1 [28:50]

For validation, we mostly only use the type system, and I mostly abuse the validators to do the input transformations, so I haven't really worked with implementing very complicated validators myself. I have no opinion on it.

Speaker 2 [29:05]

Thank you very much, Philip, and thanks all for nice questions.