jsonargparse - Say goodbye to configuration hassles

Speaker 1 [00:03]

Hi everyone, welcome to my talk Jason Arcpass say goodbye to configuration hazards. My name is Marianne Staglina. Let me start off by telling you how I came to give this talk. I work as a deep learning engineer at Omnius here in Berlin. And deep learning is just one of these areas where there is a lot of configuration options. A while ago, an amazing colleague of mine, Marisu Villegas, decided to create JSON ArcPaths for Aseromnius and for everyone else. And I ended up as one of the early users of this package. I'm very unhappy that this talk ended up as a pre-recorded video because I got COVID just a few days ago. But hey, let's stay positive, haha, the advantage of this is that I can already start answering your questions while you still watch this video. So please use our Discord room B5, B6 to ask any questions you might have. I'll try to answer them as quickly as possible. Okay, let's move on to some introduction to the topic. Jason ArcPath is open source and it helps you to configure your code by creating a CLI. CLI stands for command line interface, basically a bunch of commands you type into a terminal to tell your program how it should run, how it should behave. CLIs are great, and I tried to list some of the advantages here on this slide. For that, let's just step, take a step back and think about what are the alternatives to command line interfaces. So one alternative is what I termed here, no interface. So, I mean, you could just hard code everything directly, right? if that's the case then your program will always run in exactly the same manner it will always read from the same directories right into the same directories that's just not very useful you should separate your code from your configuration that way it's also way easier to share for example scripts if you don't have to tell your co-workers how exactly you implemented it but they just need to look at the usage of it it becomes way easier for them the other alternatives the alternative are graphical user interfaces so compared to graphical user interfaces command line interfaces are just way easier and faster to to create also clis are perfect for batch tasks if you want to rename a thousand files you are not going to do it manually via a GUI. Same goes for scheduled tasks. If you have a script that needs to run every night at 2am, you're not going to go for a GUI approach. The last point that came to my mind is environments like remote servers where a terminal might just be all you have available. These are all yeah points where CLIs are really useful and I think it's also pretty domain agnostic so whatever field of programming you're working into you're working in CLIs could be relevant for you. So to do this at least a little bit interactive and for me PyCon is also about community and about exchanging I would suggest you to take the poll in the Discord room because I'm really curious to know what tools you are currently using for command line interfaces or for configuration management in general or if you're not using any tools at all that could also be the case. Okay, let's move on to some JSON arc pass basics. For that, let's look at some code. So here is our first use case, a very simple function, which is called talk about conference and you can see it here. It has two arguments so one positional which is called the conference name and one optional argument the year which is set to the default of 2022. And then in the main part of this file we use Python's built-in argparse package. So we create an argument parser, we add the two arguments of the function, we pass the arguments and then we execute the function with the arguments. Okay, if you look at this code there's actually a lot of duplication. So the names of the arguments you can find both in the signature and in the add argument function here the same goes for the types if you do use type hints the same goes for the doc strings if you do use doc strings and you should um yeah and it would be nice if this could just be avoided and guess what with json argpars you can have that so if you would rewrite the same functionality basically with JSON-ARC pass, it would look like this. So you can replace the whole part that was under main by a single function called CLI. If you're familiar with FHIR, then you might recognize this. FHIR was just a big inspiration for the development of JSON-ARC pass. So this is the same working mechanism. So what this does is it checks the file, finds this one function that is here, it picks up the arguments, including the types and the doc strings, and it gives you a command line interface. So let's compare the help of these two functions. So for the arc pass version of it looks like this. And for the JSON arc pass version of it looks like this. So you can already see small differences. So in the jsonArcPath case you can for example see the doc string of the function and also you can see for each of the arguments what's the type and what's the default. The execution for the JSON ArcPath version is exactly the same as for the ArcPath one so we say python the name of the script and then we put the conference name so let's put pycon and we can also put a year. this is the result you will get. PyCon 2022 is the best conference ever. So in the help you can also see options related to a config because json arg pass can also pass config files. And there's a really nice pattern, which is basically print config, modify config, run config. So you can use the print config option to get a config of the format that is expected for the script. So let's do this print config. And let's dump this into a file. If we open this file, here we go. We have the two arguments conference name and year. So let's put PyCon here. Save that. And now we can run our command line interface with the config we just provided. Here we go. So, maybe you noticed that the package is called json-arg-parse, but actually the default format uses is YAML. Well, so the nice part about YAML is that You can also include comments in your config. So there's an option to print it or to print the configs, including the doc strings. So what you get is kind of a self explanatory config file which is really nice. And you don't have to decide between using command line arguments or using a config file, you can just mix it up. So you can use, for example, the config file we just created, and you can overwrite just the year argument with let's say N to 24. And you will see in the output that it took the command line argument. command line argument overwrote the config file. This is just dependent on the order. So we could do the same the other way around. So write the command line argument first here. And then the config file. And you would see that it picks the value from the config file in this case. So let's go back for a second to the implementation. So this is what we had just CLI. Next to basically besides command line arguments and config files there's a third option what you can pass and this is environment variables. For that you just need to add a few options to your CLI. Basically a prefix for environment variables and basically just to enable them then. And with that you can also control the program like this. So we put PyCon which is the prefix we set. And then the argument name. Conference name. Set it to PyCon. and the same for the year and we will get the same result okay with that we're already done with the basics. So let me summarize the key features of JSON-ArcPath you've seen so far. You can get a command line interface with really a few lines of code. So just this CLI line plus the import you've seen in MyScript. And it's possible to pass command line arguments as well as config files as well as environment variables and you can mix it up however you want. Okay we're done with the toy example let's move on to a more complex use case. So whenever your projects start to become bigger and there's more than a handful of options to configure, you should be moving to config files as a default and basically just use the command line arguments to overwrite what you're currently experimenting with. So imagine the following use case for the remainder of this talk. We want to train an image classifier. So let's say we have the cipher 10 data set and we have a convolutional neural network and we want to classify these images into different different classes. Okay and we have the the training function you can see here below. So it just takes a data set and it takes a model. the nice part about config files is for once that you get an overview of all the options you have but also they can have structure they can have hierarchy so in a config file you can put together arguments that are that belong together like they can have meaningful groups so how should you structure your config you already do have meaningful groups of things in your code i mean we do think we do spend a lot of time thinking about how to structure our code in a meaningful way and classes are one of these concepts right so all in our code all the options related to the data set or the logic related to dataset is defined in the dataset class and everything related to the model is defined in the model class. So the JSON-ArcPath approach of things is basically to just replicate this. So for the function I just showed you a config could look like this. So we do have dataset and model the two objects you've seen that go into the function. And for each we have a class path defined and the init arguments. So we can then run our program like this. So config you can see above. It's not only the config that is structured in this hierarchy, but it's actually also the help. So if you check the help for the same script, it will not show you 100 option at once, but it will just show you the highest level. it will tell you okay there's a data set and there's a model and these are classes let's well let's dig a bit deeper so from here the help will tell you okay if you're interested to see the data set for for example then you can have a look at that specifically so let's do dataset.help for the cypher 10 dataset dataset.help let's pick this okay and here you can you can see the three options you also see in the config so there's data deer there's batch size and there's augment Let's look at the types. So there's a path, an int, and a bool. So when you give a configuration file to JSON-ARC path, it doesn't just open it and read whatever is there, but it checks that the structure matches what is expected and also that the values that you provide do comply with the type you specified. So, if we try to execute the program and specify, for example, the augment parameter, which is supposed to be a boolean, with something that is not a boolean, we get an error message. will tell you okay I did expect a boolean value but you provided maybe. So the json arg pass has great support for these type checks not only for simple types like string in boolean but also for for example path. So the date idea I specified to be of type path dr so d is for directory and r is for readable. You can specify other things like it should be a file, it should be writable or creatable or whatever. And it will also check this. So let's try this. data set data deal, let's set it to something that is not readable, because for example, it doesn't exist. Then we will get an error message, json arg pass will complain that this is not a valid readable directory location. Dealing with paths is really common. And it is a great feature to already check the access permissions on startup. So imagine you run the image classification code, and your model is already training for multiple hours, and then you try to write the final model into a folder, and it's actually not accessible, that would be really bad. So it's great that JSON arc pass is handling all that already on startup. Okay let's look at the key features we've just seen. So both the config and help can be hierarchical and the config structure always replicates your code so whenever you add a new member variable to a class it will also appear in the config these two will always be in sync and the last point we've just seen is type checks json arc pass checks for a lot of types and there's many cool types defined like paths or like regex restricted ins check it out what's available there. Okay next topic is subclasses and this was for me really the feature to switch. Let's imagine in our image classification use case we're not satisfied anymore with just training a convolutional neural network on cipher 10 dataset. Let's say we want to also try a bigger dataset like ImageNet and we also want to try a cool new model let's say a vision transformer. So what would we do in the code? In the code we would basically create two abstract classes, one for dataset and one for model, and then we would implement the two subclasses for the dataset and two subclasses for the model. So this is the point when it is difficult to continue working with like homegrown config files because for the models, for example, a conf net and a vision transformer are just completely different. So I mean, architecture wise, they need completely different arguments. So the only thing they do have in common maybe is the number of classes. So I just wrote it here as a property of the parent class. Let's look at how JSON arc pass can handle this. So our next iteration of the program, let's look at the help again. Now it will show you still that there's a data set and the model argument, but now it makes more sense to look at the rest of what's printed here. So for each of the two, it will show you the available class path. So for the dataset, you can see this abstract parent class and you can see the Cypher 10 and ImageNet subclasses. You can basically run your program as before with this big config file here. So you can say, as before, config. And you can generate a config still by printconfig, just that now you need to select what class you want. So that will look the following. So for the dataset, let's pick the ImageNet class. And for the model, let's stay with ConvNet for now. And then we can still add printconfig. This will be the output. You can save it, you can run it. Since now you have different building blocks and there might be more to come, right? So you might start and try different optimizers, for example. a lot of possible combinations and it would be a waste if you create a new config file for each possible configuration of datasets and models and optimizers. So what you can do is to split this up into subconfigs. So we can have a config which has just the cipher 10 dataset options and we can have a config which has just the confident model options and you can run it like this. So you can say minus minus data set and then put the data set config minus minus model And pick the model config. So this will work. And there's another option, which is basically one config, which internally references to sub configs. That is another option you could do. So then it would be again minus minus config. Here you go, same result. From here you can continue until forever. So let's say we extend our use case even further. Image classification is not enough anymore. Now we also want to do object detection. So what you could do in the code is say we split the model into two built-in blocks. We have a backbone that takes care of generating some useful features out of the images and then we have a task specific head which does either classification or object detection. Then a config for this could look like this. So you just added one more level of hierarchy and like this you can go on forever. So that is really like the strong point about JSON-ArcPath, no matter how complex your code gets, the JSON-ArcPath config will always replicate the structure and it's very flexible. So summary of the most recent key features. you can switch between subclasses even when they have completely different interfaces that's no problem and you will keep all the benefits of before you will still have type checks for everything you can still print configs so that will stay as before and you can compose your config into multiple files if you have been using Hydra so far and you should be very familiar with this concept of composition of configs. One last point on the agenda for today. I wanted to highlight one of the major milestones of the development of JSON-ARC pass which is that it got adopted by PyTorch Lightning. So So if you're into deep learning, you will know that PyTorch Lightning is a very widely used, well-known framework. And it recently added a feature called Lightning CLI, which is JSON-ARC pass under the hood. So if you're into deep learning, please check out the Lightning CLI in PyTorch Lightning. So on this last slide, you can find all the relevant links, so please check out json-arcPath. You can find it on this GitHub site. I uploaded all the code examples I showed during this talk, and also the slides will be made available on speaker deck. i would be super happy if you reach out to me with questions about json arc pass or configuration in general but i'm also very happy to talk about nlp deep learning in general yeah so whatever it is please reach out to me on the discord channel my name is mariana steglina in case you forgot or after the conference of course you can also use twitter for example thanks a lot for listening

Speaker 2 [30:13]

Thanks Marianne. That's really good talk. So we have a couple of questions on Slido and if you have any more questions please add them or vote on the questions that already exist there. And the highest voted question is how does it decide which function to use if you define multiple functions in the file?

Speaker 1 [30:31]

Hi, great question. I hope you can you can hear me properly. So in cases like the example I showed you, if there's just one function, it will pick this one function. In case there's multiple functions, all functions get added as sub commands. So in the example I showed, you would basically then have to type the name of the program, then the name of the function, talk about conference, to select the subcommand, and then you add the arguments. So again, if you're familiar with FHIR, that's the same way of handling it. And the alternative to adding all functions is that you can also manually specify which functions you want to add.

Speaker 2 [31:21]

Okay, great. The next question, I guess it relates to the name of the package. Is it possible to use a JSON configuration file?

Speaker 1 [31:31]

would be a shame if not right so yes you can use you can use json file as well and you can also use json net which is also a superset of json

Speaker 2 [31:45]

Can you also merge multiple config files, and if so, do they have to be the same type of config file? Or could it be like one in YAML and one in JSON, for example?

Speaker 1 [31:55]

So I think by merge, you mean basically having minus, minus config and then one config, and then you put another config and they would override. That should work. And can they be in different formats? I guess, to be honest. Also you can specify a default config in the code. That's also an option. So the idea is always that whatever comes later and the chain will overwrite what's already there, so it should be possible.

Speaker 2 [32:28]

And does JSON-ARCPAR support password type arguments? Which I guess means the hidden arguments or something.

Speaker 1 [32:37]

yeah I have to pass on that question I'm not sure okay do we have I don't think so

Speaker 2 [32:45]

do we have any other questions from the audience I don't think so either so please give a round of applause to the Marianne