Oh my license! – Achieving order by automation in the license chaos of your dependencies

Speaker 1 [00:08]

Oh my license, that's probably a sentence or a phrase most of you have said at some point in your life, at least if you're doing some software engineering, and when licenses or software licenses in specific can be kind of intimidating to deal with as a developer or at least confusing for a lot of people, and this talk is supposed to get rid of at least some of this confusion. So this is a license file, and this could be your lawyer looking at you, being really serious and asking you to please take licensing, especially in the software context, really seriously and make sure that you also not produce tons of overhead for him to check thousands of license files. And how we can achieve this, this is also a topic of this talk. So maybe a little more structure of what we'll cover. The first thing is a brief history of software licenses, just because it's good to understand where we're coming from. The second thing is, okay, which categories or which mental models can we build to understand software licenses a little better? And then we'll take a look at some specific licenses to make sure that we've got the most important ones covered so you can kind of leave with an intuition of what different licenses mean in case you do not have that already and then the last thing is to think for which most of you probably came which is how can I automate my license checking and make the life of myself and also of my lawyer hopefully a little easier starting off with two sentences or three sentences about me I'm Paul I'm studying at the KIT in Karlsruhe some of you might you know the university and I'm also working part-time at QuantGo which you might have seen the logo off in some other talks, because we're also giving some other talks, and we're also all there on the sponsor banner. Who am I not? I'm not a lawyer, I haven't finished a law degree, I haven't even finished my computer science degree yet, so this is not legal advice, I'm not liable for the information I'm giving you, please make educated legal choices on your own after consulting with your lawyer, and not the person with this face right here, which is standing right in front of you, this is just to give you an intuition, give mental models, and give you some tools on hand which you can use in the future to deal with software licenses, this is not meant to be legal advice. I think that was kind of clear. Okay, let's start off with a brief history. So actually for some of you it might be kind of surprising that software licensing is somewhat of a new thing, or the fact that software licenses or software in general have copyright to them, because in earlier days we had like computer, we had hardware, and software was kind of an add-on on top of it, and it was not self-evident that the software implemented in the hardware, or not implemented in the hardware, but on the hardware, was actually worthy, so to speak, of copyright. And this started to change in the 1970s and was really solidified with the US Copyright Act and with the TRIPS Treaty, which is the treaty by the World Trade Organization, who said, OK, software actually is a literary work and therefore subject to copyright. And this kind of led to an explosion in software licenses. Why? as soon as software started to get copyright, people could come up with their own license definitions and could say, okay, I have my nice little software here, I don't care about the standardizations that might already exist, I just can go ahead and write in the file what I want people to be able to do with my software. And a lot of people started doing that. And so we had an explosion of complexity in the software licensing world, which made it a lot harder to deal with. So the next step or the next bigger step in the evolution of software licenses was people actually started caring about, okay, how can we standardize this? How can we make it so that this explosion kind of gets limited in scope? And one of the more successful attempts, I would say, was the SPDX specification, which is a project by the Linux Foundation, who started compiling a list of license expressions or license identifiers, which can then be used by other people in order to keep this is a little more bound up and a little limited in scope so that everybody compiles their own license, but they can simply look around for software licenses already out there and reuse something that is already existing. We'll come back to SPDX right now. In case you have not dealt with it, this is a brief, short introduction about how these licenses are made up. Why did we or why did I include this? Well, it's really useful in practice to be able to see, for example, like in PyPI package, a corner package or so, is this license that I'm dealing with actually valid as PDX? Because if it's not, then you can just go quickly ahead and fix it, which makes automating on top of it a lot easier for both you and everybody else involved, for example me. So the SPDX expression standard is that we have a license type. This is something like MIT, BSD3 clause, Apache 2.0, stuff that you might already have seen in your day to day. Then we can chain these different license types together, and in the end we can also have a with statement chaining an exception to the license expression. To give you some examples, I already mentioned it, BST3 clause is one of the really common licenses you see in the Python world. And then we have MIT and OpenSSL. You might not know it, but OpenSSL2 is valid as PDX. So we can show the chaining, and this will also be a valid expression in total. And then we have Apache 2.0 with and with exception. Once again, you might not know this, but this is a valid exception. So this will also be valid SPDX. And then a couple of anti-patterns or anti-examples, so to speak. One is public domain. And you might say, OK, but public domain kind of sounds like a valid license. I've probably seen it around somewhere in the Python world. But it actually is not. One quick way to tell this is because it contains a space, and SPDX license types cannot contain a space except for between the and and or. And that's really useful to keep in your head if you deal with software licenses or expressions to be able to quickly fix a lot of issues, because this is one of the most common ones coming up. Then the second thing is the AND symbol, okay, we must use the words, we can't use the symbol, and then the last one is kind of non-obvious, because this is in principle valid syntax, but this is actually a license identifier which was deprecated by SPDX, so this is no longer valid. So where are we now? Okay, we had this explosion, we had a little bit of standardization going on, and right Right now, we are at a point, especially in the Python ecosystem, where people are starting to use or adopt the standardization. For example, in the PyPI world, there was a PEP merged at the end of last year, which tries to better solidify how licenses are handled in the PyPI ecosystem. And with other ecosystems, as we'll see in a minute, we're already a little bit further, but still not quite there. So in PyPI, according at least to Clearly Defined, which is an open source project compiling packaging metadata, we are at around 60% of software packages which include valid license information which is not a lot. If you try to build automation on top of it, it will fail all the time which is not nice or you would have a lot of exceptions or a lot of cases that you would need to cover. We do a little better in the Conda Forge ecosystem which is also why continuing from now I will always assume that we operate within the CondaForge ecosystem or the Conda ecosystem in general because there we have roughly speaking 91% of packages which include actually valid license metadata and this metadata is then also valid in terms of SPDX specification. So there might even be more metadata contained in there which has some sort of licensing information but the validity of the SPDX identifiers for the 91% is also guaranteed. So what licenses are there? In the Python ecosystem, this is an example from CondaForge, you might not be or you might actually be able to read it in the back rows, a lot of these licenses have already been mentioned so far. And we have MIT, Apache 2.0, BSD3 clause, and then a couple of GPL type of licenses. You will figure out what those mean if you're not familiar in the rest of this talk. And they make up a huge chunk of the available licenses. And I only listed here the top 14, but already these are like pretty common in the standardized used licenses that a lot of people use so okay we know we had this explosion we know there are a lot of licenses like thousands how can we start thinking or chunking this down to better understand it and one really useful way to do this is to break it down to several categories and the categories I would propose or I find a helpful mental model to think about them as public domain permissive copyleft and proprietary and we'll take a closer look at what these different categories mean in a second starting with public domain public domain in general is the notion that I have software which I created and I don't want to copyright for it like I just want to put it out there in the public domain for people to use if they want to and these are like the most or more common licenses you will seen to represent this this notion of the public domain idea which in itself is not a valid license as we covered before and The WTFPL license is probably the most permissive one you can find. It basically says, okay, do what the F you want with my license, and the license text, which is not the statement written on the bottom, like the bottom statements are just intuitions for you to easier grasp what the license is about, the normal license text, are normally a lot longer, but for the WTFPL, like the license text is basically just that, like do whatever the F you want with my license or with my software. Then we have the unlicensed, really similar in the idea, in general these are really similar, but it provides a fallback permissive license, because in some countries, for example Germany, you actually cannot fully abdicate copyright for something that you created, that's not permitted by law, and therefore for these countries it provides like a fallback license for people to use, which is really permissive in case you operate within one of these countries. And then the middle two, those are MIT and BSD type of licenses, but these are just include a little bit more of legal jargon, but also incorporate the public domain ID. I won't talk about these any further. If you operate within the corporate setting, you might want to take a look at these if you want to make your software a public domain. Next up, permissive. And these are the licenses you will see in the Python ecosystem, putting it that way. I think one nice way to understand them or to build an intuition about them is by building on top of the MIT license, which essentially does the same or a similar thing as the public domain license and says, OK, you can do whatever you want with the software that I created, but you still need to include attribution to my original work. You still need to include my original copyright file, which makes it nicer for creators to not get their software taken advantage of by other people, for example. BSD then comes about and says, okay, I build on top of MIT, essentially do the same thing as MIT, but then I also include a little bit more legal jargon about limited liability, which I provide similar, like I did in the beginning, saying, okay, I'm actually not liable for this thing that I created. And then it also has this minor clause saying that you can't use contributor names to promote the derivations you make of the software which you just included. And then Apache 2.0 does a thing which is a little counterintuitive if you're not a lawyer, but I'll try my best to explain it, which is that you have a software which is Apache 2.0 licensed, and if now somebody, some contributor comes in and contributes software to that project, then this software or the software which the contributor has contributed automatically is granted access, or this the project, the original project, gets granted access to the software which the contributor provided, which becomes relevant in case the contributor has a patent on the software which he provided or on the ideas used by the software which is important so that he can't come back later and sue you for the contribution that he has made to your project which is also like a legal technicality but might be important in some cases if you do both like IP heavy software for example some examples to make this a little more concrete because your heads might be spinning now all these different identifiers all these different acronyms. So MIT license examples would be libraries that most of you probably are familiar with, especially Pydantic and FastAPI are like omnipresent at this conference, so you probably have heard of them. Same thing for BSD, like it's hard to imagine software or data science-y stuff nowadays not using NumPy or Scikit-learn or Pandas or any one of them. And then And finally, Apache 2.0, I don't know what's bugging here. Maybe we can try, oh, okay, that seemed to fix it. Apache 2.0, of course, there's an Apache project in there, but also the others, like Streamlit, PyCharm, also really present at this conference, so you've probably stumbled across one of these in your time here on the conference, at least. Next category, okay, just to do kind of a rewind, we had the first category, which was public domain saying basically I don't want to copyright, then we have permissive licenses saying, okay, you can use my software for basically anything you want to do, but there are some terms that I would want you to comply with, and now we have the heavier weaponry, so to speak, which includes the copyleft licenses, and these, instead of being built around the MIT license like we did with the permissive licenses, can be kind of nicely built around the GPL license, which kind of helped kick off the whole open source movement, so to speak, because it serves as a mechanism to keep software open source. So if you have a software, once again using my fist to illustrate this, if you have software being GPL licensed and then you have some other software using this GPL license software, this software built around the other software with the GPL license also needs to be GPL licensed. We have different versions of this, AGPL is sometimes referred to as the GPL license of the web, because it includes some more technicalities about what about code that is run on a server, because technically running code on a server does not count as distribution. So you have to be kind of careful, and it was built on top of this. And then we have the LGPL license, which stands for a lesser GPL, and therefore, like, has fewer restrictions on when you can use software, which would then force you to be open source yourself also here are some examples and you might see that this is kind of stepping away from the Python ecosystem because those mostly include like really classic Python or not Python project but more like classic open source projects less so with the LGPL but at the latest if we look at the GPL license like GCC the Linux kernel GIMP and gets it can't be more open source than that right and so yeah these are really or the GPL license really found its way to the open source movement there and then some examples for the AGPL license as well and you can also see the server-side context that this license provides and maybe also the fact that it's a little more modern so to speak than the plain GPL license. Maybe one comment to the examples because if you're really well well worse in all this ecosystem you might say okay but there's this kind of sub component of Bitwarden and it's licensed on a different term this is just to give you an intuition once again and sub components might be licensed differently so be careful there not don't take my word for it dig a little deeper if you want to use it on your own and but like the main components of the software which I present here fall within the licenses presented last category once again wrapping up or reminding you of what we covered public domain do whatever you want permissive do whatever you want slight changes and GPL type licenses okay please keep this code open as keep it open source. Last one is proprietary and now it's really time to ask your lawyer like if you see a license identifier which is formatted in this way which is the SPDX format for typical licenses like this could be for example license ref quant go and then you really want to check with somebody with legal expertise because people can write anything in these license files and can basically say whatever they want or apply terms whatever whichever terms they want. Okay Okay, so getting to the beefy part, how can we actually start automating this license checking? And we built a little neat CLI for this. It's not super complicated. Everybody should be able to understand it, but it still automates a lot of stuff away from you hopefully in the future. And we can check it out by looking at some videos. I'm actually going to take this version because it's probably a little bit easier to see for you. And content deny features three different commands, the check command, the list command and the bundle command. We'll talk mostly about the check command and the bundle command for this moment because those will probably be the most useful or relevant to you. So how is this set up? Well, in this case we have a Pixie set up. If you've not heard Pixie at this conference, maybe you will in the future. There were also some other calls. I'm sure you will find a lot of people being really enthusiastic about using Pixie as a package manager, with whom you can talk about Pixie. If you don't know Pixie yet, don't be afraid. Pixie is just a Python package manager. You could also imagine in place of the Pixie TOML just your typical PyProject TOML or whatever. It doesn't matter for the configuration. And then we have a license whitelist file, which we can take a closer look at by continuing this little video. And the license file or the license whitelist contains exactly what you would expect it to contain, which is one, a list of licenses which you consider safe in your organization, for example, and then two, a list of ignored packages. This can be relevant, for example, if you include older packages for which you are unable to correct the license identifier, or you have some internal packages which you might want to use but you're not sure how they should be licensed or whatever, you can just whitelist them with a version which is also optional in this license whitelist. And then continuing, check does exactly what you would expect it to do, which is it takes the license whitelist, and it takes your dependencies, or more specifically the license information in your dependencies, checks them against each other, and it finds some diff between them so to speak, then it would throw an error, which we will also see in a second, maybe one short add-on, because you might be confused by that later on if I don't cover it, which is okay how does condit and I actually know where my license file is and it does know because in your pixie tunnel or in your pipe project tunnel you can specify the location of this license whitelist now this can not only be a local file you can also include multiple license files if you want to and you can also which is probably the most practical solution you can include them in like a github repository for your organization which can then be centrally managed for everybody in your organization to use which is probably the most useful setup for you. So a lot of builds up, now let's check out what the check command actually looks like and also here no real surprises, it just lists you all the different packages with licenses which are not included in your license whitelist. You might recognize that for example BSD free clause was not included in the license whitelist I showed you and your typical workflow could then be something like okay we add conda deny to a repository, run conda deny check go through the list of different licenses see okay I remember BSD3 clause is kind of permissive it's probably okay for me to use my project and that's actually a good thought because then you can come back to your lawyer if you want to double check he can add it or you can add it to the license whitelist maybe the centralized version maybe the local version depending on what setup you prefer and then you can add it and iterate your way through this project and luckily in the future you won't have to repeat that process over and over again, because ideally you will have compiled a list of licenses which your organization is allowed to deal with, so you won't have to do it over and over again. That was the first command. There is something else I could show you, but I'm skipping it now because I'm running out of time. So briefly checking out the bundle command, which is used to create SBOMs, so software bills of materials for your projects which tell your clients or whoever you ship your software to with what is included in your in your project here we don't need to license whitelist technically speaking because we are only looking at the dependencies we're doing no diff checking for example and then conda deny bundle kind of goes through all of your dependencies and builds one central directory out of it including all the original license files looks like this, downloads all the different conda packages, gets the conda, gets the files out of it, bundles it as a directory, and then the directory looks what you would expect it to look like, including all the different packages with the version and build strings, and the different license files which then contain the license text. I'm not showing this here for time reasons and also because it's probably not that enlightening. Okay. That was the brief overview of our conda-denied command checks. I'm about to finish. No bad glances from that side. So this is one last thing, how you might integrate this into your continuous integration. This example, the GitHub action. Also this is using Pixie. Don't be intimidated by it. You could also set it up some other way. And this would then check your micro-member environment in case you're using not Pixie but micro-member but still using Pixie in the CI. Don't be confused by it. You're just running condo deny with some packaging setup. And ideally, if you've done your whitelist configuration the right way, then your CI will look like this, which is what it should look like, which is everything is green. And your lawyer should look like this, because he's really happy that you took care of caring about your licenses and being legally compliant. Thank you very much. And this is a little link tree containing further resources. SPDX license list is a short read, really useful to get used to the whole license topic. DR Legal offers some more in-depth information about specific licenses, also really useful resource for you to check out for your day-to-day, choose licenses, helping you choose your license for the next project. And then it is really cool talk by Carolina Suma at the EuroPython, I think it's called the conference, at the end of last year, introducing PEP 639, which was the standardization for PyPI packages in terms of licensing. with that sorry for for taking too long but happy to get some questions

Speaker 2 [22:54]

Yeah, so thanks Paul for this introduction to licenses or maybe even a more deep Investigation on it. We actually have some more time look Like if you want to show something more you can but we can we also have quite some questions, so

Speaker 1 [23:08]

So okay, let's show one more thing because it's nice to know that it exists which is in the check command which we covered You can also pipe the output in different output formats Which might be useful if you want to build your own automation on top for example in this case Piping it to Jason and opening it with the failure favorite Jason parser Just anything to keep in mind if you want to build an automation on top of it. Yeah, but that's it

Speaker 2 [23:34]

All right. So just a short reminder that we asked the questions via Slido and we got already quite a lot of them. The room is Europium if you haven't found it yet. So I will just go very fast through them and so we can cover as many as possible. So first one, does it check dependencies of dependencies?

Speaker 1 [23:52]

Good question, and I think in conda environments it does because they are included in the conda setup If not, then please correct me somebody, but I think it's true

Speaker 2 [24:03]

Okay, next one we can ask also maybe wolf was there tomorrow from pixie. Maybe he knows Even though you use the conda ecosystem. What is the pondo and the standard Python ecosystem?

Speaker 1 [24:15]

in PyPI.

Speaker 2 [24:16]

More specifically. I guess.

Speaker 1 [24:19]

And the pond off or it's kind of the nice sort of thing I actually don't think there is one existing Yes, like if you look at pi pi and what they did in the past Which is really well covered in this talk by current and suma and they had some weird license classifier type fields and pi pi packages Which try to standardize things but only create more of a mess By trying to abstract away things which are actually not capable or which you actually can't abstract away from and not using XPDS specification which is used by other software ecosystems. So PyPI in the past kind of made it harder to adapt to standardization and this is hopefully changing now that the PEP has been merged although a lot of the things that the PEP introduces are actually not binding for future software or it's not binding for a past software which is why it might take like long long time until we reach a point of decent standardization where people can start building something similar in the PyPI ecosystem and if somebody knows something in the PyPI ecosystem which I'm not aware of then please correct me or reach out.

Speaker 2 [25:27]

Right, next one is support for virtual environments using only UV planned for conda deny

Speaker 1 [25:33]

I'm actually not that familiar with UV so

Speaker 2 [25:37]

But yeah, it only uses pi PI so in that sense

Speaker 1 [25:42]

So in that sense, no, then probably not, no. At least until we get to a decent point of standardization in PyPI.

Speaker 2 [25:48]

Why is generating an SBOM relevant?

Speaker 1 [25:52]

For example, if like imagine you create software and you want to sell it then I as the buyer of the software I would really want to know if you've been compliant and using for example GPL licensed code because I might reuse your software in my own software And if you've used GPL type of license code, then my software will suddenly Legally become also or I would need to license it under GPL terms And so I would need you to provide all of your dependencies and their licenses to me So I can be sure that you are not forcing me to be open source if I kind of make sense

Speaker 2 [26:27]

Some packages state that you can use either or license, e.g. MIT or GPL. Can I really choose? Because in such a setting I would always choose the MIT.

Speaker 1 [26:38]

If they say or, you can choose. Oftentimes, what's more common is to have separate parts of the different software, which are then licensed under different terms. So it's really untypical to see, okay, you can use either the GPL license or you can use the WTFPL, because normally nobody does that. People are a little more focused on what they want you to achieve with their software, what license terms they want to apply. If they have this like messed up, then of course you can do whatever you want with it legally. I think I'm not a lawyer

Speaker 2 [27:10]

What about Creative Common licenses in open source?

Speaker 1 [27:14]

repositories, I guess.

Speaker 2 [27:15]

I guess, are they valid?

Speaker 1 [27:18]

Good question, too. I I believe and I would need to double check that these are included at least some of them because the Creative Commons are quite There are quite a lot of them if I recall correctly I think some of them at least are included in the SPDX specification But typically like I have not seen them around in the kind of forest ecosystem Maybe at all maybe a lot So yeah, it should work if they are included in the SPDX specification talking about Conde Die, but I haven't seen them around a lot.

Speaker 2 [27:53]

Which license does conda deny have and is it also listed in the output?

Speaker 1 [27:57]

Good question. It's I think if you have it as a dependency it is listed But normally you would not include it as a dependency for your main software more like in a lint environment Which you can then? Yeah, not Connect to the software which you actually ship and this by the way I think is the license for condo deny and for those of you familiar with kind of license text This is the BSD free clause because we have three clauses also a little Little hint to keep in mind shout out to Pavel who helped building the building

Speaker 2 [28:29]

Next one is, if I make a project with plugins to compose an application, would a GPL plugin infect the rest of the code, even if it's the project has a different license?

Speaker 1 [28:43]

I'm not a lawyer. Don't take my word for it. I think so

Speaker 2 [28:48]

Could you change license whitelist to license allow list for more inclusive language or is the file name required for technical reasons?

Speaker 1 [28:56]

funnily enough we already have an issue for that so maybe or we could we could change it of course it's not concretely planned for the future but let's see like if let's see we'll see where the issue takes us

Speaker 2 [29:13]

I saw projects modify a BSD3 clause license to forbid a package to be used in commercial application. Is there an OSI approved license for this case?

Speaker 1 [29:26]

I don't know, first of all, but maybe a little side note on that, which is we have a neat little add-on, which if you want to use OSI-approved licenses, you can run condo-deny-check with a dash-dash OSI flag, and it will automatically check against the OSI list, so you don't need to provide your own license list, which can be nice for quickly getting up and running for open-source projects if you want to do those. So I don't know, but a little plug for the feature.

Speaker 2 [29:55]

So we are hitting the end of the talk, but we have only three questions left I would maybe quickly go through them in the speed that like before and then we finish early enough So which license should I use if I build a startup which for an open source project?

Speaker 1 [30:11]

Go to choose a license calm and figure it out that way better than I could ever do it Probably there are also some other like there are also two other famous sites I can't recall the names, but if you google help me choose a license quiz or something You'll come across tons of free resources, which are pretty great

Speaker 2 [30:27]

Would the CI in your example fail if a non-whitelisted package gets added? Yes. And last one, are there any other software licenses relevant to cloud-hosted software beyond AGPL?

Speaker 1 [30:42]

Surely there are because there are so many licenses I'm not aware of any particular ones which I have enough knowledge to feel comfortable telling you about so Would also make sense to just check for yourself. Probably

Speaker 2 [30:54]

Thanks again, Paul. Sure.