A Medieval DSL? Parsing Heraldic Blazons with Python

Speaker 1 [00:05]

I'm really excited to be talking about the Middle Ages at a software convention. So this is real fun for me. So the reason I'm interested in this talk is kind of an intersection between my two interests of cool computer-type stuff, which I like, and gnarly medieval stuff. And this is a real fun cross-disciplinary talk. Another one of these, we've got heraldry, the coat of arms. I'm not talking about a software library called Herald. I'm actually talking about knights and shields and castles and stuff. It's this really cool intersection of graphic design, the Middle Ages, history, and then a way of doing self-representation. And there's a lot of rules as well. So I think that's why I find it very interesting. And we're going to get into how I use software with this. But before I get into it, does anyone in the audience have a coat of arms? Okay. Oh, maybe one? Huh. Interesting. When I gave this talk in US PyCon, there was a guy who had the right to use a coat of arms from his clan in Ireland. So coats of arms, there's some misunderstandings, and I'll talk about that a little bit. But has anyone in the audience written a DSL, a domain-specific language, or written a parser? All right, same guy with a coat of arms. A couple other people. More people don't have a coat of arms. This is about parsers, really. Okay, so let's get into it. Why did these exist? So if you're wearing a full suit of plate armor, it's hard to see your face and your identifying details. So that way, if you have a shield, then you can tell, oh, it's, you know, this is that king, this is this noble person. It also is, like, currently we have all these logos for all these companies, institutions. and it's sort of the Middle Ages version of a logo because they can't reproduce things picture-perfect, like, you know, pixel for pixel. They had to have, kind of, boil it down in another way. So as we're going to get into it, the individual logo of a coat of arms, like the individual way it's drawn, isn't the canonical way. It's actually defined in a language. Yeah. Oh, yeah, and the other important thing is it allows the king to hand out honors which is really important to the feudal society all right so yeah it's just like a bunch of guys in castles painting and they can't always paint the exact same image um and you also don't have like a printing press so it's like difficult to like distribute copies of exactly how it should look like so um how can you kind of like control what this logo is and have it be consistent um without those modern things and the answer is you actually make a DSL for it. And it's called Blazon. This is how this is how the English heraldry system did it. The German heraldry system had something similar, but it wasn't as distinctive. Alright, so just looking at a coat of arms, so we have our terminology. This is the coat of arms of the UK. Up there, that's the crest. The crest isn't the whole thing. It's just the thing on top of the shield. The little creature up there. Oh man. I'm sorry about that. Beneath the crest we've got the crown, and then we've got the helm, which is the helmet. The mantling, which is kind of like a cape that the knight would wear. It's got some supporters carrying it up. This belt is the order. Then here's the coat itself. Sometimes you call the whole thing a coat of arms, but this One is the escutcheon, that's the real coat, and then the compartment and the bottom, and the motto. So there's a lot of different components, and they all have different meanings, and Blazin can describe all of them, but I'm mostly going to be talking about the innermost escutcheon, or the coat. So here's an example of a Blazin. This is the Blazin that describes, in a linguistic way, the coat arms we just saw. and if you read it, quarterly, first and fourth ghouls, three lions, passant, gardant, impale, oar, armed and languored, azor, you notice that it is total gibberish. It doesn't have sentences or any way you can understand it. It's kind of written in English, but also there's all these other words in there that seem French. So we're going to parse that. It's actually a DSL, as I said. Other things about coat of arms that are interesting. Arms are inheritable kind of the way land is inheritable, as in, like, usually the eldest son gets it, or sometimes, you know, depending on the country, it's split up. But it's not the same way a name is inheritable. So if you have a name, you know, my name is Beecham, I don't necessarily inherit the arms of the Earl of Beecham of England. It's like a more restricted lineage. So don't get fooled by a lot of websites. Another thing is, I ran into this while parsing it. Blazin is a very consistent DSL, considering it was written over 500 years by warring factions in the Middle Ages. So I've had to simplify somewhat. But here we go. So I'm going to introduce us to Blazin by trying to emblazon the Python logo. So let's say azor, that means blue, background. We've got two snakes up on there, and I'm going to call them serpents because, wow, they had a concept of a python as a creature. It was more of like a dragon-type creature. So this is more like a serpent. And then we have to define the exact position of the animals. So each serpent is erect, and then they're adorsed, and then intertwined. and then there's we have to say the colors so it's kind of like these are kind of like positional arguments for each of the things about the serpents and then here of the field so this is actually like a variable reference so the field is a zor which is the blue background and then the first serpent is of the field so it's the same color as the background So it's kind of like a variable reference that they have, where you try not to repeat a color if you already set it. You just refer back to that element. And then or the sinister inverted. So great. So here, let's go into trying to build a parser. There's a lot of different parsers for Python. But unfortunately, most of the parsers, there's like 40 parsers. And they all seem like they were written in one weekend. um you know it's like uh it's kind of there's no one giant one that's awesome um there's a lot of like you know kind of fun projects people thought to make um so i ended up choosing this one called lark because it seemed like it was the most supported um and it also had a nice interface um but the nice thing about the parsers is that almost all of them use a very similar format which is like almost an academic format for for giving a parser grammar it's called the back is an hour form, and the nice thing also about Lark is that it's divorce recursion, and the heraldry language is recursive, so we need that. So here's the backest an hour form, and it's sort of like, if you like regexes, you're going to love parsing, because it's just regexes on regexes on regexes forever. um so yeah and you you may sort of recognize some of these symbols each of these is like uh each uh like we can say a lowercase letter is anything from a to z uppercase letter is anything uppercase a to uppercase c a word is one or more letters so this like plus thing you may recognize from regexes. And then if we build this little tiny example parser, we're able to parse hello world as two separate words, hello and world. And the goal here is I'm just taking us from having a big string we can't read into having it as an abstract syntax tree, which is like tokens that we know what they are and we can start to interact with them in code. I don't know what you're going to do with hello world, but now you know it's two words. All right, so for the simple ones, let's just talk about the tinctures, which are their word for colors. They separate them into metals and colors, which will come up later, and they have all these strange words for them. And then so those are all just going to be keywords, and then the tincture will be either a metal or a color. And these are important because there's a lot of rules about what colors can go with what colors. um so here these are existing arms for people um that some people like managed to like register blue as their coat of arms and no one else could have blue so that's pretty sweet um but yeah so we can parse these immediately so it starts being useful here's some more complicated fields these are called furs they're supposed to represent like the fur a king would wear an ermine and then the interesting thing about furs is that they start to have a more complex syntax. So the ermine fur over here can take two arguments for its colors, and it's like an infix. And then the ver one, that's actually like a prefix syntax. So we have to kind of support that in our parser. And this is all just totally in the language. They really use it like that. um so i think i have yeah so i have fur tincture and tincture and i also have tincture infix fur tincture and so um there's not this is the infix for i have remained because i've only seen that i haven't seen very as an infix for but sometimes you'll like download another database of codes of arms from a different area and then they you know they they use the fur in a different way or something like that. So I had to kind of keep on rejiggering this as I found more examples. Alright, so we've got, this is just like arms are a single field, field is a single texture. So now, here's like a fully parsed abstract syntax tree for a slightly complex arms. And we can just see that this is like a tree structure. And, yeah, we've got the arms are a field, the field is a tincture, and then you know what the fur is, and then you've got the two different tinctures. So if you want to start parsing that and generating an image or something based off that, or comparing arms, you now have a computerized understanding of that restructured form, which is our goal here. So if you organize your own DSL, that's when you can start using that to understand your language. So here's the next thing is ordinaries. These are like the big shapes that they would use. These are incredibly common. These are the seven most popular, but there's hundreds. Here's a bunch more subordinaries. And then these ones can take any color on them. So we have like azor, abend, very, which is the slash, very, or, and ghoul. So now it becomes more complex. We have this thing on this thing on this thing. And, yeah, we start to have more complex ones. Here we not only have ordinaries, but we have ordinaries on ordinaries. And in the arms of Dallywimple over here, we've got ordinaries on ordinaries on ordinaries because there's a saltire of the X, and then there's the rusters, which are the white diamonds, And then, oh, I guess I, they should have a little dot on top of them. So the point is that they can be arbitrarily complex. And some of these things become incredibly complex. So let's expand our little parser to be able to handle these recursive, complex groups. So we've got our ordinary. it's either a big ordinary or one of the subordinary lists and then we have any number of them any color, the name of the ordinary and then the color again and then how they're arranged and then if we have multiple groups we might use and or we might use comma to separate them or we might put one on top of the other so we're just kind of building up our little parser here and this now lets us this is a very complex code of arms so we're able to parse it with this parser so far and this has got shapes on shapes on shapes and we're able to have this kind of fully recursive charge group and then on top of that there's another charge group on top of that there's another charge group the charges are the shapes so yeah that's nice so I don't know if you caught it exactly but a charge group can have a subgroup which is a charge group so any charge group can have another subgroup of a charge group and it can be recursive like that so that allows us to get very complex they also have a lot of animals they have to draw, which are basically just a whole bunch of key words. And each of those animals has specific adjectives that they use for the shape and the attitude of the animal. And attitudes can be many different attitudes. So yeah, they have a lot of different words for what it means for a dolphin to be standing up like this, or for a peacock to have its feathers out. but and then like it kind of matters um certain things about if you draw an animal you might draw the exact position of the animal but then like there's some amount of uh artistic liberties as long as it's doing the right position so um sometimes you know you can do a you can emblazon something in ms paint you can emblazon it also you know in beautiful penmanship and drawing, and as long as it conforms and follows the rules, then it's still the same coat of arms. So here we got Berlin in Germany, and here at PyCon.de, and we're able to now parse these ones. I guess this one's got a bear rampant, which means like this, rampant, running rampant, and then the eagle is displayed. So that's the Bundessegel, and it's displayed like that. So yeah, it's actually kind of defined that. I don't know if, I don't think a bear can display, And I think a bird eagle could probably be rampant, but it would look with its feet, you know. Oh, it's crazy. All right, so getting more complex, we've seen recursive charges on charges. But actually, any two codes of arms can be combined. There's several ways to do this, but the one that became most popular towards the end was called quartering. So here's a simple quarterly sable and argent. That's arms of Hohenzollern. And here's another one, Castile-en-Lyon, which is a really fun coat of arms because it's got a castle and a lion on it. And it's Castile-en-Lyon. So they did that a lot where they do a pun, and the name of it is like what's on the arms. So yeah, to do these quarters, you just have like an entire blazon and then an entire blazon next to it, and you say quarterly. And these ones are both quartered, but quartering just means that the entire arms are represented in smaller form. It could be four, or it could be a lot more. So here's the quartered or marshaled arms of the entire European Union. And actually we can see up here there's the same coat of arms of Germany, the Bundeshegel, or the Reichsegel, or whatever it's called. And it actually looks totally different, but it's still a black eagle that's displayed, so it's the same coat of arms as the previous one, which was more of a stylized gif type. So yeah, because we already fully defined how each of these coats of arms would work, if we want to support quartering or even nested quartering, because if you notice the arms of the UK, which is still technically in the EU, it's quartered arms on quartered arms. So it's easy to support this level of recursion with just one additional rule that you can have quarterly and then have any number of quarters. And each quarter is an entire group. So it's easier to write a parser for that than it would be to draw all of them. So now this lets us finally parse the codes of arms of the UK, which is one of the more complex ones. so we're able to say quarterly and then we have three different groups so that breaks down um the first and the fourth we have um the field ghouls three lines passant gardant and then the the way they're decorated and then the second quarter um it's a golden field or a lion rampant within another charge group so it's a charge group on a charge group and so i have that over here, this is the abstract syntax tree that we parsed from that. So we've seen that there's the field, and then there's a charge, the line, and then over here we've got an ordinary, which is this flurry what's it called? Thresher. Thresher, which is the name for the border. Harold Drew's like 100% edge cases, and like weird, only the thresher can be flurry-counter-flurry, for instance. But anyway. So let's see. The third quarter, yeah, it's a simpler one. It's just the harp on a blue field. Okay, so getting into more complex, talking about these edge cases. So Haroldry's got a bunch of rules that aren't covered by our parser. So, for instance, this coat of arms, this red on black, is not allowed. It actually has a very low contrast, which is something that they cared about a lot. So if it's two tinctures on each other, it's not allowed. But if you had a metal, so metals are like yellow and white, then that would be allowed. And then over here, we actually have a metal on a metal, which is typically not allowed, except in the case of Jerusalem, and a tincture on a metal, that is allowed. So this is called the rule of tincture, and it's really important in heraldry. But to do this requires a complex understanding of which shape is on which shape, and which color is on which color, and we basically can't do it in our parser. So I just want to point out that there are certain understandings that you might, that don't belong in a parser that would belong in a validation step after. And maybe you want to, like, say, hey, I can parse this, I can understand it, it just, like, might be invalid and handle it differently. So, yeah, this, I only did the abstract syntax tree, the parsing of it. I didn't do the outputting image, because someone already did that, and it's also very, there's a lot of edge cases so I didn't do that part. I just wanted to talk about how to build a parser. And here's the reason I'm talking about this is because we've now seen this full parser for how to parse grammar, for how to understand Blazin. Has anyone ever read the parsed grammar for Python? It's actually shorter than the one I just wrote. Yeah, you just search Python 3 grammar and you can read it. and it's about half as long as my heraldry grammar, because my heraldry grammar has got a lot of kinds of specific animals and their positions. So let's look at it a little bit for Python. The reason I want people to read the grammar is because I feel like you look at different Python code and you have a feeling of how you write it, but you might not know why that's allowed. What are the exact rules that means what can you put in one spot or another spot. And it's a little bit of magic. And I think whenever we find magic, we should destroy it and learn what's going on instead. And if you look at how simple this is, it's actually very understandable. So for an if statement, if, and then there's a test, which is going to become a Boolean, and then colon, then there's a suite of what's done if the test passes. Then we have one, zero or more elifs, and then the optional else. So if someone can ask you, like, oh, is there a limited number of ellipses? You could just try it out, or you could read this and figure out how it's defined. If you have any questions about Python, here are the answers. And sometimes you'll realize that there's a simplicity in the language that you don't realize. So here I talk a little bit about what those tests are. so a test is an or test or a lambda definition and then an or test could be an and test or another and test so these are kind of defined in terms of each other but it's not very complex and if you just take a minute I don't have time right now to go through it all, but the homework for this talk is to go and read this on the train home or something like that. And you'll find out that all of Python is either a simple statement or a compound statement. And then a whole function is just a bunch of statements. And here's all the simple statements. Here's all the compound statements. That's the whole grammar. Just more of this, how class definitions work. and so here's an abstract syntax tree for this print statement so we got the start, the eval input function call it's the print and here's the first argument so you can see that this isn't getting all the way into how this is actually understood but if you can parse Python you can start to understand, you know, where mistakes could happen or if it's not parsing, how that's happening. So anyway, that's the point of this talk. It was to draw you in with Middle Ages stuff and then make you read the Python grammar. Thanks. And this is my favorite coat of arms over here. It's the arms of a Russian closed town that exports a lot of plutonium, and I think it's really awesome. So questions about parsing or about Middle Ages stuff? Thank you so much. We have some time for questions, yeah. Thanks for the really funny talk. Spontaneous question, would you be able to parse that coat of arms with your parser? Probably there's the keyword Adam that I haven't added. It's tough because when I was building my coat of arms, there's always a certain fish that only lives in this guy's hometown, and he wants his fish on his coat of arms. so as i you know i i got to about that in my database of code of arms which had like you know a hundred thousand arms that i found um i got about 70 percent that were being parsed but then there was you know the the long tail was like oh this guy needs a specific you know and this guy is an atom and no other code of arms in the world has atoms on it you know so it's it's tough to do a complete um one and the guy who there's a guy this um drawshield.net website um they have a similar project as you know more complete project where um you can type in any blaze and then it will render it for you and of course he's got the issue of like oh this person wants me to add a new image for you know yeah like i made him out of soldering iron because i wanted to have a coat of arms the soldering iron on it and then i realized like how many different items there are in the world you know they could all be on it next more questions

Speaker 2 [26:39]

Thank you for the great talk. One simple question is, is it compatible with the Italian heraldry system or is just for the, you mentioned maybe briefly that it's just for English system, it wouldn't work with German. What exactly do you support?

Speaker 1 [26:57]

So the German, yeah, I was reading about this as I was coming here to give this talk in Germany. The German system for a coat of arms, they don't have, first off, all the words are in German. So all the keywords are different. But also, it's sort of, it's just kind of like a paragraph that explains how to draw it. It doesn't have the, like, really interesting kind of, like, positional argument thing that the English French heraldic tradition has. So they have all the same images and stuff but the German one would be if you translate it, it would be like first draw a big red field then put a bear on it, then say this and this, which is just in German. It's just a description of it instead of like a really terse if that makes sense.

Speaker 2 [27:52]

But look into the Italian system because they really like recursion, so they become just a patch of random colors because you have five, six levels of recursions in the stuff they create. Another question is, the renderer that you mentioned, is it implemented in Python? And if so, is your parser compatible with it? Do you plan to... Somebody's planning to... Anybody's going to do that in Python? oh yeah like a fun

Speaker 1 [28:25]

I mostly wanted this talk to be about the parsing, so I didn't touch the rendering. I don't think that it's compatible. This is kind of... It was just an exploration to see how, basically, I know this plays and exists, but is it actually... I wanted to know, is it actually stable enough that you could write a parser for it? And then I did, and then I was kind of happy with it. And also, the more I kind of pooled on it, the more edge cases you start to find. I got about 60%, and then I realized the edge cases are anything that they thought of. But there is a good chunk that's all really standardized. Yeah. It would have been fun to go for rendering, too, but I kind of ran out of time on that. Okay, hey, thanks for the amazing talk. when you let your parser run on like real medieval blazons did you find any cases where the blazon was in fact ambiguous

Speaker 2 [29:28]

Is the impact ambiguous?

Speaker 1 [29:28]

like where your parser found two ways of parsing the text oh yeah there's also there's um there's a lot of ambiguous ones um which was tough because sometimes i would like be diagnosing an issue and be like oh actually it's it could be diagnosed in um uh parse many different ways um specifically like uh just like exactly um sometimes like what uh uh objects um are referred to by like uh different adjectives it's often really ambiguous um another fun thing is that sometimes the middle ages whenever um the heralds who are coming up with arms they would use that rule of tincture that said only certain colors can be on other colors to play a game where they would make the coat of arms really ambiguous but there's only one way to like solve it that we're like oh this thing has to be in this one position because that's the only one that satisfies the rule of tincture so they would um kind of play a game with the other heralds and make a and and make kind of a puzzle for them so you know i had to run into that too okay perfect uh thank you i learned a lot about parsing and

Speaker 2 [30:47]

housing and excited to see what

Speaker 1 [30:49]

I'm excited to see where this project goes.