Your Name Is Invalid!

Speaker 1 [00:04]

Hallo zusammen, mein Name ist... Ich dachte ursprünglich, wollte ich den Vortrag auf Deutsch machen, aber ich glaube, mit der Kodierung werden wir so viele Probleme haben and that's the good reason that I should speak in English. Could you switch the key? Ah, okay, now it works. This was just a very short list of the names I see on the post that is coming to my home on my cards from different organizations and companies. But this is how it should be written. And sometimes I even then I see this text. My name is Miroslav Šedivý. This is the first line over there. There is how you pronounce it with the IPA. And the second line is how you can type it correctly using the Compose key. I work at the Salute GmbH in Karlsruhe in southern Germany, which is the silver sponsor here at Pycan.de. We are standing behind one of the largest price-comparing websites in Germany, Billiga.de. And if you would like to come to a city where both bicycle and car were born, and where we drink both beer and wine from half-liter glasses, speak to me or to my colleagues afterwards. So, what are we going to talk about? we are going to talk about Python, so how to work with names or general strings in Python, so comparing strings, bytes, encoding, normalizing, case folding, so uppercase, lowercase, sorting, alphabetical sorting, and regular expressions, and on the other hand, how to work with names on the websites or in the databases, the concept of first, last names, middle names, and how you should proceed with allowed characters in the names. So, Python 3 gave us finally the great distinction between strings and bytes. What is a string? String is a series of characters. Each character is one of the over 1 million code points of Unicode. And this is what you have in the working memory. You don't really mind how one character looks like if it is one byte or several bytes long, because it is in the memory. You access it, you see the length of this character as 1, and it offers you over 1 million code points, so most alphabets and a lot of symbols that are used all around the world. On the other hand, we have bytes, which is something that you write into a file on your disk or you send over network, and this is something that has only 256 possible combinations. So we have to find a way how to encode our strings from the characters into these bytes. If your name is Chuck Norris, you need no encoding but anyway in python if you take the string which is string in in quotes without anything before and you encode it you convert it from string into bytes there is a small b before um this uh quotes and on the other hand if you read some bytes you just decode them and you get the string in this case uh this is only a pure ascii so the length of both the string and of the bytes the length is 12, so there is no problem, and this is how it would work actually also in Python 2. But where there comes a difference, for example, Müller, German name that contains a U with two dots, which converts into bytes using two bytes. You need two bytes, so this is the number 2 and the number 3 on the right hand. Python uses as a default UTF-8, so you don't really have to take care of that as long as you are the master of your whole code and your whole data. This works also for some other characters, for example Chinese, this is not a last name, this is Ni Hao, so hello in Chinese, and so using UTF-8 we take our two characters as a string and we convert them to six characters as bytes, and you can write it on the disk or sent over the network. Default in Python is UTF-8, but we can use also other encodings because there are plenty, plenty, plenty of encodings out there. ASCII is the basic one. In this case, this name contains only ASCII characters, so there is not a problem. If you have some German name with the U, with the RSS, you can use, for example, Latin 1, and then encoding this U with two dots converted to one byte and vice versa, you take the bytes and then decode them. There is no information about the encoding in the bytes, so you have to know what you are doing with it. In my case, my last name Shedivy comes from Czechoslovakia. It means grey-haired. Don't believe that the last name means anything. In this case, the capital S with Karen and the Y with Acute, they can convert into a single byte in Latin 2, but I cannot use it in Latin 1, and vice versa. Latin 1 is for languages like German, French, Italian, Spanish, West European. Latin 2 is for Latin-based Central European languages, Polish, Hungarian, Czech, Slovak, and Slovenian, and a few other languages. So, this is what I see sometimes when I get mail. Actually, they upgraded since two years... No. When this appeared, I tweeted them that you have all my contact data, you can contact me, I know how to solve this. They didn't contact me, but nowadays all the packages I get from them, they removed the question mark and the Y at the end, so I am just Miroslav space space Edith. You have seen that at the beginning. So, the question is, how is this possible? Because on their website I entered ShadyVee correctly, and it is printed correctly on the website but on the packages there is a question mark. Where does this question mark come from? If I take my last name as a string and I encode it into Latin 2 I will get something that is represented correctly, these six characters. If I encode it into Latin 1, because this is a company based in Germany probably this is what they are doing. If they use Python and they encode it to Latin 1 they will get an exception because the first character, this s with current, the sh, cannot be encoded in Latin 1. But they still managed to send me packages. There is a switch that you can encode in Latin 1, and then you say, okay, if there is an error, don't raise an exception, but replace it. And the default, it will be replaced with a question mark. So actually, you can raise an exception, you can replace it, question mark is the only possibility, And you can, another possibility is just to skip the character. So this is how it is possible. So please, if you can, use UTF-8. Something different is, this is some random airline company in Germany. I wanted to buy a ticket, and I wrote my last name, and they told me, you can only enter letters in the adult's last name field. Sorry, my last name consists of letters. What is a letter? you can do this in Python 3 you can write a word with some beyond ASCII characters equal to the string but you cannot name your variable a smiley or dot or question mark how does Python know about that? Does Python know all these characters with all the currents and Chinese characters you can use Chinese characters for that there is a category so if you, category of character If you import Unicode data, which is in standard Python library, then you can ask for the category and the name of a character. So, for example, I have here a list of some A's, umlauts, schaffesses. There is even capital schaffesses, which exists in German. You see it is a little bit wider. There is Dodd and Smiley. And then with this short program, I can just get the character, its category, and its name. So, what you see in the first column, it is the name, how it looks like. And then the second column with these Ls, these two characters, it is the category of the character. And if it is an L at the beginning, capital L, it means that it is a letter. And U or L afterwards, it means uppercase or lowercase. And this is how Python knows that, okay, everything that contains characters that are categorized as letters, you can use them as variable names. pay attention, if you copy random code from Stack Overflow and it doesn't work really exactly as you would expect, check whether some variable names are maybe Cyrillic letters that are very similar or identical looking to the Latin alphabet so you can have an A or A, they look the same in both Cyrillic and Latin but they are two different characters, so the variable name is different, so this is how you can actually security through obscurity. If you want to make your code obscure, you just randomly, consistently switch the characters, some variable characters for Cyrillic. Okay, and then, let in small letter a, it is the name of the character, like it is stored officially at Unicode. This is the list of all categories, so you see all the L's, it is those letters, then N's are some numbers, punctuation, and so on. There is an operating system, there is usually this character map app that allows you to find a character and down you see this Unicode 0160, this is hexadecimal, Latin capital letter S with current and this is what allows you to use other characters in your code if you are not able to type them directly. or if you want to keep your code ASCII. So, for example, what I can do with my last name, I just backslash U and the decimal code 0160, and for the Y it is backslash U 00FD. And this is pure ASCII, but it translates into my last name. Other possibility is to use the name. This is even better to read, probably. Like, you can better remember if you see Latin capital letter S with Karen, then to remember what the 0160 code means and this way you can really enter some fancy stuff from the charmap that you like and in your code you can still see what it should contain when it is printed. Case folding. This is uppercase lowercase. It's simple you have the dot upper to get it one way and dot lower to get it our other way. But there are a few bugs that don't... not really bugs, but something that doesn't work correctly with what is in the reality of our human languages. For example, the SHA-SS. If you do SHA-SS, this is the lowercase, and you convert it to uppercase, you will get an SS capital. If you then convert it to lowercase again, then you get an SS lowercase, so you don't get the sharp SS again. On the other hand, this works if you take the sharp SS uppercase, you can lowercase it into the lowercase sharp SS. And there is one more bug, but this is a bug in our Latin alphabet, and there is one of the 26 letters that is actually wrong and Python does it wrong. Which one is it? I. You see this? There is a lowercase I with a dot that converts into an uppercase I without dot. Does anyone understand Turkish here? They have I with a dot, lowercase, uppercase is I and I, lowercase and uppercase without dot is I. How should Turkish people convert their text between uppercase and lowercase if we have broken their alphabet? There is a way. I will have a look at this ICU library a little bit later. You can do that correctly, but not with a standard dot upper and dot lower. Normalizing. We have two character, two strings. The one is süß, like sweet. It can be also a last name in German. And the word two looks the same, but it is normalized. What is the difference between these two words? The first one, I've just printed all the characters. The first, süß, has three characters, s, ü, and the scharf The word 2 contains four characters, so if you compare these two words, they are not equal, the length is not equal, and if you access different characters, they are not equal as well, but when you print it, it looks the same. What you see in the word 2, the character number 2, this dioresis, it is not my mistake, it is actually really offset by 1, because this combining dioresis doesn't have a width. If you want to combine, to put some dead keys or some extra marks on your letter, you put it afterwards, and then it doesn't have a width, so it means that u and the errors it combines together, to the u with two dots. Python, in this case, is very smart that you cannot normalize and denormalize anything. So, for example, what I can do, I can take a Chaffez S and then put combining DRSs after it, but when I denormalize it, so I want to get the shorter version of the words, I don't get a character. There is no character with Chaffez S with DRSs on it. So it knows that, okay, you can combine only characters that are possible to combine, because otherwise you would get something like this, which is a Stakeoverflow answer for a question, how can I use regex to parse HTML. And you see here at the end, this is actually some random character with a lot, a lot, a lot of combining characters afterwards. So then another topic is alphabetic sorting. These are some letters, imagine these are the first letters of some names. When you sort them using Python just like this, first You get all the SK capital uppercase, then SK lowercase, like A-O-U. Then you get the Umlauts uppercase, then the Shaf-S-S, then the Umlauts lowercase. Then some letters, characters from Central Europe, like the Esh. And the capital Shaf-S-S comes at the end, because it was added to Unicode a little bit later. So this is not how you want to sort alphabetically. And I don't really... My last name is Shedivy. I don't really enjoy at conferences to be always at the end of the alphabetic list of the speakers, because in Czech and Slovak, S is between S and T. But I don't know, in German it's probably different, in English it's again different, in Hungarian it's probably different, we are going to have a look at it. But if you use this, this is probably not what you want. If you want to sort alphabetically in the German logic, logic. You import locale, then change the locale to D-E-D-E, UTF-8, and then we sort it, and you see that the A's, they land just between A and B, and the uppercase, lowercase letters are also together. On the other hand, when you do something with Swedish, the umlauts, umlaut and the O, with the circle, they are at the end of alphabet, after Z. So if you sort the Swedish way, it will look a little bit different. In Hungarian, there is C-S, this Č letter, this Č sound, but it is C-S, and alphabetically it comes after C, so C-V, like Cvikli and Čipelž, they are in that order, because C-S is not between C-R and C-T, it is after C-Z. In Czech and Slovak, as I told you, Č is between S and T, Č is between C and D, and we have this letter H, C-H, that is between H and I. and this is also sometimes problematic because usually in Czech and Slovak if you see CH it's H but sometimes there are composed words that one word ends with a C the other one begins with an H and then it is not a H these are two C and H letters next to each other so even this doesn't always work correctly French there are some rules some French official dictionaries that use this rule that they sort everything alphabetically but when it comes to accents they sort them starting from the end of the word. So you first sort all the last syllables, then the before last, and then the other way around. The problem with locale is that it is connected to the process. So if you change the locale, the whole process has been changed. So it means if you have some process on web server, and there is some user that wants to see a list of the names sorted the German way, Swedish way, Czech way, and then if you set locale, then the whole process will be changed. This is not what you want. That's why it's better to use this ICU, International Components for Unicode, and then you create an instance in that language, and then you can sort in that language without changing the locale of the process. And this is also what can be used to uppercase, lowercase the Turkish characters. Now, regular expressions. let's say we want to extract from this text the word München how can we do it? If we extract all letters A to Z then we get only the M and the HEN we ignore the Ü when we use the backslash W then we will get even the numbers but we don't want the numbers so we want to find a way how to extract everything that is a character for that we will have to use the third party library regex that works the same like the standard RE, but that allows you with the backslash P in the curly braces to include the category of the letter that you want. So, for example, the capital L is everything that is letter. With LU, you have uppercase letters. With LL, you have lowercase letters. And this is how you can do it. Okay, I came here for Python, but I stayed for the names. How many of you have any problems with encoding or ordering or first name, last name? Okay, I think that I hear all the PyCon participants that have no ASCII names. Happy to see that. Who of you have simply first name, last name? Okay. First name, middle name, last name? First name, patronymic surname, patronymic last name, patronymic surname. Okay, this is more the Spanish and Portuguese way of doing the things. There are also, for example, in the northern countries, for example, Iceland, has this concept of the first name of the father, own first name and the first name of the father with the son. So, for example, if someone is called Sigur and his father is Johan, then he's Sigur Johansson, but they never call him Mr. Johansson. his name starts with an S so alphabetically sorted after Sigur and he is Mr. Sigur Johansson last name, first name, any Hungarians or East Asians here so again, this is not last name if it comes first any kings or popes here Vinetu no, just simple name Yeah. And then imagine all these von und zu and all these aristocratic titles. Sometimes they are part of the name, sometimes they are sorted, sometimes not. And even worse are all the doctors, because if you are a doctor in Germany, you get it as a part of your last name, but it's when there is a first name that it comes in between. So now imagine that in all the languages it's different. Very different. but maybe if you are here at the conference you have seen we have sorted you according to the first name actually we took the whole string of your name and then just okay this is your full name no matter whether you are doctor or first name last name and anyway you you order your name it will be just one string or one field in the database and there is something like how should we call you this is probably i think the best way to to store names because with all the first name last names, it simply doesn't work. Okay, there isn't. Except Mr. T, he said, okay, what is your first name? He said, first name is Mr., my middle name is period, and my last name is T. So don't, when you do something with the names, don't assume anything. Don't put random limit on their length. So if there is something like, type the fourth letter of your last name and I know there is someone here with three letters last name there is someone with two letters only ok, so there are even longer names on the other way or this way I think in this case they did a mistake in his name in Wikipedia and nobody noticed it, even himself don't use top words if it is an offensive word in your language is probably a normal name in other language. Family members don't have necessarily the same family name. Yes, all the male members of my family are called šediví, the females are šedivá, because it is an adjective in Czech and in Slovak, as I said, grey-haired. And also in Czech and Slovak we put ova at the end of female names, if the last name of the man is a substantive. different transcriptions from non-Latin alphabets so for example Anton Pavlovich Chekhov there are plenty, plenty of in every language they transcribe the Russian name differently the other way around as well in my passport I was in Russia in the past few years twice and I have two visas for Russia and my last name is written in Cyrillic alphabet differently in both ways so if there is some Russian organization that takes care of me, they see me as two different people, persons, probably. Chinese, the same, and other languages as well. Men, they can change their family names too, so if you ask for the maiden name or name, it's probably not what is applicable to men as well. And one letter name is probably not an initial, so Benoit de Mandelbrot is B, is actually only a letter B and not an abbreviation of a name. So probably you should expect that everything that is printable is fine and is a part of letter, because you cannot say that only letters are allowed, because there are some dashes, there are some apostrophes. So there are people who think that they have problems with their names, even if they are completely ASCII. if your computer if your program, your application has problem with Mr. Null please see me after the talk there are ways to get Null into the database without problems I have seen somewhere someone told me yesterday that someone in USA wanted to have a car plate with Null so that he cannot be entered into the database if he should get a speed ticket The problem was that he got all the speed tickets of everyone that couldn't be, where the license plate was not readable. So, if your database has a problem with null, you will probably have problems with these names as well. Okay, how about names of streets and cities, because cities and streets are sometimes called after people as well. So who knows what is the most common street name in Germany? Ausgang? No. You can find it quite often on the streets. If you park your car somewhere and remember, I park here. No, you are not going to find it again. On the other hand, there are a few names of streets in Germany that you don't find them in Germany, but you find them, for example, on American lists of companies, directories of companies. for example Hauptstrabe you will find it quite often there are places so you don't have to limit the length of the city at least three characters no, don't do that, because O, this is somewhere in Scandinavia Y, somewhere in France the habitants are called Y and there are also other extremes oh no, ok, of course, this so if you have a question, what is your mother's maiden name or where were you born. At least six characters should contain uppercase, lowercase and digits and so on. No. Okay, and there are places like Hlamvajrpuch, Vingeh, Gogerach, Vindreboch, Quantasilio, Gogogoch. That is even longer. Or, okay, this is fiction. A very famous Polish movie where the main person called Grzegorz Brzezinski Kiewicz and the place he pretends to live in is Chrzancizewo Sicepowiat the face of the German soldier looks like that for the whole two minutes after that. And even sometimes you even don't have addresses. So in Iceland you just draw a map and the letter will arrive. I am almost in the half of my talk now. But now I'm going to go really fast because this is just a list of falsehoods programmers believe about names, so I'm going to let you read it all of that, because every time you try to write a program that assumes that, ah, okay, of course, this has to be like that, no, it can be everything. Okay, the last, no, people don't always have names. So, what to do? Please, respect your users' names, and don't break the locale when program, so import this ICU third-party library. When you work with bytes from the network or file, read these bytes as soon as possible, convert them to string using the right encoding, and on the other hand, when exporting, convert them as fast as possible, as late as possible. UTF-8 is cool, Python 3 is cool, and BQL2, please. And please, telling the user your name is invalid only because of laziness, ignorance, and spitefulness prevents you you from handling their input. It's not cool. Be nice. Thank you very much. Thank you, Miro. Do we have questions? Microphone, please. I just want to add that in Portuguese we have the mother name and the father name, but in a different order than in Spanish. but there is no difference between how a mother female name and father's male name sounds so it's not like in Slovakia I also have my grandmother's surname anyone else has their grandmother's surname I think my grandmother's surname is 21 percent one I have a maiden name anyone has no name. I have not seen such a name which is without names. Anybody has questions about their own names? Excellent talk. Have you ever got a chance to work with Indian languages? Because there are some characters in the language letters the characters so does this applicable like ICU and Unicode libraries does this applicable even for Indian languages in which language is Indian Indian yes like we have multiple languages like we have some two thousand languages in the country if you have if you have if you occur if the characters of these languages are in Unicode and they are defined category as letters so you can use them you can pass them actually as letters even if you someone use this Japanese letters as smileys was the she she I think then these are letters anymore questions anyone everyone wants to go to the lunch I guess one question for me any advice about how to approach sorting where we when we don't know the local you probably you can use this you do us the English sorting or any sorting because then it will be always better than the primitive Python like the simple part Python sorting that sorts is according to Unicode because at least English knows that's the uppercase lowercase should go together and very probably shaft SS will be somewhere at the end of the alphabet but at least sure will be hopefully somewhere before sharp is this so you said that we should know what the what the encoding of the bytes is that we're loading yeah what do we do when we don't you try there are there are some encodings that can be detected. UTF-8, for example, has some security mechanism. But for the others, if you know which language it should be in, then you can actually tell, okay, in German, I allow only these characters, so there should be no other. But nowadays, If I write something in German, I sign it, automatically there is a character that is not in German, so I will probably break your program. But with Finnish, you have probably no problems. Okay, I think that's all the questions we had. Thank you, Miro, again. Thank you.