Encoding Charactersets - may the force be with you

Speaker 1 [00:05]

Thank you. So many people not living in a Unicode bubble and interesting in character sets. My name is Martin Herrmann and I'm working as consulting and giving workshops. But let's start with a little story. All pictures are from mid-journey, so at the very end you also have the prompt if you like. Imagine you have a friend, nice haircut, but unfortunately she's captured by an evil person. I think you know him. But we have a little friend with an escape plan, but unfortunately he has a plan and it's somehow gaveled, and we don't know what to do, and if we do the wrong thing, someone evil is waiting for us. So good to have a universal translator, but unfortunately this one also gets a Unicode decode error because he has an invalid continuation byte. And if you receive bytes or open files, there's no such thing as plain text. Only bits and bytes. Nothing else. And there's a really brilliant talk from Dylan Bitty on YouTube talking about this topic over an hour. And if you open a file and you guess the wrong encoding, then you have to look up in some tables. For example, take the first byte, look up in an ISO table and seeing a character which is probably not a German one if the message is German. So maybe this one is right and there's some special text or it's wrong. So guess we have the correct encoding in this case and it's a two byte word and we can look it up on another table, a UTF-8 table, and then we get the letter U. Which is a letter and which is not a letter is also quite a big topic, and you can talk about this by a little bit. Betty talked about that, I think, 10 or 15 minutes, what is a letter. So this Here we have at the beginning of each byte we have information in UTF-8 which tells us this one is a two byte word. And the two first bits of the second byte is this is the following byte. And if you take all the other bits and put them behind and calculate the number, it's D6, and this is the Unicode point, and now you have the correct U, which is the first letter of the message to rescue your friend. And if you decode this D6 like we had before with an ISO encoding, then we have the U, which is O with diuresis, and if you try to decode this in UTF-A, we have this unexpected end of data because we only have one byte and the first bit tells us there must be exactly two bytes for this letter. There are a lot of decode errors like no definition for this bytes or start byte exists in definition but unexpected end and things like that. If you want to deep dive for the first bits, define how long the word is to decode your letters. And UTF-8 is specially built for Western countries where all ASCII characters, which just need seven bits, are put with a leading zero, and then you know you only need one byte for all all ASCII letters. So it's very efficient for storing Latin letters. But not very efficient, for example, for Chinese letter, then if you have such a character set, usually you take UTF-16. So like the master would say, incorrectly detecting the encoding all times, impossible it is. But you can try. And that's what nearly all your IDEs are doing or your Notepad++. You have two possibilities. You have a configuration file or you guess. And if you try to guess, that's a really cool package down there on PyPy which is a CharSet normalizer. And if If you read a file with from pass from the CharSet normalizer, you have several fittings and there's also a best function for the probably best encoding and you get a lot of information about what is encoding, it's UTF-A, what languages are inside, is there a bomb byte, a byte order mark which try to solve this struggle in the file and the string representation is a text like this one opens the fifth door and luckily you will free your friend. And now we have some knowledge about the file, we can open it, but this is probably a bad idea. Why? Because in Python 3.10 and in Python 3.12 we have slightly differences in the default encoding which is currently OS-depending. And in 3.15 is accepted PEP 686 which makes UTF-8 the default mode. So if you're working on Windows, this will be a major change in in your Python code base. So like Tim Peters tells explicit is better than implicit, use the correct encoding the next year until maybe one day everything is encoded in UTF-8, maybe. And if you're writing Python source code, this is also defaults to UTF-8, format was ASCII, and if you want to deep dive, so UTF-8, you can choose strange variable names, the pi is the best one, for example, but you can also use letters which looks like an A but are not As and things like that, so I don't know if this is a good idea. And if you deep dive into string representation, so how Python represents strings inside the compiler, this is a completely different story. This is flexible string representation. And with UTF-8, it's not all signs are accepted, like the snowman. This will lead to a syntax error. And my boss is still like he still likes to read all files with his VI editor in ISO mode. So in all the most of our files we have the prefix line which tells our encoding is in ISO and in this case the Python interpreter looks the first line and interprets all letters in this file as ISO. This works pretty well with Python, with PyCharm, and I had no success in running this one in Visual Studio Code, and I didn't find any plug-ins which works with this one. So you have to configure all your code base if you want to do something like that one, if you have a boss like me. So how is it going to go from bytes to string and to bytes? you use the functions bytes decode or string encode, and if you have bytes 3, 3, A4, then you can decode this with, for example, UTF-8, and this is a small letter A with diuresis, so you have a byte stream, and you use the decode function and use the correct encoding, otherwise you get problems. So if you have 3A4 and you decode it in ISO, you have two letters which is the A with a tilde and the currency symbol, and this one are called Mojibake, so always when you have text which is not intended to be text, then you have characters like this, I think you all have seen them somewhere sometime. And the other way around, you have a letter, for example, A, then you have the encode to make a letter into bytes, and we have the same letter, two different encodings, and we see we have E4 and 33A4 to the different character sets. There's a beautiful Unicode package in Python, and starting with default, you can use Unicode letters inside your program, like the first one. You can use the Unicode hex number, or if you know, very pretty, you can use the Unicode name. We have a better one on this next slide. And you can use the Unicode ID as an integer. And with those code examples, we have the first four letters of the Greek alphabet, alpha, beta, gamma, and delta. And the other way around, we have the Unicode data package. And if you have, for example, the Python snake as a symbol, you can print it, of course, and you can get the Unicode data name, which is quite nice, which is snake. Or you can get the Unicode ID with an odd function and transfer it usually to a hex. So if your little friend is talking to an ISO database which is inside, probably not the best idea, then what happens? The DB driver detects the char set and converts everything from ISO to UTF-8 and vice versa. And if you only have round trips, this works pretty fine. And I have seen projects, it also works if you do Greek letters inside, if you put if there are bytes, if you put German one, this will also work unless it's not even ISO. You can do strange things with such round trips. But when you have a letter which is inside Python, and there's no corresponding letter in the ISO, then you get a driver error. So invalid byte and code set conversion. And you have to deal with this one. And therefore, you can use the encode function with the errors replace command. And this is replacing all bytes which cannot be encoded in the desired format. And each character set has its own replacement character. Usually it's the question mark and UTF-8 is a special character for that. And with this byte stream you can decode this to ISO 8859 and then you can store the information. But in this case, of course, you have data loss or information loss. Okay. So maybe it's a better idea to use JSON because JSON has an easy specification, looks like. JSON exchange in open SQL ecosystems must be encoded in UTF-8. Of course, if you are the only one in your project, you can use whatever you want. But that's the definition. The definition goes further on. The encoding supports a full Unicode character set. Good. Those characters outside the basic multi-language plane, which are the first 65,000 letters, there is something special. So however, if escaped, those characters must be written using UTF-16 surrogate pairs. I haven't seen that before this presentation ever. I don't know if you know that. And let's have a look. What does it mean? If you have a dictionary with pictures or with emoticons, and you dump it like with JSON dump with a little s, so dump string, then you see a surrogate pair, UTF-16. And this works with every JSON-compatible readers and writers. But if you don't like this, if you want to watch in your JSON file and want to see the correct UTF-8 configuration, you can use ensure ASCII to false, and then you see the emoticon. That's not recommended. That's a possibility if you want to see, if you want to have a look in your JSON file how it looks. Okay. There's a really mighty package in Python, Python Codex. There are defined about 96 languages, 10 BOM constants, so byte order mark constants. There's a surrogate handling. There is translation for EDNR, base64, UUN code, UUD code. You can talk about the kinds of encoding available in the world, I think, a whole week. So this one, if you're looking for translation of different bytes. If you are going to talk Pyth, if you're going to talk REST, for example, or to friends, talk REST, which is others, then typically REST is talking in JSON, and JSON we had before, but you can also send text and then you have to look if you are talking about the encoding, that's everything we heard before, and the language which is on top of it. And the most problems I had in my life is when people sending you messages and saying saying, hey, here's a message, it's UTF-8, but it's encoded in something different. And then the only thing you can do is running the bytes, looking at it, and try to reassemble them, and this is pretty much not nice work. But if you do it and then you can show, hey, you're sending me this and this byte, and it's not UTF-8, this is, of course, this one and this one, then you're usually a great hero sometimes in your project. So it's worth going down into bits and bytes sometimes. Okay. I'm pretty fast. So we have a lot of time for questions if there are and if I can answer them. To an end, you must come. So I learned typewriting in 1986. I don't know if all of you are already born, and I don't know if you ever wondered how strange the letters are on the keyboard, and that's because the hammers used to get clamped when you're typing really fast. When I learned I was not very fast, and the hammer gets clamped, and then you have to touch in it, pull them back, and then you can write further on. And so The letters are built in this way that really fast typers don't have clamps very often. Poor little boys like me in 1861 going to, I don't know how to put that on the machine. We still have another thing, which is line ending. If you are going on a typewriter, you have to return the carriage to start a new line. But if you start typing, this one is on the same line as before. So you have to feed the role, and this is the line feed. And this is still the result in line encoding, which is in Windows, carriage return line feed. In Linux and Unix, it's line feed. And an old Mac I heard it's just carriage return. And Python has a wonderful thing I didn't even know before this presentation. I know how it works but not the name. It's the universal new line mode. And this universal new line mode converts everything into backslash n. And if you're writing files, it's always dependent. And on Linux you write backslash n and on Windows you write backslash r backslash n into your files. And if you are going to produce Windows readable files on a Linux machine with Python, then you don't change the file itself. I think it works, but you don't do it. So you just have the backslash n in your file contents and with open context manager you You put the new line, backslash R, backslash N, and then you have the backslash R, backslash N in your files. I didn't know that either before I tried all absurd things. So pretty easy. And with a new line in the context manager, you can define what kind of new line you have. And yeah, we have three books I'd like to recommend. The first one is perfect coffee table book. It's all about printing 256,000 or something letters of the Unicode alphabet. You can put it on your coffee table to impress all friends which are not working in IT. The second one is about deep diving into Unicode. So what's with the first two bits in a byte, how is it with how many bytes belong to one character and so on. And what's the difference between UTF-8 and UTF-16 and things like that. And the third one I don't have to present you because this brilliant guy was also giving a speech yesterday. This is my favorite book. Thank you. Okay. And one last recommendation. This is a really cool website if you have Mojibakes and you don't want to install Mojibakes scanners and things like that. You just put your Mojibake text into the line and it's decoded and then you have the correct, the one I tried, the open the fifth door message already working. I think this is a really cool website. Okay. With this link you can get the presentation and all code snippets. The one I tried. And I hope you enjoyed the presentation. Thank you so much.

Speaker 2 [20:28]

So we have couple of questions, let's start from the online one which encoding do you suggest for JSON rest api's?

Speaker 1 [20:36]

UTF-8, definitely, if you are in Western countries. Oh, so, yeah, UTF-8.

Speaker 2 [20:43]

Okay, the next question is when should I work with byte mode instead of text mode when working with strings?

Speaker 1 [20:52]

So, when you should work with bytes, I never work with bytes explicit, but I always work with bytes implicit, because if you open a file, it's only bytes. There's no letter or char sets, whatever. If you open files, it's always bytes, and if you have byte stream, you always have bytes, of course. But explicit, I don't work with bytes nearly never, just to show things.

Speaker 2 [21:22]

Okay, thank you Martin. Any other questions from the audience? Please raise your hand We still have about 10 minutes for questions. So just feel free to ask anything Are you sure

Speaker 1 [21:38]

And everything answered.

Speaker 2 [21:41]

Okay, then we are probably done yes, and you can find Martin in the hallway anytime and ask your question a round of applause