Strongly typed datasets in a weakly typed world

Speaker 1 [00:03]

Hi, I'm Marco from and I'm going to tell you something about types. Actually, it's quite surprising, right? I mean, we are like having Python developers all over the place and I think most of you think about types but not about strong types. So, why I'm going to tell you this. The problem is I don't care about your types if you have a Jupyter notebook and if you do some data exploration and stuff. but I'm responsible for data engineering in production, and there you face different problems than like in research code, for example, because you have life cycle management, updates, and long-term data preservation like for months and years, maybe. So, why are types actually important? Let's have a quick look at small, tiny, and like small examples. and types can be like influence what operations you do and what the operation means. So if you look at the very first example and you have like, I don't know, a boolean, an integer, and a string, a plus operation might lead to very different results. Even more surprising are that some results, like if you have an object series in Pandas, for example, and you do like a negation operation on that, you get like a minus two, like what the heck is that? And you have like different other examples where your type actually influences your results and you want to be sure in production that this doesn't happen like all over the place, like types change and you get the results back that you didn't expect. So if you look at the stack of like maybe many like data science and machine learning companies, you think, okay, yeah, I just have pandas there. But that's actually not true, it's not just pandas, because you also have NumPy, and pandas is straightly integrated with NumPy, but there are differences in type systems and differences in how they behave all over the place. And then you don't only have NumPy, but you also have PyTorch, for example, and that works slightly different in some cases. Then, on top of that, you may now have Arrow, as Wes introduced in his keynote, because you may want to interop with Java and JVM or with databases and so on. Then you read data from databases because you need to get your data from somewhere. We have already developed Turbo-ODBC, it's one ODBC driver, but basically you map the ODBC type system somewhere in your Python world. And then you actually come to the point where you want to store your data and you sort it into packet data sets. And there, you have a completely more or less different approach on how types are preserved. Now, in the following slides, have a quick go through all these type systems. Very quickly, just to illustrate that all the authors of the libraries put massive efforts into what they think their types should be, how should they behave and how they should be converted into each other. But before I do this, maybe something about dynamic versus static. So here you can see basically two domains. So you have this thing that you have in memory and basically what you will also have in your Jupyter Notebook, it's like a dynamic domain. You have it, it runs, and you throw it away afterwards. It's also the part that you upgrade, libraries change, you optimize your codes, and so on. But at the end of the day, you need to store it somewhere. And that's where you have a static domain. And this is what you are going, the part of the code and also the part of the data that's going to bite you next year. Because this year, for example, this year, you preserved like, I don't know, strings for timestamps, and next year you wanna, or now you upgraded to proper timestamps, and you don't preserve any strings anymore. And you probably don't want to like, as shown before, just use plus and minus operations all over the place because that means something very different for strings than it does for timestamps, for example. Now, NumPy has a type system, and I'm going to upload the slides later and share it on Twitter and other social media so you can get it as a reference. So NumPy has it, Pandas has it, and Pandas even has now an extensible type system. PyTorch has it, Turbo-ODBC and the ODBC system has it. PyArrow has it and Parquet has it as well. And now the question is which type system do you want to keep as a ground truth and how do you think about it? Now, let's enter a perfect world for a moment and see how many people may think things should behave. Let's say you have Pandas and to store your data, use Dask, so if you've listened to some talks earlier, you've had the tutorial that Uwe gave, you may know, okay, it's just to take a data frame and you can just store it, and then you get on your, now as a simplification, on your hard drive, but in production it can also be like an AWS 3 Blob store or like an Azure Blob store, and you get a folder there starts data frame, and then you have a common metadata file, and I will explain that to you in a second, and you have partitions where your data is distributed over. Now let's have a look into one of these partitions. Okay, so you can basically load the data in. You can use PyArrow to do so, and you get an Arrow data frame, or an Arrow table back. Now it has some one month of data, it's like January, so it has 31 days. But if you look at the names and at the types, you already see, wait, in the original data set, I had a month, right? I had a month column. Where is that gone? There is no type information for that left. So how does Dask know how to recover all of that? Yeah, there's this magic file called common metadata, and that's basically the crown truth for the whole data set. Basically, there you have recovered your month information, and if some people may get a little bit confused about this underscore, underscore index thing, it's basically index information for pandas. But there, everything is stored. Now, before I go on, I will quick have, want to have a look at the arrow type system for a brief second because this is a type system that is very strong and very easy to reason about. So you have date types in different bit widths, and you have unsigned, signed integers, floats, and so on, and you also have some structured types like lists. For example, you can have a cell in your data frame where you actually preserve a NumPy array. And then you have dictionary encoded types, for example. All of that, I will come back later, why I think this is important. Now, why do types change over the place? I mean, many people think, okay, I load in my data, and types will just be preserved. Like, it's basically no issue. I mean, it worked all the time, and basically there's types are types, and I do operation on this, I still get the same type back. Now, there are multiple reasons for this, and here are some examples. For example, TurboDPC has this wonderful feature where you can say, hey, instead of, if I fetch data from the database, and my numbers are very small, why should I allocate 64 bits for every integer? I can automatically choose a way smaller type and it's memory efficient and makes things spacing fast. Then pandas, that's a question that came during the keynote is actually if you merge data frames together and you have a missing value, your integers get casted over to floats and you have a similar approach for Boolean operations or for Boolean types, for example. So Pandas converts types. Pandas also has another ZIP type system. You basically have categoricals to store your strings more efficiently. And even though all the operations that work for strings work on categoricals then as well, if it's a string categorical, the type that Pandas reports back is category. And basically, this is what then later on would map to what Arrow knows as dictionary encoded types because you basically, instead of storing the string all over the place, you basically only say, oh, I have a unique set of strings, in that case, one, two, and three, and I only store like a pointer or like basically an integer that says, oh, that's the first string, that's the second string, and so on, and can be very memory efficient. Then NumPy also converts types all over the place. So for example, you have a large array or matrix with just integer data in, and you get a mean operation, then it says, okay, I cannot express the value of the mean as an integer, so I automatically convert it to a float. Now, all of this sometimes seems to be fine, and sometimes it leads to problems. And I wanna show you some examples where I think this is fine, and some occurrences where this might lead to heavy issues. Like for example, let's get a very simple one in and say you have a 16 bit integer, you convert it to a 46 bit integer, a 64 bit integer, everything is going to be fine. You can just have more values in the target space and everything will map just easily. So if you do it the other way around, obviously large and very small rallies will be cut. So your conversion will destroy your data. Now, a trivial example, if you do like a same to the same conversion for 64 bits, everything is fine. But now I come to the conversion that, for example, Pandas does, and why I think that might be very tricky for some people and also for production use. So even though the value range of floats is way larger than what you could express with integers, There are some cases where your data gets destroyed. So now I've artificially constructed something that may look like a very large number. It is also very large, and the thing is, if you convert it back and forth between these two types, you get a very small difference. Basically, you lose some bits of information. Then is this okay or not? And I actually don't wanna blame Pandas, because Pandas sets up on NumPy and NumPy is numerical data. So for numerical information, that loss is really, really small. So it's, I would say, ineligible. But if it's like an ID number that you got from the database and you want to use it for group by operation, unique counts, merges, and so on, you probably don't want to be like these two numbers to be the same, even, or like different. You don't want that you get ID collisions all over the place and weird things happen in production use. So I would say this is, for us in production, a complete no-go. If it is numeric, it's better that you have a float, but if it's like an ID number or a hash or whatever you can imagine, it should stay an integer. Then you have similar things, for example, for Unicode and binary. Even though a binary is just very common, you can basically put every bit sequence in there, and Unicode maps completely fine to this. If you then look at the results, and you now have München, which is the German name for Munich, and you convert it into binary, and now see, okay, it's in UTF-8 encoding, and it doesn't seem to be the same. It's the same content, but like semantically, you mean something different here, and we don't want to have these conversions because we want also to preserve the semantics of our types. Now, what does it mean for type preservation? I said some data types are obviously compatible and others are not. Now, if you look at these error types, we could make a simplification to the whole type system. You have massive amounts of types here, but some of them are just basically the same. All these integer types are basically the same, just that you store a more or less compressed version of them. So if you say I only need eight bits because I have small numbers, that's semantically the same except maybe for hashes you can find occurrences where this is not the same. But for IDs, for example, it's the same as if you store larger integers. And the same goes for dates, times, and also for floats. So every float in 32 bits maps perfectly into the larger bit space. Now, getting back to this DAST data frame and why we are so worried about how it works or why we want to take care about this is because the ecosystem out there is waste. We basically have DAST, which is an awesome tool, and we also have Hive and Presto that can read these data sets, and we want to be compatible because I don't want to write one tool where I only can consume my own data. why can the Python stack not write data that can also be consumed by JVM, for example, unlike SQL engines that's on top of it? Because it's a very powerful system. Now, you still need to write something in this common metadata, so the question is, what do we write there? Because we don't want to lie about our data. We just want to be sure that when things are compatible when we change for example things over time that everything evolves with it so for example like the id number example now you have like i don't know an id for a few countries and you are enough with eight bits are these eight bits because you have 200 something countries maybe maybe it's not enough but maybe maybe it's a it was enough once and then you get more countries over time maybe like because humanity changes and history and so on, and now you just need more bits to store country information. Then what do we do? In the first month, it was okay. In the second month, if we add more data to our data set, obviously we don't match the original data type anymore. So what do we do? I know some people like functional programming and like Haskell and so on, so this is the slide for you now. And basically, we defined a very simple thing. So we defined all the, we said, okay, we have all these different data types that error gives us, and we do a very simple conversion. So for integers, floats, date, and time, we say we don't care about the bit width, and if we just write the widest type that we can find and write this into the common metadata. For lists, we do a recursion, and for dictionary types, we just strip the entire dictionary information. We say, for example, if you had a categorical in pandas, we don't care it's a categorical, we care that it was a string categorical and we preserve the string information. So, if you want to have some takeaways about this, it's basically that we say, okay, you have a dynamic runtime and we don't care about types there or we don't care too much because it actually works and it works in most of the cases. But if you store the data, you wanna make sure that it is readable and writable even in the future and your types can evolve and you have some freedom. So, for example, for bit width, we say you want to allow some variance and you want to not lie to all these DAS constructs and to the whole open source world. We just store the maximum with type. For categoricals, we do something similar and we allow compressed and uncompressed data, so we don't care if it's dictionary encoded or not, and we store the uncompressed type. And this is basically how we think we are compatible with the open source stack, and how we think also others could do it. So it might be very different for your company, or even for your project, and we invite you to have an open discussion about this, because in the end, all these tools should work together, and you don't want that you open your data with Dask or with Hive, that it gets completely messed up, and yet you read and write things that doesn't work with your stack. So I know that Dask currently has a slightly different implementation of this, but we want to encourage people to actually be aware that some of the assumptions that are made there might be dangerous for production use. And we're actually planning to open source quite a significant amount of code around these data sets in the future. So if you think you want to work with us and have ideas about this whole thing and how you think data preservation should behave in production use, just either contact me or one of my colleagues or come to our booth out there and talk to us. So, then thank you for listening and if you have any questions, feel free to ask them. Is there any question? Thank you for this very great talk. And actually this is an occurring problem that I think everybody of us faces. What I was wondering about, what are you now doing about the mergers and the joins? Do you now have a solution, how to keep the IDs as integers? Okay, so for integers, so there's, we don't have a production ready solution for this. We know that Pandas now, as they support these extension types, they also support one type that is an integer that supports null, like missing values. And for that, it would actually work to have this kind of operation that also preserves the type. For many other cases, we often find that integers don't appear as a value column in mergers. For example, you more often merge on the integer itself because it is actually your ID and your group by operation actually works on that one. So we don't face this implicit conversion that often. And if you think it should be a numeric type, you often just start with a float and say, okay. Actually, yeah, most of them are like whole numbers, but it's actually like a numeric information. Is there any other question? All right, so thank you very much, Michael.