A deep dive into the Arrow Columnar format with pyarrow and nanoarrow

Speaker 1 [00:06]

The first practical point, so you see a link here to the Github repo material, either link or the QR code. There are some instructions in the read me to install if you want to follow along with the code examples. So you can install the packages. Small note for people using conda, if you want to run all examples, you need to pip install PyArrow because people already has the latest PyArrow from two days ago, Kona not yet. Small note. Normally the binder link should also work if you want to run it online. And if you downloaded the material already earlier this morning, you might need to do a Git pull or download it again because we made some small updates. Then to our slides. Just a quick introduction to ourselves. So we are all, yeah, I will do the agenda afterwards. We are all three working at Volton Data. We are working on the open source side of the company, working on Apache Arrow, mostly PyArrow, but also related packages, non-Arrow, UBC, Arrow-related open source technology. Photon Data itself is a company working on composable data platforms, also accelerated by GPUs. That's a very, very brief summary. I'm Joris. I'm from Belgium. Apart from Arrow, I'm also one of the developers of Pandas and involved within the geospatial stack, so GeoPandas Shapely. So that's a.

Speaker 2 [01:58]

I can try this one, so you don't have to move. Hello, my name is Alenka. I'm a mathematician, but after a couple of years of using open source tools, not even knowing how everything works, I started contributing to open source with Apache Arrow, which was interesting. And now I'm all into open source, really like to talk about that, so if anybody's interested in starting contributing, talk to me, I'm happy to share information.

Speaker 3 [02:28]

Okay, and well, I'm Raúl Cumplido. I'm also an Apache Arrow committer and PMC. I've been involved in the Python community for some time. I was 10 years ago, indeed, in EuroPython on the first very Python, PyCon Berlin, I guess. It wasn't called PyCon Berlin yet, but yeah, it's, you know.

Speaker 1 [02:52]

Okay. That's our slides. Maybe just quickly the agenda here. Yeah. So we'll introduce various parts of the AeroColumnar format. But let's maybe just go to the slides directly. Then I will go there. So you will see here if you open JupyterLab to follow along, there are a few notebooks. I will start with the first one here, intro. So the topic of the tutorial is the arrow columnar formats. So therefore, very brief, like, intro to arrow itself and the general project. So maybe a question. You can raise your hand if you think you have a very, like, quite okay idea about what is is the Apache Arrow project. Few people, but certainly not the majority. Apache Arrow is also very, like, there are many different aspects about Arrow. But where it started, the original starting point of Arrow is illustrated by this figure. And I will maybe zoom out a little bit. It's not too important that you can read what is on the exact boxes. It's also a bit outdated. It's a figure from when the Apache Arrow started in 2016. Actually, not entirely sure about the year, but something around that time. And the problem that it wanted to solve was that there are a lot of data systems, let's call it like that, that interact or that you have to work with, that all have their own data representation internally, that all have their own conversions to all the different kinds of data systems. And so you have a lot of deserialization, serialization overheads, a lot of reimplementation of things. And that was the starting point of Arrow. Let's define a standard, a common way to represent tabular data so that at least we can reduce this overhead. And if everyone can talk in the same memory formats and can talk to that, we can make those things much more efficient to move data from one system to another. That's the core of Arrow, is this format. How is tabular data, like data frame, a table, however you want to call it, and you load that into memory, how is that laid out in memory? That's the core of Apache Arrow. That's the columnar formats. That's what we are going to focus on in this talk. But the Apache Arrow project also grew into a lot more than that. So in addition to the format itself, there are also specifications about how can we move this data from one process to another. There is inter-process communication, RPC protocols. There is also interfaces, how can we share this data within the same process, but between different runtimes, between different libraries. something we will show at the end of this tutorial. There have been implementations of Arrow in a lot of different languages of the formats. So for all those languages that are listed here, there is an implementation that implements the format itself within the Apache Arrow project. There are, of course, also projects outside of the Apache project itself that will implement Arrow or is compatible with Arrow. But these are the languages for which we insight the project we have in implementation, but with varying functionality. Because some of those projects, and PyArrow is one of them, also started to have a lot of other functionality on top of just this memory format. So for example, the PyArrow implementation, which is based on C++, can also read and write a whole bunch of file formats. So PyArrow is mostly used probably for reading Parquet files, or certainly in the early days. In-memory analytics processing, like the Rust implementation, the Data Fusion component, which is now moving to a separate project, but that was essentially a whole career engine on top of the Arrow Rust implementation. So there is a variety of levels of functionality in those implementations. But the core of it are the specifications. And yeah, that's the columnar format. We are going to focus on the columnar format itself. We have a few code examples, but don't expect that you'll learn a lot of new code today. It's mostly about the mental model about Arrow. The code examples are mostly using a tiny part of PyArrow and nanoARO just to create an array and inspect the data that we created. PyARO is the main bindings to the ROC++ library, the main implementation in Python. In addition, nanoARO is a newer package. And the main motivation by nanoARO is that pyARO is a big dependency, can do a lot. As I mentioned, it can read Parquet files. It can do CSV files, other org files, read from a whole set of file systems, do a whole bunch of compute functions. So it can do a lot while Nano Arrow is really focused. If you just want to receive some arrow data and import arrow data or export arrow data just around the memory format itself, that's what Nano Arrow does. just that and provides bindings in Python, in C, in R as well. So that's the scope of NaNoWriMo. We will use it because it has some nice functionality to inspect our array to see how it's laid out. Last thing for me for the introduction, so it's the arrow columnar format. So it's for table with columns or data frame with columns. But it's also columnar in the way that it's laid out. So if you look here at this small data frame, it's four columns with some interior string and flow data type. You can store the data. This data is stored in memory. You can do that in a row.

Speaker 3 [09:41]

RAW FASHION

Speaker 1 [09:42]

fashion and so many of the traditional databases are row based where the data of a single row is stored closely together in memory but yeah if you want to do an operation on a certain column that means that it is less efficient and you will have to jump if you just want to sum all the the first column for example it's yeah not nicely to get every memory, so you have to jump, so it will be less efficient. On the other hand, for traditional databases, for transactional database, where you mostly want to work on inserting, updating rows, it can be a very nice way to store your data. But columnar formats, and most of the more modern analytics-focused databases, but also the data frame libraries, will all use a columnar format, And that means that the data for one column is stored together. And that's the main idea of a columnar format. And that ensures that you can, yeah, what I mentioned before, data is together in memory. You will get a better memory locality. You can do, like, SIMD optimizations to efficient processing. In general, yeah, processing compilers work better when data is close together. So that's the aspect of the columnar formats. And so for the rest of the tutorial, we'll go through some of the different layouts, the different types that we have in Arrow and how it's actually laid out in memory. Okay, so then we can go to the first set of layouts.

Speaker 3 [11:34]

Okay. Okay, so yeah, we are going to start with the primitive layout. One of the ideas of the tutorial at the beginning was that even us arrow committers have some problems sometimes to understand how these formats are stored. So it was an exercise of learning how they work, et cetera. So hopefully we will do a good job explaining everyone how those work and yeah I mean I'm gonna start just yeah porting as Joris said num nano arrow by arrow etc and I'm gonna start with the fixed size primitive layout So the fixed size primitive layout representing data that has the same width, so each value has the same width. In this case, we are having a couple of columns here, integer and float, and you can see that But well, integers in this case all occupy the same space and memory. And in the physical layout part, you can see that we have several buffers. That's how we are going to be showing all the different formats. We are going to be showing how these buffers are stored in memory. I'm going to start just with the first column on this case. I'm going to omit the first buffer, which is empty. We will see why this is later. But in the case of the fixed size primitive layout, the thing is that we only need one values buffer. We don't need any offset. We don't need anything else to represent because we know each value how amount of memory they occupy. On this example, for example, we are going to represent four bytes per each value. So in 32s, we create an array and we will see that it's an in 32. With PyArrow, we can inspect the buffers. I will go a little bit later through how PyArrow and NanoArrow are used. If we inspect how many buffers do we have, we see that there's one with none. We will see later. But this is a single buffer with all the values there, yeah, 20 bytes, blah, blah, blah. And if we use nanoarrow, we can pick the array and inspect a little bit more, and we can see, for example, that we have a single values buffer with all the data that we created. Okay. How we are doing this with PyArrow and nanoarrow? Well, first, with PyArrow, when we create any type of array, we can inspect the number of buffers that we have, so you can do that with any type of layout, and we will see what are the buffers, their address, and if the buffer is mutable or not on this case. So I mean this is basically the same that we were showing before. As we were saying with primitive, as we don't need to say how much each data occupies, it's just a values buffer, and in this case it's just four bytes each value, so we know there There are five values because the size is 20 on this case. Yeah, so yeah, we can see the first buffer again and we can inspect the value of the buffer. Yeah, as I was saying, we can see that each value occupies like four bytes. We can see the, well, the values, the one, the three. I mean, this is an hexadecimal, but yeah. Yeah, and then, yeah, the different values that we've created with the buffer, each one occupies eight, four bytes. I mean, it's just, in this case, with NumPy, it's showing on int8. If we say that these are int32s, we'll just say the same arrays.

Speaker 2 [16:03]

Hello, yeah, the interesting thing is that the buffers don't know the data type Right you have information about buffers as role mentioned, but you don't know the data type, but in nano arrow Yeah, it's gonna get nicer. So the buffer in pi arrow has the basic basic information How the length and how much space it occupies? That's it. I think that's it. Yeah

Speaker 3 [16:30]

Yeah, the difference with nano-arrow, as you can see, is that the buffer knows that it's an int32, because really we are not using a buffer, we are using a buffer view where we also store what is the data type that we are storing, et cetera. That's why we are using nano-arrow to view, because it's a little bit nicer. Instead of seeing all this hexadecimal with all this padding of bytes, et cetera, we can see that it's an int32, and that it has five values, 20 bytes, et cetera. So we can just pick the initial data buffer and then if we pass this to numpy, we don't have to see that they are in 32 because the buffer view already is sharing that it's in 32 instead of just one byte. Yeah. Okay. So, well, PyArrow has the concept of missing values or nulls. So that's where the other buffer that we were seeing that was null will appear. So in PyArrow we have a bitmap buffer that represents where the nulls are. So in the case, I mean, it's an optional buffer. In the case of a fixed width where we have all values, we don't require a bitmap buffer because we don't have nulls, right? But if we have nulls on our column, as is the case on the float, we have this validity bitmap buffer that is telling us on each bit which element is valid or not. The validity bitmap buffer is ordered with least significant bit. So that's why, I mean, if you can see the first bit here is representing the first value. This is a valid value. Second value, third value. Oral valid values. And the third value is not valid. It's null. So we can skip when we are parsing the data, so we know which values are valid or not. As I was saying on the first example, that's why when I was inspecting the buffer, it said that there's, well, we already see that it's the validity, but this is not occupying anything basically because it's optional, so we are not having this buffer. Yeah. So yeah. I mean, we will see it here. We have the floats array, and then when we inspect, as I was saying, we have, yeah. One thing, as you can see probably, I mean, if you inspect, this is ordered the other way around to map correctly, but in memory we are really storing with least significant bit, so that's why they are swapped in the example. So in memory, if you would go with C++ and see the memory and what is the value of this validity bitmap buffer, you will see this example, but with NanoArrow, we are already just swapping that. So in this case, it's a single byte because, I mean, with one byte, as we are having only five values on our column. We are already having enough bits to represent what are the values and if they are valid or not. But if we just make this bigger, I don't know, just by multiplying by two, for example, we have a bigger array. Then we will see that it will occupy two bytes on the validity and then it will have, instead of 40, the double because we will have double, sorry. So in this case, we have two bytes for the validity buffer, and we have the values buffer with more values. And we just pad with zeros because we don't care what it is in the end. So, yeah. Okay. Yeah, this is the value of the validity bitmap. In this case, in hexadecimal, it's two bytes because I increased the values. But, yeah, so it's not a small array of five. It's ten now. But, yeah, that's why it's two bytes instead of one. Yeah. Yeah, so two values, the two bitmaps, and then I don't know what we have here. Yeah, just the ones and the zeros that we can just unpack the bits from NumPy with the buffer and the ints and we have to say that the bit order is little, otherwise we should see, yeah, the difference. Okay, yeah, there are several libraries like NumPy that does a difference between none and null because they don't have the concept or indicator for missing values as none is a specific value for the floating point specification, that's a difference with ourselves, so if you have a null value in our validity bitmap buffer, you will see that none is already it's a valid value, because it's a valid value on float, so that's why we have it's still marked as in the validity map as a valid one. Okay, so I'm talking a lot and I'm nervous because I'm not used to do that, I like to be at home on my cave coding, so I think some of you can relate. So I'm going to go and yeah, I mean, I would like you to take a look and I mean, date times are also specific with value, so we don't have to have other offsets or arrays that that we will see in the future. So yeah, if someone can take a look and tell me what is the type of the array or yeah, how this works. There are some hints but yeah.

Speaker 2 [22:48]

And while you're thinking and thinking hard and trying out all the things you learned, we have a slide of four questions and we're going to answer them while we have the tutorial if it's connected to what we're speaking, okay? All the general questions we'll take later. And we'll also find if you just raise your hand and, you know, interrupt us, okay? Because it's nicer that way. We see that you didn't float away and, you know, in the cave in Spain.

Speaker 3 [23:19]

I mean, Spain is nice.

Speaker 2 [23:20]

Nice. Spain is nice. So the question, this array, if you run it and you check it, it's a data type, what's its memory layout? Anybody? It's broken. Yeah, it's broken in our case, so we need help. How many buffers?

Speaker 3 [24:07]

How many buffers does a column date time has?

Speaker 2 [24:16]

I hear some typing. I hope it's connected. Yeah. Is there any missing data in that array? Because I don't see the whole one. No. No missing data.

Speaker 3 [24:33]

Three date times.

Speaker 2 [24:33]

Three times. Okay, so does it have a validity bitmap buffer?

Speaker 3 [24:34]

Okay.

Speaker 2 [24:40]

No. we're still under the i mean the hint is also we're still under primitive layouts so you have a data type data time what would it be similar to which kind of type in memory yes an integer if you have an integer column it has validity bitmap if it has missing values this one doesn't and a data

Speaker 3 [25:27]

Yeah, so a date time is another example of a fixed width if the type contains a byte with Attribute it means that it's a fixed width because we know how how long it takes in memory So yeah, we can we can have it in this case. We don't have any None, but if we add an on then the bitmap The the buffers will change and then we will have a second buffer with the values as I said it's optional. If you have nulls, it will appear. If you don't have nulls, then I mean we at Arrow like to save memory. I don't know why. So I guess that's why people like Arrow. So yeah, we avoid having a buffer whatsoever. So yeah. Okay. So yeah. This is probably pretty uninteresting so far because we have some ints and floats and we have things that are slightly more interesting I guess. If we move to variable length binary strings or things that we don't have affixed with and we don't know how much are they going to take in memory then we have different ways of saying, okay, how long this string is or how long this binary is, et cetera. I think we all can understand how this is going to be. So for us, we store all the string and all the binary together in memory. As Joris was saying, that's one of the things for the column format. So we have to say where the value starts and where the value ends, right? So the difference here, there's some information about what's the difference between a binary and a string. For us, in the physical layout format, you're going to see that it's exactly the same. The only difference is that strings are valid UTF-8 binaries, so they can be represented as a string. That's basically the only difference. But, yeah, that's basically it. So, yeah, another example. We have some string data in this column. We have several strings. We have a missing value. So as in general, you will see this a lot. We have the valid bitmap buffer, where we represent which values are valid, right? As we were saying, so the three first values are valid. We have a missing value, and then we have another valid value. And then we have all the values together concatenated in a single buffer. That's the data buffer, where we have the Python data conference Berlin. And then we have an offset buffer. The offset buffer is going to represent where each value starts and ends. So we know that the first valid value will start at 6, at 0, and then 6, and then from 6 to 10, 10 to 20, 20 to 20, because it's missing. But we still need to have that, because the offset buffer, it's always represented as an int32 in this case. we will see how the large binary strings or the large binary or large string are represented but we know exactly if we want to go to the X element where it is because we know how many in 32s we have to skip so we go we see how long it takes then we go to the values buffer and we can get directly to the information I hope it's clear So, yeah, that's an example of binary. As I said, it's exactly the same layout. We are using some strings. We are seeing some hexadecimal, because at this point, we are seeing binary, so we don't know what values they are. But if we show the bytes from the hexadecimal, we will see that it's basically the string. If we directly use the type for a string, we will see the string when we show the representation of the string array. The same buffer that we had before, if we see the buffers, as I was saying, now we have three buffers. We have the validity bitmap buffer, we have the offset buffer, and we have the data buffer. We represent them. The first one, as I said, the validity bitmap buffer, so we know that we have three valid valid values, one missing value, and a fourth valid value. And if I take a look at the offsets, it's exactly what we had on the example. So we know that the first value is from the zero. It's gonna have six characters here, gonna occupy. And as we said, this is int32, yeah, the values buffer, all the values. And if we expect with the column with nano arrow, again, we see that it has a length of five strings, it has three buffers, the validity buffer, where are they, and the data offset, and the data. So yeah, binary type, as I said, is exactly the same, three buffers with the validity bitmap, the offsets and the values, and if we inspect, we have exactly the same. I mean, yeah, okay. So at this point, can someone tell me what might be the difference between string and large string? I kind of hinted, so. So we have this offset buffer, which is 32 bits, I was like for a second, I'm confused. That means that we can represent values up to a point with 32 bits. I think it's a bunch, but yeah. So if we want to have data which is larger than two gigs, we require the other one because the value is going to be bigger than two gigs, so if we want to say that this is coming from this position to more than two gigs, we require a larger string. So instead of int32, we use int64 to represent what the large string here. So sorry, here it's just the normal string, as you can see. we are creating a buffer and here we are gonna create a large string buffer. Yeah, can also someone tell me what's the difference between these two? What's gonna be the difference between these two that I'm gonna create? Apart from, I mean, yes, it's gonna be in 64 but what does that mean? Lots of zeros at the end. Lots of zeros at the end. So each one of these values, so this is occupying one byte, right, because it's the validity bitmap. This is occupying 24 bytes because it's, I think, 6 in 32, so it's 6 by 4, 24. And this is 26 bytes because it's just the values buffer. Yeah? Only the offset will be twice as big? The offset will be twice as big, yeah, that's correct. Because we are going to be using 64 bits to represent each value, so the offset is much bigger. But we can represent that data, which is, yeah. Yeah? I have a question. What would it look like for the integer types when there's missing value, the buffer still deserves space for that value? column it looks like that that does not happen

Speaker 1 [33:35]

Does this not happen because it doesn't have any offset?

Speaker 3 [33:37]

Does that mean that if you fill in a missing value in a string,

Speaker 1 [33:39]

value in a string column all of

Speaker 3 [33:40]

string column, all of the subsequent

Speaker 1 [33:41]

of the subsequent items need to be copied.

Speaker 3 [33:47]

I'm the release manager.

Speaker 1 [33:52]

Sorry, no, but it's a very good question the that's one of the things with if you have a fixed width Layout like if a fixed size type like in 64 That means that yeah the the length of the the data buffer is like predefined like you have ten elements There are four bytes with three or forty bytes That means that the place where there is a missing value which is masked by the validity bitmap still needs to be filled in And arrow, like the arrow C plus plus limitation and pi arrow will by default use zero, but the specification doesn't specify that. It can be anything. So you will see, I think we saw it before with the integers or also here with the floats, like the case, like the last, one but last value here was none. And when creating that, we filled in zero. It could have been any value. But indeed, it still takes space in that data buffer. On the other hand, for the string data, here you have a missing value as well, so it's indicated here by the zero here. But at that point, it still occupies space in the offset buffer. So, again, you could, in theory, you are allowed to put any value that is being masked by the validity bitmap. So in theory, we could have put here, like, a long string. It would be useless, of course, but that's... So the default that Arrow uses is an empty string because there is no increase in offset, so essentially that means you have an empty string at that place. If you would view this array without its validity bitmap, then you would have an empty string in that place. can you speak in the yeah so the offsets are uh yeah monotonically increasing right so is that a requirement or couldn't you also save space by actually saving in this case zero six four ten like just the length okay but it would be slower because every time you would into some, right? Yeah, I suppose the main reason that we use offsets and not just lengths is that to know the... Like, if you're the 10th element, you need to know where in my values, my string buffer, then you need to sum the previous values to know where in the data buffer you are. Because now you have a direct index. Like, if I want to, like, give me the third element in this array, then I go to the third element in my data offset, and this is the index that I use inside that. So you have a relatively cheap access. You don't need to compute anything except first accessing the offset, like your index into the string buffer. But for this type of string data, it's indeed a requirement of the specification. The offsets need to be strictly increasing or equal for empty strings, but strictly increasing. But that's the next topic that's where we will continue with, there is also a newer string data type.

Speaker 3 [37:21]

Yeah, also you would have to go through all the missing values to find out In which point you have to go to the offsets and then find the value. So yeah, I Knew the answer, but I didn't catch the question Yeah, so Yeah, we we have added some new layouts this Yeah for strings Those, yeah, are used or have been used in DuckDB and Velux and we are going to see what is the difference with the existing strings and why those are interesting. So yeah, we have this example, for example. We have a column here with some strings. And again, we have our validity bitmap buffer. The difference with the previous one, as you can see here, is we have a views buffer. And in this views buffer, we have several information. For a short string, which is smaller than 12 bytes, we can have the full data on the views buffer. So this is the case for short strings. So for example, in this case, we have a 5, Then we have a short here, in this case we have 12 and we have the shortest string, right? So if the length is less than 12, we can have the full data on the views buffer. If the length is higher, we are going to split the information on the views buffer where we will see what is the length of the string, what is the prefix or the initial part of the string, and then which buffer or which values buffer contains the string, and then the offset inside of the values buffer. So why is this interesting? We can do, for example, if we want to do comparisons for strings, et cetera, we don't have to read all of them. We can just take a look on if the length is the same and take a look on the initial data, so comparisons are much faster for the initial part of them, and then we can read the rest of the string if we want to. So it's easier for this type of operations, and yeah. For the values buffer, we can have several. So for example, if we here concatenate or have a different values buffer, in this case we are having a new column where we have a new added values. So yeah, we can have different values. So I'm going to just show it because it's easier with code usually. So yeah, what I was showing, we are going to see the different, again, it's the same example as we had before, the column, we have a missing value. If we see the buffers, we are going to see three buffers. We are going to see the validity bitmap, the views buffer, and the values buffer, and if we start taking a look at them, again, validity bitmap, I'm not going to go through it again. But then if I see the views buffer, we are going to see the values that I was suggesting. So we have this string, the first one is 21 bytes, so we know that here we are going to have the first, the prefix for the string, then it's in the first views buffer, and this is the offset, and then the next one is going to be five, so we know this is going to be the full value of the, these two are going to be the full value of the string, et cetera. So if I go and see the views buffer, here, as I was saying, the first information, the first four bytes, we can see that it's a length of 15, it starts with a string, and then, as I was saying, that's the part where we have the, in which values buffer it is represented or is stored, and what's the offset. In this case, we have a five and we have the short string already represented, et cetera, and then the values buffer with all the rest of the values there. Here we've done a little bit of work. So what we are doing is we are picking the views buffer, getting just the first 16 bytes, which are the length, the prefix, the views buffer and the offset, and we can see that if we represent the first one as an integer, then the four strings, then two integers, we can see the values that I was suggesting. In the second case, as we know that it's maximum 12, even though it's length five, we just fill up with zeros, because this was the short string, et cetera. And then if we concatenate an array to the first column as a string view, we will see the example that I was suggesting, that now we are going to have one more buffer, because now we are going to have two values buffer, and the views buffer is going to point to the second one so if I do this with a little bit of magic we just can see that the first one is just a prefix with what's the value this is the full value this is the missing value full value again just prefix prefix

Speaker 1 [43:09]

There is one relevant question in Slido that is asking, what happens if I want to change a string in an array? For instance, if I want to concatenate something to one string in the middle, how do you update all the buffers? That's certainly relevant also for this type. So concatenating, it's not exactly what is being asked, but what we showed here in the example is actually a concatenation of two arrays. And in the case of a string view, we don't need to, like, the two buffers that represent the actual string data, that will be, in most cases, occupy most of the memory, can actually just be kept. Because the only thing that we need to do is to, yeah, concatenate the views buffers and ensure that the index to, if it's the first values buffer or the second values buffer that is being used, that that is updated in your fuse. But that's a cheaper operation. So you can see here, so the last string here points to the second string buffer while the other point to the first. While with the default string type that we had before, where you just have offsets and your data. Whenever you change a value in there or you need to change a value or you concatenate, that means that you always need to copy and combine your full data buffer as well. So the data buffer with all the strings. Because that's how this layout works. It's always in exact order of your data. Not only for concatenating, but let's say that you are ordering your data frame and you need to switch orders of values for this string data type because it needs to be nicely consecutive, you need to actually rewrite your data buffer with the string values so they are in the correct order, while in the views case here, you could essentially just reorder those views because they point to a certain buffer and a certain offset in that buffer. It doesn't need to be nicely increasing order. It can just be out of order given that we separately store the length and the start and in which buffer it is. What's the purpose of the prefix? The prefix, as Raul mentioned, a typical use case for that is to shortcut, like a short circuit in like comparison operations, like if you want to compare, you have two string arrays and you want to check where values are equal, you can, like, because the views are like fixed width, it's much cheaper to access those values, you can quickly check the prefix, if that is not already equal, then you know the two strings are not equal. And it's only if the length and the prefix is the same, then you actually need to look up the index, go into the full string buffer and look up the full string. And that means that you can avoid doing this lookup in your variable length string buffer for every value. That's a typical, like, there are other places in other algorithms where you can use the same or a similar trick, but that's a typical example. I saw is CPU attribute in buffer, maybe can you give some more details, the difference, how it's stored? Like, in buffer, yes. Yes. What's the difference on underlying data structure? If it is true or false? So the arrow format can live on any device. So you can have arrow data on the GPU as well or on other devices. So this is CPU is essentially that. So meaning that the data lives on the CPU and that's ROC++ can access the data. If you interpret this pointer, yeah, essentially what it means that this pointer points to an address that the CPU can address. Because if it would be like a CUDA device pointer, then yeah, it would crash if you would try to access that from the CPU. So yeah, that's essentially that. So RO is not limited to a certain hardware device type. But that's mostly works for CPU. It has a little bit of functionality for CUDA, but mostly limited to CPU.

Speaker 2 [48:15]

And also the layout is is the same. I mean Yeah

Speaker 3 [48:23]

Yeah, it's where the memory address is pointing. Okay. You want to continue? Yeah. Okay. Yeah, so, I mean, this example, as you see, we are creating some binary which are fixed with, and I wanted to, yeah, ask the audience if you can tell me what's going to be the difference with the binary or string that we've seen before. Basically, how many buffers do we have? Because we like to see memory.

Speaker 2 [49:46]

Correct. Yeah, yeah.

Speaker 3 [49:47]

So, and it won't have a bitmap either because there's no null value, so it should have a single buffer with the data. So, if we inspect, we should see that we don't have, yeah. So, that's the difference between fixed width. we know that each value is going to be on four bytes, so we don't need the offsets because we know exactly where things are. It's the same as when we are storing ints or timestamps or other information.

Speaker 1 [50:29]

Okay, there are a few questions on Slido, like we will try to, if you have time in the end, answer more general things. There are a few questions, yeah, that I want to answer now. So one question was, what changed with data that it's now smarter to look at columns instead of rows, or was it always smarter, but we just didn't realize it? So that's a question about the row-based memory layout memory layout or the columnar. To be honest, I don't know the exact history there. I think why databases were using row-based. There are certainly benefits to row-based as well, for transactional use cases, and use cases where you don't necessarily care too much about the analytics performance, like the typical analytics, like take the average of a certain column. but you're more interested in like inserting, updating things, then there are many good databases that are not columnar. Yeah, the recent year, there is more.

Speaker 3 [51:42]

Yeah, in the past we, I mean, it's now when we are doing a bunch of analytics and a bunch of information with data, but in the past we wanted to just store transactions and new users and what was going on and storing this data. It was just easier on a, I mean, appending data, it's easier just on a row-based format because we just have to store all the information so we don't have to regenerate columns, etc., so yeah, that was the initial.

Speaker 1 [52:07]

One other question now. So someone asked about I installed the requirements and the model NanoARO has no attribute array. That probably means that you're using NanoARO 0.4 and we need the development version. So in the readme, there is a line how you can, using a special PyPI index to install the latest NanoARO.

Speaker 2 [52:33]

I will continue with nested types. A lot of information until now, huh? Everybody okay? Okay. So if we sum up what we learned, we have the simple things, we have integer types that have one data buffer and maybe if it's missing, you add the validity bitmap, then it gets a little bit complicated if you use strings because then you need an offset to know where it begins, where it ends. But if it's a fixed size, you don't need that again, so we're back in the primitive. And now we get to the nested type. Nested means that there's an object in an object in an object, right? And in Arrow, we use the child array's name and a parent. So we have a parent array that can have one or multiple children, right? And then this child can also have a child array, which are all arrays. Before it was like buffer level, like array and the buffer, but there is like a first array which is parent and then it has child array. And that child array can have buffers also, right? So this is the relationship that we use in Arrow. And also the parent array is the nested type, but the child array can be a different type. It could be primitive, integer. It could also be nested. It could also be a type, like a list, okay? So this is an extra thing that we'll learn now. So the first simple case is a list. It's like one, so if we're looking at columns and we have one slot, and in that slot we can have a sequence of values which are all the same type for the whole column. Yeah, and for the slot. So, yeah, the offset, similar as in the string type, you need the offset to know where one slot begins and where it ends, because the values are stored again together, all together in one buffer in the child array. Does that make sense? Yeah. So, you have a parent, which is a list, yeah? And that list has one data type of the value. So here it's an integer. That's a child array, not a parent array. Parent is a list, and the list only holds the information of the missing slots, not elements in the list, but slots, and the offset where the first slot starts in the child array buffer and where it ends. So the child array will have information about the child, which is, again, a validity bitmap if in case one of the numbers would be missing, like not the whole slot. And then the values buffer all the values together for all the sequences, one after the other. So this is the diagram. Is it useful? The diagrams are useful? Okay, good. Thank you. So this is how we construct it. Okay, this is how it looks like in PyArrow. So when we inspect the column, the list, what does it say here? Oh, yeah, it will return all the buffers. So the list has two buffers, if you remember, and then the child has also two buffers. So the first buffer will be if the slot is missing or not. The second will be the offsets. The third will be if the individual elements of the slots is missing or not. And then the last one will be the actual data. So do we have a lot of time? Maybe I'm just going to skip these things. Because it's, yeah, we're just going to, like, here we take one buffer after another. And we kind of put it back in with NumPy so we can see. Because, like, okay, buffer address, we don't know what it is. So you can use NumPy to actually get the data in. This is the validity bitmap. This is the offset, which if you check, it should be the same. Did we mention that the offset always has one element more than the column? Did we mention that? I don't remember. Does it make sense why? Because the end, we need to know where it ends, and it's like one extra. Okay. So this is the validity bitmap for the child, which is missing. So we check if it's really missing. It is. I mean, it's none. None means that there's no memory needed for that thing. Okay. And this is the values. And I really like nano arrow. This is just beautiful. We have all the information together, right? So you can see a list has children, just one in this case. And then you have information about the child, which is itself an array. Okay? And one thing I should not forget when we get even more complicated is that even the nested types are columnar. So we don't store even the nested types are just column by column. Okay? All the information, all the data. Oh, exercise. Okay. I think you'll know this one. It's kind of similar than it was before. So if we take this second example, and if you run it and check the buffers, what's the difference between the first, this one? So I'm going to let you run it and check and, you know, have hints what happens with the offset buffer. So the difference is here in the definition of the type. There's one difference. It's It's not the same type. It's still a list, right? And also the array is a bit different. So the difference between the first one and the second one, it kind of, the slots look like similar length, right? Is that correct? Am I correct? If I check the first one? Like the first one was length three, then it was four, and then it was empty, and now we're going to a second version of the list which would be fixed shape, right? So this one has a shape of three. So each slot has the same length, and that's how we call it in PyArrow. We say, okay, this is length three. So if I say, because I like NanoArrow, but this keyboard is new to me, okay. So did anybody find any difference in the buffers? Nobody. Yes, thank you, who was that? Good job, thanks. So yeah, it doesn't have an offset because it doesn't need it. We know the length of each slot. So similar than before in the question before, which was great, we have to fill it somehow. We don't need information for the offset. But then if something's missing, we need to put something there. So the validity bitmap has, like, too much zeros. And also the data, as you can see, it has three zeros there. That means that that's missing. So it's just the default values there, okay? I think there's another thing I have to mention here. No? Okay, thank you. Thank you, Joris. Joris is our beautiful person. Yeah, question. I love questions. I didn't think it through, but that basically means if I have a lot of null values, the fixed size might be bigger than the variable size. I don't think, yeah, I mean, yeah.

Speaker 1 [60:54]

Yeah, that's correct. It's a trade-off.

Speaker 2 [60:57]

Trade-off. Trade-off, yeah. So yeah, this is our diagrams, representation of fixed size list. You can see there's no offset, and yeah, it fills in the values, correct. Any questions about the lists? Okay, good. List and large list, similar then for the string, if the offset needs to be bigger, need a large list. That's basically it. So if you check the offset, this is the previous one, and this is the large one, you have just a bigger offset, okay? What's the time? No, I'm going to skip this one, it's similar to the string view, except it's for the list, so you can have, on the left side you have columns, but the buffer, the values buffer, I'm sorry, camera, as you can see, it doesn't really match, right, oh, thank you, it doesn't match, you see, the values, because normally in Arrow, everything that's written, it's It's from bottom to, from up to bottom and it has to be consecutive. And here it's just mixed. Right? Do you see the difference? Yeah. Thank you. So this views, this list view can give you ability to make out of order lists. You have a buffer and you say, okay, from beginning to middle I have one slot and then somewhere in between the other one. So again, you don't have to do things with buffers. If you already have something, you can just use that and not make another memory copy. OK, skipping to struct. So lists, a struct is also an stdata.

Speaker 1 [62:58]

Sorry.

Speaker 2 [62:58]

Sorry, maybe.

Speaker 1 [62:59]

Maybe quick, because there was a question about the list. Why is it necessary in the offset buffer to start with 0? You can see, yeah, so just one, whatever.

Speaker 2 [63:12]

The diagram, right? This one, you mean.

Speaker 1 [63:15]

Um...

Speaker 2 [63:16]

All the offsets. No, no.

Speaker 1 [63:17]

No, if you go back, any output that you have there is fine.

Speaker 2 [63:17]

Okay, offsets.

Speaker 1 [63:22]

Yeah, here. So the data offsets always start with a zero. In our example, it starts with a zero, so can't we just leave it out? No, because it's not required to start at zero. It's only because the way that we create our data, like with small toy examples, the default, it starts at zero. But you can have, like if you would slice your data, you take a slice of a list array, the only thing that it does it takes a slice of your offsets it doesn't need to slice the child array and then your offsets will not actually start with zero so it's not a requirement it's just in our examples

Speaker 2 [64:00]

It's a good question. So I think it would be worth to have an example with sliced. It was you Very good to have an example with sliced array because I think this is we should know that. Thank you struct Okay, it's not as fun fun part. There's gonna be a really interesting example afterwards that you're gonna have to Help me with okay. So the struct is a nested types, but Like in the list, we had a list of values, like a sequence of values. Here you have a sequence of types. So there's a field, a new thing to learn. You have fields, one, two, or more, and each field has a child array. Okay? You have one field here, which is a name. The field name goes into the schema, which is a description of your data structure. It's not in the buffers. And the values of that key, so the values of the array, go into the buffer. You can have multiple ones. So this example is, oh, yeah, the child arrays, yeah. So here we have two fields in this example. The child arrays are separate buffers. So if you have five childs, they have separate buffers. They don't have to be together in memory. Okay, and all the values for each field need to be the same. So if you have two fields, first is here is a string, they all have to be strings, all the values. And the same for the second, which is ID. So the struct, this example, so the parent type only has the validity bitmap. No other information. All the other information is stored in the child arrays. So here this strut has two child children arrays. One is a binary string. And the other one is fix site primitive which is just integer type. I think that's enough of information. You can have three, four, five. So it will all be stored like a field as a name and then the value as an array. Does it make sense? Is it clear? Okay. So here this is an example of the construction of this diagram's data. This is how it looks in PyRO. So the buffers, again, you have struct buffer, which is the validity bitmap, and then you have three buffers for the string type and two buffers for the integer type. A lot of buffers. Okay? And if you use nano arrow, it's a really nice way, isn't it? It's really clear. You have two children, first one is string and the second one is integer. It's really clear with all the buffers listed. Okay. This exercise, I like this exercise. You have to create this nested example, which is a struct of a list, and a struct of a list of a string, and then the second field is a list of a fixed size list. Okay? Which is, it's nice. You can get creative with these types in Arrow. And my question is, this example, how many buffers does it have? And I will wait for the number. How much time do we have? We have 20 minutes. and I mean yours is still up if I mean it's worth listening to yours so you should be quick and we'll go together just like if somebody has any idea you can have also you can help yourself with some code that you learned oh the hints yeah I'm sorry I just want to show that oh that one and the hints we have hints ah this is so easy with the hints isn't it I mean, you can use buffers method and just list them and just count.

Speaker 3 [68:26]

Yeah, they asked me this morning and I got it wrong.

Speaker 2 [68:32]

Anybody, just a number, or we should go together? I don't give out beers. I don't know if... Yes? Bravo. Yes, 11. Very good. One for destruct, right? And then you have the first list, which has validity bitmap, offset, and one child array. The child array is a string, right? It has a validity bitmap, an offset, and a data buffer, which we are at six, which is destruct and the first child. And then the second child is a list, which has a validity bitmap, an offset, and a fixed-size list, which is a validity bitmap and a child, which is an integer. But it's a simple integer with only two. I don't know why I counted. You said child. Okay. So it has two validity and the data, final data. 11. Very good. Okay. So we're at the last, then I'm off to have a map, but we're at the map. A map is, it's interesting, it's similar, I mean, it's, to me, I needed some time to have a real clear difference between a map and a struct. A map, so here what it says, represents nested data with a key value struct, so you have a variable number of key value pairs example on the the keys are movies dark night this is actually not my example it's from the doc strings that somebody contributed thank you very much really good example so the the movies and then the this is the keys and the values are the ratings from one to ten um so the difference between the struct and the map is the The first thing, the struct names are saved in the fields. This is the keys, which is in a schema. In a map, the keys are a separate array, right? So this is dark night, dark night. It's all one array of keys. In the struct, the keys are field one, one key, field two, second key, and they can just be string because it's a descriptive, it's in a schema. In a map, the key can be a number, it can be string, it can be whatever. And the values are also a separate child just like the map. It can be a string again, it can be an integer, but all the same type. So value has to be one type and keys has to be the second type. And another difference is, as you can see, a slot can have multiple elements, a slot. But it can only be one key array and one value array. Because in a struct, you can have multiple fields with multiple values. With multiple child arrays. Did I do well? Kind of? You did great. Thank you. So a map has always two child arrays. A struct can have ten child arrays. map always have two child arrays. It has a key child array and a value child array. And this is the example. Okay. You can run this. You have the notebooks. You can run this yourself. We added a comparison of how the struct and map converts from PyArrow to Python. Because it's not really easy or simple to understand. Just run it. There's descriptions. Does it become a dictionary or a list of tuples or what does it become and what are the options to convert that? So have a look at that and I'm going to leave it to yours. Thank you very much.

Speaker 1 [72:35]

There are coming in a whole set of questions. I will try to answer some of them at the end. Or also already some of them answer in the last notebook. So we have seen different layouts. We have seen in the primitive layout, we have seen some example of types using those layouts. But there are, yeah, that's only a subset of arrow that we have seen. Yeah, this diagram will not be very readable in the back. I think zooming in also doesn't help because it scales again. But I will do my best to quickly explain it. So this diagram has an overview of all the types that are using a certain layout. So, we have seen the fixed size primitive layout, so just one buffer of data and you know how many bytes each slot takes, so that are typically the numeric data types that we have seen, like int floats. Also some others, like temporal types, like timestamps, dates, times, duration, those are typically stored as integer values. In the case of timestamp, for example, it's like the number of seconds since 1970. And so it's stored as an integer. And that's also using the same physical layout in memory as an integer array. Same as for the fixed size binary, decimals is also fixed with a certain byte width. So in this case, there is a version with 128 bits or 256 bits. The bool the same, the only difference there, bool is also a fixed width, but bools essentially just use the same idea of the validity bitmap, so it's just bits. So for example, in numpy, a boolean array is a byte, like it's one byte per value. The boolean array in arrow is one bit. Using the same logic of the validity bitmap. So, for example, a bool array with missing values has two bitmaps, essentially one the validity bitmap and one the data buffer. There's also a null layout, which essentially means there is nothing. It just stores how many values are in that array, and they are all null. I won't go into detail. We have seen the variable size binary layout for binary and string data, and then the newer variant for this with the views instead of the offsets. Then we have seen all the nested types. So we have seen lists, fixed-size lists, a version of list view, destruct. A few of the types that we haven't seen are... There is also a union type in its own layout. What is a union? Essentially, I have an array, and I know that it's either an interior or either a string. It's one of the two, and so it's like an abstraction around that. Essentially, what it does under the hood, it stores all the ints, all the strings, and then it keeps track of my first element is a string or my second element is an int, and that's the way that it stores like a union type. The map that we have seen, which essentially under the hood, it's not a separate layout, so it doesn't really fit in the diagram here because it's actually just a list layout with a child array being a struct. Dictionary and runtime encoded are essentially two types that I was thinking I can quickly show the dictionary encoding. So in this case, there are also nested types because they have a child array, but because of the encoding, they can do some compression, like some size reduction of your data. So in the case, like dictionary encoding is used if you have a lot of repetitions in your data. Let's say I have ABA, so a string array. I dictionary encode this array. Encode. So, this results in a dictionary array. And then a dictionary array, what it will do, it will store the unique values separately in what it calls dictionary. And then the actual data, so which has the same length as like three values in this case, are then indices into that dictionary array. If I would look at this with nanoARO and inspect it, you can see how it's laid out. I didn't import nanoARO here. A row as NA. You can see that the data itself, the buffers of the data itself, you have the validity and the data, which are my indices. And it has a separate concept of a dictionary, which is the child array, but it's, yeah, in the format, it's a dictionary is a separate concept from a child array, although like logically it's very similar. It just stored separately in the, or mentioned separately in the layout. So that's dictionary encoding. Run-end encoding is essentially if you have a lot of repeated values that are next to to each other. You can say, like, I have 1, 1, 1, 2, 2. And then you just say, I have 1 three times and then 2 two times. So essentially, that's another way to encode your data. That are all the building types. Maybe you have, like, you want to define your own type of some application-specific logic. And Arrow also has a concept of what it calls extension types. Extension types are nothing more of I'm going to use one of the existing types, but I'm going to add some custom interpretation. Like I have integers, but I know they represent something else. Essentially, timestamp are integers that represent the seconds since 1970, for example. you can have your own custom interpretation of the data. Examples are like UUID, you would store it as like fixed size binary, but you can annotate the type as I know these bytes should be interpreted as a UUID. Some examples of building extension types, so there are certain extension types which are like standardized by the Aero project itself, and they are called canonical extension types. And those are the fixed and variable-shaped tensor data. And currently there is discussion about also standardizing this UID and standardizing JSON as just a text blob, but where you know that the text represents some JSON value. There are also a few community extension types, and one of them where I'm involved in myself is geo-arrow, where we are going to represent geometries in arrow formats. And essentially what you have here is, yeah, you have different ways that you can represent geometries. But in the end, a geometry is like you have a point, which is an X and Y coordinates, and you maybe have a line string, which are a collection of points. And you can, in the end, store this data very easily with arrow using, like you store your X's together your store your Y's together, but then you can use like a list of lists to determine like like I have a polygon which Has rings and each of them is a list of a list of like you can make very nice Use of of this nested data types that we have seen a question The geo-arrow project has some library packages that implement conversions between them. It's like well-known text is a different way to represent so you could also store your geometries in like a text in a column of strings. But that means like the main advantage of like the native arrow, like the more native arrow way is that if you just want to access the coordinates, like you want to know what's the min and max x and y values. For the text one, you need to parse your text, while in the case of the arrow, using like a nested type of just floats, the whatever nesting you have of the list of list of list, your final x and y is just two arrays. I have one array of all my x's, one array of all my y's, and can just calculate the min and the max of both arrays. So you get a very nicer, yeah, layout to actually compute things on than and something like well-known text. The idea would basically be to just use well-known text to get the data into PyArrow? Yeah, that's certainly possible. The last thing, I don't have much time anymore, but I want to address a few questions as well that were related to this. So in the examples, we have often been passing some, like created some data with PyArrow and pass it to Nano Arrow to inspect it, there have been questions about why do we do that or which one of the two should I use and is there a conversion happening. So maybe just checking the questions again. So the first thing is Nano Arrow is a very small library and only deals with the formats. So if you, for some reason, like you receive arrow data from somewhere and you just want to, for your own application, I want to get the buffers, I just need the memory address so I can pass it to some other C++ or Rust function that just needs, here is the data, then an arrow is sufficient and you don't need to rely on something like PyArrow. PyArrow comes with all that additional functionality. For most end users, PyArrow is what you should use. The reason that we use NanoArrow here is because it has a bit more, because it's lower level, it shows the real layout of your data, which for the purpose of this tutorial was actually useful. Most end users will want to use PyArrow. We have been passing NanoArrow or PyArrow data to NanoArrow, or it could also be the the other way around. And that actually happens just zero copy. And that's not only for PyArrow or NanoArrow, but that can happen, that works for any library that implements what is shown here, the ROC data interface. And that's part of the arrow specification that allows libraries within the same process to just share data, because all those buffers are predefined how they are laid out. You can just say I'm a Python library and I can pass to a Rust implementation of Arrow. Here is my buffer, here is the memory address of my buffer, this length is that type and they can just interpret that data without any copy. And that's when you pass data from PyArrow to NanoArrow or the other way around. They will just share the pointers to the data and the data type and they know how to interpret it without any copy. So that's the C data interface, this is not specific to Python, so it works with any language that can implement a C extract or a C form function interface. I don't have time to go in detail here on the actual struct, but essentially you will recognize things that you have seen in, like, the nano arrow, so you have animation, like, you have the, yeah, the buffers, children arrays, and also the length and null count, but yeah, essentially the buffers and the children arrays. I had a small example here to show that it's actually zero copy. It's something you shouldn't do at home to mutate buffers under the hood, but to prove that it's zero copy, so I create a string array with PyArrow, I pass it to NanoArrow, and you can see now it's the same strings, but it's actually zero copy. To prove that, I take the string buffer, so the third buffer of my PyArrow array, I convert it to a numpy array, this is something that also happens zero copy because of the buffer protocol of Python. Then I mutate my buffer, the string buffers, and it's in bytes, so I'm converting here the exclamation mark to the interior representation of those bytes. And then you can see that actually the first four bytes of my PI array is now exclamation marks and And the nano error array has the same mutation. But yeah, certainly don't do this at home. That was our material. Yeah, I don't know if I'm allowed to answer a few more questions or if we have to stop. Maybe we do three more minutes, so pick one or two. We have 12 questions. We will not be able to answer them all. Let's do three more questions, and then we end. Please keep seated, and then we go for coffee, OK? Yeah. So I will then just go from the most voted. Is Arrow supposed to replace Parquet? No is the simple answer, because it's two different things. Arrow is the in-memory layout. Parquet is an on-disk file format. There is some confusion, because you can also dump whatever Arrow has in memory to disk, and you also have a file format. But the Parquet is really optimized with a whole bunch of encodings and compressions to have efficient on-disk storage, as small as possible. That's not the use case of Arrow. But they work very well together, because when reading Parquet into memory, Parquet is also a columnar file format, has more or less the same data types as Arrow, so it matches very well together. When working with data in Arrow backend, does Arrow automatically choose the best representation, for example, for strings, by looking at the content. Arrow itself doesn't do that because Arrow is also, like, there is no such thing as, I mean, Arrow as an implementation. I mean, you have only, like, you have separate implementations. And I can answer that question from PyArrow point of view. And PyArrow doesn't do that. It will not. So it has a certain default string type. Like, if you don't specify the type, you will get the string. You can ask for a string view. But I would say, in general, this the PyArrow wants to be a low level building block and it's the application, if you would build a data frame library on top of PyArrow, it's them that could decide, okay, I see the user has strings, maybe in this case I should rather use a string or a large string or a string view and they can hide that complexity. PyArrow itself doesn't do anything smart except like infer the basic data type. How much more space efficient is our format compared to others? Yeah, I'm not super familiar with other formats in the same space, like, or other data system or databases that have their own formats. I know that, for example, the newer data engines had their own string view because it was more efficient but that's now being adopted by Arrow as well and being standardised. Maybe the last question, is there any way that one could pass the Arrow memory pointer to another process? That's not possible. But it's actually possible to pass it to Java. So within the same process, you can use a CDATA interface as well to pass data to Java or, like, to the GVM within a Python process. But to actually another process, we typically have the IPC protocol where you have one serialization step. But because it's all, like, the data, how you pass it over to the other process is exactly the same layout as it's in memory. It's very cheap. Like, of course, you need to do a copy to send it to another processor or to send it to another node or cluster or another machine over the wire, but because it's exactly that format that you send as it's loaded into memory, you get a very performant way of sharing data between processors. Very good. Thank you, Alenka, Raoul, and Joris. Let's give them another round of applause.