Exploring Zarr: From Fundamentals to Version 3.0 and Beyond

Speaker 1 [00:05]

So yeah, welcome everyone. This is 23rd of April, 2.10pm and we are at PyCon Germany and PyData Berlin 2024. The title for my talk is like exploring XAR from fundamentals to v3 and beyond. I'm not gonna lie, this is the title that was suggested to me by chargpt. So yeah, I wrote the proposal myself and I thought I couldn't figure out a title so I was like just give it to chargpt and see what it says. I was like okay, let's go with this. And yeah, I take yes on the section where you have to say that you've used RGPD. So yeah, I didn't lie. So yeah, my name is Sanket Verma and I take care of the community and the open source software of XAR as their community manager. I've been in this position for more than two years now. And if you like this talk, please check me out on Twitter. There's a QR code for that. The slides are basically on this QR code and you can also go to this short link if you want to follow along. I'm just going to wait for like ten seconds before I move ahead. Everyone got it? All right. And I also uploaded the slides and the notebook to my GitHub repository. So if you want to check out the notebook and stuff, you can also use this. But there's going to be another link for the notebook at the demo session. So yeah. Okay. Moving on. So we'll be mostly talking about, like, going from, like, fundamentals, basically, like, what is XAR, and which will cover, like, basics using illustrations, like, neat graphics. Then we're going to go towards, like, what's new in the XAR spec V3. And then I'm going to talk a little bit about the community, like, how does we form the community, how do we accept proposals, and how do we actually work on the specification. And we actually have a nice process which is known as ZEPS, which is similar to PEPS. It is known as ZAR enhancement proposals. And there's going to be a small demo of how the things are working in ZARv3. Starting with the basics, so basically arrays are the container of items with the same data type and size. And the number of the dimensions and the items are basically described by the shape. So here we have a 1D array, which has a shape of 7, then you have 2D, which is 7x7, and then we have a 3D, which is 7x7x2. So this is how you basically organize your arrays into container items, and it's necessary to have the same data type across all the items. So basically this illustration was made by this person, Trevor Mann, so yeah, huge shout out to him. So yeah, I mean, let's say if you have a data set that you want to work with and it's small enough to basically load into your memory, store into your memory, make changes to it, it's fine, right? Let's just say you have 100 megabytes of data set, you can easily work with it. But what if the data gets too big and it's 1,000 gigabytes or something like that? You won't be able to load it, store it, or make changes to it. And that's where basically XAR kicks in. And what XAR actually does, it basically divides the big array into small equal parts. And this process is known as chunking as you can see and the small equal parts are known as basically chunks and we compress those Each individual chunks and we have like compressors which are like state-of-the-art compressors from various Organizations and actually there's a library known as numcodex which is a dependency of czar which is also hosted in the czar developer github organization and which basically helps to compress all these small chunks into it and When your array is compressed and divided into small parts, it's basically a ZAR array. And the retrieval basically works in a way that, so let's just say if you want to select the chunks which are borderline with orange selection area. So the ZAR will only load the chunks which are in this area, the violet one. Not the entire array. So let's just say if you are, have like a thousand gigabytes of array and you want to load only the, I don't know, a few gigabytes of that, and ZAR will only load that array and you can store it, load it, make changes to it and save it back to it. It saves a lot of space and headaches when you're working with big data sets. The assignment of the keys and the values basically work similar to Python dictionaries. The value here are basically the keys. The keys are basically the values which are assigned as 0.0, 0.1, And the values are the actual data encoded in binary format. So this is basically how stuff works in XAR Python. So in Python you have this mutable mapping function, set item and get item, which basically over here you can see like the .00, this is the first chunk in the array is assigned this binary encoded value. And when you want to retrieve this particular chunk, the get item functions call for it and it basically gives, you input the key and you get the value, it decompresses the chunk and you get the actual value. And you want to make some changes to it, you store it back, you compress the chunk, then the binary value of the chunk is changed because you changed it and it's stored back into the main array. So this is essentially the basic working of how the chunking and the mapping works in the XAR Python ecosystem. You can organize multiple arrays into hierarchies which are known as groups. So this green column is known as basically this green block is known as the metadata which contains the actual information about the XAR array which includes the data type, the chunk shape, the size of the array, any special attributes you want to add to stuff. So this was basically the version 2, like how things are looking in version 2, but there's actually a neat graphic which shows how we evolved from version 2 to version 3. So coming back to the specification, so basically specification is actually based on a specification which is a technical document which lays out how your metadata should look like, what's a chunking strategy, how your hierarchies are basically organized, what are the attributes, and what type of data types do you have. And it's basically a technical document, anyone can read it, it's over here if you want to have a look, and it contains version 1, 2, and 3. And this is essentially based on which the XAR software is written. And it has implementation in all the languages. So, I mean, it's evident enough that the Python community is big and large, so the XAR Python library has the most number of users, but we also have implementation in C++, C, Rust, Julia, JavaScript, Java. And implementations are basically growing day by day. So I think there are three Rust implementations as of today. Two Java implementations, one Julia, and I think three JavaScript implementations. So the community is growing really fast. So basically the thing is like the version 2 was working really well for us. And things were going good. But like the question comes back like why did we start working on the v3? I think the first commit goes back to 2018 or 19, if I'm right, yeah, when the work started for the V3 specification. And there are multiple reasons why we started working on the V3 specification. So the first thing was the interoperability. So V2 was version 2 specification was originally designed with the ZAR Python implementation. So it basically relies heavily on the NumPy machinery and fundamentals. And which basically the all the other programming languages kind of feel like left out. So what we did is like we slimmed down the core specification and make it more language agnostic so that any language can implement it. Then there was the high latency storage. So the V2 specification was developed keeping in mind the local file storage. But there are certain operations which have like high latency when you're dealing with cloud object storage like Amazon S3 or Google cloud service or Azure. And so basically this was basically removed, this high latency storage operation, the high latency per operation time was reduced by consolidating the metadata into a single document. And earlier we saw that there was like multiple JSON documents and right now they are basically consolidated into a single document. And the last part is like which is really exciting one is the extensibility. So over the years, ZAR has gained a lot of traction in the various domains like genomics, geospatial, data science, bioimaging, and nuclear. So people in these domains are using ZAR heavily for their storage needs. And all these domains have various feature requests or various needs that they want to add to the ZAR. And V2 specification didn't have any mechanism to add this feature into the specification. So that's why we started working on this new concept, which is known as extension mechanism, which allows you to add new features to the ZAR without touching the core specification. And this is one of the most exciting part of the V3 specification. So yeah, let's look at some of the major design updates that we made. So we researched the whole JSON metadata. So JSON metadata that was in V2 is completely different from what it looks in V3. And so this is what the V2 looks like. So this is a simple XAR array, which chunks ordered in a manner. And you have the Z array, which contains all the information of the XAR array, like the data type, chunks, shape, name, and everything. And Z-atters contains basically the custom attributes that you want to add. So let's just take, for example, this is a data set of an image of a tissue or cell taken from a microscope named X. So you can add this information into Z-atters. So this is what the V2 looks like, and this is what V3 looks like. So what we did is we consolidated this z-array and z-atters into zar.json, and there's one small difference if you see here, these z-array and z-atters have dot in front of them. So basically they were hidden, you can't see them unless you do like ls-a. And zar.json is actually visible in the directory list. And all these arrays are basically organized in multiple directories, like individual folders based on what the shape of the chunk is. So you can see the v2 and the v3 comparison is like v3 is more structured and more consolidated as compared to v2. So this is what a single array looks like and this is what the groups look like. So I think you guys already saw this and this is what the v3 looks like. So in v3 there's a difference like every so there's like a top level jar.json which contains the attributes and the node name. And over here, at the bottom, you see there are two individual arrays. This is similar to the ZAR.json that we saw here. So basically containing two individual ZAR arrays and having grouped them together into a hierarchy which is known as ZAR groups. And you can organize multiple arrays in this manner and which could be as deep as 100 node levels or maybe more than that. So it's like a tree structure. Yeah. So we also added like the explicit support for features via extension mechanism. So I'm just going to give an example of what it looks like. So these are basically the various metadata fields inside the XAR array. And these are like array type, like how the chum grids look like, what's the encoding, what are the codecs, and the storage transformers. So this is the actual metadata, what it looks like. So you can see, like, the data type over here is like float64. And the chunk grid is like regular and encoding and codex. So all these various fields could be manipulated using extension mechanism, and basically, for example, if you want to manipulate this data type field with extension points. So if you see, like, over here, the data type is float64, and over here we have defined a new data type which is known as date time and its configuration is unit in nanoseconds. So whenever you propose a new change for the ZAR core specification, the metadata field will change and all the associated properties with it. So one of the most shining examples is extension mechanism in sharding codec. So sharding codec basically allows you to group multiple chunks into a shard which is known as a collection of multiple chunks. You might be thinking why do we want to group multiple chunks in a shard? Why can't we just leave it there? Because when you're dealing with cloud storages, sometimes the chunk numbers go in the order of 10 to the power 6 or something like that. And when you want to do input output operations and you want to read these number of chunks, it's going to get really hefty and the latency will increase many folds. So basically sharding allows you to read and write through these multiple chunks in a very efficient manner. And this, I mean, it's similar to like what we saw in the XAR dot, like the XAR array. You have a XAR dot JSON which contains the metadata file, and we have shards which basically groups together the individual chunks into small, small packets. And every chunk is a compressible unit, because as we saw, like, XAR basically compresses every chunk, and every shard is a storage unit. And if you want to read more about the proposal, like how it was proposed and how it was accepted, accepted, you can go to this QR code. Moving forward, and, yeah, so, just example, like, how would this look like in the sharding codex? This was the general metadata, and this was the codex, so sharding was implemented via codex. This codex was gzip, and this field would get modified once we enabled sharding in our data set, and this is what the sharding codex would look like. Basically, you expand this field into something like this, which which basically states everything, like the sharding index, the chunk shape, the codec, and everything. And this is the CRC32C checksum, which basically validates the integrity of the individual shards. Yeah, and more extensions are basically coming soon, so if you want to have a look at what the current extensions are proposed, you can go to this link, and the most exciting or the recent upcoming extension is the Zeb 3 which is known as the variable chunking. So currently the chunks are basically in a regular shape. So as you've seen earlier, the chunks are divided into equal shapes. But variable chunking will allow you to have, like, chunks of various sizes. And this has, like, various applications in the field of, like, genomics and transcriptomics data. I'm always confused with that name. So yeah, it's still in process, like the POC is ready, and we are waiting for the voting phase. When the voting phase is passed, then this will be actually included into the Python implementation of ZAR. These are a couple of things that were renamed. If you compare ZAR v2 metadata with v3, chunks are replaced with chunk grid, dimension separator is basically replaced with chunky encoding and the separator is changed from dot to forward slash as we saw in the, okay, yeah, over here. So the separator is basically changed with dot with, so initially it was like 0.1, 1.1, and now it's like foo slash c slash 0 slash 0. So that would be the address of the first chunk. And if you have 0 slash 1, that would be address of the second chunk and so on and on. Okay so this was basically the brief of what the V2 specification looked like, V3 specification looked like and now I want to focus a few minutes on what the community is like and how we basically reached this point where the V3 specification was accepted by the broader a community of users. I take care of the community and I think in my mind the community is the mix of the open source software and specification, users and developers, basically contributors and maintainers, governance and council, mix them all, easy to use, it's like a good thing but a bad thing also and you get something like that. So we have a large and diverse active community. But there's this way that we want to achieve the standard format. So Czar is basically used across all the different scientific domains. But achieving consensus is a very hard thing. You need to basically go through various processes. And that's why we needed a structured way to solicit and process the feedback. Because the discussion dates back to 2019, and there was a lot of stuff which I needed to solicit and present it to everyone. And this basically led to the creation of the Zepp process, which was written by me with the help of the others in the community, and how we did it, so it was mostly like reading a lot of the existing processes and how they work, which includes the PEP, and NEP stands for NumPy Announcement Proposal, and STAC is something which is used in the geospatial world. I have previous experience of dealing with basically making processes for the community organization so that helped a lot and also understanding the needs of the community which basically led to me like creation of the ZEP process and here's a blog post if you want a deep dive and if you end up making a process for your community or a project this blog post is really helpful like how I went through all the stages and how we made this. And how do you adopt a ZEP? It's simple like we have this representation from every other software which our base zones are and everyone gets to vote like if this feature should be enabled if this feature should be added or not, and you kind of like lay your sword, like speaking in a figure, and the zip is adopted. So just quickly going over the demo. So this has the link for the Google collab notebook. I have it here on my, so I have a normal notebook, and I can just go over it. So basically, currently, the Zerita, which is the derivative of ZAR, has the POC implementation of ZARv3. And it is basically currently ported from Zerita to ZAR, and the main release is due somewhere around the first week of June. And you kind of create an array in a normal way, and you mention the store, and you mention the shape of your big array and the chunk shape. So basically your array will be 16 by 16, and your chunk shape will be four. So it basically will let you have like four chunks in manner, like when you divide four by four by 16 by 16. And you mention the codex, which codex you want to use, and if you want to add any attribute to it. So this is what the array would look like, and the JSON would look like, the metadata JSON would look like something like this. And you can also open your array using the open function, which is similar to like array.open. And if you're doing the, if you're opening a sharded array, sorry, if you're creating a sharded array, you need to mention the shape of the chunk, then you need to mention the shape of the big array, the shape of the chunk, and also the shape chunk of the sharding codec. So this would basically result in having four sharded, four shards which will have, like, arrays of, like, two bytes of size. And this is similar to looking at the metadata is somewhere you'll see the it's not neatly organized but you can see the codex would somewhere mention yeah, okay, over here. The coding name is sharding index. So the fields would change when you change the configuration of your array and how you want to store the metadata. It is similar for creating a group. So I basically stored a group into multiple hierarchies. You can see there's one array, two levels down the hierarchy, and you can see there's group number one, group number two, and group number two actually contains the array. So there's like XAR.json at the top, and XAR.json at the other node, and XAR.json which is the actual XAR.json for the actual XAR array. And if you want to navigate inside the group, the top-level metadata would look something like this, which is like the XAR format and the node type. The other metadata, which is still node metadata, which would look like the XAR format 3 and node type group. the third one which actually contains the array which looks something like this which has all the information and you basically open the group using legsr.opengroup. It sounds cool but how does it look like? If you want to see, basically if you scan this code you're going to go to this link which is the oh, sorry. Yeah. So this is actually a V3 data set of tissue of the mouse's brain. And this was basically mentioned in a study by, I mean, you can cite this study. And this whole data set is basically in V3. And this is basically rendered in Web Gnosis. So Web Webgnosis is a visualizer tool which lets you visualize ZAR arrays in an interactive manner. You can interact with it, you can make changes to it, you can annotate it and stuff. And there's also another visualizer which works really well because Webgnosis has already implemented V3, so that's why it's able to support the V3 datasets. And there's also another visualizer by Google which is known as NeuroGlancer. It also has implementation of ZAR V3, so you can actually visualize your V3 datasets into NeuroGlancer. And it's going to take some time to load it, but it will look something like this. I think we are towards the end, but we have biweekly community meetings, ZEP meetings where we discuss the ZEP proposals, CoDev meetings and Office Hours. So if you want to learn more about it, please visit it, and here's the link at the bottom. And I think that's it.

Speaker 2 [23:23]

Thank you. I'm not seeing any questions on the slider, but maybe there's questions in the audience Don't be shy

Speaker 1 [23:49]

Yeah, thanks. Very interesting talk. Probably a question you get a lot, but how does it compare to HDF5 in terms of reading speed? Yeah, so ZAR has a native support for cloud storage. And, I mean, I haven't used HDF5 that much, but in order to get the cloud's inability in HDF5, we need to jump through, like, many hoops. V3 was basically developed towards keeping the cloud storage in mind, because all these domains which I mentioned earlier, like genomics and geospatial, they have humongous data sets. So just for example, if I go to, I have this, I maintain this small web page, which is r.dev slash data sets, and come on. And you can see there are data sets which are worth, like, one petabyte in size. And these are stored in a Google cloud service. So it helps, it actually helps and basically this functionality which I mentioned sharding which basically enables like less latency input output operations when you're dealing with cloud storage. So I think that would be the most highlight, spotlight point I would mention over HDF5. But HDF5 is great. So I mean if you go towards the, so this is the actual documentation of XOR and it's going to take some time to load. So basically the design of ZAR was motivated by HDF5, so. Go ahead. I'm not personally using ZAR, but I have colleagues using that to store satellite images in the cloud. But it's really massive amount of data. It's massive amount of data, like almost terabyte of data and I fear that there is quite stress of having to reprocess all the data to move to this new version 3. Okay, so basically I was in the meeting yesterday where we were discussing how would we lay out the path for converting the existing V2 to V3 data sets and we are basically working towards a command line tool which would help you to convert the data sets from V2 to V3. But I suppose the scale of your data sets are in maybe 100,000 gigabytes or something like that. So I think we should talk about this, like how we're going to get to towards the v3.

Speaker 2 [26:17]

Got it now. Sorry. I had to log in as the admin couple of minutes, so These may have not been asked of course how to manage the problem of the increase in size of the metadata Was that already asked? Okay? So here is the question so how to manage the problem of the increase in size of the metadata file when your data set is increasing over time

Speaker 1 [26:44]

Okay, so I think yeah, I'm gonna be honest, but I think this hasn't come up so far where the problem of increasing the problem of the large metadata metadata has Occurred so probably this is the edge case that we still need to discover. But yeah, it's still unexplored for now. I mean usually the size of the metadata files are not that big so if I can just And these are actually the array and things I showed you, and this is like what, 498 bytes, and it's like JSON file, so I think it's edge case, and we'll see when we get to it.

Speaker 2 [27:31]

Cool, thank you. This may have already been asked, what does cloud-native mean here?

Speaker 1 [27:36]

Cloud-native means your format can be stored easily on the cloud and retrieved easily. And you can make operations to it via using the library itself and not using any property or third-party tool.

Speaker 2 [27:57]

This breaking change means that all existing data sets using v2 have been have to be converted to the v3 compatible format

Speaker 1 [28:06]

Not exactly, I mean, you can still support v2, but there's an option for converting from v2 to v3. It's not going to break your existing data sets.

Speaker 2 [28:18]

Thank you, how customizable is the compression of the files? Can you apply different compression techniques?

Speaker 1 [28:24]

Yeah, definitely you can do that and actually, you know, we have moved towards the Sorry not this one We have actually moved towards the new Specification which allows you to lay out all the codecs So currently we have listed these codecs which are BLOSK, bytes and CRC20 and you can also see sharding over here So if there's any new codec you want to add to the current specification and you want to enable it for your use case you can just create it over here and we'll be more than happy to assist you. So it's basically like giving the power to the community and letting you guys do the work for your use case.

Speaker 2 [29:04]

Alright, we've got one more. How is the relationship to OME czar? Will it also move to v3?

Speaker 1 [29:12]

Yes, so OME-ZAR is actually a convention built on top of ZAR. Think of something like ZAR is the parent and OME-ZAR is like a children. For the uninitiated in the audience, OME-ZAR stands for Open Microscopy ZAR and this is a format which is suitable for data sets which are coming out of microscopes. So OME-ZAR is a convention of ZAR and the second part is OME-ZAR will be moving to v3 but that will be coming right after the ZAR Python moves to v3 which is somewhere around June. There's actually an issue somewhere, I mean, if the person who asked the question who can just go on Google and search ZARv3, only ZARv3 migration, they would see a whole issue and all the discussion.

Speaker 2 [29:54]

Great, we're at time. Thank you so much. Let us thank our speaker