Open Table Formats in the Wild™ - Reloaded: Vortexing Ducks over Floating Icebergs

Speaker 1 [00:03]

Good morning. We are going to have the next speaker. I'm Sumaya Nalukwaga, an IT professional, and I'm the session chair. So our next speaker is Franz, and he's an open source and Python enthusiast. He's a father of three girls, and he majors in psychology. He plays chess as a hobby, and he's a former competitive player of the ultimate frisbee. He likes cooking and baking sourdough bread. Let's have Franz take us through the next session. Thank you.

Speaker 2 [00:39]

Yeah, also very warm welcome from my side. Really happy to be at PyCon again and to talk about open file and table formats. And this year it's going to be all about Apache Iceberg. And for good reason, I think. It's fair to say that Apache Iceberg has become the most widely used and adapted table format out there, which also becomes, if we take a look back last week, there was the Iceberg Summit in the United States, It's a two-day conference, in-person conference with more than 700 participants solely dedicated to Apache Iceberg. And one thing that really makes me happy is to see open-source-minded people forcing or driving the development of Iceberg across many different companies, such as AWS, Google, Microsoft, Snowflake, Databricks, companies who actually come together and collaborate on an open standard that will hopefully really improve the life of data engineers in the future. The aim of today's talk is to cut through the hype and also to see what is really the status, the current status of Apache Iceberg. So I hope by the end of the talk, for one, you will understand why it's called Vortexing Ducks Over Floating Icebergs. And for the other, I hope you will get a little guide on when to use Iceberg and when not. I divided the talk into three main chapters. First of all, we will talk about the past in a sense that we want to understand what was the original goal of Apache Iceberg, for one. like what was the motivation, and then understand how Iceberg actually solved all these issues that we had before. Once we have a good understanding and intuition of how Iceberg works, we want to move to the present and see what works well and what doesn't work well. What are the cases where Iceberg still struggles, for example, streaming workloads and also AI applications requiring random access. And once we have a good understanding of what doesn't work so well, we move then into the future, and we will see competitors to Apache Iceberg and Apache Parquet. For one, there's Vortex. I'm really excited about Vortex. And also there's Duck Lake. And also Duck Lake is also very exciting technology. And it's not like that they will remove Iceberg or Parquet, but it's rather like they provide blueprints and idea on how Iceberg can or may improve in the future. So let's start with the past. As a little warm-up, let's travel back in time for 10 years. Let's make it year 2016. This also marks my start as my professional career in the data science and data engineering world. And when you remember, if you've been around during that time in the data engineering landscape, basically there was Pandas and Spark. With Pandas, it was still the pre-1.0 era. With Spark, it was still the pre-2.0 era. There was no declarative data frame API, for example. And things like DuckDB, Data Fusion, you name it, they didn't exist. Rust was not even a thing. When you take a look at the data science landscape back then, well, scikit-learn was the gold standard. But the entire neural network era was just visible on the horizon. TensorFlow and Keras, they just had their first initial release, and it took PyTorch another two years until they had their first release. So it was like true, honorable classification and regression problems. No neural network, high complexity thing of things. And from the infrastructure side of things, Hadoop was still a thing. Hadoop was on its peak. And also I remember myself working with Hadoop and also working with Parquet. Parquet as being an open interoperable storage format, solving many issues back in the day, but not solving all issues. So once you started to scale your workloads, once you started to get more complex use cases, you came across issues such as, well, those are just files residing on HDFS, but I don't have like a true table abstraction on top of it. Moreover, if you care about performance, there was no first-class primitive of metadata residing next to Parquet files, which did not exist back in the day. And for one solution, there was Hive. Hive actually predates Parquet, and Hive provides so-called Hive tables as a logical table extraction on top of existing Parquet files. And also what Hive provided is like the central repository of metadata, in a sense that for each and every partition that you have within your Parquet table, you store min and max values, which allow your query engine to improve the query plan and to be more efficient to prune data. However, this little B got also overwhelmed once again when complexity rose and also when you scaled out. It was just too much of a puck here. Why? For one, storing metadata only on the partitioning level was just not sufficient. Imagine having a huge table, and within one partition, there are still thousands and thousands of files. Still, your query engine has to issue a lot of S3 calls, for example, and to get all of the meta information. Why don't you just store all of the meta information for every niche and every file? That's what Iceberg does, for example. Also, when it comes to evolution and flexibility, if you want to change the schema or the partitioning of a table in Hive, this typically means you have to rewrite the entire table, which is an extremely expensive operation. With Iceberg, it's different. It becomes a metadata operation. And also, when it comes to asset compliance, imagine multiple writer applications trying concurrently to write to a Parquet file or a Hive table. Typically, this meant some sort of corruption of your table, which is just bad. And this was the original motivation what Iceberg intended to solve. And kind of the Iceberg then mastered all of these Parquet files. It provided order, it provided structure, it provided certain guarantees for us developers. And now we want to dive a bit deeper into it and understand how Iceberg does this. So first, we start with the data layer. So now we want to together build an intuition of how the metadata architecture of Iceberg works. We start with the data, and this is like unchanged comparison to Hive. We have the little parquet guy down there with the data files, and then we enter the metadata domain of Iceberg. The first concept we want to familiarize ourselves with is the manifest file. You can imagine a manifest file like an index holding references to each and every parquet file, along with metadata, min and max values, along with a schema, and also along with partitioning information of this per-key file. And now this is the change to a hive. We now track each and every file, not just on partitioning error, like partitioning layer, but also on the file layer itself. And a query engine can already leverage this manifest file to prune. So it knows, well, I have all these per-key files, and now I don't have to read them because this predicate excludes like 90% of my data, and I'm done. The next metadata concept we have to familiarize ourselves with is the manifest list. And yet again, this is like an index containing references to all of the manifest files. And now we don't store all of the metadata information, but we store the partitioning information contained in the manifest files. And this is like yet another possibility for a query engine to prune. I can read the manifest file, and by the partitioning information, I can say, well, I only have to read 5 out of 10 manifest files. The next thing is a snapshot. A snapshot is closely related to the manifest list. You can think of it as a one-to-one relationship. Each manifest is directly linked to a snapshot. And a snapshot is just one point in time which describes our iceberg table. And this snapshot then is contained in a metadata file. So yet another metadata object that we have to deal with in the iceberg world. And now all of this resides on Cloud Object Store. This is really important. All of this resides on Cloud Object Store. But we are not done yet. When it comes to asset compliance, when it comes to atomicity and isolation, We need yet another layer, the catalog layer, which is responsible to pointing us to the most recent metadata JSON file. And this becomes important. Let's assume we now add new data in an append-only operation. So we add some new data, new data files. Then we have to write manifest files pointing to those data files. Then we add a manifest list pointing to the existing manifest files. We don't have to rewrite them. And then we also add a new reference to the new manifest files. We then copy the old metadata file, modify it, and then add our new snapshot to it. And now at the point that we have to change the pointer in our catalog to say, well, the most current version now is the new metadata file. It's not the old one. So there's already some complexity to it. This still might be a bit abstract, so let's take a closer look at the metadata file, what's in there. I took the freedom just to minimize it really for presentation purposes like not a valid metadata JSON file I just want to convey a few concepts for one we have the like the format version at the moment the most recent iceberg version is v3 verified in June last year in this case like the most used version is v2 at the moment we have like a table UUID like an identifier we have the location for the table and then this is important the last sequence number the sequence number is like this monotonically increasing commit log. If it says two, it means, well, there have been two versions of this table. If it says hundred, then it says, well, it's the hundredth version of this table. It's monotonically increasing. And then, yes, of course, we have information like when was it last updated. And then we have these three attributes down here, which I want to take a look a bit more closer because they tell us how Iceberg handles schema evolution and partitioning evolution. When it comes to the schemas, it's fairly simple. Just take a look at the first one, schema ID 1, sorry, 0. And with those fields, we only have one field, order ID. We start simple. We just have a table with one column named order ID. And then I realize that's a bad idea, at least need something else. So I change my schema. And then instead of having just the order ID, I also add my event time. And you can see those schema IDs, they are also monotonically increasing. In our manifest files, I can then reference whether a parkey file adheres to a schema ID 0 or 1. The same, in principle, applies to the partitioning specs. So a partitioning spec defines how our data files are partitioned. And once again, I started really simple. I said I have no clue about the query pattern, so I just don't use any partitioning at all in the beginning. But then I realized, well, this table is getting bigger and bigger, and my queries are getting slower and slower. Hence, I have to partition it somehow. In this case, I'm even using a transformation. So I say I want to have an hourly transformation on my event time column. And this is also a metadata operation, which is really nice. Because in contrast, if you were to partition on an hour of the event time in Hive, you have to do the transformation on the data level. And this now becomes a transformation on the metadata level, which makes it really nice for us as developers. You don't have to rewrite data or add data in this case. And again, we see this is a spec ID, and then this spec ID can be referenced by the manifest files. And now, lastly, taking a look at those snapshots, once again, this is like a history, just a list of snapshots, and very importantly, we see the sequence number here, the chronically increasing commit log number, and then we also see the manifest list. As I said before, each snapshot has a one-to-one relationship to a manifest list. So I hope I didn't lose you yet. back to the architecture and what else is specified in the iceberg specification, what else is defined what functionalities are in there, for one we have optimistic concurrency control, it defines what happens if multiple engines try to write to the same iceberg table how do they retry, how do they behave we do have table maintenance operations such as snapshot exploration compaction and often file deletion, these things are really great, with Hive you had to compact your data yourself. There was no proper algorithm for this. Also, often file deletion, let's assume you're writing to an iceberg table, or to originally a parquet table, and then all of a sudden your write application, for whatever reason, crashes. There are some files in object store which are not deleted. With often file deletion, there's a mechanism specified on how to delete all this non-referenced files. We do have partitioning and schema evolution. We talked about this. We even have type widening. You can say, well, At the moment, it's an int, but it should be a float in the future. That's possible. And then we have two modes. Importantly, I'm just quickly explaining them copy and write. This is like the standard operation. Imagine a GDPR use case. You have to delete several rows from a large data set. And with copy and write, this means you have to rewrite the entire file. Parquets, they are immutable. They're non-changeable. And if you just want to remove one row, you have to rewrite the entire file, which is really expensive. So in contrast, what you can do, instead of deleting the original file, you just add a new file with deletion markers, which you then merge on read. So while reading the file, you merge those deletion markers and you get the current state. And this is like a concept which was popularized by the Hudi community. They introduced this. Now, Delta Lake also supports this. And it's currently, I think, the main mode of how those table format clients work. And as I said before, deletion markers, that's exactly what Iceberg V3, you've got deletion vectors. It's like rowing bitmaps stored in Puffin files. With the V2, we have positional deletes and equality deletes. And there's actually even more to it. So you already get the idea there's lots of functionality specified. And now I'm coming back to this chart again, and we will see that on a catalog level, there are lots of technologies also involved in the Iceberg spec. So for the catalog functionality, So pointing to the metadata JSON, you could use a Hadoop file implementation. It's just a file sitting somewhere and pointing you to the most current metadata JSON. Then you could use the Hive Metastore using the Thrift protocol just to store this pointer in a transactional database in the background. You may use the JDBC implementation to just also talk to a transactional database storing this latest pointer. You maybe also could use more specific interface, such as like an interface purely for AWS Glue. to restore this pointer. You're getting the point. Well, why not have a standard interface that decouples those specific implementations? And that's exactly what the REST API, the Iceberg REST API, is about. And that's really great that the community came up with this idea to decouple all of those client implementations with the actual catalog implementation. And what's also important about the REST API is that REST API catalogs, they take on more responsibilities, such as metadata management and also conflict resolution. instead of having the client do all of this. And then we take a further look at the metadata file. It's in JSON. Manifest list is in Afro. Manifest file is in Afro, like row-wise binary format. Data files can be in Parquet, ORC, and Afro. ORC stems from the original Hive community. And then everything of this resides on ObjectStore. And now it feels like, whoa, that's quite a lot to do, quite a lot to implement. And this is what I call just complexity, implementation, amplification. The iceberg is happy. Yeah, I can do all of these things. But all of these engines, they are kind of crying, they're suffering because there's so much complexity spread around the developer community. Each and every one has to implement this. If you contrast this with a transactional classical database such as PostgreSQL, all of the complexity resides in the server process. And now we're shifting all of this complexity in the client. I think this thought is really provoking. And you could also say, well, interoperability has its price. You can also put it this way. Anyway, so now we have a good understanding of how Parquet works. We saw the metadata architecture. We saw some of its additional features. Now let's move to the present. First, what works really well with Iceberg? First of all, batch workloads. They work really well with Iceberg. That's the original idea of Iceberg from Netflix when Netflix was a heavy Hive user and they realized, well, there are some issues, as we said before, and then they invented the Iceberg format and for batch workloads, it works really well. Schema evolution, partitioning evolution without data rewrites, that's nice. We got asset compliance by the catalog layer. We get interoperability because so many developers were forced to implement the specification and we also get time travel, as we saw with different snapshots. What doesn't work so well? When I prepared the proposal back in November last year, incremental processing was still difficult. Or you could say it was defined in a specification, but it was not yet implemented in the clients. That's why the incremental processing part, as being the missing primitive, still appears on here. I don't want to go into too much detail because of time constraints, But I will leave it in here as, like, if you want to use the slide deck, you can use those visualizations. Essentially, with Iceberg, we too, CDC is not possible. CDC is just, well, you've got one table on the left, one table on the right. In order to understand what files or what rows have been removed, well, you can execute this innocent-looking SQL, but behind the scenes, this is really expensive. And also, you can't even distinguish what rows have been added or updated. You just don't know because we have no meta information. There's like no unique row identifier here. With V3, this has changed because the specification has introduced a new thing called row IDs and the last updated sequence number. And using those two, you just can leverage the metadata, like those metadata columns, in order to define what has been removed, updated, and added. And those are more lightweight operations that you can use and leverage. and interestingly the spec was ratified in June last year V3, Amazon they announced we support V3 in November and by the end of November 25 even though up until today Athena does not support V3 because Athena is based on Trino and Trino does not support it yet so be careful with AWS when using FHD V3 however with all Spark related services it just works fine Snowflake, they announced a public preview of Iceberg V3, including raw lineage in CDC in March this year. And Databricks, they followed in just last week with V3 support. So it's now a feature that I think you can rely on. And it's a feature that you really need. It's often the case you have some raw data residing in Iceberg as being open and interoperable, and then your downstream jobs, they want to incrementally only process the new arriving data. And this was kind of hard before, and now it becomes more easy and more developer-friendly. But now let's go to the streaming workloads. We can see the iceberg now is in a hurry. I was afraid that there are so many of these small parquet files coming. How can I handle those? And let's assume a streaming read. A streaming read, how is a streaming read defined? What is typical or characteristic for a streaming read? Well, when we talk about streaming, we typically talk about low latency. And when we say low latency, I'd say, well, there might be different definitions. I select, let's say, a response in 200 milliseconds. This should be like a good target. On a catalog level, what does a client do in the first case? It goes to the catalog and asks, well, what is the latest version? And a catalog can be a transaction database, so the response should be less than 100 milliseconds for sure, maybe 20 milliseconds or so. But once we know what is the most current metadata JSON, we know the pointer now, we will consult the cloud object store. So we have to read the metadata file. And this will at least take us 100 milliseconds. Typically, metadata files, they grow. On average, iceberg tables can be easily multiple megabytes. Once we have the metadata file, we know which snapshot to load and which reference, which manifest list to load. Yet another round trip, another 100 milliseconds added. Once we know the manifest list, we can now load all of the different manifest files in parallel and then adding yet another 100 milliseconds to the overall sum. And once we know the manifest files, we can then finally load the data files in parallel. But this may also take more than 100 milliseconds because with Parquet, it's also a shady protocol. First, you consult the footer, then you go into the row groups. This will also result in several round trips. And this is easily more than half a second. So without proper optimizations, like caching and stuff, no way you will do true streaming reads with Iceberg. On the opposite side, now we flip it. We are not reading data, but rather we are writing data. Let's assume this is like our startup easy Iceberg table, and now we want to add only a single row, because streaming ingest, or like a few rows to add to Iceberg table, what happens if we just add a single row? Well, you already know it. We have to write the data file, we have to write the manifest file, we have to write the manifest list, we have to write a new metadata file, and we have to switch the pointer to cloud object store. So this will also take a lot, and when you think about adding lots of those single roles, this is what is called the write amplification. You get plenty and plenty of those small metadata files in the object store, which then you have to compact, and then maybe the compaction contents with the actual write operation. So also for this, Iceberg is not really well suited at the moment, in my opinion. Okay, streaming, we talked about streaming, now we come to AI applications and point lookups. We can see this grumpy parquet, this AI application with the robot arm trying to extract a single value and the grumpy parquet says, no, I'm not giving you the single value, either you take me in the hole or nothing. That's the access wall, this idea. Why is that the case? Just as a reminder for us, how does parquet work in general? Let's assume on the left side we have an in-memory representation of a data frame, and then on the right side we have the on-disk representation of Parquet. How does Parquet work? First of all, we partition our in-memory horizontally, so we get row groups, then we partition those row groups vertically in column chunks, and then once again we partition those column chunks yet again horizontally, and we get data pages. Those data pages are important because they are the smallest addressable unit in Apache Parquet. Let's do a closer look at data pages. Let's assume we have 20,000 rows of mostly 0 and 1s with some repetition, and we get this decimal representation, and then there are a lot more values. We can also do this in the binary representation with an int32 type, but we don't want to store this on disk because it's just a waste of storage. What Parquet does, it uses some encoding, some type-aware encodings first. For example, we don't need all the redundancies of all of the zeros up here, we just wasted space. So in this case we can use, for example, bit packing or run length encoding and bit packing, and even before a dictionary encoding, we can even cascade them to make it a bit shorter. And then for us, important to remember, Parquet was born in a time when network bandwidth was really limited. So Parquet tried to optimize to really slow, a really, really low space, a really low disk size. And then that's why they also added a compression on top. So we already applied our content-aware encoding, and now we apply some type-agnostic, bit-oriented compression on top, like ZStandard or Snappy, to make the file size even smaller. And this was a good idea. It was a good trade-off in 2013, but it might be not a good trade-off today. So what happens if I just want to access a single row here in this case? This becomes really hard. First of all, that's that's block compression. I Can't access a single value. I either have to compress everything to access the one value. I can like do nothing about it But we do not preserve a type order or a byte order in this case. You know, I can't just access it and even if we disable block compression on parquet, which is possible, we still are in trouble with the Default hybrid run length encoding and bit packing encoding so in this case just imagine We transpose those bit packed values here on the top, so we get a binary representation Or us to read we also get the decimal representation, and how does the encoding work these hybrid run length bit pack encoding? When we want to do a point lookup, let's say we want to get the eighth element once again The encoding works this way it sees well There's four times one so I do a run length encoding because I can use the redundancy to minimize storage And then I do bit packing because those values change very often and afterwards I do a run length encoding once again But by intermixing it I'm forced to decode this stream right from the beginning It's not like I can say just give me the eighth element. It doesn't work I have to decode the all the data first in order to accept single value So once again this makes it makes it kind of hard for us to do real random access or point lookups apps. I didn't say why this is actually necessary, but for AI applications during training and inference you typically want to shuffle and randomise the data, or at least for inference you just want to get this very value and compute a target value out of it. So we can see that streaming workloads and also random access are two use cases for which Iceberg are not so well suited, and now we will see alternatives to it, like what can you do instead. And first First we will talk about Vortex, the parquet of tomorrow, we will see. First Vortex just doesn't use block compression at all, we get rid of it, we don't need block compression anymore. And secondly, well they changed the picture in a sense that, well first of all we don't use run length encoding and bit packing anymore as a default, but rather we are using the BETABLOX scheme. BETABLOX is a paper published by a Munich research group in 2023 and it's best described as adaptive cascading time-specific and lightweight encodings. What does it mean? Well adaptive means before you choose an encoding you sample the data and based on the sample you decide for a proper encoding. Secondly cascading means that once you have encoded an input and you you get some resulting output, you can apply once again and encode on the output, which is the same what Parquet also does. It typically does first a dictionary encoding, and then those keys get once again encoded via run length and bit perking. Type specific means that we don't use just general encoding schemes, but rather we are very specific. We can leverage certain commonalities or patterns within our specific types. We will shortly see an example for strings. And then lightweight means we still allow random access and those encodings are parallelizable. We want to reduce data dependencies between connected points. So if you imagine, like in the example before, you have to deconstruct, you have to decode this entire stream. This also makes it really hard to be parallelizable on modern CPUs and GPUs, because you have a dependency that you introduce. You want to get rid of these dependencies to make it parallelizable. So this might be still like I'm telling you some voodoo, like it's all done in the background. It's some magic. How does list encoding work? So let's take an example with the FSST encoding. Let's assume we have the raw data on the left side. This is our corpus. We see some web addresses here. And from those web addresses, we now build up a dictionary, but not a dictionary that works row-wise, but rather a dictionary that works symbol-wise. So we look at each and every row and then realize, well, this http dot slash slash disappears quite often. So this becomes a symbol in my symbol table. And also like this www dot is a symbol that we can use. The ending dot de is used fairly often. And then we can reconstruct the original content within those compressed form. And it's just like a new idea, and that's what Vortec is really strong about. They incorporate the database research from the last 10 years. Also when it comes to floating point numbers, They have this LAP encoding when it comes to integers. FastLens is a really popular paper, really worth a read. For me as a non-data, like a non-computer scientist, really hard to read, but I really enjoyed the ideas presented, even though I understand roughly 30% of it. But anyway, let's come back to our data page, the original data page with Parquet, and compare it to how Vortex works. First of all, we don't have any compression anymore. And secondly, Vortex supports zero-copy memory mapped reads. What does it mean? Well, it basically means that the encoded data is dumped right onto disk. It's like a memory dump when you use Pickle and Python. You don't have to do any transformations anymore. You just read it from the disk, and then you can start processing it. There are no more transformations involved. And what's also very exciting, I think, is that you can operate directly on those encoded values, Like modern engines, such as DuckDB, they support it that you don't have to decode the values first. They're original, like in 32-memory format. You just operate directly on those encoded values. Think of, like, the strings you saw before. When you do a group by count, there is no need to decode those strings. It's enough to have those numbers standing there for the group by count operation. And it's also, like, really clever. And also, you get a zero-copy error integration. And error is, at the moment, the standard for in-memory columnar representation and all of the major engines supported, which makes error, once again, really flexible. So what is the claim? What is the result of all this? For one, they themselves argue, well, we have 100 times faster random access. We have 10 to 20 times faster scans, like in a typical analytical batch workload. load, we have five times faster writes while we're having a similar compression ratio. And that's a really strong claim, isn't it? You can check out the benchmarks yourself on this Watex site. It's a really detailed website. They also include benchmarks regarding LensDB. Yeah, and I think it's exciting to see what can happen if you use or if you leverage the knowledge from the last 10 years where Parquet might still have not developed as fast. And also there was In 2023, a paper by, amongst two guys, Andy Pavlo and Wes McKinney, they evaluated columnar storage formats, like in current times. And in the end of the paper, they have eight lessons. And funny enough, basically all of these eight lessons, they appear in the Vortex design. I guess for no accident. Keeping the encoding scheme simple, limit the use of block compression, centralised and friendly random-to-random access, Affinity to modern in-memory formats such as Arrow And encoding algorithms that are parallelizable All this is being implemented in Vortex Or specified in Vortex, I should say Great Now let's move on to Duck Lake The return of relational databases Remember this guy, getting overwhelmed Streaming workloads, not working so well with Iceberg Remember this guy, being happy but all of the clients are crying and screaming, well, too much complexity for us to implement because of this. And now with Duck Lake, it looks a bit different. We just put the duck in the middle, and we're done. Just a joke. Instead of storing all of the metadata in a cloud object store and having the complexity of asset compliance put on the catalog and cloud object store layer, why don't we just use traditional databases, traditional transactional databases, such as Postgre or MySQL, They have a track record of being really production-ready when it comes to asset compliance and when it comes to handling small metadata. So Duck Lake somewhat did the obvious, and it's so intuitive to do it this way instead of storing metadata in the cloud object store with high latencies. And funny enough, when you think about Snowflake, Snowflake has the same design. For their metadata, they use FoundationDB, a transactional distributed key value database. And then for storing the actual data, they used micro-petitions, very similar to what Parquet is on Cloud Object Store. So it's kind of the same idea, the same architecture, with the difference Duck Lake is open source completely, and Snowflake, you can't use Snowflake just to this. Of course, it's the entire platform and enterprise solution that you get. But the principal idea of separating metadata and the actual data in the transactional database and then on the cloud storage with the actual data, that's the same idea. And I think this is really striking. And once you come across this idea, you wonder, why did they store metadata in the object store in the first place? Well, there's a good reason for this. They argued once metadata itself becomes big data, then the Hive metastore, the transactional database in the background, becomes a bottleneck once again. But I think you need to have a lot of metadata to have your transactional database become a bottleneck. And then even then, nowadays with the managed transactional database solutions on Snowflake and Databricks, they are scalable. They are serverless and scalable horizontally. So this bottleneck should not exist anymore. And there's an interesting blog post by Duck Lake from, I think, two weeks ago. They did a benchmark. Yes, they did a benchmark, of course, comparing a streaming workload between Duck Lake and Iceberg. And they claim that they are two orders of magnitude faster for the ingest part and three orders of magnitude faster regarding their queries. And how do they do this? Well, they say, we have data inlining. If I have a very small change to my table, I don't need to write a new parquet file. I just inline it in a transactional database table. So all of these changes are just recorded in a transactional database. Instead of writing a parquet file, writing a manifest file, manifest list, metadata JSON, updating the pointer, all of this is done in a transactional database, which makes it late and fast. And when you read it, you just combine the information from what you have stored in a transactional database for all of those small changes, and then you also read the data from the object store. And this explains this three orders of magnitude faster query because you're essentially comparing a transaction database read in contrast to many small files in a cloud object store. Then this makes sense. And importantly, just on Monday, Duck Lake, they released their 1.0 version marking their first stable release. They say production ready. If you look at their website, There are many companies that at the moment are using Duck Lake already in production. And I had to put this here. Hannes Muehl-Eisen, the author and creator of Duck TV and also of Duck Lake, they had a Duck Lake release video with Mother Duck last year. You can see the name and the link down there. And there are two quotes, and they're a bit spicy, but I think they really make a point. He said, for one, one of the things in Duck Lake that we managed to do is to cut, I want to say, like 15 technologies out of the stack. And we saw this before. I think he has a valid point there. And the second one is maybe even more harsh. He said, I think Iceberg's problems are conceptual, fundamental, and baked into specification, which is an entirely different way of being wrong. And in this case, he addresses the central idea of doing like asset compliance metadata, small CRUD-like operations on a cloud object store. This really defeats like an anti-pattern, you could say. I'll just leave it there, yeah. But maybe the question, what do we do with all of this information now? We have seen Duck Lake, we have seen Vortex, and they seem somewhat superior to the existing solution with Iceberg and Parquet. And, of course, the community is not sleeping. The community, the Iceberg community, is really, really active. And last week, as I said before, the Iceberg Summit took place, and there were several talks talks Addressing all of these issues for one the the Julian lead em. I don't know how to pronounce him properly. Sorry he's one of the original co-creators of pake and he gave a talk on the like changing requirements over time like 2013 when pake first was created We had different hardware constraints like limited Network bandwidth and also nowadays we have like 16 times the amount of CPUs available GPU threats have increased a lot So things change, and also the use cases change. Who would have thought that you want to do streaming on Iceberg? This is kind of a stupid thing in the first place because it was not designed for this. And also random lookups on Parquet, like, no. Why do you do such things? That's like the clever question in the first place. Also now we see there's a talk on decompression bottleneck as we covered before. So using patched frame of reference and LAP for floats to improve the decompression in Parquet. And then also this, Will Manning, the CEO and co-founder of Spiral, he's the guy who also came up with the Vortex idea in the first place. So how to integrate Vortex with Iceberg. Iceberg just also released a new file API, which makes it easier to introduce new file formats to the Iceberg spec. We also have a talk on rethinking Iceberg's metadata structure in v4. It's the essential idea, instead of having three commits to Cloud Objects, so just have one. Single file commits, the same way as Delta Lake does it. Improve this one. And also, one talk about the Iceberg REST specification. I touched this in the beginning, but it's also an important concept. The Iceberg community is moving forward to doing more and more of the metadata work in the catalog layer. So it's not the client, the reader client, doing all of this, or the writer client, but rather it's the Iceberg catalog who does the heavy lifting in the future, reducing complexity on the client side. So all of this is also getting integrated now in the Iceberg specification, and the community is discussing it. And there are many more parts we could talk about regarding Iceberg, but I think that's enough for now. When it comes to my personal opinion, and that's my personal opinion, if you require interoperable batch workloads, I think Iceberg is the go-to technology at the moment. I wouldn't know what should be a reason not to use Iceberg at the moment. If you're on Databricks, well, you can also nowadays use Iceberg. You're not forced to use Data Lake anymore. That's like an easy decision. When it comes to scalable, low latency streaming, I would not use Iceberg, to be honest. If your requirements are like, I don't care about latency, like in a sense of 10 seconds, okay, then you could use Iceberg, then it's fine. When it comes to AI workloads, I would rather use Lens. or Iceberg with Vortex, even though this Iceberg with Vortex is like rather in development, whereas Lens has a track record of being used in production. And when it comes to Duck Lake, I think, I really think it's technically superior from its design and architecture, but it's not as mature yet. And if you read the announcement, the 1.0 announcement, they list the different clients that support Duck Lake. And for one, even saying, well, it's wipe-coded or agent-decoding, They have this kind of self-humor, which I really sympathize with. So this is not so production-ready. But I think in the future, you should look out if you like a startup, a rather small company with strong technical expertise, this could be something to look for. But I guess large enterprises, they will still stay away from it for now. And that's it. I hope this vortexing ducks of a floating iceberg has become clear. and I hope you enjoyed it.

Speaker 1 [41:26]

Okay, thank you so much. You really gave an amazing talk. So we are going to take questions, and the first one is a non-technical question. What have you used to create such nice iceberg characters in your slides?

Speaker 2 [41:41]

Gemini. Banana.

Speaker 1 [41:43]

Okay, so the next one is, do the client implementations such as polars decouple the users from the choice of file format EG? Could I use polars plus Iceberg plus Vortex, or am I stuck with Parkway Twin using specific implementations of Iceberg?

Speaker 2 [42:02]

I didn't really get the question, but I think it was asking whether when I use Polars, do I need to use specific catalog implementations? I think once Polars also supports the REST API specification, then you're free to use different REST catalogs such as Polars, Nessie, Snowflakes Horizon, Databricks Unity Catalog, or even AWS Clue. But essentially, each and every engine at some point will need to support the Iceberg REST API in order to talk the Iceberg protocol and then to read and write Iceberg data. And I'm not sure about the current state of Polars. Maybe someone knows about Polars here in the audience? No. Okay, you can look it up.

Speaker 1 [42:47]

Okay. Would you recommend using Iceberg in Databricks? Are there problems to keep in mind?

Speaker 2 [42:54]

Well, there are some problems with Iceberg on Databricks. As long as you stay within the Databricks platform, I think you're fine. But as soon as you want to, for example, write Iceberg tables to external catalogs, so let's assume you're running Spark on Databricks and you want to write to Polaris or Lakekeeper, this just doesn't work yet. So they do a lot of advertising, being Iceberg-ready and blah, blah, But writing to an external catalog doesn't work at the moment, which is kind of a big disappointment to be honest

Speaker 1 [43:30]

Okay. Is it possible to define how all data should be migrated when schemas change on Iceberg?

Speaker 2 [43:39]

Can you say this again, please

Speaker 1 [43:40]

Okay, is it possible to define how all data should be migrated when schemas change on Iceberg?

Speaker 2 [43:48]

How old data should be rewritten

Speaker 1 [43:51]

how data can be migrated.

Speaker 2 [43:53]

Ah, migrated, sorry. Yeah, well, in the best case, data does not need to be migrated at all. So it just remains like a metadata operation, and you don't have to rewrite it. But that's part of the iSpec specification. Like, this shouldn't be your responsibility as a developer. This is the responsibility of the client doing this.

Speaker 1 [44:12]

Okay. Do the client implementations such as... Okay, now I've gone through this. Can you use Iceberg on Vortex files?

Speaker 2 [44:22]

I assume you can but rather in a development state as we just saw on the iSPEC summit. There was a talk by Will Manning Outlining how you can use iSPEC together with vortex

Speaker 1 [44:33]

Okay, how do you see the future of Iceberg and Delta by considering the Databricks acquire tabular?

Speaker 2 [44:40]

Mm-hmm. To be honest, I found it a bit confusing that Databricks acquired Tabular because they are like the original developers of Data Lake themselves. But I guess they also kind of realized that Iceberg is the driving force at the moment. So if you look at all other larger companies, maybe except Microsoft in the beginning, they were supporting Iceberg. And they also have to pick up the, like, let's say, Iceberg paradigm and Iceberg shift. and I like they themselves say that in the future there will be one format we don't know yet which one this should be but I guess it will be iceberg at some place and also when we look at iceberg re3 and delta lake they become really close like the deletion vectors the feature and iceberg re3 this is like compatible with delta lake so they're converging more and more so I think in the future it's like with all of these technologies in the beginning you have like a lot of different implementations and ideas and then there will be they become a phase of consolidation and we agree on a certain standard and i assume this will also take place in the next years

Speaker 1 [45:43]

Okay. How is schema evolution handled in streaming plus CDC setups in V3 follow-up? Do you think future Iceberg versions might be able to reduce latency?

Speaker 2 [45:56]

Well, regarding the first question, I don't know. Regarding the second question, once the ISP4 specification will introduce single-file commits, then we will reduce latency, yes. But you will never get as fast as Duck Lake, for example. As long as the metadata remains on the cloud object store, compared to a transactional database, you won't become faster. But then catalogs may become more intelligent. Catalogs, they can cache the metadata JSON information. So if you put more of your complexity in the catalog layer, which then again is backed by transactional database, so it's again the same idea of Duck Lake, then this will become faster.

Speaker 1 [46:34]

Okay. What does Iceberg provide that Delta doesn't have?

Speaker 2 [46:39]

I think hidden partitioning is a feature that Delta Lake does not have at the moment, or at least it's not as mature. Apart from that relative path, that's a feature that is also going to be added in Iceberg V4. That means if you... Those manifest files, you remember, they have absolute path. So if you copy your Iceberg from one street bucket to another, it just doesn't work, because those paths, they're absolute. We have Delta Lake that's different. They use relative paths. So Delta Lake supports this already and Also the entire streaming story in Delta Lake since it's called Delta was way better and it's also better integrated in Iceberg In Spark and in Spark streaming

Speaker 1 [47:24]

All right. Thank you so much. For the rest of the questions, please, you can contact him after the talk. Let's give him a round of applause.