Scaling Data Processing for Training Workloads at DeepL Research with Rust

Speaker 1 [02:27]

Hi, everyone. Good afternoon. Thank you so much for attending this session, and I hope you've been enjoying the conference so far. Today, we have two lovely speakers who are both staff research and data engineers at DeepL. We have Jonas Deden and Johanna Gorgon. I'm going to let them introduce themselves in a bit, but just before we start, a bit of housekeeping. Please give them a warm welcome, first of all. Thank you. Thank you very much. Now, please kindly put your phones on silence and kindly don't change rooms between the sessions. And if you have any questions, please submit them via the talk tool and we'll read them after the session. So thank you very much. Please may proceed.

Speaker 2 [03:17]

So welcome to our talk. It's sadly the only Rust talk, this PyCon, so I hope we make it really work for you. So basically, this is a little bit about a small story of how we used Rust at DeepL research to really improve the situation a bit, comparing to an existing C++ ecosystem and a very particular use case of us. And we generally want to convince you to use Rust also in whatever Python needs you have. So who are we? So my name is Jonas Detten, and that is Johanna Görgen. We're introduced by, we are staff engineers at the research data platform team. So basically, you can see this AI-slot-generated image. That's basically a depiction of what we do in the data craft in DeepL Research. Basically, we're a big platform for all data needs for DeepL Research, basically. And one small task of ours is that we also provide data tooling, for researchers, for the data ingresses, for the model trainings, like we also do our own models entirely from scratch sometimes, and have our own data centers and everything and basically they need tooling to do the data ingress. Okay, so a little bit of background around file formats for model training, as I said, and then a little bit of a story how quickly we were able to rewrite an existing library with existing Rust tooling and provide it as a Python package to our researchers, and then a slot by me about generally how super nice Rust is for Python interop, and then also I want to show a little bit on a very high level, it will be very quick, lots of very dense slides. You can download them, don't try to make photos here, just download the slides, basically about what tools we use in our data tooling.

Speaker 3 [05:18]

not working? Okay, now it's working. Okay, cool. Yeah, so I'll start by giving just a bit of background around why we even came to need to use Rust in the first place. And this has to do with the data formats that we were using for model training at DeepL. So at a really high level, our training pipelines were looking like this. They were this distributed workload where we had quite a lot of workers on a shared cluster that needed to access one or more data sets at a time in a kind of sharded fashion. So each worker needed to access either kind of a straight series of rows in the input data sets or it sometimes wanted to access kind of random indices in this data set. And the problem that we were running into is that we had really huge training data. Often these training data sets were on the magnitude of terabytes at a time and the workers were having gigabytes of RAM available. And due to some implementation details I'll get into in a second, RAM was really becoming our bottleneck so we were unable to scale up these workloads to their full capacity because basically finding the rows that we wanted to read in was causing us to go out of memory all the time. So why was this? The status quo at the time was that we had a proprietary file format. In our defense this file format was dreamed up I think like a decade ago and made a lot of sense at the time but now there are obviously some industry standards that would help you work around these things. At a high level we had a table of contents in our file format and our file was just one large usually multi terabyte file and the table of contents pointed you to different gzip blocks within the file that that contained the actual data and the row data was UTF-8 strings with JSON dicts inside. So anyone who knows data probably can see a bunch of issues here. The table of contents was really the issue that was causing us to go out of memory a lot in our training pipelines. That thing was often gigabytes large when we had really big data sets. And on a latency front, reading each row required decompressing the GZIP block, UTF-8 decoding the data, and then JSON deserializing the actual row data. And often if we actually wanted only just a segment of each row, we still had to load the whole row into memory and JSON deserialize it. Many of you might be thinking, okay, there's some ideas out there for making this better. And we thought the same. We landed upon trying out Parquet. For those of you who don't already know it, it's a columnar file format. It's a nice format because it's also self-describing and has a really expressive set of data types. And those are always encoded in the schema in the metadata of each file. And standard tooling also supports multi-file data sets. So this got us away from that situation where each data set was just its own gigantic file, which is good for things like object storage, where until recently there was some limitation on individual object sizes. So that was also running us into problems. And there are a number of other benefits with Parquet, like column projection, meaning you You can load in pieces of your data by column instead of always having to load in entire rows if you're only interested in something. Columns are compressed separately so you can choose a compression scheme that makes more sense for each data type. You have predicate pushdown so you can do kind of filtering on your actual data using some statistics headers that allows you to again not load everything into memory unnecessarily. And again the type system is just way more expressive than like what you're limited to with JSON. So this is some results of our initial benchmarks. As you can see, we really crushed the proprietary file format right out the gate. Parquet gave us a way better latency to the first actual training data point. And the runtime of doing some kind of basic operation like counting lines went way down when we used PyArrow with, like, standard Parquet tooling. This only has to be kind of a metadata-only operation, so you should really never have to iterate through all of your data. Yeah, the middle blocks are kind of different, slightly dumber ways to actually do this. But in any case, Parquet was always exceeding the existing format. So yeah, from that, you might think, like, let's just migrate all of our data from the proprietary format and then move all of our tooling onto PyArrow, which is, if you're You're not aware of the standard Python library for interacting with arrow data, arrow being the in-memory format that's usually used to read Parquet, which is the on-disk format. But this caused immediately a bit of trouble for us. So when we were doing some initial benchmarks, we found that just even iterating over row data with PyArrow to kind of read in large data sets doing exactly what we needed to do in our model trainings was having a kind of worrying memory footprint and even sometimes looking like it was leaking data, leaking memory. With a little bit of tuning, we found that we were able to turn some knobs on the underlying memory allocator that Arrow, the underlying C++ library, was using for managing memory, and we also found that the OS, when a firm limit was set, was often able to reclaim memory, but still the footprint was a little worrying, given that in these benchmarks we were even releasing references to the actual batch data we were iterating over, so this was really just the overhead of reading the files, reading the data without actually counting the size of the data itself. Then when it came to even writing the data, things got a little bit more tricky. As you can see, this is in the scale of gigabytes. What we were trying to do here was repartition a data set, which means basically take a data set that's full of, in this case, files of 10 gigabytes each and make it a data set of smaller files or files with different sized row groups, which are like units in Parquet. And we found that, again, Arrow was not releasing memory until it got up to 10 gigabytes. And this was kind of unacceptable for our training workloads. And again we saw what looked like a memory leak when writing out rows, and again here we were also releasing references and trying to do everything we could to keep the memory down. So yeah, we ran into just quite a lot of trouble using the standard tooling, and it was at this point that we thought, let's take a look at Rust maybe. We've heard good things about memory safety in Rust, and there was already... So just a little side note here, at this point in time, both of us had really not used Rust before for any sort of production workflows. And this project was done entirely at a time where there were not really coding agents to make this as easy as it might have been today. So we really did have to kind of get up to speed on the ecosystem at the time. And to give just a little bit of context around what we were really trying to do here, on On the right, you can see what exists in the Python world. So pyarrow is the library that you usually would interact with arrow and parquet data with in Python, and it's built upon arrow, which is a C++ library underneath, which also the project exposes bindings into some other languages, too. This is where all of the leaks were pretty much happening, or at least the kind of tricky memory footprints. On the Rust side, what was already available is a crate called Arrow RS, and this is kind of a rather low-level library but was exposing basic IO and computing with Parquet and Arrow data, and it did, importantly, already have what it calls a feature, a PyArrow feature, which allowed it to export basically pointers that could be understood on the Python side to kind of read in Arrow data in a zero-copy way. So what was really missing for us was basically just the Python bindings layer. We wanted to be able to expose our library to, like, our whole research department is using Python, so it was important to have something in Python. And we also needed, like, data sets as a first-class citizen, so kind of directories of individual Parquet files, which wasn't yet a concept in Arrow RS, but did exist on the PyArrow Arrow side. and so yeah we needed to do a little bit of work on the rust side as well as expose a bindings layer with a little bit more like metadata operations and such as well and so our library became named disco parquet just so you know we'll be referencing it as such for the rest of the talk and briefly just wanted to point out the timeline because the important thing here i don't know if it's big enough for you to read it in the back but the important thing here is that essentially we began hacking around with Rust on one day and then already on the next day we had like a POC of a very basic version of this library that we could use to run benchmarks and then within two weeks we already had a first release on an internal package registry and then in another two-ish months we were already running the first model trainings using Parquet with this new library So, the timeline was really fast, I think, for two beginners, basically, to expose this already really memory-optimized wheel. And that was a lot due to the ecosystem, which Jonas will get more into later. But I think it helped a lot that there was kind of like one canonical way to do this. Which I know isn't so much the case maybe in other languages where you want to create bindings to another language. So now I'll just give you a quick comparison, what were the outcomes before we get into some more technical details. So we are the orange line here. Our library immediately really cut down the amount of RAM needed to iterate over rows in batches. This was kind of, I think, from the first benchmarks that we did, and we honestly didn't have to do really so much beyond exposing the read functionality that was already there in the arrow RS crate. With writing, we also had a substantially lower memory load and no leaks. Really we were able to just write with holding always the smallest amount of data possible in memory. And then recently we've been also working on exposing object storage. And this is an important slide because it also really shows the tradeoff that we've made. For us, memory was really important. It allowed us to scale up our training pipelines much more and use smaller instances, but as you can see on the left, PyArrow crushes us on throughput. We basically even tried to tune it down to doing the minimum amount of prefetch of files and the minimum amount of pre-decoding of data, and even then, it's substantially faster at getting through Parquet data. And wanted to just also shout out this rightmost bar is belonging to Arrow 3, which is actually an open source library that's also based on Rust, which I think is definitely worth checking out. Provides a bit of a lower level API than what we have achieved with our package. But since we're not open source, we want to give a big shout out to this project. And although the memory load here doesn't look so good on the object storage side, it is quite good for Aero 3, I believe, on disk reads. Just a tangent on that front. And then to kind of wrap up the whole story, here's an end-to-end comparison of what our actual training pipelines ended up looking like in terms of RAM load. And you can see that the more you scale up the workers, the more of an impact it has to reduce the memory load of actually reading data. And this allowed us to just really scale up. So I believe that's it for the story time. And now Jonas will go a little bit into how we achieved this and what was so nice about it.

Speaker 2 [18:13]

This will be a more general section about generally how to do Rust things with Python. Basically, I don't know if this is way too small to read. I'm sorry for that. Basically, this is a small comparison of what you can do on the C++ side. We also previously had a lot of C++ custom libraries that we then binded to Python. We had a lot of pain with that. Now we saw this Rust thing and it's super nice. on a C++ layer, you have your build frontend, which nowadays, hopefully, is UV. Then you have six different build backends that you can choose from. Then you have a different interop layer, nanobind and pybind, common examples there. Then you don't really have a package manager in C++. There are new modern things, such as Conan, which actually is written in Python, or VC package, which I don't know which language it's written in. and then you need a meta-built system like CMake or whatever, and then you call finally into the compiler, and it's like a huge pain. And with Rust, basically, you have uv, mature, and pyro3, cargo. That's basically it. Like, on the right side, I just pasted the minimal code that's needed to basically spin up an already working Rust Python library. It's literally just, like, three commands. Yeah, and, like, one use case we also had is, Okay, we want to introduce parquet as a new fancy file format into our research department But people need something to just like look at a file on the command line, right? And we also saw like okay PQRS. It's like this random rust tool that also does this. It's basically a CLI tool Okay, we can build that and we are actually like patched that a little bit for our needs and then we can like okay Call it somehow, but then you have to like first build it somehow and we can have a can ship a rust binary directly But and this is a little bit of a pain like we have Python users. They really want to do Python So what actually can you do in this in this ecosystem? You have this rust library that doesn't know Python at all. You just simply add a PI project toml with these random things It's basically just mature in metadata things and then you already can build this as a wheel and this will install into UV and Exposing the binary that every Python user can directly just use so There's not even Python layer in between. It's natively shipping the Rust binary as part of a real. That's super great. More general things about Rust. And this is a very generic example. You can also do wrong code in Rust. It's not this magic language that solves everything. This simple example is basically a three-element vector. And now I want to access an element that doesn't exist in this vector. In C++, this is literally undefined behavior. The compiler is allowed to do anything here. And I actually tried that out with Pybind, Nanobind, Siphon, apparently, and other things, too. And I got random integers. I got segmentation faults, random stuff. It's literally allowed to do anything. And here, you get this really nice Python exception. It's basically just a panic exception. And even it says directly in which line of code happened, and it also exactly says what the error is. You have a vector of length 3, and you want to access element 10. That doesn't exist. This is the error. You don't want to panic all the time. Usually, you have fallible IO or something. You can do a retry system or something like that. You don't want to immediately panic and crash the entire system. So in Rust there's actually this result enum that's used everywhere in the language. Hopefully some people that already know Rust will know this concept. And what I have here is just like a very simple function that I just coded that tries to open a file that doesn't exist. And this is just bare Rust code. And the only thing that, like this would return a result. But pio3, this binding layer between Python and Rust, basically exposes a pi result enum. So I just had to add the two letters here, pi result. And what this actually does, like it takes this Rust error that doesn't know anything about Python, and then translates that to a final found Python exception. Looks super nice. with C++, things can happen here. I don't know what. Also, it's super trivial to code your own type conversions. Whenever you cross the function, the language barrier, you have to do some conversions. Rust doesn't know what a PyList is, and Python doesn't know what a Rust vector is. This is literally taken out of PyO3. This is everything it needs to get a Python dictionary and translates that into a hash map and the other way around. It's super easy. It's like five lines of code for each direction. Another super important thing, GIL, so the Global Interpreter Log. So ideally, you want to always release the GIL when you do some fancy computation in Rust, because you're not using Python anymore. So you want to allow other Python threads to run. In C and C++, this is either just the macro or some closures, but they don't really check whether the things that happen inside the section of code where you have the gil released actually do something illegal. In Rust, this is actually completely checked on a compile time level. In the above example, I get a PIE string, and I try to access that while I have the GIL detached, and this actually is a compile-time error, I cannot make this code even execute. And in the other example, I'm just retrieving a vector of integers, so this is like a pure Rust thing, and I try to detach the GIL, this just works, this compiles, because I'm not touching any Python objects inside with a disclosure, and everything works. I don't know how many people know what the Python API is and what the stable Python API is. It's basically like a feature that you can build a native library. And instead of building it for every Python version that's used in your company, you can just compile it against the stable API. That means every Python version starting from that and above can actually consume this library. In Rust, this goes all the way down to 3.7, which back then in 2024 was important for us. And in C++, you can do nothing with PyBind and with Nanobind. You can only use 3.12 as the minimum. And again, also on the Rust side, this is super trivial. You just simply enable this feature there, and then it just does it. And on C++, it's a little bit more hassle. Thread safety. Say you have a Parquet file writer in your library, and now you can write for that write. If you were allowing multiple threads to write into that object in the same time, the output is not really defined. It can be garbled. It is the same as multiple threads writing to the standard IO or something. It also will just be merged. It will be garbled output. In Rusty, actually, if you have an object in your struct which doesn't implement the trade sync and send, and I will explain what this is, then you have to put this unsendable thing there, otherwise it won't compile. And whenever you try to use it, it will actually throw an exception as soon during the runtime it detects that this object is used in a multi-threaded What is send and what is a sync? Basically it just says send is a trade that just says I can move this object from one thread to another. Sync is another trade saying I have an object statically somewhere and I can use it from multiple threads simultaneously. And if I just now wrap this thing in a new text, so basically I put a lock around it, And then this suddenly is implementing sync. And I can remove the unsendable thing on top there. And then this is all it needs to make this stably work in a multi-threaded Python function. And again, this is all compiled. Do you have some water? Okay. Okay. Like, UV is all the rage, right? actually has an equivalent thing. It is both a package manager. It's a package manager. It can be used to set up new projects. It actually interfaces with the compiler. You can use it to publish things. You can even search for packages. You can run tests. You can run benchmarks. You can generate documentation. All of that basically is one tool. And this is the standard tool in Rust. Also, I put it on the right side. It is very similar to UV. You can also just say add, and then it adds a dependency. You can even say bench, and it will just benchmark your library. It will benchmark your tests, basically. It is super nice. Yes, stub files. Yes, type checkers are all the rage right now in Python, right? Yes, that is also a good thing. Whenever you write a native library in something like C++ or Rust or whatever, you're exposing things to Python that don't really are Python native. So it needs some information, which methods it has, which function, which members, some classes exposed, and stuff like that. Previously, this was really, really bad on the Rust side. Basically, you had to always write the stub files you own. In C++, you actually have methods to automatically generate those. Nowadays, this is a very experimental thing. this was released this year or something, you can actually have ways to automatically generate your stub files, which was super nice. OK, then another thing about what general tools we also use at DeepL Research, we basically, like this is the OG thing, right? Like Pandas, Dask, stuff like that. This is written either in like, not pure Python, because it's calling NumPy and stuff like that underneath, but it's basically this Python, Siphon stuff. Last year, DuckDB was also quite a big thing here at PyCon, which we also use here and there at DeepL Research. But really, we are now concentrating on the lower right corner. So Polos, Polos even has a paid offering now, Polos Cloud. We can do distributed Polos. DataFusion is basically the Rust equivalent of DuckDB. and I just put some benchmarks. Again, look at the slides if you're interested in concrete numbers. But basically, the thing is, it definitely kills Spark. It kills Pandas. It's super fast. Daft also is a super interesting library. It's basically the Rust equivalent of Dask. So Dask is basically distributed Pandas. So you do Pandas syntax, but on multiple nodes in parallel. And Daft is basically like, yeah, polar syntax, but also distributed, and it also like completely kills Spark in performance and even is able to execute things that Dask doesn't even, like it just OEMs in a lot of benchmarks in the upper right corner. And I heard that a lot of people love Spark here, and Rust also has some things for that, called DataFusion Comet, for example, and Lakesail, both of these things are basically plug-in replacements for Spark. So instead of spinning up a Spark cluster, you spin up a cluster of these. And then you just speak the PySpark API to these things. And it will just do the same computations, but way more efficient, like with, I don't know, like a tenth of the RAM in some benchmarks, and with basically no overspill to a file system and stuff like that. Also, another thing that we also use is we want to introspect Parquet files in a web browser. For that, we use the Parquet Viewer. It is a tool that basically is WebAssembly combined with Rust, combined with Data Fusion. So basically, you have this tool that you can just open in a browser and enter a file, a URL, a S3, whatever, or even a local file. And then you can just actually speak to it in natural language, like this says, show first 10 rows that have an answer longer than 25 characters, and answer is just the column here. And then it will actually translate that into SQL and executes that in your browser. We are a multimodal company, so DeepL voice is a big thing. We wanted to deal with audio data. We also have vision models for PDF translation. So we need something to introspect, like pictures, images, videos, stuff like that. And for that, nowadays, we also look into SmoothSense. Just yet another thing. Maybe you can try that out. It's like all of this is open source and not connected to us at all. And then the biggest thing interesting for us right now is, OK, Parquet is great. It really can solve the text-based model training super, super nicely. But for the multimodal things, yeah, there are a few problems. The first one being, now we want to, as Johanna also mentioned, we want to basically randomly gather rows from tens or dozens of different data sets in a super shuffled fashion. So for that, we ideally want to have O of 1 random row access all the time. With Parquet, this is actually not what happens on a different... It actually decodes a lot of different stuff to actually get your row. Parquet also doesn't offer a way to do in-place updates. It's also super inefficient when you have extremely large schema tile, which can happen in an AI context where a researcher wants to just add hundreds of different annotations with some model. And also, it doesn't support embedding searches. For example, say you have pictures and you want to generate embedding sort of theft, and then you want to search for something like give me a tree or give me a bird or something, and Parquet doesn't have a native way for that. I looked at Iceberg and Delta Lake, and for our needs, this doesn't really do anything interesting. thing. I see use cases for that in other companies, but I don't think it solves anything for us. We are looking into the fancy new research around novel file formats, and three out of four of them are actually based on Rust, so that's a good thing. No memory leaks. There's is this future file format thing in the top left corner called F3. Basically, they do some really, really crazy stuff. It's like a research project from a university, like a Chinese and, I think, a US university. And yeah, I don't want to go into too much detail here. Basically, they want to really have the encoding layer as part of the file. Like, you have some web assembly in the file itself that defines your decoder, okay. Vortex, tomorrow there will be a talk about Vortex, which I definitely will attend. Like this is like a super fancy file format backed by the Linux Foundation and I think also Microsoft and others. Basically it says we try to be as composable as possible, so basically the file format itself is not opinionated about anything. You can literally build Parquet with Vortex. You can build any of these other formats with Vortex. It's like you can define, okay, I have a row group in my file format. You can basically do anything. LANs is also an open format, but backed by a company actually called LANsDB. It's really made for like AI workloads, multimodal things. I will go into detail in the next slide. and there's also Nimble, a file format written in C++ backed by Meta that I unfortunately don't really know that much about. But at least in my opinion, these are the four players for the Parquet 2.0 format. Okay, so maybe you guessed Lance is the thing that we're actually looking into heavily right now. So why is this so interesting for us? So Lance tries to be both what iSpec and Delta Lake are and also what Parquet is. It has a metadata layer and also a file storage layer. That means it has ways to do in-place row updates. You can also do time traveling. You can see history. You can do schema changes and all of that. You can also do atomic writes because you can just first write a data file somewhere and then register as soon as that's actually completed, the write completed, that with the data set. such that other readers can actually see that there was an update, and only if that update actually completed, it will actually read the file. Okay, the O of 1 row-wise random access is super interesting for us, and potentially the most impressive thing about this file format at all. So basically with Parquet, as I said, you have the problem, you can't really look at a random row, because you first have to jump into a row group, then inside that, jump to data pages or even iterate data pages, decode a whole data page, and then try to find your specific row. I mean, it's not really made for random access. It's very apparent. But with the new AI workloads that we have now, this really gets more important. And Lance really solves this by having multiple metadata pointer layers, basically. So you immediately know, just from looking at the footer, at which data page a certain row is. And inside this data page, you even know exactly what the byte span is that corresponds to your specific row. So you don't have any memory overhead at all. And it's super fast, as can be seen in both of these benchmarks, actually. Here, they tried to do... This is officially from LanceDB themselves. They had a web service or something which had to access a random row. and it just tried to benchmark how fast that works. And Parquet had single-digit requests per second, and with LANS they were able to do like 1,000. It's like three orders of magnitude faster. And this also can be seen on the right benchmark. This is actually from a paper from the F3 guys. So they compared all of these five events that I mentioned here in the slide. This is a random access time. Retrieve a random row and measure the time for that. And parquet is even beyond the plot there. They are just the numbers written on top of the plot. It's like 600 something here. And Lance is able to do the random access in like 5 or 10 milliseconds. Okay, and that's basically it. Thanks for listening.

Speaker 1 [39:05]

Thank you so much. That was a really insightful session. We've got some questions that we'd like you to answer. The first one is, did you evaluate Polars or DocDB for your use case, and how do they compare with your library?

Speaker 2 [39:20]

Okay, like these are DataFrame libraries or like DuckDB is more of a query engine. I would call that actually. What we really need is as low level as it gets in terms of error record batches access. So basically we want to have the Parquet file format and just want to decode that into raw error data. We don't want to have anything in between and we want to tune this as much as possible. And for that, Polus actually uses RRS. DuckDB is C++-based. I think they have their own decoder for Parquet. So probably DuckDB is also... At least better than PyArrow. Polus probably is also comparable to our performance in memory load, potentially. We haven't benchmarked it, to answer the question. But we really need raw like our record batch access, so it's not really interesting for us

Speaker 1 [40:21]

Thank you. Another question says, did you switch to the open source library throughout or did you solve the issue on your side?

Speaker 2 [40:30]

We had a look whether we can switch to that library and initially had a really, really fast development. Our library was already a little bit ahead. As things develop in a company, you don't always have the time to contribute all of these features to an open source library. So unfortunately, we had to continue with our own library. But we are actually thinking about how we can either bring our features to this open source library or whether we can just publish our library open source.

Speaker 3 [41:06]

Yeah, just to add a point to that, I'd say not ahead our library, but rather that we were kind of having a different focus in the sense that we were really focused on producing metadata for, or interacting with the metadata of our data sets and just needed some different APIs that it looked like the Arrow 3 library wasn't really focusing on, which was really just much more focused on streaming data. And on that front, it is really good. But yeah, like he said, it just was kind of a subset of what we needed.

Speaker 1 [41:40]

Thank you. Did you try to run your training with PyArrow or did you see the memory issue and fix it in anticipation of breaking the training?

Speaker 2 [41:50]

The initial benchmarks that we did were all with PyRO. The first slide with the first benchmark release, I think. And then we also tried our model training pipeline with PyRO. But the RAM load basically led to this thing exploding in like 10 seconds. And then we really saw, okay, we did it. We spent, I think, I don't know, at least weeks on trying to fix this somehow. Even delved really deep into the C++ code, actually raised GitHub issues. And then we just decided, okay, we have to proceed somehow and we develop our own library.

Speaker 1 [42:32]

Why did your initial approach lead to memory leaks in PyArrow? Is this an unavoidable thing with huge Parquet files or did you discover a bug in PyArrow?

Speaker 3 [42:44]

I would say it's kind of a fundamental difference in terms of how the underlying library handles memory. Like with C++, they're basically having some memory pool abstraction and trying to, I think, in the vein of getting as much throughput as possible, keeping memory available and not always releasing memory right away. But with Rust, it's like a language feature that memory is basically freed up when it's not in scope anymore. And so whether it's a bug I think on the writing at least potentially a bug But maybe on the read side where we could kind of tune it to at least be stable It's maybe just kind of a feature of the library in favor of getting a lot of throughput

Speaker 1 [43:31]

Thank you. Do you plan to open source Disco Arrow at some point?

Speaker 3 [43:37]

That's a great question.

Speaker 2 [43:38]

Yeah, I mean, I touched this already, basically. Yes, if we get the necessary priority to work on this, it's also a little bit of a capacity thing. We're just two people that developed this library and also have loads of other things to do at DeepL. Unfortunately, but also fortunately, because it's very fun work. And also, one additional point, it's not that easy in Germany, actually, to publish a library on your name open source that you used company hours to develop the thing. Yeah, so this has to go through our legal team anyways first. And we did not do this yet, sorry.

Speaker 3 [44:21]

Yeah, we're not ruling it out, but so far.

Speaker 1 [44:24]

Thank you. Are per-K files efficient for joins? It seems a similar use case to a random access. Or are these use cases unrelated?

Speaker 3 [44:33]

Yeah, not really efficient for joins, so actually lance is something that we're looking into for use cases like that. Parquet doesn't really have anything like a scalar index marking specific rows, so for things like joins, you usually do have to load quite a bit of data into memory or build some sort of layer on top of your parquet data that assigns unique IDs and knows what to do with those, but lance is quite good for, since it has scalar indexes, doing things like joins with the random access support that Jonas was talking about.

Speaker 1 [45:07]

Thank you. And there's a question that I like here that says, what are the resources you recommend to get started with Rust?

Speaker 2 [45:13]

Oh, good question. I read two books, like the Rust programming language book, and from the same author, Rust for Rustations. I think that definitely is the start. And then also the Pyro 3 documentation also is like a book, like for some reason Rust creates called the documentation book. I don't know. It's extremely extensive and goes into very low detail of how Rust libraries interact with the Python interpreters is really fun to read.

Speaker 1 [45:47]

Thank you. Do you have any resources that you want to recommend as well?

Speaker 3 [45:51]

Yeah, I was using some resources on the O'Reilly learning platform. I think there were just some tutorials, but I can't really remember the names of them right now. But also just a shout-out to the standard library of Rust. One thing we wanted to mention but didn't is that generally if you want to kind of explore Rust under the hood, unlike maybe some other low-level languages, you can often just read the actual Rust standard lib, and it uses all of the same concepts that you as a developer use, like enums, traits, structs, like everything is under the hood, the same stuff that you actually develop with as an end user of the language. So it's also just really understandable to go and read the actual, how it functions.

Speaker 1 [46:38]

I think we have time for just one more question. It says, comparing PyArrow and disco per K performance, do you know why the throughput of PyArrow is so much larger?

Speaker 2 [46:49]

The reason simply is we are a single-threaded library because we just want to read the bit

Speaker 1 [46:49]

Yeah.

Speaker 2 [46:57]

of data that we're interested in. PyOR, of course, is multi-threaded, so they actually have multiple readers that work in parallel. How that works actually in detail is one way is to do horizontal scaling, so basically you have per row group your columns and then you spawn just a thread per column that reads that and then it merges that to a final record patch again or you can just do it vertically so you just have one thread per row group and that just yeah like searches through all of that of course that has way more memory load because you have to pre-fetch all of this data keep it in ram and stuff like that and our library is just fundamentally different it's just like here is a little bit of data that you want to get and please read that and nothing more

Speaker 1 [47:40]

Great, thank you so much. Let's put our hands together for our lovely speakers here. And that brings us... That brings us to the end of the session. Thank you so much, everyone, for attending.