Distributed file-systems made easy with Python's fsspec

Speaker 1 [00:07]

Hi, everyone. I'm Barak, a founding engineer at LickFS. It's an open source platform for managing data at scale. I'm filling in for Einat Orr, our amazing CEO. She couldn't make it. Today, we are going to explore how a fantastic Python library called FSPEC can easily and dramatically simplify the way we interact with your data. This is me. And a roadmap for this session. We'll start with why. Why it matters. The data shift to the cloud. We'll introduce FS-Pack as a solution for this problem and go into the concept and features it provides. We'll see how it integrates with popular data libraries and how it can be extended by taking lakeFS as an example. We'll wrap up with a short demo and key takeaways. Let's start with the simple code. This is very familiar to any developer. There is no credentials, no networking. It just works, assuming the file exists locally. But local storage is limited. Cloud storage changed the game. We now work with massive data sets that doesn't fit on a single machine. The cloud provides for us redundancy and availability, while ensuring that there is central control that bring security and auditing, for example. But this flexibility introduced a new friction. Let's take this code for example. This is how we read a file from S3. Although it works, the code is very verbose and very specific for working with AWS. Nothing here is similar to the code while working locally. Here is the same example using the Google storage. Entirely different code. Every storage system introduces its own way of doing things. This inconsistency forces us to learn new ways and sometimes reimplement the same logic working with different storage systems. So distributed storage is great, but working with it can be harder than it should. What we need is a consistent interface, a way to work with storage no matter where it lays. a single way for working without reproducing the logic and just keep working with our data no matter where it lives. This is where FS-Spec comes into the picture. FS-Spec stands for File System Specifications. It's a Python project that defined a standard interface for working with file systems. It doesn't matter if the file system is local or in the cloud. The core idea is very simple. build one consistent Python API to work with any storage. In the heart, there is a high-level function called open that you can provide a path toward your data, and it can read or write from that location. How does it work under the hood? The code just uses the standard FS spec functions, nothing fancy. Based on the path you provide, it automatically picks the right back end and provide you a unified interface to work with the storage. The back end handles all the low-level details, the communication protocol, the authentication, every specifics related to the underlying file system. Of course, this is a classic. We are solving a problem by using another layer of abstraction. This slide shows a list of some of the APIs that the FSPEC interface provides. Looking through the method, I'm not going to go over each one. The idea is just to give you a simple example of how simple and familiar the interface that FSPEC provides. You can see it mimics the Python OS package. If you worked with local files before, it should be very natural. So what are the key benefits of using FSPEC? Simplicity is huge. You write your code once, and just by changing the path, you can work with different storage systems. It provides your code consistency, it saves you mental overhead. But there is more. FSPEC provides additional capabilities above your unified file system. It can provide you caching, compression, transactions, and more. It's already integrated in some of the tools you are using, like Pandas or Hugging Face data sets. FSPEC being extendable and open source can also adapt for new storage systems and provide you with community and developers that also know or work on your storage. Let's take a look at some code examples side-by-side just to get an example of how it is used. Here is the code we saw before, reading a file from S3. On the left side, you'll see we will need a client. We will need to call get object and deal with the response. In this case, we need to decode text because it's text file. On the right side, you will see that we are using the FSPEC open and just point to the data and everything else looks very familiar, like working with your local storage. Familiar and simple. That's the basic idea. Writing files, same story. We need to get a client called put object in case of x3 and make sure that we are passing the right parameters. And again, very specific to the storage we are working. On the right side, again, we are using open, this time we are passing the write mode and write our content, and we are done, clean, simple, and consistent. What if we need to read only part of a large file? Using the S3 version, we will need to construct a range header where we need to calculate the bytes we need to read, very specific to S3. With FSPEC, for example, you can see that we are using the standard file system calls like seek and read. This is exactly how you will work with local file, but this time it will translate those API to the relevant underlying storage for you. So although FSPEC directly has some cases, it's really nice to know that there are many data tools that already use it behind the scene. Let's take, for example, Pandas. Pandas integrates directly with FSPEC. It means that it's super simple to work with different data storage. In the example, you can see that we are reading a CSV file from a remote storage and then saving it as a Parquet file to a different storage. There is no specific vendor libraries or API here. We are just telling Pandas to where to read from and where to write to, and everything is handled under the hood by FSPEC. The only thing you need to make sure is that you have the dependency back end libraries installed. In this case, it's S3FS and GCSFS, because we are working with those two storage systems. In this example, we are using FSPEC S3 file system in order to build some toy data loader. The key thing here is to access the file system using S3, accessing the S3 file system. And if you are going through the code, we are listing files in the bucket. We are iterating each file and using Pandas Read CSV, passing it a file-like object from the file system. And we are asking for 100 rows here just for the demo's sake to read only part of the file. In this example, if the CSV file is huge, we are probably reading only the first part. Although this example hard-coded S3, we can, while we develop this code, we can just write file as the name of the file system and everything here you can develop and debug locally. This is where FSPEC really shines, especially when working with Parquet. Under the hood, when you are using Pandas to read a Parquet file, It uses the engine as a pyro. If I remember it as two engines, it can be fast parquet and pyro is the second one. Pyro can make good use of FSPEC, efficient access to our storage. Here is why. When you're reading a parquet file, at the footer of the file there is a metadata. Pyro will take the metadata first and fetch only this part. It will go over the request. In this example, we are filtering data on the vendor ID and we are asking for specific columns. In this case, it will figure out which blocks, which groups it will need to fetch, and it will access them using FSPEC and download only them. This result in big memory and performance in cases you are not reading a complete data when working with your data sets. Okay, but there is more. We mentioned that there are additional capabilities Fspec. Let's look at one. Caching. This works exactly, quite interesting. The really cool thing about Fspec abstraction is that it enables you elegant composition. The feature we are using here is called URL chaining. We are chaining a string into the URL. And the basic idea is that FSPEC allows us to build a layered file system. We can take our file system that we are working on and put another layer above it. In this case, we are layering cache as a file system. Let's go through the code to understand what it means. At the first part, we are reading 100 files directly from S3. Same code as we saw before. We run the same operation twice just to show no caching is involved yet. In the second part, we prepend to the string the simple cache. Simple cache itself acts as a local cache. What it does, the first time we are accessing the file, it will download the file completely and any other call to the same file will serve it from the cache and not hit the network. You can see here that there is Call animation where you can see that my laptop is very far from the actual bucket. I'm working with and You can see how the cache will work when we hit the second iteration In general FSPEC provides a couple of Implementation for caching layer it can be block cache and file cache and bytes cache each cache system has its own pros and cons and there is no one-size-fits-all of course and the best strategy will probably depend on your access patterns and the file size you are working with so far we've seen FSPEC provides unified interface for our storage system and give us extra features like caching but what if your storage system doesn't support FSPEC or maybe you want your own caching layer The beauty of FSPEC, it's designed to be extended. You can create your own back-end just by inheriting from the best classes that FSPECs provide. It's very flexible and, for instance, if you're building your own file system and it's read-only, you don't have to implement all the right operations. At the end, you just register the file system name and then you can see all the stuff we've seen so far working with your file system. As part of the presentation, I wrote a toy spec file system. Let's switch just to show how it looks like. Let me see. Okay. Let me know if the font works for you. So toy spec is a file system we added. We have here not a lot of code. It's a single file. Let's look at how we use it first to understand what it does. It's not complicated. The idea is just to give you a feel and look of how a file system is implemented. This is a read-only file system where I'm taking dictionary and implemented a key value where the key is the file name and the value is the storage this is where we initialize the toy file system and here you can see all the methods we saw before where i can list or open a file or access the information of the file system or even read the file using pandas all from our file system that we just instantiated and the idea again it's to just reflect the data we have in memory in a dictionary and so it's a read-only for now and there are two classes I'll jump to the end it's just a 120 lines and the last line here is just a registration part let's see the main entries that can look interesting the toy file system that inherit from abstract file system the name of the protocol which is the toy because it's toy spec in the initialization we are going through the content and just most of the code here is just normalization i'm normalizing the path to have a slash and then in the content i'm making sure it's bytes and this is the implementation you see it on the screen and the get content is just fetching the data from the dictionary and it is used by the toy file itself. The Toit file is when we are asking for open. When we open a file, it's just instantiating the handle that represents a single file. And if we will go to the Toit file implementation, we can see that except from passing details about the file in the path itself, we just need to implement the range function. Because using range, I can read the whole file, part of file, all the methods that the base class will need in order to implement read-only file can be implemented using the single function. All in all, there are, I think, seven methods here. The biggest one, let's see, except from open, we have exist, is file, is there. And the two others are info, which just reflect the information about the entry. Is it a file, is it a directory, and the file size. And listing, which this is the biggest one, but it's not very effective. It's just going over the dictionary and trying to implement for you the listing by cut out and stripping out the relevant path. I hope it helped understand what it means to implement. FSPEC. Sorry, I'll switch back to presentation. So, as I mentioned, I work at LakeFS. LakeFS has its own FSPEC implementation called spec, LakeFS spec. It wasn't written by us, it was contributed by the community. One company called Material AI wrote it. Now, LakeFS itself isn't an object store, it sits on top of one and provides versioning capabilities. We'll take a look at LakeFS spec and how it's enabled access to our LakeFS while unlocking some powerful features. But before we do that, But let's take a step back just to understand what Lekafest does. Lekafest is an open source project that helps you manage your data as code. Very similar to how you handle your source control with Git. It sits on top of your existing object store and add versioning capabilities such as commit, marriage, div, tag. If you look at how it fits to the current ecosystem, its experience is like working with a thread. difference is that in the path where you access lakefs you add you add your branch that you are working on lakefs provides full flexibility in equal while interacting with it it can be used through the ui api cli let's take a high level look at how lakefs does it behind the scenes that will help understanding better the the cool feature that the implementers of lakefs spec did using it so like if it's managed metadata over your data your object store each commit represent the state of the data at the specific point in time every time you add delete or update files and commit the reference it's reference all the data that the previous commit had and the changes you made this enables a very fast branching operations without copying or moving any data on your storage and it's unlock powerful workflow like a referencing all commit for reproducibility creating branching for working in isolation and experimentation you can diff between changes between branches to see changes and merge data the same model with different the same model as you work in your code but for your data now that we understand what lakefs provides let's see lakefs spec and how it enables lakefs to perform all the things we saw before in the code example you can see that we are loading files and iterating and listing and the only difference here than the code that we saw before is the protocol that we are using which is lakefs and of course the branch that the data is found at. Let's explore a specific feature with FSPEC that aligns with one of the lakeFS capabilities. Transactions. Transactions are crucial for consistency and data integrity, especially in distributed workflows. FSPEC provides semi-atomic transaction operations it enables you all or nothing you the transaction will go through or not at all looking at the example you will see that we are initiating a transaction and writing two files one is the data file and one is the metadata this is a good example for it's a good use case for understanding why you will need a transaction if any operation fails an exception is triggered the transaction will not touch your uh will not show on the storage that that means that it's all or nothing the files will appear only only if the two operation will succeed at the end but it's still semi-atomic currently all the main cloud providers do not provide atomic cross-file operation that means that in a point of time when the transaction will be complete There will be a second where only one file will show and the other will not, although this is a complete transaction and it will perform the operation at the end or move the data at the end, it's still semi-atomic. However, LakeFS spec addresses this limitation by using transactional guarantees by LakeFS capabilities. So, how LakeFS spec actually pulled this off? the answer is quite elegant lakefs has native branching capabilities and it will use them to perform atomic file operation when a transaction block begins it creates a temporary branch based on your target branch instead of tracking changes it will include all the file operation on your temporary branch when the the block will be completed it will commit the data to the temporary branch and merge it back to the main main branch that means that it will look like an atomic operation from the main branch view the main branch will be on commit a switching to commit b that means that the two files will be appear at once it's clean and safe so let's see it it live. Let's start a new instance of lakefs. This is lakefs. create a repository repository is equivalent to a bucket and we're working with the live networks so let's hope everything will work and we choose where to store the data on the underlying file system because we are metadata layer and add some sample data to it so creating a repository is just it it doesn't do anything accept a metadata operation but because i ask for sample data it will also upload data like the readme file and examples so here we can see the objects in the main branch and you can click on a file and we integrated the duckdb as part of the interface so you can even query data while you inside the ui and let's make some changes let's create a branch based on our main branch And in this branch I'm going to delete a file and upload a file. By the way, you can also just check the physical address to know where the data is stored, but this is the main issue with LakeFS. It's the main core idea that it will manage the versioning for you. Let's delete this object and let's upload some data. and here you can see the uncommitted changes that we made on the demo branch you can see one file is deleted and one was added and we can commit the change and we can see that the demo folder and does include sorry does include the example parquet that i uploaded and the lex parquet is gone not like in our main branch and we can compare those branches and now let's do a transaction using lakefs spec just to see how it update our storage and for that i'll create a new branch Let's go over the code. It's very simple. Not a lot of data. I'm creating a data frame, a metadata, it's a demo, and I'm creating a file system using lakefs. And here is where it's the new code that you can see that here I'm creating a transaction based on the repository and the branch I'm working on. It's PyCon, and I'm working on branch demo two. in this code i'm going to write two files the data csv that i created up up there and the metadata and i'll commit the data and let's make it interesting i'll just wait here for the data and i'll wait here for the metadata and before I start it I will go to demo 2 and okay there is nothing here let's run the transaction data is written and I'll refresh just to show there is no magic and nothing is here and then I'm writing the metadata still nothing is here because we went over the slides and we know that Behind the scene there is a branch that holds all those changes. We can go into branches and you can see there is a Transaction branch that is created in the background and we can see the data is there and of course if I am Let the transaction complete and a commit and merge will happen. We will see that there is no more Side branches there is the target branch where we wanted the data and the data is here And, of course, for everything that works, we just need to check that something will not work, and we will raise an exception during the write operation. And in order to watch that something, okay, let's first create a new branch because we already made this branch dirty. I can also revert the change, of course. So demo3 is based on main. The files are not here. Let's run the transaction. And something bad was happened, as we know. And there is nothing new on this repository, on this branch. And there is no temporary branch. But usually, in some scenarios, this is because the options I used called delete always. delete always, make sure that the temporary branch will be deleted, but in cases you have a complex workflow and you want to explore the data during a failed transaction, we can use it like that. And then even when the transaction fails, you can see that nothing changed here, but still we can explore what went wrong because it will leave us the transaction branch. Okay, so I'll switch back. Wrap-up and takeaways. Cloud storage are powerful, but they're tricky because they will bring you different APIs and different behavior using different storage systems. You still want to use them, but pay attention. FSPEC simplified that by providing a unified Python interface. We saw how it is integrated into other libraries and other libraries that use it under the hood hood enable you a simple way to just specify the ui and access your data fspec is extendable that means that you can implement your all file system or support new ones it's in open source you can see other users having the same troubles you have and developers we took for example the lake fs fspec which was com contribution from the community all that makes a key part make fspec key part from the data modern tools and I hope you definitely understand look into it to understand it better and even use it thank you

Speaker 2 [28:37]

Thank you, Barack, for this very clear presentation. There are a couple of questions here on the Slido. Do I need S3FS if FS-Spec is already capable of reading files from S3?

Speaker 1 [28:52]

from S3? Yes, it doesn't include this implementation. It includes the mapping. So for each popular cloud vendors, it holds a dictionary saying which protocol is implemented with which package. It will throw an exception, say this package is missing for you. This is for all the non-cloud providers. Also, if you are implementing your own file system, you can either register it as part of your project and work with it directly, or you can open a ticket and ask them to add you as a file system to their dictionary.

Speaker 2 [29:24]

Okay, thank you. I think there are two questions more or less the same. How is authentication handled? That is, how do I pass credentials?

Speaker 1 [29:35]

Sure. So there are two options. Basically, the default is the default of the SDK. So, for example, if you're using S3, S3 by default will use the local environment variable, then fall back to the credentials. So the same chain will work because it falls to the default of the SDK that the backend system provides. There is always an option to when you are instantiating the file system, you can also pass storage options and then pass your own credentials. So you can, for example, at the same code, create two file systems, the same type of file system, passing different credentials. So, for example, LakeFS also provides S3 gateway. I can instantiate file system S3, and instead of talking to AWS, I can pass a specific backend endpoint and specific key and secrets, and it will talk with LakeFS.

Speaker 2 [30:25]

Can I use FS-Spec to read documents, PDF, docx, from my Azure blob storage without downloading those files first to extract the text?

Speaker 1 [30:38]

So, Azure support, yes, there is a package also for Azure. The library that reads the file, it depends on you. So, if you are passing it to a library that can use the file like interface or you are reading the content and provide it to the library, it will work.

Speaker 2 [30:57]

Okay, can one connect FSPEC to a custom cloud provider or is this mainly the same as we have already seen in the toy example with the own file system?

Speaker 1 [31:10]

I'm sorry?

Speaker 2 [31:11]

I just repeated, can one connect FS-Spec to a custom cloud provider or is this mainly the same as we have already seen in the toy example with its own file system?

Speaker 1 [31:23]

So FS spec is the spec itself. It has a catalog of back-end implementation. All the famous cloud providers are supported. And like I saw as an example, LakeFS, this is how you can provide something new to the table. This answers the question?

Speaker 2 [31:42]

I don't know.

Speaker 1 [31:42]

I don't know. Okay, so this approach, I can help you with that.

Speaker 2 [31:49]

Is there also a way to detect changes to a file and are different protocols supported, for instance SFTP?

Speaker 1 [31:59]

So SFTP supported if we are talking about the tech changes you mean getting metadata or info from the Remote storage. Yes, there are method that can provide you information and so like the info message you can invoke them also on remote storage systems and SFMTP if I remember correctly is supported.

Speaker 2 [32:22]

Could FS-Spec be used to open MinIO for other cloud storage providers, not only be AWS compatible, any directions towards it?

Speaker 1 [32:34]

it yeah it works with minion no problem as i described before like where i can work with

Speaker 2 [32:34]

Yeah.

Speaker 1 [32:40]

fspec directly through the s3 interface so instantiating a file system called s3 or by going into i can show that if you want and the part that i'm using choose content i need a shortcut for this So, if we look at AWS config, I don't know if a lot of you are using it, but you can define services, and in services, you can enter custom endpoints. So, as I said before, you can also, inside your code, instantiate the file system and just specify that you are working with custom endpoint and point to different location, in your case, MinIO, so MinIO can be a custom endpoint and you can also specify it from inside the code. I usually just add another profile in my S3, this is S3 native, it's not something I invented, and you just set the profile and it works. Also, for this example, I can just say my profile is lakefs, sorry, and then if I'll use ls, I'm getting information from this talk, from the lakefs. This is the basic idea that will work with FS-Spec, just working with Minio.

Speaker 2 [34:07]

Can and should I use FS-Spec for accessing large data, bigger than 10 gigabytes, stored in buckets on S3 in performance-critical systems?

Speaker 1 [34:17]

so again it depends can I

Speaker 2 [34:20]

Can and should I use FSPEC for accessing large data stored in buckets on S3 in performance-critical systems?

Speaker 1 [34:20]

should can Okay. This line is very important. Yeah. So, as I see it, again, it's always when it gets to performance critical because I'm an engineer. Every time I'm going to performance critical, I'm removing all layers. So, it depends on the use. I mean, if the idea is to download the file the fastest way as possible, and this is the only thing that the system you should optimize for, don't use any abstraction layer. Go download the AWSC implementation and do it directly if you can. I think for a lot of times, the idea of using caching or partial read, especially in large files, or the capability to read the same code and manage code in a consistent way, usually it's a trade-off. So you need to decide what you want. Something easy to maintain, clone codes or something like that, or you want to go on a path where you remove everything and ask the provider which API can download this. Do I need to send the USB key to get the file?

Speaker 2 [35:36]

Okay, clear, thank you.

Speaker 1 [35:36]

Something like that.

Speaker 2 [35:39]

How does FS-Spec implement caching? Does it detect file changes on the storage system?

Speaker 1 [35:45]

So there are a couple of file system implementation. The symbol cache doesn't detect anything There is a file cache that can use hash to understand if the file was changed But there are many implementations that you can use and so I suggest just go over the documentation and see each one what they can provide

Speaker 2 [36:04]

Is it possible to host git repositories on github or gitlab?

Speaker 1 [36:10]

This is a question for me. I don't know

Speaker 2 [36:12]

I don't know.

Speaker 1 [36:13]

it's a

Speaker 2 [36:14]

It's in here.

Speaker 1 [36:14]

you know

Speaker 2 [36:15]

Well, never mind.

Speaker 1 [36:16]

Yeah, I think the git repository. Yeah. Well, I I know there is a github file system implemented for FSPEC if it's help I don't know if they implemented gitlab as a file system, but github there is a file system in FSPEC

Speaker 2 [36:31]

that's maybe the question and then the last question what are the main

Speaker 1 [36:31]

That's maybe the

Speaker 2 [36:36]

differences between lake fs and dvc

Speaker 1 [36:40]

Oh, there are many differences, as you can see, of course the scale and lakefs is a server implementation, it's a service, it acts like, for example, Minio R3 versus DVC where you're working as part of files where the changes are part of workflow, different guarantees because as I showed before, if the use case is atomicity at the bucket level or your repository level, it's a different way to look at DVC because DVC does, as far as I know, copy the relevant data you're working as part of your workflow. It doesn't guarantee right atomicity. Help you version your data, also like FS, but, for example, I can run Spark over that data and let it read from branch and then I can revert those changes and just compare what changed. I can't do it with DVC because there is no backend for DVC it will just help you get the file that you need at the specific version

Speaker 2 [37:44]

Thank you. That was the last question. Thank you, audience, for all your attention and all your questions. And thank you, Barack, for the clear presentation.

Speaker 1 [37:52]

presentation.

Speaker 2 [37:53]

Thank you. Thank you very much.

Speaker 1 [37:53]

Thank you.