Dataframely — A declarative, 🐻‍❄️-native data frame validation library

Speaker 1 [00:08]

All right. Thank you, everyone, for joining the session today. In the next 30 minutes, Daniel and I want to talk about DataFrame.ly, a Polar's native DataFrame validation library that we've built internally for over a year now and which we finally open-sourced last week. Before we talk about DataFrame.ly, though, we want to briefly introduce ourselves and the company that we work for, QuantCo. My name is Oliver. This is Daniel. We're both based in Munich. We both studied at the Technical University of Munich. I did my master's in data engineering and analytics there. Daniel went a bit further and did his PhD in computer science. We both joined QuantCo within the last three years, and at QuantCo we work on topics across machine learning, data engineering, and general software engineering. And the project that we work on at QuantCo, we help companies turn data into decisions by combining economics, engineering, and AI. And to this end, we maintain dozens of data pipelines for our clients that each process billions of data points. The pipelines that we build are heavily built on top of open source software, and as a result, we at Quantco are also heavily contributing to the open source community. So much about us, though. Let's talk about DataFrame. So where did we start? At the end of 2023, we had some breathing room in one of our longest-running projects, and we wanted to rewrite the data pipeline that has accumulated plenty of baggage over the last six or seven years. And while doing that, we naturally needed to understand the old data pipeline really well to replicate its behavior. And this is usually how we felt when going over that. This is a pretty accurate description. So the data processing logic was really hard to understand, which made the entire refactoring effort much harder than we thought at the beginning. For rewriting the pipeline, we then chose Polars because we generally saw massive performance improvements compared to our old pipeline that only used Pandas. And we generally grew to love Polars at Quantco. But one of the main challenges that we wanted to have a solution for was how can we make the pipeline easy to understand and how can we make it easy to debug going forward? And this helps us answer why we will data frame it. As already mentioned, we noticed that, in general, maintaining code without the knowledge of the data frame contents is really time consuming. When you see a function, you usually see a data frame is coming in and the data frame is coming out, and you never know what's actually inside this data frame unless you step into a debugger, which is usually time-consuming. Further, erroneous and unexpected data generally leads to costly pipeline failures, and even if you discover that during testing, if some operation fails very far down in the pipeline, it usually takes very long to debug where the root cause of this was. It might have been at the very beginning of the pipeline where you make some very core assumptions about your data. So generally we realize that validating assumptions about our data pipeline or about data frames in our pipeline increases the pipeline robustness, because at many points in the pipeline we check how does the data frame that we have validate or satisfy all of the assumptions that we have. And furthermore, type annotations for the data frame contents help us greatly improve code legibility, because we can now, if we use type annotations correctly, we can see what content the data frame has going into a function and what content it has going out of a function. And this is how DataFrame.ly, our declarative data frame validation library, was born. So while back in 2023, so two years ago, the options for validating data frames were much We're obviously not the first people who have thought about validating the contents of DataFrames. If you want to validate your Polar's DataFrames today, you might have already heard of Pandera with the option to use Polar's DataFrames or Petito. So maybe as a quick show of hands, who in this room has already used Pandera before? All right, that's a few people. Who of you has used Petito? Okay, half a person. And who have you used any other DataFrame validation library before? All right. So that was maybe five more people. So not a lot. But then I am happy to tell you about DataFrame and maybe encourage you to validate your DataFrames going forward. So DataFrame has both soft and hard benefits. First of all, it integrates really well with Polars because it fully leans into the Polars expression API, and thus it feels very natural to use. On the other hand, it extends the existing options with many more advanced features that make it useful to use in complex data pipelines. And in this presentation, we want to talk about the basics of DataFrame.ly and also about some of the more advanced features to tell you why we think that it serves so well to build complex, robust, and legible data pipelines. For this talk, we want to use a running example to illustrate this, and this example revolves about hospital claims, because insurance claims are something that we both deal with in our daily work a lot. So the general idea is that we have a relational data model where, on the one hand, we have hospital invoices that each contain an invoice ID, which identifies the invoice as a primary key and then some associated data. With each of these invoices, there are associated a number of diagnoses. So in a traditional relational data model, we represent this with two tables. When we use DataFrame code, naturally each of these tables is represented by a DataFrame. And the question is, so in order to write easily understandable code and code that processes this data reliably, we want to tell you that all you need is import DataFramely SDY. So let's dive into DataFramely. The first thing that you want to do when you start using DataFramely is you define a schema. Schemas generally allow you to define the names and the data types of all the columns on your DataFrame. So in this instance, we introduce a new schema, the invoice schema, which inherits from DataFramely schema, and we define all the columns that we want. So we want an invoice ID, which is a string. We want the date when the patient was admitted to the hospital when they were discharged. We want to have a time stamp when the invoice was received by the insurance. And we want to have the amount that is written on the invoice. So this is already quite helpful as someone who would read the code, because you now know exactly what kind of data you can expect or to some extent you know what kind of data you can expect in this data frame. So we can go a little further with data framing, though, and we can use parameters for the data types that we annotate for a column type or for our columns. So to make our schema even more useful, we designate our invoice ID as a primary key as I told you before, and we designate all of our other columns as non-nullable. So very similar to what you would do in SQL, for example. We also have some more advanced column-level checks. So for example, we can say that for the amount column, we only want positive amounts. So anything greater than zero. If the amount is zero or negative, there's surely a faulty invoice. However, this is not all we want. Oftentimes we also want to define constraints across multiple columns. And this is where the DY rule decorator in DataFrame comes in. So with DY rule, you can define constraints across columns with general, with native polars expressions. So, for example, we can say that the patient has to be discharged from the hospital after he was admitted. So that's a very natural constraint. Anything else doesn't make sense. And we also want to enforce that the invoice is received after the patient was discharged from the hospital. If we get any data that validates these constraints, there's surely something wrong, and we don't want to deal with that. In general, you can define as many functions as you want, annotate them with DY rule, and thus use standard or native Polar's expressions to express what data you're interested in and what data you're not interested in. All right, so once we have defined our schema and put all of these assumptions about our data in code, we obviously want to verify that all of these assumptions hold for the data that we process. And the first thing that we usually want to do is use the validate function. So let's say we have a data frame with all of our invoices, then we simply can call validate on our schema and receive a data frame that is guaranteed to be validated once this function returns. If it doesn't return, then there's something wrong with the data. Some of the row or some row in the data frame doesn't satisfy our schema. And this method raises a rule validation error that provides some error details. So for example, it might tell you that one rule failed validation and in the column amount we see one row where the min exclusive is violated. So we have some row which has a negative amount or amount equal to zero. While using validate is very useful for development and in some cases also in analytical environments, hard failures are often very problematic in production. After all, if you have a data frame with 10,000 rows and one of these rows violates your schema, you still want to process all of the other 9,999 rows. So in order to remedy this issue, DataFrame.ly provides the filter method. And this filter method is very useful because it partitions the rows of your data frames into good rows and bad rows. So those which pass validation and those which don't. To be even more lenient, you can also set a parameter named cast equals to true, which automatically casts your input data types to the types that you expect in your schema. The most important thing about this filter method, though, is that it never raises an exception. It always returns. In some cases, good, which is the data frame that contains all your good rows, might be empty, but it always returns. And the failure object then allows you to dive deeper into the issues. It can tell you what kind of rogue data is in your data frame. And you can do that with a variety of utility methods. So, for example, you can see with counts which of the rules that you defined how often did they fail. Per occurrence counts tells you how often failures co-occurred in certain rows. And failure invalid allows you to retrieve the input rows that actually failed validation for further debugging. So to illustrate this a little better and to make it more tangible, let's look at an example how this works in practice. So let's assume this is our data frame of invoices. We have three invoices. all have their own primary key, and you can already see that the amount column has an issue. So the first invoice has an amount of zero, and we're not interested in that based on our previous schema definition. So if we use filter with cast equals true, we then receive the good data frame which correctly filters out the first row that we're not interested in, and since we said cast equals true, we also automatically casted the data type from float64 to a decimal. On the other hand, we have the failure object, where with failure invalid, we can recover the original row that failed validation, and this helps us debug why something failed in the aftermath. So, with schema's validation filtering, we now have everything on our hands to trust the contents of our data frame at runtime, and that is already very nice. However, we also want to document this in our code, and this is where so-called type data frames come in. And Daniel will tell you more about that.

Speaker 2 [13:02]

Thanks, Olli. So about typed dataframes. Most will agree that static type checking is a great way to detect bugs without executing the code. And we strongly advocate using MyPy for static type checking. However, if you're passing Polar's dataframes around in your code base, your type hints will typically just be Polar's dataframe or Polar's lazy frame if you're using lazy frames. DataFrame.ly provides more advanced type hints, and this leads to, first, better code readability, and second, safer DataFrame-related code parts. Validate actually returns a DataFrame with a static type hint, including the schema. So here, we're validating the invoice schema on the data frame invoices, and we're getting back a generic type hint of the DY data frame type with the invoice schema as the generic type parameter. Importantly, the runtime type of the data frame does not change by this. This is purely a typing construct. It doesn't change the runtime type. I guess many of you are familiar with the concept of design by contract. In our experience, writing functions with DataFrame.ly is more like defining contracts. We get more formal, precise, and verifiable interfaces. And if you define checks in your schemas, in your input and output schemas, you can actually verify that the pre and post conditions, which are the rules in your schema, are satisfied in the function that you're calling, if you're calling validate or filter before calling the function and also at the end of the function. If you use intermediate schemas, you can also check for invariants within the function body. If you skimmed through the feature matrix that we showed earlier, you may have noticed that one distinct and very powerful feature of DataFrame.ly is the ability to validate interdependent data frames, so groups of data frames. And we already saw the one-to-many relationship between hospital claim invoices and diagnosis in the diagram. Now let's create a second schema for the table of diagnosis. We have three columns here. One is the invoice ID. It's also part of the primary key for this table. We have the diagnosis code, which is also part of the primary key, and as you can see, we can also define a regular expression for this column. And then there's a third column, which is isMain. It's a Boolean column, and that shows us if that specific diagnosis is the main diagnosis for this invoice. For the set of diagnoses for an invoice, there should only ever be a single main diagnosis. We can implement a corresponding validation rule using the group by parameter and the DY rule decorator. Here we're grouping by invoice ID, so we get all the rows for that specific invoice ID, so all the diagnoses. And then we can define a polis expression that implements this validation rule that ensures that there's only exactly one main diagnosis for an invoice. Now that we have these two data frames or these two tables and these two schemas defined, we can capture these two interdependent data frames in what we call a collection. A collection can take members, and these are data-framed schemas, and we can perform validation on top of these collections, and I'm going to demonstrate that in a second. Similar to schemas, the validation rules can be defined on the collection level. However, in this case, we cannot use dyrule, which we've already seen for the schemas, but we'd rather use dyfilter. The difference between dyrule and dyfilter is that with dyrule, we return a Polar's expression that evaluates to Boolean series, but we cannot do that for the collections. For the collections, we must return a data frame where the columns must include the common primary key across all of the members. So in this case, we want to have at least one diagnosis per invoice, and we select the distinct invoice IDs from the diagnosis table, we join it with the inner invoices, and we return that join result and then we can inner join the result of this filter method with all the members. So that will then eventually remove all the columns excuse me all the rows where we do not have at least one in one diagnosis per invoice. As you might have guessed you can also call validate on a collection and if you call validate on the collection, then you...

Speaker 3 [17:54]

you get

Speaker 2 [17:55]

get a validation error if there would be at least one row that would have been removed by the validation rules, because either you have some filter rule that is violated, or you have some schema-specific rules that are violated, and those rows should not be part of the result. So validate would then raise the error, and similar to the schemas, calling filter would give you back the good rows and the bad rows, and the bad rows are part of this failure object which allows introspection of the reasons behind the validation failures. The last topic I want to talk about is testing. Testing data pipelines is often not trivial, and while you will probably run some sort of end-to-end test on a full data set once in a while, writing unit tests can be extremely useful during development and also to localize faults in your pipeline code. But when writing unit tests for part of a data pipeline, for instance, a feature computation function, you often need to create data frames with some content that you want to test against. And DataFrameD provides means to generate sample data that can be used for unit testing. The first useful feature is that you can create empty data frames with a valid schema by creating calling create empty on a schema. The second feature is that you can sample random data on a schema by using the sample function and just passing the number of rows that you want to have. This will also take care of sampling rows that are valid with respect to the custom checks that you defined and we use fuzzy sampling for that. Also what we find quite useful is that you sometimes want to override certain rows with certain values inside a data frame, and with sample overrides, the overrides parameter that you can see here, you can do exactly that. You can override certain values for certain rows in certain columns using this parameter. Notably, you can use sampling data not only on the schema level, which I just demonstrated But you can also do this on the collection level. So you can use sample on the collection level and then sample multiple data frames at once. Finally, we want to share some experiences how DataFrame.ly has affected our work on data pipelines at Quantco. First, validating data frames has greatly improved the legibility and robustness of our code. Thanks to DataFrame.ly, we are much more confident in our rewritten pipeline than we were in the original legacy pipeline. Second, the statically typed APIs define contracts that increase code correctness and quality. Data scientists at Quantico love data-friendly and are starting to use it to define schemas even for single-feature computation functions, as the APIs that are defined with data-friendly are just crystal clear and really easy to comprehend. It also reduces the risk of one data scientist exclusively knowing certain parts of the codebase, so people are able to work on different parts of the codebase, even though they haven't originally written that part of the codebase. Filtering has made introspection of pipeline failures more efficient and effective. We've demonstrated that we can safely soft validate data frames in production by filtering and then logging the invalid rows. We've seen the different methods that are available on a failure info object. And finally, setting up sample data for unit tests has become much easier. And actually our tests are now much shorter and easier to review and to comprehend. So now, if you're tired of explaining your data pipeline over and over again to your coworkers, we hope that we've motivated you enough to give DataFrame.ly a try. It's open source as of last week, and we would love to hear your opinion on it.

Speaker 3 [22:25]

Thank you, guys. And we have a couple of questions, more than a couple, to be honest. And let's start with the first one. The data frame has to fit in memory, I imagine. Any way one can validate bigger data sets yet?

Speaker 1 [22:43]

So I don't think this or any any computations are an inherent or would be an inherent constraint for data framely So if you can if you can use polars to process your data frame out of memory Data framely also supports that So essentially what data framely does for the validation is at first builds a lazy lazy frame with all of the computations And then collects them If polars supports that So ports collecting this data frame out of core or with a data set, yeah, for a data set that's bigger than the memory, then it just works as well.

Speaker 3 [23:24]

thank you so as you can imagine there are a lot of pandas adopters here so there are questions regarding is data framely comparable with can you use it with pandas

Speaker 2 [23:39]

us? No, no.

Speaker 3 [23:41]

And is there any options for the Pandas adopters to use this kind of validation that you can suggest as an option?

Speaker 1 [23:56]

Yeah, so currently no, so at least not via DataFrame.ly. So DataFrame.ly only supports polars because we at Quantco are heavily invested using polars for any new data pipelines or for any new code that we write, really. So yeah, right now we purposefully excluded pandas from this package to reduce complexity. So you'll have to use polars, but that might be a good idea anyways.

Speaker 2 [24:27]

Just maybe one addition. I think for Pandera, the feature matrix for pandas is actually also more filled than it was before. So these were just the Polar's features in Pandera. So for pandas, there might be more features in Pandera.

Speaker 3 [24:40]

So take a look for pandora, maybe it is an option The next one is also about adoption of another frameworks Have you looked into narwhals to maybe extend data frame lead to multiple data frame libraries?

Speaker 2 [24:57]

Well, when we started working on DataFramely, Nowels was quite small. I think they started last year in February or so, and we started a bit before that. And for us, the benefit of investing into Nowels right now is limited because new pipelines, we generally want to write in polars. And therefore, we can avoid the extra indirection and the feature lag that would come with an extra library. But I think in general we are open to this suggestion and you feel free whoever asked this question to create an issue on our Open source repository and we can discuss there

Speaker 3 [25:32]

Good. The next one is regarding the disadvantages of strong validation. So do you know about any that you can share? And so, for example, some overhead of time or defined schemas and so on.

Speaker 1 [25:51]

I think if you think about defining a lot of schemas in a lot of places, so we only talked about enhancing code readability with schemas. You can drive this to the extreme and annotate absolutely every function that you write with these type data frames and then validate every time you pass something in or return something. At this point, it's debatable whether it enhances code readability. So in our experience, verifying or validating DataFrame content is especially useful at application boundaries or team boundaries where you interface with other people and want to communicate what's inside the DataFrame. You shouldn't use it for every function that you write for a DataFrame. So much about the legibility part. And then regarding the runtime impact, obviously, validation has a runtime impact. But generally, since everything here is or everything that Data-Friendly does is native Polar, so it's really, really fast, in our experience, the time penalty that you pay for running validation is absolutely worth it, given the benefits that it has.

Speaker 3 [27:08]

Great, thank you. And also people found out that you have the version 1.0.0. So is it an indicator that it is production ready?

Speaker 2 [27:22]

Well, we are using it in production, and therefore it's an indicator it is.

Speaker 3 [27:26]

Yeah, that's a great answer Okay In order to validate a lazy data frame does data framely first have to collect it Might it cause a bigger computational pause if it needs to do that?

Speaker 1 [27:44]

I didn't get the second part of the question.

Speaker 3 [27:46]

might cause a bigger computation pause if it needs to do that so is it need to collect the data frame before doing something with it

Speaker 1 [27:55]

So, you can pass, so you don't have to collect a data frame before you pass it to DataFrame. DataFrame itself has to collect it at some point, because especially if you call validate, DataFrame has to raise an error if your DataFrame contents are incorrect, so you have to collect your DataFrame contents in order to evaluate that. Also, if you use the filter method, at this point, it was different at some point in the past but at this point filter also returns an eager data frame so somewhere within the function body data frame you actually collect one of the main reasons why we do that is if you access the failure object and the data frame with the good rows essentially you would have to collect twice if you wouldn't collect within the filtering step because then you would have just two disjoint computational graphs so yeah currently it returns where it does return a an ego data frame and collects somewhere

Speaker 3 [28:56]

Okay, good. So could you show examples of how the value object looks like? Does it simply return a problematic rows or also show the rules which were violated?

Speaker 1 [29:11]

So, I think, in general, the failures object is really just a standard Python object that provides you with a couple of utility functions, so counts returns a dictionary that maps rules to integers, co-occurrence counts returns a dictionary which maps essentially lists of rules to an integer. And then failure invalid provides you with a data frame. Failure object can't do more than that. I think it also allows accessing the list of rules which were generally included in the validation. That's everything that was useful for us for debugging. Obviously if anyone has anything more in mind, feel free to approach us or open an issue and we're happy to discuss that.

Speaker 3 [30:04]

Okay, great. So there are also a couple of questions. I'll post it in the correspondent channel of Discord. So yeah, people are pretty interested in your product. So that's a great success. So thank you. Let's thank once again our today's presenters.