Data as (Python) Code

Speaker 1 [00:07]

Thank you very much. All right, Data as Python code. Data as code is not an abstract concept, it's quite the opposite. It is a way of working that I first introduced in my company around two years ago and then I came up a name for and a paradigm for distributing version data as code is a definition I spent quite some time to find in preparation for this presentation today. So my goal today is to enable you to work in this way. So let's start. Short introduction about myself, Francesco. I love outdoor sport, as you can see from the picture. I cannot avoid mentioning I'm a physicist. I am especially proud about my research career that I was one of the pioneers using deep neural networks in in quantum monte carlo i currently work as machine learning engineer at eon energy deutschland that i feel fortunate to represent today and i also have my own munchkin academia company where i try to teach agile and clean code with fun to researchers specifically computational researchers All right. Whenever I need to explain something, I try to remember these words from Confucius. I forget what you tell me. What you show me, I remember. What you make me do, I understand. Now, in this format today, I cannot come to each one of you and make you do something. So please forgive me, but the best I can do is to show you something. And I hope you will remember, because then you will have a chance to practice by yourself, because all the examples we will go through are actually also available on GitHub, and there will be a QR code that will lead you exactly to the folder for each example, so you can replicate. And throughout these examples, we will come to certain what I call meditation checkpoint, which hopefully will motivate you why data as code is hopefully a good idea. I also think it's important to make clear what I will not talk about. So, for example, I will not talk about unstructured data, only tabular data. And I will not talk about stream data, only batch data. This doesn't mean that data as code cannot work in this scenario, it's just I never work with data as code in this situation, so I don't want to go into this area. So, it all started from this. So, in between my research career and my current career as an engineer, I worked as a data scientist for quite some time. And one thing that always struck me is the fact that sklearn has some data sets and you can load them as Pandas DataFrame very conveniently, right? So you pip install scikit-learn and then load and bam, you have some data. And I always thought that's so convenient, pretty nice. There's not only wine data, there are also other data sets available. And yeah, today let's do this. Can we replicate this? Can we do it with our own data? How hard can it be? So let's do it. Simplest possible implementation. This is also a sort of crash course on building a Python package, in case you have never done it. So we create a folder structure like this one, and we need some data, right? So we have this energy parquet inside the source that contains our data. We have a manifest.in where we specify that in our package there should be this parquet file included. And pyproject.oml, we just specify the name of the package, the version, dependencies. And then the load.py, which is basically a pandas read parquet. Nothing harder than that. And I need it just for convenience. Let's expose this load function. All right, that's all. That's all you need. Then once you have done this, you can use Python build to build your package, your wheel in this case. And you can install your wheel wherever you are in your virtual environment. And then once you do that with Python, you can import load, run this load function, and bam, you have your data. Easy. However, you see, here there's a problem. We are embedding, like, the parquet file into your wheel. I mean, this wheel will become quite fat very quickly. Let's avoid that. But this is very easy. So what you can do is put some data in the cloud, get rid of this manifest and parquet file in our folder structure, and change the load. For example, in this case, I use, again, Pandas and rely on ADLFS that lets you read data directly from an Azure storage. In this case, it's a public storage, so anyone can access it, so you can replicate this example. And then, yeah, don't forget to add the IDLFS dependency. If you do that, you build your package, and then you send this wheel to someone else, he will be able to do literally the same. So do you notice any difference in the code? No, there's not. And this is very nice. We already see one nice feature about this data as code, which is we can move data around without impacting the user. So whether it's in your wheel, embedded, it's on an Azure storage, it's on a Google Cloud storage, it's in Snowflake, it doesn't matter. The user doesn't care. It gets up on this data frame. Pretty nice. Nice abstraction. However, like in reality, most of the time you have to do with sensitive data. So you probably don't want to expose them like this. Again, simple solution. just, for example, on your Azure storage, you don't make it public, you make it private. So you change the storage options, anonymous, false, and you give access to the data only to the Azure identities you want to allow. And so the difference at this point is that when the user installs this package, first needs to do a Z-login, so identify himself, and only then can load the data. If you fail this or you are not allowed, you will not be able to access the data. So this shows that in this way, with this approach, so if you are pushing this wheel into a PyPy index, you're basically saying, hey, there is some data, so data presence is public, so to say, but data content is private, which is, I think, reasonable. Then another thing as we can add at this point as a data scientist, I always hated this hard-coded columns, I don't know about you, in my code, it was always very ugly. Can we maybe add these column names to this package? Let's do that, how hard can it be? So we add a schema file and inside of it we introduce a class schema and we put these attributes corresponding to the column name. and we may be also expose the schema and when you then load your package you can load also the schema and what you can do at this point you see I am loading the data frame the F and then I'm selecting just one single column notice no hard coded string so the column name comes with the package again nice so in this This way, we can avoid hard-coded strings, as I say, but we can also allow the data producer to rename the row columns without impacting the user. Again, an abstraction. Can be useful in some cases. Well, since we introduced the schema, what else can we do with it? Are you familiar with Pandera? If you don't, look it up, it's very important. We are going to use it. So for example, let's add column description and types. So this is our schema. We are now using Pandera. It's sort of pydantic for data, OK? Something similar. And you see, source, now we don't put just the column name, which is a SIEC name, but we put a serious string. This is saying this column is made of strings. And this field, the alias, corresponds to the row column name. And then we are also able to put a description. And we do this for all columns. So now, at this point, we are sending to the user more information. Don't forget to add Pandera into the dependencies, of course. Otherwise, it will not work. And at this point, the user will be able to inspect the schema and really look at it and get all this information that we put. So this is also, I mean, quite nice, right? So now we are shipping data and metadata together to the user. It's like one other thing that came up to my mind here is one of the cool things of Python, right, is this help that in the interactive mode you can ask help about a class. This is something similar. You can get information about your data automatically. Okay. What about some more custom constraint? Like can we put even more information and build some data quality? Sure, why not? So again, using Pandera, here, for example, for the source of energy, I'm using this keyword, isIn. So these strings are allowed to take only one of these values, nothing else. Nullable false cannot include null values. And we can do this for all columns. And we can also add some custom check. For example, here, for each location in here, I know that the source total must be greater than any other source. And I'm checking it. It's a bit convoluted, the code. It doesn't matter. But I wrote a check for that. And now, when the user takes the schema, not only can inspect and see all this information, but also has this possibility, at least with Pandera, it comes automatically. Schema.validate. So it can check that the data frame fulfills these requirements. So this description that I put about the data is not just information that is not true. You can verify that it's respected. And this is already really nice. However, what wouldn't be great is if the user received the data and has the guarantee that the schema is respected. That would be, like, a next step. And for this reason, I introduced this CLI open source tool that you can find on GitHub. MIT license, so very permissive. That also simplifies the building of this data code package. So at this point, you need just the usual load and schema file and an additional requirements that contains only the dependencies that you need. and the building is a little bit different now so you need just to create your usual virtual environment install DAC, this CLI tool together with your requirements and then what you can do is use this command DAC pack, specify your load schema package dependencies package name, package version and this, when you run this command will do two things Number one, run schema.validate, and number two, build the package. So, if you build your data as code wheel using this, you are guaranteed that the schema is respected, which means that you have automatically quality check in your data. And here we come to our first meditation checkpoint, so data docs and data contracts. So, data, so documentation about data are like code comments, cannot be trusted. If someone writing a markdown, oh yeah, this string is made out of floats, yeah, sure, that's what you write here. But can I trust it? The difference here is that with data as code, schema can be trusted because it's executable and executed code. And for what concerns data contracts I hope you are familiar with the concept So there are many implementations around Many are coming with a vendor So I don't know AWS as its implementation for example And they come with a vendor lock-in in the end So you use it only in a certain environment that they give you And in other cases there are attempts of being too agnostic So, for example, they push the idea that the schema should be defined as a YAML file, which I think comes with a lot of problems because not all checks can be written in a YAML file. I mean, for example, the e-zine or the nullable, yes, but this custom check I wrote about the data frame that you saw afterwards, you cannot write in a YAML. And sometimes it's necessary. So, per contrast, data as code does not require any special tool. So no lock-in, as you have seen, you can use this Datascode CLI if you want. Don't use it. There's no reason why. And Datascode is also tool-specific, the tool of your choice. So, for example, here I made an implementation with Python, but you could do it with Go. See, it's the same. I use Pandas. Why don't you use Polar, Stask, Spark, set of Azure Storage SAs, Snowflake, Delta Lake, whatever you like. All right, so this is, I think, already, we achieve already a lot, right? We have built-in data quality now. Now, what about data versioning? I mean, nowadays, if you're serious about any data pipeline, you must version your data, right? So let's see here. So suppose you have your data warehouse, let's say, and every week you take your snapshot, so your immutable data set, one per week, so week one, data set one, week two, data set two, and so on. Then what is the role of data as code here? Well, pretty simple, actually. For each data set, you make a new release with a new package. We have the only difference is the URL. So you see the point, slightly different data. I mean, the path is slightly different, the version of the data. So, in this sense, the test code is a way to distribute version data. So, the version of the data is your problem. You have to do it up front. The test code helps you just in distributing it to the user. However, here, like, maybe you notice, have you ever seen a Python package whose version is V1 or V2 or V3? I didn't. Usually it's semantic versioning, I mean, use semantic versioning. And here we have, actually, an opportunity. So semantic versioning. What is semantic versioning for data at this point is a good question. So in software, semantic versioning has a clear meaning, right? So first number, major, you increase it when there is a breaking change. Patch is when you fix something. So same functionality as before, just something was not working as expected. minor when you add a new feature without breaking change and now for data what do we can we use the same pattern well why not so major let's say major is a breaking chain so maybe when you remove the column or change the column type or change the constraint in a word if you change the schema a patch is when you fix something so maybe you release some data but you realize that some values sorry we're wrong so they intended content is unchanged but actually has been fixed and finally minor is when you make a new release for example a batch data a new data set that's that's for me a minor that's my opinion and so and this is the part that you usually automate so whenever like you have a a pipeline that every week make a new release, you make new minor releases. Okay. So let's see an example so we get a bit more of a feeling about that. So suppose I have my energy data set that we used before, and I make a first release, 1.0.0. Then I have some new rows, so I want to do a minor release, 1.1.0. By the way, for this DAC, the CLI tool can help you in finding the next version for the automation, can be helpful. And then we decide to change the schema. Okay, we go to version 2.0.0. Great. Then we are here, and then we suddenly notice there's an issue. In our data, there were a lot of numbers that shouldn't be there. And maybe we make patch releases, notice, for all of them. Because because usually you track your repository with Git, right? So you can go back, for example, even to the version 1.0.0, make a patch, and release it. So you can release patch for all of them. And then maybe you move data to the cloud for the version 2, at least, and you make another patch release. Okay. Notice that from here, then, you could also go on with new minor releases, both for version 1 and version 2 in parallel. You could do it. Actually, we have done it, so I can guarantee that it's possible and it works. And here we arrive at the second meditation checkpoint. So semantic version introduces a new language. And there's this saying, your thinking is limited just by your language. So now we have new possibilities. we have a way to communicate breaking changes the data, so your data pipeline is likely to break if you increase the measure of your data communicate fixes, which means please pull them as soon as possible, rebuild your model, redo the inference whatever this data set has been used for and enable a safe way to introduce breaking changes and long term support as I show you you can maintain in parallel version 1 and version 2 so you say you are the data engineer you make it release version 2 but you know that data scientists need some time to adapt so you continue releasing version 1 in the old format so they can the pipeline keep working like that and then when data scientists are ready you they start using version 2 pretty nice and now let's let's go to the last part which in my opinion is the real reason why data is called this is useful so let's go again to the situation where there's data scientists which is a model and now data scientists art in the dependency of this model this data is called package and it's specify okay my mother works with data is called energy version one only version one so if not not with version 2. So then when you have your, say, model training pipeline, okay, and the engineers try to run the model against this data-scaled energy data set, it will work with version 1, and next week with the version 1.1, it will work fine, no problem. But if it will try to do it with version 2, it will fail. It will not even be able to start the running, because in the same virtual environment, he He will not be able to install model and data as code, which means data, at the same time. So it's a way of the data scientist to protect itself from unwanted data. So this, in principle, brings you the guarantee that you won't have any broken pipeline anymore if you work properly. What does it mean? the model must be properly tested, right? So it must always work with version one. That's the guarantee that you have to give. And there are ways of doing that. Now I can't go in depth in this because it will take another half an hour basically. But if your schema is also able to create some synthetic data, like an example, that you can run your code against in your test, then you can come to the guarantee that your model will always work with this version of the schema. And you can do it by yourself, or if you prefer, there are actually very good frameworks, again, Pandera, Hypothesis, and they can help you in that, and you can work with code written in this way, with this decorator, Hypothesis.given. And this is basically the test you have to write. You don't have to write anything else. And it will automatically generate many random data frame and check that your model runs against that. So if you do that, then you have the guarantee. And here we come to the third and last meditation checkpoint. So I think in data science, the real difficulties come from the friction between data and code. It's similar to REST API, when you receive the JSON payload, and God knows what the user will send you as a JSON, right? And the same with data, like which data will I get this week, who knows? And here, like, comes most of the problem. But now if we make this twist and we start treating data as code, then now data is not integrated into the software engineering. We have shifted the problem, and we are basically talking about software compatibility, which is a, I mean, well-known problem that's been addressed for many, many, many decades. So we have a good solution for that, and it's the one that I explained to you, right? Yes. So that's it. Thank you very much. If you want to read more about Data as Code, this is the website. If you want to connect with me, maybe please add a note so that I know why you are trying to connect with me. Thank you.

Speaker 2 [24:36]

I was just raising the five minutes left card, but you finished, so we have more time for questions. That actually gives me the opportunity, when I was talking about social event, I forgot to introduce you, saying you are a proud generalist, refusing to specialize in one specific area. So, what makes you say that, and what advantages do you have as a generalist?

Speaker 1 [25:02]

Well, I don't know about the advantage, but it's just how I am. I mean, I consider myself, first of all, as a human being, and I think it's part of being human to be curious, so I always want to learn about new things. That's all.

Speaker 2 [25:14]

I respect that because these days I hear a lot about these T-shaped personalities or M, like specializing in one or several areas, and you say, no, I'm just going to be a judge.

Speaker 1 [25:26]

I mean, eventually, I guess in life, you end specializing into something, right? But it's not my drive.

Speaker 2 [25:34]

That's nice. So, among the questions, there is one, got few votes, and it's, the metadata maintenance looks quite heavy. How would you convince and onboard your colleagues for that?

Speaker 1 [25:54]

I don't think it's heavy at all, because I say these minor releases, which are the automated one, come with always the same metadata, the same exact schema. And it's true that, especially at the beginning, when you try to set this schema, you will have a lot of discussion. Data scientists, data engineers, and business will have to talk about it. By the way, maybe I take the opportunity also to mention a fun story about that. so we implemented this and we work for inference data for example we were receiving some CSV from the business we agreed on a schema with them and we implemented and then every week we were getting this data and using it for inference and then what was happening very often is that business were changing the rules without telling us and then we were coming here It's not working anymore. You broke something. I say, oh, yes, we had some changes, but, you know, we forgot to tell you. Oops. Yes, oops. You know how we solve it. We basically automated that whenever this check was failing, business were automatically receiving a message in Teams saying, hey, the pipeline is broken because you are not respecting the schema you agreed upon. So let us know once you fix it. These are the details of the failure. Please fix it, which is like pushing away responsibility. And it's the advantage of data contracts. And so maybe this also motivates. It's a bit of an effort to come with a schema, but it definitely pays off.

Speaker 2 [27:34]

That actually is a nice segue for some of the questions around that, the collaboration with stakeholders. So one of them is about making the data available within the organization with your approach now. Wouldn't it be easier to use a data catalog? So question and answer.

Speaker 1 [27:54]

answer yes of course this is in a way is an alternative to a data catalog if you

Speaker 2 [27:54]

Yes.

Speaker 1 [28:00]

want or you can build a data catalog on top of that you decide what else can I say it is as it is but I think maybe one important aspect to keep in mind is that because I received this question is that data is code makes sense for people who are coding. So don't expect business to use this, right? I mean, people who work with Excel, they will not take any advantage from this. So always keep this in mind when you are thinking about your data catalog.

Speaker 2 [28:36]

Okay, so I was wrong with the time left, obviously, because it seems that we have only one more minute.

Speaker 1 [28:46]

I have one question for the audience. I hope I will have the chance.

Speaker 2 [28:49]

Okay, then take the one minute.

Speaker 1 [28:49]

Okay. I will share the questions with you.

Speaker 2 [28:51]

I will share.

Speaker 1 [28:53]

Maybe less than one minute. If anyone is deep on behind the curtains of the mechanics of PIP and PyPy, please reach out because I would need some help to go a bit further.

Speaker 2 [29:08]

So that's new so if you have questions come to him if you have answers come to him as well Thanks a lot again for your insightful presentation