The pragmatic Pythonic data engineer

Speaker 1 [00:06]

Hello, hello. Good afternoon. Thanks so much for your valuable time. Hope you had a great lunch, a great morning. This first starting. And today, the idea is to advocate a bit about the data engineer, specifically with Python. I won't talk about the tooling, but more about concepts. And of course, Python, how we can make the things more Pythonic. first of all some disclaimers here just to let you know that I'm not endorsed or affiliated to any conference on foundation this is a very condensed overview about what I wanted to say you have like just like 40 minutes so I had it too short sometimes I will be a bit rushed but like feel free to hands up your hands ask any time even you have like a QA in the end if you don't understand or have anything, just hands up. Unfortunately, I planned like a live encoding, but unfortunately, I cannot make it. The time is too tight. It's not about operational data or platform itself. It's just about the components, the opinions on my own. And of course, feel free to reach me out after the talk any time here or in my social contacts. Introducing myself a bit. So, I'm Brazilian. I've been living here in Berlin for almost eight years. I write Python code for more than 20 years. I started Python 2.2 when I read this book first, when they were released. Since, like, 2011, I've been organizing meetups, conferences, mostly in Brazil. I had a pleasure to organize like the Python Brazil in 2015. Also, I'm inviting you. So if you want to travel to Brazil in October, you have a local hour Python Brazil edition in Rio de Janeiro. It's October 16th, 21. You'll be more than welcome to go. In the last 10 years, I've been focusing on data engineering and data ops. I've even worked for several markets, real estate, food tech. I spent several years at Microsoft and GitHub. And recently, I joined the two delivery heroes, so I could see different point of views how to manage data in different situations, in different scales. And in my free time, I really love cats. I tried to hunt the spicy foods in Berlin. I learned to paragliding, and now I'm learning to paramotor, so it's how I try to spend my free time. Today, I wanted to discuss a bit more philosophy, why to discuss data engineer fundamentals in general, and then you're going to expose anatomy of data products, pretty much generic, but any data product has, like, a specific anatomy. You're going to talk about the data formats. That's very important when you need to link those components. Or flow management, how you manage several steps of your data, how you distribute your code to run at the scale. And most important, and sometimes people neglect it, is, like, about data testing and validation. and why you needed to discuss this kind of of topic first like being pragmatic is the key here because you want you to solve problems and solve problems doesn't depend on silver bullets why because when you go like to blog posts to medium you tend to see like a whole fascinating those companies solve their problems let that scale they link several components and they make a lot of magic and they release like a product as an open source and they started to to make a lot of things on top of that but people don't really think sometimes that like being pragmatic is like what problem i needed to solve what they scale off my data so practical thinking and to solve those problems and a real world that's what i'm trying to to show here and i'm still advocate that of course you have several tools several companies doing a really nice job but like what you call the traditional methods is still uh it's still a thing that you can rely on it's for a small or large company i see it for example in a work for big techs in general i mean like working like in a petabyte scale and then traditional methods still can handle in a very affordable way ways to to manage your data and the first thing you needed to answer is why do i need like a data engineer basically because you have a lot of mass information flying around the world and then you need to structure it you need to organize it and then you need to make accessible for for the people how you make it it needed to be reliable you needed to be efficient from the point of view like affordable costs because you cannot spend like millions of dollars to spin up several clusters to process data so you have several techniques, you have several ways to make it like in a low cost but also you need to think how to scale up in the future. Don't believe like in silver bullets, try to think and try to make this puzzle by yourself because other things is that your data mostly doesn't fit in your machine. Even you have like a super machine, so your data might fit there, but you have a problem to process there, so you need several aspects of your data you need to fit into to have like to access our data, to process our data in an affordable way. And the most important is like data is part of your business all the time. If they are not part of your business, most probably data is your business i see a lot of sponsors here that the data is the business so you have like that cliche for 10 years ago that the data is the new oil the new energy whatever so imagine that like treat your data like uh you treat yourself like it and you are in a python conference like of course i needed to say like why do we need to choose Python here? First of all, the vast amount of resources that you have. You have several open sources. They are well-maintained. You have many tutorials. You have a very diverse community that spreads the knowledge, meetups. It's a growing ecosystem. Every month, you have a new tool, a new technique, someone trying to make something different. Most of the tooling that's not Python is well integrated with the Python ecosystem as like a generic tool for example who works with the data engineer probably has at least one component from the Apache foundation in your pipelines in your architecture and most of these components are right based on Java, JVM or something but most important that the Python itself it's on top of most of those tools and also that our house assistance from the cloud providers also provide like native python integration if you need to generate like user-defined functions or if you wanted to run small jobs for any most biggest providers let's talk about like the anatomy of our data product it's pretty generic any data product has a similar approach first of all you have sources for you have like a data product you need to have a data where the data flowing one there's several ways for that it can be like an IPI a database it can be like images and structured data streaming data you have your app you have your website so users are interacting with that they are collecting events and then we have the sources this is a very first part but for that we need like a common format something that can be translated and be efficient that you can consume one data structure validate and then you need to transform it any transformation step relies on a data storage. It can be ephemeral, can be like a distributed file system, can be in memory, independent, but they need to be stored somewhere. And then this transformation happens because you get your data, you make some cleanup, you make some change, whatever you want to do, but in the end, they need to be stored. And in the second, you need, for the third, sorry, So you need to serve it somehow. Probably you put in a database that the people you queue that, or you expose over an API, or you send it back to another system. Independent of that, so you need to have like this transportation layer, this serving layer. And to manage it, everything, so you need to rely on orchestration. So when you link all the points, we need all the tasks, every step that you need to make your data. And, of course, you need to observe it. You need to monitor it to make sure that everything is running smoothly so you don't have errors. If you have errors, you need to be alerted. And then this is the anatomy of basically any data product in the world, of course, with your fancy names, fancy products, fancy tools. But like our body, everything is an arm. Everything is a leg. So for the data product, you have like those steps. So, I'm proposing here in the next 10, 15 minutes, it's like to imagine to build a data product in our mind. So, as I mentioned before, the first thing is to define a data format, so how you transfer the data between all those steps. There's several choices. People use, for example, JSON, Parquet, CSVs, or they choose. For this moment, I'm suggesting here a Pythonic way to do it with PyNero. PyNero is a project from Apache that's like writing in C and parts in C++, but they have like a wrapper over the Python. So it's a high-efficient like analytics platform in memory. This is very important that avoids serializations between the data formats. it's like it's very interoperable. They have a lot of interoperability because we don't need to serialize the data between them. It means that, like, for example, if in your pipeline you have consuming, like, an IPI or something, so you can push, send it to, like, to, or create, like, your PyNero structure, and then they will be available in memory for you, and then you can work as much as you can. But, like, as you can see, like, PyNero can be, like, a data exchange format. It's a memory format. They are network formats because, of course, you navigate between the network. It's a storage format because you can choose how you want it to write to make it to the storage. It's an IO format because you have, like, this transfer between the CPU, GPUs, and memory. It's a carry engineer. So, if you wanted to perform some analytics, they provide APIs for you. Also, they are partition managers. So, if you have a massive amount of data, you can partition your data. You can manage it. They work pretty smoothly with data frames from Pandas, from Polar. So, if you have a legacy code, if you have another step in your pipeline that uses those tools, you can easily use PyNero. So in a very Pythonic way and also they have like a vector computing So if you are doing like things with GPU and all this nice stuff is that's come up So Pynero is very affordable for for it It's very easy to use just the PPI style Pynero So you have it's unavailable in your virtual environment Just push and here is a code sample for for it. Basically Basically, you can push any data. Here, I create a list of dictionaries with a structure, and then you can define a schema. These schemas can be very detailed, so they support a complex type, list, dictionary, nested formats, and in the end, you can use the table or from PyList to create your structure. So it can be from list, from dictionary, from external files from data frames so you have like a vast amount of options for for for work and also as i mentioned so uh you can also partition your menu you partition your data so in our case here what you are suggesting is basically you get a lot of this data from the talks and then you can partition for data for example and you can see that for example that you generate Farparker files for you. So it's the way. So when we needed to read this data set, so they will come to the data for the PyNero, their arrow formats. Again, it's pretty, pretty, pretty easy to make and then makes you more reliable because when you are defining, we are talking if you are stakeholders, you can define the schema of your data format, like basic validations, and then you make sure that the data that's flowing to your pipeline into your system is like at least pre-validated at some point and the second step is like to orchestration and observability so in fact this is like the tool that most of the people use there's other choices in the market but then Airflow. Airflow allows you to use, like, Python codes to create, like, graphs or tasks that link to each other, so you can schedule those tasks, and they also provide, like, a set of tools, like, including, like, a graphic UI that you can see, that you can see your, what you call the DAG, I'll explain the next slide. they have like dozens like hundreds of integrations with several services several libraries so you can customize by yourself if you want for any type of workflow imagine again you need to go to any API push that data do a snapshot to our database do the snapshot make some cleanup make some analysis and then output this data to other places, so Airflow works straight for you. But the most important is that they also have internal mechanisms that allow you to observe it, so they collect metrics that you can use like your tooling for metrics like primitives, data dogs, whatever you want to use, you can link on top, and then you can have graphics understand the whole health of your ecosystem, your platform working. And then some of the user cases like ELT or ETL, depends who you call it. So automated generator reports, if you needed to generate reports on a daily basis, for example, backup tasks, snapshotting tasks, and as well as the uses for training models for machine learning. Some of the concepts that you need to keep in mind. When you say in Airflow that you have a DAG, basically you have a Python code. Python code, how you structure, so you define the configuration for that context, how many operators do you have. Operators is a unit, a YERC unit that can do everything you want. Connect it to a service, execute a query in a database, write to somewhere to make some analysis, to execute a command a operator can be extended it can be customized as much as you can and then you have the tasks basically tasks are the graph itself so how you link one operator in each other in a context of an attack and then you have like a sequence that means like directly a single Basically, it's a graph with just one direction, so you can configure it. But one of the most powerful features in Airflow is the templating. So if you are using it to use engine templates for other projects, you can use the same concept like in Airflow, so you can create your macros. You can standardize like commands, queries, files, everything is on a template. So it's pretty much useful. And Airflow works with five main components. You have what you call the metadata database. It's a relational database that controls all the aspects of the Airflow. And then in the core, you have the executors and the schedulers. Scheduler basically is the crown job of the Airflow. Basically, you can configure how your DAG you run. So you configure the rules. if each hour, if 30 minutes, if daily, if every Monday. So it's just up to you. And the executor is the key who manage how you create the workers. The workers is the code of the operator that you run, and specifically you need to work. Depends how you deploy it. For example, if you're deploying a Kubernetes, so they will spin up a pod for you, specifically for that piece of code. Then you run that code in the pod. that you get the result, this is a worker. They have an accelerator work or they can execute in the same machine. And as I mentioned, you have the code that makes everything happens. That's what you call it the DAG. And in the end, so when you deploy or use Airflow as a service, you have a web server and a UI that you can interact with. It's a bit small, but I will try to explain a bit. And in this part, so when you create like the DAG context we import, so you create the context for a DAG, so you can define the scheduler, you can define when this DAG starts, the name of the DAG, some DAGs. and for the second you have the operators you instantiate the operators as i mentioned to you have a hundred uh hundred of choices so you can connect for example to azure amazon google third-party tools you can create your own operator custom operator basically you chain it the third one is if you wanted to create a template if you want to for example to execute a complex command line for specifically serving a remote server, for example, you can create a template for that, use the parameters and they will render it for you in their flow. And the magic happens on the last line here, where you change, where you put your tasks to work together. Like you say, this task depends on this task, that depends on this task, if one task fails, don't run the next one or run the next one and it turns into a graph key UI for you. First of all so when you deploy your DAG to Airflow pretty much easy probably gonna see something like that for example our PyCon DAG that you generated if you get into the DAG probably you're gonna see a kind of graph for that so and each step is one worker each worker has your results that has our own log so and it's completely monitored by the airflow now that you have like a data format not now you have like your orchestration sometimes you need to think how to scale up this project along the time and the choice is This is Spark. But this Spark is writing in Java in a Python conference. But the most important here is talking about PySpark. That's the wrapper on top of the Spark. And why? Because it's a real-time and large-scale processing. I will explain how it works behind the scenes. Because even if it's a Java or Scala tooling, what is executed in the workers is a Python code. And it's because you are here. Because there is like this magic with the Py4j here and supports all the workloads from Spark, including the machine learning. So, if you are doing machine learning with Spark, so you can use PySpark. And the concept here is very simple. You have like the Spark context. That's the code that you are writing. So, you are getting the data from somewhere. You are doing transformations. You are doing several tasks on top of that. What's behind the scenes the PySpark does? They serialize your code, your Python code, through the Py4j. They distribute this serialized Java object to the workers. You can have, like, any worker you want. So it depends on the capability of your cluster. And the worker serializes it to the Python code again. Execute everything they need, they shuffle in the data, they coordinate our data, they get the result, serialize it to Java again, and serialize it to Python again as a result. This is the magic behind the PySpark. But, it's pretty much easy, like take this as an example. Spark works with the same concept data frame as Pandas, like a very rough way. So here you are creating three lines in this data frame. And let's say, for example, that I wanted to create a function that announced the talks for all the data that I have inside my data frame. What do you do? You take this method that you serialize in a Java object, that you spread it to the workers, that you get the data that you create. imagine that you have like a million of rows, then you divide it between the works, then you apply the function to this, the function to all the, the function to all workers in Python, get the results, and then you show up for us here. Basically, it's a map operation. The magic behind it is like Python, Java, Python, Python, Java, Python. But you don't need, like, if you wanted to use, like, a more affordable API, they do the same. But, for example, you can do, like, a filtering, for example, like, in a very Pythonic way. So, you can use, like, the native objects or you can use a SQL. You just need to register a temporary data set. And in the context of Spark, this will be like a viewer table, something like that in memory, that are available in the Spark cluster. And then you can execute in a SQL. So very complex SQL queries if you wanted to do. And then, for example, you're going to see I put it's not good, but it's similar. And here, it's like creating a session, a Spark session. read the same parquet files that you wrote like in a pine arrow a few slides ago and then you can see that you have like a well-defined scheme or you can show up for the list test your data as you test your code so sometimes you neglected it but like it's very important to test your data avoid you're working so especially if you are in a small team and here you have like one framework called the great expectations that's a very platonic framework that's for the data quality you have several ones i will explain about the pydentic after but like one of the concepts of the great expectations is the collaboration between tech and non-tech people like when you define like your data what you expect of our data of course you have some expectations about that so should have duplicators or not or missing values a range of values negative numbers whatever you want and then how it works they work basically on the data frames from the pandas or the spark i put the two options here but basically you can read your your data source whatever you want for any format in our case we are using you are using our perky files generated and here I say that I don't want that this column all for cannot be new and then they gives you all the statistics based on that you can consume it is output and define your rules for example let's imagine in our our flow you have a nice step for data quality so you check our data and in case you have some problem in our data you can emit some monitoring some alert directly from the airflow linked with the great expectations all the cycle ends and by the end so you have the pidentic that also is a data validating tool but different from the great expectations they are more focused on your business. It's a data validation because you know your data, you know your business, and they have like some concepts. The base model, so when you need to create a validation on Pydentic, your data class, so you have like a Python data class, you need to extend for a base model. That's the subclass, and they have like error handling by default. When you instantiate a new class, let's say, a speaker that you show in the next one, if you use any wrong value, they automatically throw an exception for you, and they make the data validation for you. But the nice thing is that they use, like, the Python types annotation for that, so you don't need to do any other magic. Annotate your data like this. We have, like, a class called the speaker, so for properties. And then if you create like a wrong value, for example, the room here, the literals, I just accepted three values, I put one wrong here, probably you're going to have like an exception true for you. And then the question is, great expectations or pidentic? Both are validation tools and you decide for use. But keep in mind, great expectations is about what you expect from your data, like the basically validations and well-defined. It should contain new or not, range, negative numbers. And pidentic is more about business validation, let's say, because you know your data, your business. Work both together to understand. But like the last one, it's just not about data quality. It's just to save time and reduce errors. Because you can understand your data from the start. And then you can perform, for example, regression tests, performance tests for it. And by the end, it's like our data anatomy, more or less, how you can link those components. PyNero here, that you can navigate through the sources. So you have a data format, a common format that is affordable for most of the scenarios. The transformation, Spark, PySpark here, that make the transformations, they manage the data storage, and between you might have several validations for that. For serving, so PyNero also is a good choice. imagine you are working like in a huge project and then most of your people from the analytics are trying to to to make some some some analysis on your top so you can just provide an up and narrow point and for them and they can manage because they can use appendix if they want they can continue to use spark or any other tooling and to manage everything a passion airflow for for it and a big thank you for attending your value time here so if you want to contact me my linkedin my github is bsao just like i did dropping my message or if you wanted to talk directly my telegram is bsao zero and thank you so much

Speaker 2 [31:34]

Thank you so much for your talk. It was really insightful. So I have a couple of questions. So the first question is, can you recommend how to organize DAGs in Airflow if there are 50 plus of them? Are there folders? Does it make sense to enforce naming convention or tags?

Speaker 1 [31:57]

I'm sorry.

Speaker 2 [31:58]

Sorry, so can you recommend how to organize DAGs in Airflow if there are 50 plus of them? Okay.

Speaker 1 [32:05]

Airflow, in general, they have a standard for that. So any of your DAGs, by default, are inside the DAGs folder. So you have your project. You must have a DAGs folder by default. If you want to customize, you have some configurations that you can do. But the DAGs folder is the default. If you want to create custom operators, you have the plugins folder. So by default, you can change it as well. But in general, when you need to organize an Airflow project, you have mainly two folders, the DAG one and the plugin one. And then you put your code inside there. In general, you pack it in a Docker container with this structure. And when you install your Airflow, you have like the Airflow command line that you can perform several tasks on top.

Speaker 2 [32:57]

Thank you, so I have a next question, so it's from a Stefan so the question is Python to Java Java to Python serialization marathon looks tedious How much of a pain is that in practice example with respect to debugging? Sorry

Speaker 1 [33:20]

So, how much it's painful. How much it's painful. No, it's not painful because it's a problem that is already solved by the community. So, people working hard behind those open source projects, especially the PySpark or the Spark, they already handle all those problems for you. Specifically, in the newest versions of Spark, you don't even need more to configure several variables in your environment. So, if you wanted to use PySpark, PPI install PySpark, just need to have like Java installed in your machine. And all the magic behind it will happen transparently for you. Writing your Python code, when you execute, even a debug or in a command line, the magic to transform, it will happen transparently for you. No painful anymore.

Speaker 2 [34:12]

Thank you. So the next question is, how do you test your airflow in DAGs?

Speaker 1 [34:18]

On Distribute?

Speaker 2 [34:19]

Yeah, the test, test your airflow index, like the question.

Speaker 1 [34:19]

Aha. Airflow as an operator, as I mentioned, so when you create an operator, basically you extend on a class called the base operator. Like in PyIdentic, you have like the basic model. Similar approach for Airflow, so you have like the base operator. The base operator has some methods that you need to implement. How to test it? Unity test is because you can mock parts of your code. It's pretty much easy. introduce unity tests for airflow is very easy because an operator is a class so you can test at the class you can you can predict like what is the input what's the output you can mock parts of them for example connections or other outputs but overall testing airflow is a similar product of TDD unity tests

Speaker 2 [35:18]

Thank you. So I have one more question. So how do you handle validation error in production?

Speaker 1 [35:26]

Thank you.

Speaker 2 [35:27]

How do you handle validation errors in products?

Speaker 1 [35:31]

In production, so you handle the same way in your pipeline, so most of the people are, it's a good practice, so when you process your pipeline, in the end of your pipeline, probably you have some task for the test, for the validation, and that moment is how you test in production, because in production, your pipeline is running, so you are input or output your data to somewhere, and then you test there. For example, for great expectations, they have output with, like, information for you that you can store it in a database, for example, or you can send it to an observability platform. For the Pydantic, most probably you have, like, exceptions being handled in your code, and based on that, you can manage it. But, like, it works for production or development or staging, whatever, who you use. Thank you.

Speaker 2 [36:28]

Thank you, so I have like another question. I mean so the next question is Here what is the difference between streaming and eventing in incoming data?

Speaker 1 [36:48]

Streaming and events, they are pretty much similar, but the most different is that like you are streaming, you are consuming for some topic from some message queen. It's involuntary. You're just pushing. Event intends that like your external systems informs you. Like for example, one very common use case is CDC. For example, new techniques, you want it to be informed from your database that some role changed. do like snapshots from the whole database every day basically the database informs you they send a message to your system for a cafe car or any other machine system to say like this role change it they were updated deleted inserted and then in your pipeline you can just uh you can just manage in a row level but they are pretty much similar Thank you very much.

Speaker 2 [37:41]

Thank you, Robson Jr., for your talk. It was really an insightful talk. I'm sure we have a lot of questions. So after the talk, you can connect to Mr. Robson for the rest of the question. Thank you so much, everyone, for today. Enjoy the rest of the talk. Have a nice time ahead. Thank you so much.