Fight your garbage data: implementation of a pythonic data quality monitoring framework in PySpark

Speaker 1 [00:05]

welcome everyone to this session and just a few quick reminders before we start with the talk our Q&A that we have planned at the end of the session will be held via the talks app you might have heard already but I'm just going to repeat it so if you have any questions put them in the talks and then I will read them out to the speakers in the Q&A session of this talk and Also, you can rank the questions if you don't have any question on your own or you see interesting questions Like I will read them out by the ranking who gets the most likes gets read out first So just keep that in mind and always like to share on social media, of course as well hashtag PyCon.de But then now I have the honor to introduce you to the next two speakers Rostislav Kraso and Joshua Finger and they will present a talk about fight your garbage data implementation of a Pythonic data quality monitoring framework in PySpark. Please welcome them with a warm round of applause.

Speaker 2 [01:17]

So, yeah, thank you, Sonja, and welcome, everybody, to our talk. We're excited to have you all here and that you found your way up here into Helium, the third floor. Today, Rostislav and me are going to talk about a project we did together last year, the Data Quality Monitor. And we cover both technical and organizational topics, like how to choose the right data quality framework that fits your project and how to drive data quality improvements in your organization. so let's start with a small introduction my name is joshua finger i'm a data engineer based in karlsruhe my background is actual full stack software development but at the moment i'm building robust data pipelines and there as a data engineer i'm responsible for the full stack like from investing local sensor data over transforming it in databricks and then serving it through elastic search to power our data products so yeah my tech stack is databricks on Azure and Elastic Stack at the moment. And with that, yeah, I work for InnoVex. We are an IT project center. Maybe you also see some other talks from us on PyCon. And yeah, with that, I'm going to pass it over to Rusty.

Speaker 3 [02:28]

Welcome from my side as well. My name is Rostislav Kasov. I work with data since the beginning of my career. I started as Oracle Database Administrator, then I moved to Hadoop, and now I work on Azure. In my current position, I'm responsible for development and maintenance of the IoT data lake of Rational. Rational is a manufacturer of kitchen devices in one specific segment of kitchen equipment. We are based in Landsberg in Bavaria and produce around about 100,000 devices per year. And in this particular market segment of combi steamers, iCombi and multifunctional cooking devices, iVario, we are global market leader. And since a couple of years, we also offer our customers digital products. This is quite new for the company and it's really interesting to see how a hardware manufacturer is moving to a digital provider as well. And there, we offer our customers a platform called Connected Cooking. If the customers connect the devices, they can use digital products and we can collect data from the connected devices. This data we can use for use cases like predictive maintenance or data reports for the customers and also internally to improve long-term data quality. In this talk, Joshua and me will present how we built a solution to monitor data quality of the data sent us by our devices in order to improve the quality of our digital products. We will start with the definition of data quality and then we will present you our initial motivations for the project. Then Joshua will present you how we chose the right data quality framework and how we implement DQX based solution. And finally, I will share my thoughts about how you should change your organization to make data quality a native citizen in your data environment. Let's begin with the definition of data quality. Probably everybody of you in the room agree garbage in, garbage out, like it was 10 years ago. It's still the case, even if you use advanced generative AI models and in my opinion it's even more important to have good data because with more complex models the explainability decreases and you need to rely on the data even more than before. And if you try to define data quality it's quite easy on the high level. On the high level data quality is the degree to which data meets expectations of its consumers to drive correct actions. But this definition, even if it's generally accepted in the industry, you cannot drive any improvement with this high-level definition. To drive improvements, you have to dig under the surface, and there you find out that data quality consists of multiple different dimensions, like for example accuracy. Accuracy describes if your data matches the physical reality. For us, we are processing data from cooking devices, for example about temperature, it's important to know if the temperature in the cabinet was 100 degrees or maybe 105 or 110 and how far you match this real physical temperature, it's an aspect of accuracy. But there are many more of them and then I try to sort them out for responsibility and And we can see that some of the dimensions for data quality are completely the responsibility of the data source, like accuracy. There's nothing you can do about accuracy if you get the wrong data from this temperature sensor. You cannot fix this magically in the data platform. But you could check it for plausibility in the data platform. For example, this temperature sensor, it should be never below minus 10 degrees and above 400°C, because this temperature cannot be reached with our devices. And also there are some aspects which are more business related, like relevance. To ensure relevance that your device sends relevant data, it's not a technical question, it's a question about data governance, how you link your data projects to your business. Since we are moving in the context of IoT devices, we also have some specific considerations to keep in mind. First of all, the connectivity. Our devices are spread across the world. They're using customer network to connect with us and they have all kinds of issues with firewalls, ports, proxies and so on. And in the end, the connection is unstable. Then, our devices are built for 10 and more years of service, which means we are currently collecting data from devices 15-12 years old and back in time they were not built to create good data, they just send us some debug logs and we have to process data from very old devices and very recent devices all simultaneously. Also they run different software versions, because even if you provide regular software updates for our devices, we cannot enforce customers to update their devices and we have to deal with every single software version we ever released. And last but not least, we are only exploring huge use cases, what we can do with this data and there our data quality needs changes from time to time. And this all leads me to the conclusion that in my case, for our data, we simply have no chance to have perfect data we have to deal with data we have and we have to at least measure how good is it and that all lead led us to this project the project goal was to build a reasonable tool to analyze multiple data streams for our device this tool shall help us to quantify data quality durations and we wanted to focus on data quality dimensions which are where data platform can help us to drive improvements like completeness, plausibility, uniqueness and consistency. Other dimensions of data quality are out of scope for the project. And our motivation was to first of all quantify data quality deviations. And if you do it on large data amount, you easily can find statistical correlations. So you can see which software version cause more or less data quality issues. also good for data consumers so they can know if a particular data set have a good or bad data quality for the use cases they do and last but not least especially if you work for a manufacturing company nobody understands really the data quality maybe a 2% of people and it's hard to talk about data quality but if you talk about completeness plausibility uniqueness it's easy time to explain and therefore I wanted to have some set of matrix where can communicate them to explain data quality there are also two different approaches we have chosen to monitor data quality only this means in our data architecture we have multiple stages probably everybody of you have multiple stages maybe you call them differently we call them raw processed and serving you might call them for example bronze silver gold and we process data from the row layer with the original device data to a reusable process layer and we monitor it. After it was transformed we monitor the data and detect data quality issues. In theory it's also possible to ensure data quality by changing transformation by writing out the data to a quarantine table. So if you find malicious data you write forward it to a current set of tables and then you fix and re-ingest it with the right approach you could ensure data quality is a different thing but from my point of view it's not possible to do it for IOT data because you don't have the full set of rules for all software versions you don't know all the use cases so we decided to go with the monitoring approach on the left side so it's in documentation it's called we monitor data quality deviation post transformation instead of detect them within the transformation and for sure we wanted to build on a established data quality framework because there are a couple of them in the field and in the beginning to be all only new great expectations but then we have to research and find out there are many more and where to evaluate them in the first phase of the project. Our criteria were first of all accessibility for domain experts, because our device is quite complex. I have many colleagues and they are all specialized in some kind of thing, for example, one team is making climate control, like control of the energy, microwave heating within the cabinet, team developing autonomous cooking and the next team developing cleaning processes and all of them have very specific domain know-how. This is something you cannot centralize in one central team and it was crucial for me that my domain expert colleagues can maintain the rules and therefore we needed some kind of domain specific language for checks because mostly they are not fluent in Python. So if my checks would be in Python, they wouldn't understand them and couldn't help me a lot. Next crucial criteria was ability to define and reuse custom checks, because most of the frameworks bring built-in checks, but they're only suitable for simple things like not null, not negative and so on. But we also needed a way to implement and reuse custom checks. And for sure we wanted the solution to be integratable with our tool stack, we use Databricks as compute engine and the solution shall run natively on PySpark and also use code first approach so we can reuse our CI-CD pipelines. And last but not least, the cost is always important if you're processing large data amount, it should be natively supporting an efficient engine like Databricks and also we wanted to avoid any additional license cost. And with this criteria, I'll pass over to Joshua, who will explain you how we have chosen DequeX as data quality framework.

Speaker 2 [13:09]

Yeah, thank you, Rosti. So, yeah, let's take the evaluation criteria that Rosti just presented and throw them on our selected framework candidates. So, as Rosti already teased, we have great expectations, which you might already know, and Soda as established players. And then we also tested DQX from Databricks, which is like a newer, lightweight data quality framework. So, let's start with great expectations. So, in great expectations, checks are called expectations. You define them in Python code, and then they can be exported and imported as JSON, which, yeah, it's not really a domain-specific approach, language approach in our case, in our understanding. It feels not like the right thing to do. Like, great expectation wants you to define the checks in Python code. And we want to, for example, YAML-based. format so custom checks so we can extend the pre-built check-in checks of the of great expectations with custom checks that's either way done with python code or an sql so that works perfectly fine then there is a pi spark integration available for great expectations so that would have been good for our case and last but not least there are two different types of licenses that you can buy for Great Expectations. So on the one hand side, there's the cloud platform, which is a SaaS service, which you can buy. And then there's also the free open source Python library that you can integrate into your project, which would have been the chosen one for us. That's called Great Expectations Core. So what about Soda? In Soda, we really like the domain-specific language for checks. It's called Soda CL, Soda Check Language. That's YAML-based. For custom checks, they are only available in SQL. And actually, if you want to write one custom check and then reapply it on another, for example, a column of another value, you in soda core you have to rewrite the whole check again if you want to parametrize it and for example use it for multiple columns columns at once then you have to buy a solar license so that's kind of confusing so yeah if you're not if you're not happy with the features to get from soda core you can buy another license or you can maybe write your own yaml parser that then It creates dynamically a whole big YAML file with all the SQL statements, but that seems a bit confusing for us. And also for Soda, there's Python integration available, as you would expect of a step-less framework. So let's talk about DQX. In DQX, we really liked the domain-specific language. It's similar to SodaCL, also YAML-based. But you can also define your checks in Python code. So, there are two options, three are flexible. Custom checks can be either way defined in SQL or in templatable Python functions, which I'm going to talk about later on. And this is really a feature we like, because here you get the templates for free and they work out of the box. So, no custom code required there. Yeah, obviously it's by Databricks for Databricks, so there's the Databricks integration available. Would have been confusing otherwise. And yeah, it's totally open source, so no additional costs involved, just it comes with a Databricks open source license. That means you are only allowed to use it inside of Databricks. So to sum it up, for our requirements, DQX was the clear winner. But yeah, always depends on the requirements and on your specific business case. And just to mention it, if you maybe have other requirements, also it might be worth looking into DLT expectations and Unity Catalog data profiling for Databricks. These ones were just not flexible and not programmatic enough for us. And on Snowflake, there might be a good lightweight alternative to DQX, which might be the data matrix functions worth checking out. So, let's talk about our winner, DQX in this case. If you can't apply it in Databricks because you are on another ecosystem, we really liked the core features of DQX that can be easily applied, especially in the era of assistant coding and assistant coding, like for your own little lightweight library. So just take the core features, apply it to an own lightweight library and you're good to go. So how does this lightweight approach work? First of all, we have like an input table we want to check. Then we load our checks that are defined, for example, in our case in the YAML file. We apply the checks and then at the end we get an output table where one row is, for example, then marked as error. So how does this work? into the yaml file first so the yaml file is straight it's pretty straightforward in dqx there are row level and dataset level checks so they just differentiate in the input the check gets so yeah and at the end they all output on row level based in the qx that's how it works here we can define the criticality level and then we call the built-in function which is called is not null and not empty so let's look inside the code what this function actually does it just evaluates one column if it's null or if the string equals to empty and if this evaluates to true the check hits so that's no magic here and how can i set up dqx for example in database notebook works like this I can install it via pip just make sure to pinpoint the exact version as if you not do it you may get breaking changes as I did when I rooted the example and also if you want to you use the LLM features which are going to present later on also make sure to install that one then you have to restart Python and yeah do all the DQX imports. In this example I'm going to use the bakehouse samples database from DataFrame from Databricks and for this example we set one column the column city to none where customer ID is two million and three so our check later on can also find the yeah the specific row Then we apply the check like this, so we specify the YAML file, we apply the checks and then we get a new data frame which contains good rows and bad rows, like bad rows with errors in it. And there you see you get two new columns, like error columns and the warnings columns which are built by the QX and in the errors column you have an array with all the errors we got for this specific column so if you zoom in on this error we get a list of python dictionaries here you find again the error function we just called it's not not empty we have like dynamic method error messages and the dynamic name that is built inside of the check function and then we can also do a little bit more with this so we can also add more for example more metadata which is done like this in the check here you find the metadata again and you can also specify the check to only work for specific columns which might be very useful and then if you want to reuse this one check for more columns you don't have to rewrite all this yaml syntax again. So you can just specify a YAML list, which is done like this. And then you apply the check for multiple columns. So that was pretty useful for us. So there's one core feature we're still missing. It's called the custom checks. So if these checks don't fit all our requirements, we can write our own checks, which I've done on this slide. So here we had in our case we had an operation hours counter, Betriebsstundenzähler, if you don't know the word, and it should always be increasing. So that's why we wrote a custom check which is called is monotonically increasing. There the column parameter is mandatory and the other parameters you can define by yourself. In this IoT context, it was really important to sort and partition all the columns to get the right ordering. And then we could, with a lag function, always compare the previous value with the current value. And the check evaluates to true and hits if the previous value is greater than the actual current value. And then we also build an error message by ourselves and throw the error. So then how can these custom checks be used? That's also a great thing. They can be used like any other check in DQX. So the syntax is basically the same. You just get more arguments. You refine the order column and the partition column here. and yeah also these which columns we want to apply the check on so yeah that pretty much much rounds up all the core features which we really liked and now i'm going to talk a little bit about other things you also get for free if you use the qx like for example dashboarding so if you install it on your on your workspace you also get free dashboards these are basically two sql statements that renders to one pie chart and one table with the error details which might be a great starting point to start your own data quality monitoring and then you can also put like alerts on top of this and then also if you don't want to write all these complicated yaml files by yourself you can just use a dqx feature for that so there's a feature called data profiling which takes in also an input data frame data table and then generates makes some proposals for useful rules or maybe not that useful um because as you can see uh yeah it's pretty much always uh it's not null check or here it's uh isn't in range check which checks franchise ID if it's between 3 million and 1, 3 million and 44, which is not dynamic at all. So that might be good to understand the syntax, but not that useful. But luckily, we can use Dutch language models for that. So we can use the foundation model in Databricks and just prompt it with this prompt like generate all useful checks for my table for this table and we're going to use again the sample bakehouse sales customer table and if we do that we get these checks proposals which are actually well it's just a subset but i really like them so here the regex match function is used which is also already built inside of tqx and that checks the mail address within RackX if it's valid. Or for example, the gender column is checked for multiple valid entries, which also is a good idea in my opinion. And you can also alter the prompt. If you already have some checks in mind you want to check, you can alter the prompt and say it's like, yeah, I want to check column one or column two with this check. So that might be worth checking out. So, how did we implement these DQX features in our implementation? Basically we packaged our whole project as a Databricks asset bundle, which some of you already might know, we like asset bundles. Sadly they require, so asset bundles do all the deployment stuff inside of Databricks for you, which is a good way to do it. And they require these Databricks YAML file to be at the root directory of your source code, which is not really flexible and not ideal in our case, because we wanted to have different domains and different workspaces where we can deploy it to. So we wanted to have multiple Databricks for YAML files. And we solved it like this, that we placed them in a subdirectory and then wrote a little script in the make file to create a dynamic sim link to swap them out on the root level so yeah that's like our workarounds to work around these uh database abundance limitations maybe you have like another great idea for example dynamically recreating the yaml file and then we also tested all our functionalities like we wrote checks we wrote tests for all the checks and all the different jobs we created and into the source folder we had this one folder where all our yaml files lived checks folder then we wrapped dqx and some helper methods and then we had the actual jobs that load the tables and process the data so how did we process the data and as i said we load the day our tables on a daily basis there we use the change data feed feature of databricks where we get all the latest inserts updates and deletes for this table and then we use our checks i just talked about to find all our data quality issues um and actually we used some as you saw earlier we used some complex transformations so like for example this custom check goes over different rows and it's quite cost expensive but at the end we were quite happy with the performance but there you also always have to be like careful which checks you define and for example if you do data set level checks it might be not that efficient and then we had these row level outputs for each row we processed and that's in our data context it wasn't really that was not ideal as it isn't really interpretable so we had one device this which has for example multiple serial which has for example multiple software versions and multiple batches so we have like maybe 300 1000 entries per device that's why we had to make the output interpretable we had to apply some custom aggregations so we group by obviously in the serial number of the device and then also the software versions and we counted DQX errors so we counted all the errors like how many errors occurred for this device for the software version and then also which kind of errors occurred for this device so we had an interpretable output and yeah this interpretable output then was persisted in the Unity catalog and yeah that pretty much rounds up the technical details and now I'm gonna hand it over back to Rosti who's gonna tell you how to use this output and use it to improve data quality into your organization

Speaker 3 [29:56]

So now I probably presented you the outcome after we implemented all the stuff So as before we have our usual processing chain for data We have our devices the send us data We put it to the raw layer and we transform it to process layer. This is this part is unchanged And now we regularly apply Data quality checks with the QX they're running as part of our processing ETL based on the delta feature of Databricks, change data feed, and we write out every data quality deviation we detect in a special set of tables and offer those tables for our data analysts. So our data analysts, when they work in our use case, they can decide which data quality requirements they have for this use case, and they can read our findings from the shared tables. And also, I created a dashboard based in Power BI in my case, so that I can analyze all those data quality deviations, and it's quite easy to correlate my findings to software versions in our case, because every software version has small changes, and the majority of data is correct, but with a large data amount, I can find the needles in the haystack and document that. as a bug so basically in our case most of the data quality issues are related to software bugs under certain conditions which occur quite raw and this is hard to test in our test environments but they occur at customer side and based on my data findings i can document good bugs and my colleagues can provide new software version which fix the issues and And we called the solution Data Quality Monitor. Not very creative, but understandable. And this tool allows us to analyze data quality issues. It's great transparency. We can compare different software versions to each other. And it creates a feedback loop. So we can inspect our data and we can adapt our software to fix the issues one by one. It's a good solution. I'm quite happy with this. But it's obviously reactive. So we have to first have the data quality issues and then we can detect it and fix. It takes some time. It also requires me a lot of efforts to analyze the issues and to discuss them. And if you think about data quality as organizational aspect, there are a couple of things you can do to embed data quality in your whole organization to make it kind of native citizen in your environment so that your data producers think about it and you don't have to react on data quality issues. And therefore, I have summarized my recommendations for you. And it's all about data ownership because data quality is part of data ownership. And there you have some challenges, typically, especially in companies where data is not your main business, like manufacturing companies. Your data producers are not aware of data quality and they are not aware of the issues related to data quality. Because mostly we still have central data teams that deal with data. They can detect issues, but they cannot fix them because they have no access to the data producer software. And in order to tackle this, I would recommend you always to bring your data producers to the data so that they can analyze their data on their own, try to build your tool stack as usable as possible so it's not enough if you have some environment for data engineers who can work with Python you have to provide some usable set of tools so everybody in the company who somehow produce data can access own data can use it for own use cases and therefore have to work on trainings documentation and improve the skills of the people one by one. And also, especially in large organizations, you may have the case that a data producer produces data, but is not aware of usage of data somewhere else in some other department. And there you should make the data usage across your company transparent, make clear which data is used in which use case, and how your company, how your customer benefits from it, because this is something which gets lost, especially in large organizations. Obviously, the key competence to improve staff is communication. There, you will find out that it's hard to explain data quality for people outside of data bubble. Also, first of all, it's hard because it consists of many dimensions and they are also very different, but also it's a moving target. If you have good data quality today, you might find a use case tomorrow where data quality gains is not good enough so you have to move your targets and this is something which also not easy to understand and they're my recommendation try to start with an impact of bad data quality so think about how your data quality impacts the outcomes like reports for the customer like predictive maintenance outputs and argument with this impacts and also try to avoid term data quality I don't like the word data quality, I also don't like data governance, because both terms describe everything and nothing, and you should try to argue with concrete dimensions. Like everybody can understand completeness of data and accuracy, it's very easy to understand, but it's hard to understand data quality. And to sum it up, we would like to break it down to four takeaways, two of them are technical and two of them organizational and you always should think about both parts because only the best technology solution doesn't help you in the long run. First of all, if you are looking for a data quality framework, you should not only consider them large frameworks like great expectations, you can also research for lightweight data quality frameworks because in the end they avoid complexity. You should start with few checks, because if you're analyzing large data amount, you always will find some technical issues in your data, but you should start with few checks where you exactly know what is the impact, because then you have a chance to get priority by data producers to fix them. And then two long-term organizational aspects. First of all, you should involve your domain experts in creation of those data quality rules because probably you are not aware of every edge case in the data in your company as data engineer. And in the long run, strong the idea of the main ownership in your company, bring the data to the people who produce the data, to the departments so they can take responsibility in the long term and then your data quality will improve as well. So we are done. Thank you for your attention. And we are happy to discuss this or other topics in the conference later.

Speaker 1 [37:34]

Well, yeah, thank you Rostislav and Joshua for your insights and sharing them with us and we have some questions from the audience The first question I have is are there any checks on multiple rows or even relations between different data sets? For example number of rows in DS1 must be greater than the number of rows in DS2

Speaker 2 [38:00]

So, maybe let's go back to the one slide where we had the custom check implemented. So, yeah, actually it's pretty flexible. Like in this, this is a row level check. So, in this row level check, we always have, already have like a comparison between different rows. So we compare the current value, which is this one, the current row, with the previous row, which is done inside this lag function that saves the previous row. And also, if we want to have relations between different data frames, I think this can be done with dataset level checks. But we didn't use them. And they were also quite new. And last year, in our data, this wasn't required. But we had like more complex checks. We also built like one dynamic expression where we compared not only two but multiple rows Depending on the YAML file. So that's pretty flexible and we really like that Just be careful that you don't write any checks that are very Cost expensive like take a long time to execute

Speaker 1 [39:18]

All right. Thank you. And another question. What's the volume of data you are processing?

Speaker 2 [39:25]

So, yeah, I think I'm not allowed to talk about any exact, like, sizes. But, yeah, like, the whole tables are in the terabytes. And then these checks are only run on specific rows. Like, we started with some simple checks. And this is just a subset of the daily subset. So, like, a small, small percentage. all the data and yeah that's I think all it's like in the gigabytes then that's I think all I can say about that

Speaker 1 [40:04]

All right, and then we have can you share any success stories where you prevented an incident that has been caught by the Czechs?

Speaker 3 [40:15]

I mean, to prevent, it's quite hard. It's a reactive approach, right? But we could detect, for example, an issue with one data which we have to write for regulatory affairs. So our device is cooking food, and our customer says the protocol that the food was in certain temperature area for a certain time to avoid some of the issues. And there we had rarely cases where the timestamp was not valid. was negative and we could detect it based on the solution and we could see in which software version happens and so I could provide my colleagues many occurrences of this particular issue so they could fix it in the end but it's not possible to prevent an issue with this framework obviously a test occur first and then we can detect it

Speaker 1 [41:07]

Alright, next question. Does it support any unsupervised anomaly detection, for example deviation in value compared to historical values?

Speaker 3 [41:16]

Right now not. Right now it's based on our checks which we define with our domain experts. But we are already thinking about how to learn from our historical data to detect anomalies faster. So we would probably build checks based on historical data and let them verify by our domain experts.

Speaker 1 [41:37]

Alright, we have another question. Would you say DQX is a lightweighted framework?

Speaker 2 [41:45]

Yes. Yeah.

Speaker 1 [41:46]

Yeah. Okay, well, on to then. Data quality is a moving target. How do you manage data API changes and going down data schema changes?

Speaker 3 [42:01]

a good question for sure. So we try to stay backwards compatible, so our device, if a new software version is deployed to the fields and it's backwards compatible, so we can only process new stuff. And if you have breaking changes, it's all about communication, so we establish a link between data team and the embedded software team and also i'm working on to our own innovation of software tests so that the software team can unit test or system test a new software version and we can detect breaking changes in the system test but it's hard to achieve

Speaker 1 [42:47]

All right, do you know if DQX is available in terraform to replace asset bundles? Asset bundles are built on terraform, but much more limited.

Speaker 2 [42:59]

I think that's like kind of two different topics, like So DQX is like the data quality framework and Terraform is like an infrastructure as code. Yeah, library and so Databricks Asset Bundles can it's also maybe that's the mix-up with Databricks Asset Bundles. So the Asset Bundles are kind of infrastructure as code at least they also use Terraform underneath and it's a good extension to Terraform because actually you can provision a lot of things in Databricks with Terraform and then use the asset bundles to package your projects and deploy them to different workspaces

Speaker 1 [43:50]

Okay, we have two more questions. Is the integration of the dashboard and other Databricks supported frameworks in MLflow, for example?

Speaker 2 [44:03]

Actually, I didn't use this, so I would have to look it up on the documentation. Like for example, we didn't use MLflow yet, so maybe, but the TQX dashboards are really quite simple. It's only like some SQL statements, so that wouldn't be a problem, I think.

Speaker 1 [44:26]

All right, so we have another final question here. Someone wants to know, are there any guardrails for the checks that domain experts can set up, and what is the process to integrate new checks into the pipeline?

Speaker 3 [44:38]

I basically set up a Confluence page and ask everybody I know, hey, describe to me how you check your data and how you ensure that this particular file is correct or not. And I have kind of inbox there, and then we sit down together and prioritize them. So what is the impact of an issue? And important issues are going into this framework. And then we're discussing how many occurrences we see and what is the impact and how can we fix it so it's all about communication

Speaker 1 [45:11]

All right. Thank you. And then we're done with the Q&A. Thank you for all your questions. Thank you for your talk. And give him another round of applause. Thank you. Yeah. Well, next up is lunch. So we do have lunch on this floor, too, like in that direction. Yeah. Enjoy lunch. And I'll see you later.