Building Trust in Your Data Pipelines with Observability
Data pipeline observability addresses the problem of "black box" pipelines, where engineers only discover failures through stakeholder complaints or cryptic error messages. To build trust and transparency, a three-pillar approach is used: metrics, alarms, and logs.
Metrics provide numeric measurements of pipeline health over time. A robust foundation relies on the "four golden signals": latency (duration of execution), traffic (input and output record counts), errors (counts of dropped or failed records), and saturation (CPU and memory utilization). These metrics are visualized on dashboards to perform explorative data analysis, allowing engineers to detect anomalies, such as unexpected spikes in specific event types or silent data corruption caused by hidden bugs.
Alarms transform monitoring from a pull-based approach to a push-based system by triggering notifications when metrics cross defined thresholds. Effective alarms must be actionable, reliable to avoid false positives, and contextual, providing clear implications and documentation. To prevent alarm fatigue, noise is reduced by disabling non-actionable alerts and fine-tuning thresholds.
Logs provide granular, time-stamped details of internal pipeline states, acting as breadcrumbs for debugging. Best practices include using appropriate log levels to avoid noise and implementing structured logging—converting unstructured strings into JSON format—to enable robust querying and filtering. Together, these pillars allow engineers to identify a failure via an alarm, analyze the trend through metrics, and pinpoint the root cause using structured logs. Tools such as Grafana, Prometheus, AWS CloudWatch, and Datadog are commonly used to implement these strategies.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Data Handling & Data Engineering and was classified suitable for intermediate domain by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
This talk explores how observability can be applied to data pipelines to improve reliability, data quality, and confidence in complex data systems.
The talk begins with an introduction to observability in the context of data engineering. It explains the three core pillars: metrics, alarms, and logs, and discusses why observability is particularly important for data pipelines, where failures are often silent and correctness issues may only surface through stakeholder complaints.
The first section focuses on metrics. It demonstrates how straightforward it can be to instrument data pipelines with basic metrics using Python. The talk then discusses which metrics are worth monitoring, adapting established concepts such as the four golden signals to data engineering use cases. A concrete example based on a near–real-time event processing pipeline illustrates how fine-grained metrics can reveal systematic failures for specific event types.
The second section focuses on alerting. It addresses the challenge that engineers rarely have time to continuously inspect dashboards and therefore rely on alarms to surface important issues. The talk outlines what makes a good alarm, emphasizing that alarms should be actionable, reliable, and provide sufficient context for investigation. A scenario with excessive and noisy alarms is used to illustrate alarm fatigue and a strategy how to get out of such a situation is described.
The final section covers log messages and their importance to reason about how a pipeline ended up in a specific state. It discusses why logs are often difficult to work with in data pipelines, as they may contain a mixture of critical errors, informational messages, and low-level framework output. The talk introduces structured logging as a way to add context and make logs easier to search, filter, and aggregate. Examples include monitoring the distribution of log levels to uncover hidden issues and using centralized logging to identify dependencies between pipelines that are otherwise hard to detect.
The talk concludes by emphasizing how the three pillars of observability build trust in a data pipeline.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:09]
Ah, now it's working. OK. Good to get adjusted to that.
Speaker 2 [00:15]
Hello, everyone, and welcome back. Today, we have the pleasure to have Stefan Dienst. He has four years of experience working on a variety of data platforms, ranging from classic ETL pipelines and data warehousing to near real-time stream processing. And today, he's going to explain to us how to build trust in your data pipelines with observability. Warm welcome to Stefan.
Speaker 1 [00:46]
Thank you very much for the introduction. Hi, everyone. Picture a Monday morning. You're a data engineer. You start your workday, and you check your emails, and you see this. Dashboard is missing data. Hi, Stefan. I just noticed dashboard is missing data. I wanted to show the progress later in the board meeting. Can you please fix this ASAP? So instant high urgency. see here. Maybe you have some overview of your pipelines. You check that and you see this. Lots of failures. You click one of them and you see this distributed transaction coordinator failed to establish quorum consensus. So very daunting error message. If this would be me, I would be stressed. Like lots of things racing through my head. Like was this because of the deployment last week, can I fix this? How quickly can I fix this? And what happens if I can't fix this? And I've been in this situation multiple times, and it sucks. And this talk is all about making it suck less. So by using observability. So instead of receiving such emails, we should get an alarm which tells us first that things are on fire, so we could act quickly. Instead of being unsure where to look for information, we should have a nice dashboard with metrics to tell us everything we need to know. And instead of being unsure how to debug our pipelines, we should have good log messages in place that we can use. So, summarized, this talk is about building trust in your data pipelines with observability. So, what is observability? Everything, of course, in the context of data pipelines. One sentence definition could be the ability to inspect the internal state of your pipeline at a specific point in time by collecting and displaying signals like metrics or logs. That's a lot. So for me, it means making the invisible visible. So your pipeline, it's just a program. It maybe runs daily on a schedule or once it's triggered by an event, then it does some fancy data operations. And if you're lucky, you end up with new correct data in your database or object storage. But if you're not lucky, you need to look inside this black box. You need to make it transparent. And exactly for this, I like those signals, metrics, or logs. And in this talk, I want to focus on three, what I would consider core pillars. First one being metrics. This is all about what are the hard facts about my pipeline. Second one is alarms. Is something out of the ordinary happening? Should I be worried? And the last pillar is logs. What exactly was my pipeline doing? And in the following, I will go through those pillars, one by one, introduce them on a basic level, and then share what I would consider best practices and some experience I had with them. And this won't be about any specific tools, this is really just a general approach how you can use observability. If you have never used observability on any of your pipelines, you will leave with a lot of information what you could do to make your life easier. If you already have a solid foundation, you may pick up one or two new ideas or feel motivated. So with this, let's get started to the first pillar, metrics. So what is a metric? A metric is a numeric value measured over time that tracks the performance, health, or behavior of your pipeline. So think everything that you can measure in your pipeline and put a number behind it. So maybe you want to measure did it run successful, zero or one, or the duration in minutes, or or something technical like the memory usage, or something very business specific. Whatever, you need to measure it. And let's look at an example how we could put this inside our pipeline to track something. Imagine we have a pipeline that processes user events, and you want to count in one run of this pipeline how many account-created events are in this batch, like this specific event type. You first obviously need to do the measurement. You need to count it. So we do some data frame operation, filtering it, and then you have this value. And then you would need to push this metric. And this means that creating a metric event, one individual measurement. And here, whatever tooling you are using, the data model is more or less the same. You have first the name of the metric that you want to push a measurement to. Here, user events counts. And then you have the option of giving it dimensionality, like in the form of text or labels. Here we would count the specific event type, a count created, and then you could also count all other events if you wanted to. Then you need the specific value you meant as count. And then you need a timestamp. There's this timestamp data, like time series data. When was this value measured? And that's it. And with every run of your pipeline, this code would be executed, and you would get a new measurement. And because the raw values are a bit uneasy to work with, What you most of the time do is you do some plotting. You make a dashboard for it. And here we could, for example, just plot this metric. Over time, we always got how many of those events were in our pipeline. For this example, around 500. And maybe if you let us run for a few days, suddenly we'd see something like that drop to zero. Maybe the platform is just not popular anymore. No one wants to create an account. Or there is an issue, and you must investigate. So setting up a metric should be easy. The hard part is then what to actually measure. And here I can't give you some comprehensive list. The first recommendation is it's very individual and will change over time. So maybe start small, but iterate. If you set up a new pipeline, don't go overboard and try to measure everything. Try to focus on the following questions. What would you like to know if there's unexpected behavior? This should be the first place that you look into when something happens. For me, this is like, was there any input data? Was the data stale? Did some filter return zero records? How were values distributed? But it could be anything. Another recommendation is, if you're using some tool or framework for your data pipeline, check the documentations. Maybe it already exposes a lot of metrics, so you don't have to do this manual work of setting it up. And you could just get stuff for free. It's always worth checking this out. If you have to do this manually, what I always like to refer to is a model called four golden signals. And here, the key idea is that there are four kind of signal categories. And when we have a metric in each one of these, we have a solid foundation. The first is latency. How long does it take your pipeline to do what it's supposed to do? One obvious example is duration. Just measure how long it took your pipeline to do what it's supposed to do and keep track of this. Maybe if you initially deployed it, it took like one hour. But with more and more data coming in, maybe it took two or three hours. And maybe at some point, it will take too long to do what it's supposed to do. And then you must intervene. Second category, traffic. How much is your pipeline processing? Think like number of input records, number of output records. These just point out obvious errors. Like if your pipeline ran for one hour but produced zero output records. It's usually not the behavior you want. Third category is errors. How many records could your pipeline not handle in the expected way? So in a perfect world, there would be no errors, but usually there are always some records that you just have to drop or where you have some work around doing stuff. And here, suggestion would be like, make this transparent, like track this in a metric. Then you always know if like the errors are silently increasing and you have to take another look if what you're doing is actually correct. And the last category is saturation. How full are the resources of your pipeline? This is like all the technical things, like how much memory is your pipeline consuming? If you have some stream processing pipeline running 24-7, you should have a look at this. Because if the memory is steadily increasing, you have a memory leak, at some point you will run out of memory. You should intervene before this happens. or CPU utilization. Maybe you're paying for a very expensive node, but you're not using it. So all kinds of things. But if you have one metric for each of those categories, you should be already set to go and have a solid foundation. Now, how do we actually use those metrics to build trust in our pipelines? And here, my recommendation would be to treat the dashboards for your metrics like an explorative data analysis. So really, we as data engineers, we create these pipelines to generate insights into various kind of complex systems. And in doing so, we're creating complex systems ourselves in the channel of pipelines. So I would recommend to use the same approach, asking questions and answering them with data. So if you can answer them with the metrics you already have, make a plot for it. If you can't, create a new metric. Let's demonstrate this with one example of a previous project. Here, our team had a pipeline that processed user events near real time from a video on demand platform. So things like account created event, account deleted event, subscription renewed, subscription canceled, just to name a few. And once we set up our pipeline, what we did is we looked at a distribution, like real time processing with micro-batches. So we looked at how many events are in a micro-batch and what kind of event types are usually in there. And then if we plotted this, we saw like in blue, we had more account created events than account deleted events. So like expected, platform is doing fine. And the same for subscription renewed in orange and subscription canceled in purple here. We also had a look at the latency. We wanted to process the data as fast as possible, and had a look at it. Then we let this run for a few iterations, and we got some rough idea of what our pipeline is dealing with. And now, after having run this for several months, suddenly, on every Monday, we noticed a different behavior. So this happened. We got more events. Latency went up. Fine. But the distribution also changed. There were far more subscription canceled events than under normal conditions. So this could be completely normal. So maybe everyone canceled on Monday for whatever reason. But here we investigated, and it turned out that the team that was responsible for managing the contracts, they did the reprocessing on every Monday of every cancellation. So we were receiving duplicate events that we were processing. And with a look at that, we were doing fine processing duplicates, so there was no error, and the latency was also good. But we were getting, yeah, I would say like trust in this pipeline, because we understood under those changing conditions, we were still doing what we were supposed to do. A different example was error counts. So for every micro-batch, there were some events that we could not handle for whatever reason, and we would put them on a dead letter queue. And there we also had a dashboard looking at distribution. Under normal conditions, there was a low error count, and no specific event was dominating. But again, letting this run for a few months, we suddenly saw this. errors, and the account deleted event was responsible for almost all the errors. In this case, we had a hidden bug. So we were relying on a specific timing expectation. We were always assuming that we would get an account created event before the corresponding account deleted event, which seemed reasonable at the time, but with things changing, this was no longer the case. So at this thing, we were actually producing garbage data and pushing this to the stakeholders. And without this dashboard, we wouldn't have noticed. No one thought about this. These are only two examples, but I hope I could kind of convince you that this is a useful approach of building this trust and maybe even uncovering hidden bugs. And a very nice side effect of this is you will end up with very insightful dashboards of your pipelines and in this project we had a weekly meeting where all of our engineers got together and one of us moderated walking through those dashboards and then we discussed what is normal what is extraordinary and just great for knowledge sharing like everyone gets on the same page and understands what's actually going on also you get the visual feedback of what you're doing so once we discover this bug and fix it by enforcing the order we could see after we we shipped it in our dashboard, that this error count was dropping, and it was not popping up anymore. And this is really fun. This is great feedback. And with this, I'm done with the pillar metrics. Let's go over to the next one, alarms. So once you have nice metrics and nice dashboards, and you look at your pipelines from time to time, you get this kind of idea what are the normal ranges for your metrics to be in and what are some threshold that if it crosses it, you get this feeling of worrying and needing to investigating and finding out what happens. The thing is, you don't want to look at your dashboards 24-7. And this exactly is what alarms are for. So what is an alarm? An alarm is a rule defined on one or multiple metrics that is automatically evaluated and can be in three states. Either in OK, so the metric is in the okay range everything is fine can be in an alarm state so it has crossed some value some threshold you should worry or there can be like no data like there's no new metric event to evaluate the it's just uncertain what is what the what the rule evaluates to and the idea is now that whenever this alarm changes state you get a notification so you move this from this pull approach of you looking at the dashboards and searching for things that are wrong to this push approach that you don't have to do anything and just check your emails and stuff like that. Let's look at an example how you could configure such an alarm. So first off, you would give it some name. For example, no output records produced. And then the most important part is this rule. Here, for example, we would look at the metric, num output records, and we use the max value of the last 25 hours, and if this evaluates to smaller or equal to zero, we would say the alarm is in a firing mode. We expect that the pipeline always produces at least one output record. Then we would also need to trigger this assert state, like if there is no data, you would say like treat missing data as alarm. If we don't know how many it produced, we assume it just produced zero. And then we would also get an alarm. Then, very important, who to notify if this alarm changes state. I could just put in my email. You could also like your Messenger, Slack, or MS Teams. Important thing is just that you use something that you actually pay attention to. So email may not be the best thing. And then description, like the ability to give this alarm additional context here, just some string, no output events have been produced, and so on. So in the end, setting up, configuring an alarm, it should be easy for you. The hard part is, again, what is a good alarm? And for me, a good alarm has three attributes. The first one is actionable. You should not be able to ignore it. There should always be something that you do once you get this notification, be it stop whatever you're doing and investigate the bug, or find out if it's a bug, or communicate with someone, find out how big is the damage, and then raise this as an issue, ping another engineer. Just something. If you don't do anything, it's just noise. Second attribute is reliable. The alarm should only trigger if there is really an issue at hand where you need to do some action. If it's like often producing false positives, and you often spend your time looking at stuff which everything is just fine, there's nothing to worry about, It's just annoying and wastes your time, and you won't take it seriously. And the last one is context, like in this description. Everyone on the team should know what this alarm means and what the implications are. So, for example, if the number of output records is zero, what does this mean? Is the dashboard not showing the recent data, or is it not showing any data? The implications for the stakeholders, how severe is this? It's like, how quickly do we need to act? What's the impact? Also, if it's like a recurring bug, you may have some process how to recover from this, so you should link documentations to it. It should just be like some self-contained information that everyone on the team can work with. If you are in a situation where you have lots of alarms which don't have those attributes, you may end up in alarm fatigue. Alarm fatigue describes a state in which engineers are overwhelmed with triggering alarms that they just ignore them. I've been in a situation once, and for me it was just normal, every day checking this channel, three or four alarms. Scrolling through them, nothing to do, that's just how it is. It's really, yeah. And the bad thing is that, at least for me, I missed important alarms. Then I got emails like this because I was scrolling through this channel, but I did not take an important alarm serious. and this totally sucks because you spend time on observability you build your metrics your dashboards your alarms and you're still getting those emails i at least want to avoid and so my approach of getting out of this is like two steps first one is reduce noise so everything that you just scroll through and ignore just turn it off yeah and the second one is then divide and conquer like turn on the alarms again one after another but make sure that they have those three attributes and if they don't fine-tune them like make them reliable maybe you find out that one alarm was not actionable so why does it even exist so just drop it and if you're lucky and this works you may end up in a situation where you on the morning check your channel and if there is an alarm you know exactly what to do and if there isn't you really trusted everything went correctly because you have alarms for every extraordinary thing and if there is nothing extraordinary you don't have to worry and can go on with the next task on your hand and that's all on alarms let's move over to our last pillar logs so what is a log message a log message is a message that states what happened in your pipeline at a specific point in time let's go directly to an example Imagine you have one pipeline where at one step you want to drop older data, data that is older than 30 days. So here you would define like this cutoff date, calling daytime now, doing the time delta on it. And then you would do this filtering operation on your data frame. And you would end up with a cleaned data frame. And now a log message, you would just write what you did. You dropped rows older than the specific date. And then what is the outcome? Remaining X number of rows. and compared to like metrics the logs are all about those nitty-gritty details like really fine granular you can the idea is that you put them like everywhere important like when your pipeline starts the different sections if there's branching there you need to know what branch that your pipeline take if there are very important very important filtering steps where things may go wrong put a log in there and in the end in theory you could like log just everything what your pipeline is doing, but then you would produce a lot of noise. So the idea is that you exactly strike this right balance of logging just enough of information that you can reason about the behavior of your pipeline, so that in the end, you can understand how your pipeline ended up in a specific state. So this is like, for me at least, the goal of having log messages. And with this, it fits perfectly together with the other pillars. So imagine you have a pipeline where we have a metric on the number of output records of your pipeline. And then you create an alarm on it. If zero records are produced, you get a notification. And if this happens, you should be able to just open up your logs, read them top to bottom, and reason about how this could happen. Like, how was this state achieved of producing zero records? And maybe in this case, you will find this message dropped rows older than 2031st first remaining zero rows. So, you know, okay at least in this operation After it there was zero rows and then you look at the date and you notice okay This seems weird. Like why is it a future date? This can't be like the pipeline did everything correctly but somehow it had a wrong date and In the end is all about like leaving breadcrumbs for your debugging Quest and if you're lucky you will go on and go on and find the next one So how do we write good log messages? For me, recommendation would be first, think about this goal. If something breaks, what information do I need to find out what happened? It's the hardest one, the most difficult one. I often get this wrong. I log too little. But if you then find yourself in a situation where you look at the logs and you don't find what you're looking for, this is the perfect opportunity to write the log messages that you should have written in the first place. So it's an iterative process, at least for me, not perfect. But after a time, the pipeline is completely covered. Second one is use correct log levels. So I once was in a project where pipelines were producing like 50% error log level messages, while everything was fine. So once things were really not working and I checked the logs, everything seemed on fire, and it was hard finding the real error log message. So use things sparsely and with caution. and the last recommendation is reduce noise so if you open up the logs and you scroll like for five minutes and it's annoying it's exhausting um you're logging too much maybe um you have some tool that um writes like a lot of info log messages maybe you can change the logging behavior that only warnings or errors are visible so in my opinion to have like trust in your pipeline When we are logging, you need two things. You need to log what is important, and you need to be able to navigate your log messages efficiently. And for the last part, I have one final recommendation, and it's structured logging. Structured logging is this idea of going from this unstructured string to logging something like a JSON with structure. So everything of the log message turns into a field. I have to do this quickly now. In the end, you have then, for example, one timestamp field, one field with level for the log level. You have what logger it came from, and the original message, and stuff like this. And you can write additional context, like what pipeline produced this log message, and so on. And in the end, this gives you the ability to write robust queries on your log messages to answer questions. You could filter after the fact for the correct log level. You could even build a dashboard for the distribution of log levels of capturing very hidden bugs or stuff like this. In the end, it just gives you more filtering options. And with this, at the end of my talk, I want to quickly summarize. When I have a data pipeline with metrics that give me insights into how the pipeline performs, alarms that fire when I need to act, and logs that I can easily navigate and help me understand what happened, I trust this pipeline. And with this, thank you very much for your attention.
Speaker 2 [25:43]
Thank you so much for your wonderful presentation, and there are some questions, and indeed in the end you mentioned about the noise and how to filter, and actually some people are wondering how to make sure that the number of dashboards doesn't become unsustainable, and how to reuse them to reduce the development time.
Speaker 1 [26:01]
Yes, dashboard crawl. If it really goes all out of control, the approach I like to do is the drill-down approach. If you end up having too many dashboards, create an overview dashboard, which has the essential things that you once or twice check a week or something like that. And then if you want to go deeper, you just click links in this dashboard, which gets you to another dashboard where more detailed information is. Really building this hierarchy of dashboards This at least has helped me to not get overwhelmed with the number of dashboards.
Speaker 2 [26:37]
And if you ask, is there any tool or platform you would like to recommend to work with Python?
Speaker 1 [26:44]
Yeah, I don't know.
Speaker 2 [26:44]
Yeah.
Speaker 1 [26:46]
Like, I've worked on Datadog, AWS CloudWatch, and Grafana and Prometheus, like, those three kind of platforms, and they all have their pros and cons. It really depends. Like, it's really hard to do. Like, it's also, like, a cost option, like, how much money do you have available and how much engineering power do you have? I would recommend to start small, like, cover the essentials. So it's maybe really just a lambda that captures an event of failure and sends the chat messages enough if you don't have much time of building such a platform. So it really depends.
Speaker 2 [27:23]
And do you also monitor the content of your events? What if you suddenly receive the same payload for all events, for example?
Speaker 1 [27:31]
In this specific case, we actually did, but covering duplicates, this would be like on a different level, more like an ID-based, so not like after the fact with the monitoring. But in the end, if you can filter stuff efficiently and you have the cost under control, the more information, the better.
Speaker 2 [27:54]
And how do you differentiate between different kinds of errors for your alarms that remain actionable or not?
Speaker 1 [28:04]
Sorry again, please.
Speaker 2 [28:05]
please uh how do you differentiate between different kinds of errors that you're
Speaker 1 [28:10]
So, I did not mention this, but often you can also give this severity attribute to an alarm. Like, is this a critical alarm, or is this just a warning, or things like this. I don't know how useful this is, because I prefer to have every alarm matter and be actionable. And if you get to this level that it's really just some information, for me this should not be an alarm, but things may differ. You could change the routing, like routing different channels based on the severity that you only really take this focus on this critical alarms in a specific channel. This would be like options, categorizing them.
Speaker 2 [28:46]
And how do you deal when observability actually puts extra time processing for the data pipelines?
Speaker 1 [28:53]
Like actually making the pipelines slower or like additional load on the engineers?
Speaker 2 [28:59]
It's slower in terms of time processing.
Speaker 1 [29:01]
Oh, in the end, in my experience, it should not be the heavy work. This should be easier things. You could batch all of the metrics in one request at the end of some processing step. If most of the time is spent actually measuring, then you're doing something wrong, I would just say. It shouldn't happen.
Speaker 2 [29:23]
And how do you handle different instances of your pipeline like dev, QA, and prod? Will they all send the same alarms or not?
Speaker 1 [29:31]
The obvious thing is using different channels for the different environments. So you can mute dev and pay only attention to QA, maybe. This is one option. But I also like the flexibility to be selective there, that I only declare alarms on prod or something like that. Also, the environments, they may change with the data, so there's rules for the alarms to evaluate. probably won't hold for dev, QA, and prod. So I like to have this flexibility to be selective there on what environment I produce what and how I parameterize them.
Speaker 2 [30:09]
Thank you so much for your answers, and thank you again for the presentation.
Speaker 1 [30:13]
Thank you.
Speaker 2 [30:13]
Thank you. There were still some questions, so don't hesitate to...
