Advanced Observability with OpenTelemetry and Python

Speaker 1 [00:05]

It's a lovely crowd, it's a lovely conference. Haven't been here for three years. Yeah, we have a big history with PyCon Germany, so I feel like super happy about being here. I will tell you in a sec about this little story. So that's me, generated by MidJourney. I live in Munich, I organize events, and I run a software agency, which is, of course, focused on Python, mainly. A little story with PyCon Germany is that it was quiet in year 2015, and we had nothing to do. I just moved to Munich, got my first contract there, working with new friends, and felt like we need to organize something for the Python community. So we organized PyCon Germany, like this conference, but like ten times smaller, probably, in Munich, the university, and it was really like a child steps there. We had to agree with everybody in emails about coming, we had to beg every single sponsor to give us something, and now I'm here five, actually more, seven years later, I see how things run to perfection, makes me super happy about what the organizer did, so big thanks to the organizers. Now today we speak about OpenTelemetry and we have a very diverse set of people on every conference so I know the hardest part here is how do we target people who came here to know what is OpenTelemetry and people who are already using it and wondering how can you push it to the maximum. This is a difficult thing. So first of all I will say what this talk is certainly not about. This is not about how to build a fancy dashboard. For this, you can consult data dog tutorial, whatever. For me, this is, to be honest, this is way too much. This is just to show off, and the talk is not to show off. So for me, this is just pure visual trash. It is not about New Relic. New Relic is cool. I mean, all of these things are cool if you use them right, and this talk is not about how to use them right. We will not build a fancy graph of microservices and say, hey, look, the spans fly here, it's nice. No. Trash. It is also not about New Relic Insights, although it's cool, and although I understand that these gouges, they look like in a race car, super sporty and stuff, but it also looks like if you put this AliExpress nitro gouge on your Seat Ibiza, or I don't You don't need it. Trash. This is also trash. It's so old school. Five years ago, it was cool. Now, from this, to get real insights about your application, I doubt it. This is more to show off in the office. We understand, yeah, to impress the clients, employees. But if anybody of you actually ever check this, if you have it, because I know half of you has it, Too old school, better put some PlayStation 5 and enjoy cyberpunk. Way better use. So this talk is not about how to show off. It is about how to get actual insights and observability from your application. Would it be small or would it be big? What is telemetry in general? Let's start from this question. I was new to the world of telemetry when I learned about the world of RC, radio controlled toys. The simplest is the RC car. This is already some, I would say, nice kit where you have your RC control and the car, but the data goes into just one direction, from the controller to the car. Why? Because you see the car. You don't need all of these fancy dashboards to see the speed. If it crashed into the wall, it was probably fast, maybe too fast. You see what things happen, so you do not need really feedback. And telemetry is the feedback. So when do we get the telemetry? Drones. It's a much more complicated thing. If the drone is not in your line of sight, you need to see something, right? And the most important in the RC car, if the battery runs low, it just stops. In a drone, it can be more complicated than that. So the drones, for me, introduced the word telemetry for the first time, FPV drones. And they do not put this fancy dashboard like we had in New Relic or whatever. It would be impossible to fly. And why I like the example with drones, because they put actually things that are important on that screen slide. The battery, most important, and some tracing data from the accelerometer, gyroscope, stuff like that. Actually things that matter. Probably GPS coordinates, the distance from you to the drone, things that you actually want to know when you fly it. So they only put these things there. So then it's an important question. What is your application? Is it an RC car or is it a drone? Because if it's an RC car, maybe you're on the wrong talk, maybe you're not there yet. But if it's a drone, then it's actually time to introduce some telemetry. In fact, we already have some telemetry in our software as well as in our hardware. What is the simplest form of telemetry? These lights, LED lights, if it blinks, it works. It is some sort of feedback, right? You see that if your Wi-Fi router blinks, means all good. And you know, this MacBook is an even more advanced tool for beginning with telemetry, because if the fans spin, it's complicated, it's a really good sign, things work. In the old ways, we had just one way to receive feedback from the application, and that was logs. Like, a very good, simple log file, log message here when something went wrong and you want to know what. And here we all need to admit that we started with something like this. Info ASD, ASD, ASD is the simplest form of feedback, and you can call it telemetry in some form, of how you get feedback about something going wrong in your app. Now a more advanced use case is when you log something like here. And I know that half of you had it, so... Then we got even smarter, and we introduced something like this. Before doing something, and then after with timestamps, so you could actually see how long it took. I also know that we all used it. And now finally when our brains went to absolute perfection, we introduced this, adding some information about the user, like user ID, order ID, whatsoever, because these are things that you want to know, if something goes wrong, you want to know what exactly went wrong, how long it took, and to which user or order or whatever it was connected. This is telemetry, done in the primitive way, but it is what this talk is about. Now of course, jokes aside, when the things get really complicated, I had to bring one of these fancy microservice architecture pictures here just to make it look serious. So this is Uber. Uber are pioneers of tracing because they introduced so many things, they contributed so much to open source, including to open telemetry, that for me their engineering blog Log is source number one for the data. So if you have architecture like this, you cannot really log this user IDs, order IDs and times anymore. You need something more reliable because requests will go through the multiple services, and you do not know what went wrong and where did it went wrong. So you need to bring order, and one step would be to bring in something like Kibana that would just parse your logs, and you could query it by time, like something went wrong At exactly this time, if you can query Kibana and see what exactly went wrong, you're good. You might not even need the OpenTelemetry fancy stack yet. To summarize the introduction, so we cannot understand a complex system if it's a black box. If it's not a black box, if you know your app, if you have two or three developers in the team, you do not need to make things more complicated than they are. You can see the logs, perfect. And a big advantage of logs is that your colleagues can also understand logs. You are in vocation, something crashed. They log into servers, they see logs. They know at the same minute what happened, most likely. But only given that they can see inside the system. If the system is this, they cannot see inside the system. And then we need to introduce the tools that we are discussing today, which is OpenTelemetry. First, can we even trust it? What is OpenTelemetry? So OpenTelemetry is like a project that resulted from merging two big tracing projects out there. It's OpenTracing and OpenCensus. They have been developed also by big companies, but they are two absolutely different standards and absolutely different tools to do the tracing as well as to analyze your data. So when they decided to merge and I think that OpenCensus was Google, OpenTracing I don't even remember. When they decided to merge together and to have just one format for all of the applications to send their logs, their traces, and their telemetry data, the metrics, they had, of course, to take some compromises because it is really a merger. There is not much in common with either OpenTracing or OpenCensus, yet we are using advantages of both worlds because it turns out that these two projects are complementary. They do not do the absolutely same thing. They do things differently, and that was a good thing because when engineers came together, they came up with a really cool and standard way of sending your telemetry data from your application. Now, as a result, OpenTelemetry exists in, I think, more than... The support of OpenTelemetry exists in more than 10 languages. By support, I mean SDK, standard agents, agents for most popular frameworks, as well as huge support of third-party software, visualization software, which I will show you later. Now, let's start with the basics, and we go from the basics in OpenTelemetry to the uses of how can we get something more custom in Python, and then we go to the final level where we see the visualizations, I'll show you a little demo, if it works, we never know, of course, let's hope. So what are the signals? Signals is like atoms, this is the basic piece of OpenTelemetry, and this refers to many, many types of telemetry out there that the framework can collect and process. It's a fundamental component, so to say, in the open telemetry, the smallest one. The main signals are, for instance, the traces. Like, traces can track the path of a request going through various services. Like, on that terrible Uber microservice map, the trace would go the whole way. Trace represents some user action, let's say, ordering something. And then you can trace a whole chain of events happening there when you order something. Then there are metrics. Metrics are quantitative data. It's like characteristics. A typical metric would be request time. A good thing, they do not need human attention, like 24-hour human attention, because you can just say when something, when this metric is over this value, something is wrong. This is a key to good use of telemetry data and automation because not all signals can be a trigger. But the metrics can and then finally logs spoken up about logs now. This is more Descriptive more detailed information about what has exactly happened, but these advantage you need to read them Automation won't work as good so Open telemetry is not just one. I wouldn't I don't even know how to call it like set of Protocol set of api's that is out there to collect traces metrics and logs it's a common one a benefit is just that the approach is unified I would say this is a benefit number one because looking not just about the products itself like not just about the amount of agent agents available out there but also about the companies developing this and these are like as I said uber google companies that have some experience in this I can be sure that this will survive next I know five years ten years because the effort is really joint in this case and there is a big part of the community also involved in it. I wouldn't trust them in this one company tool because yeah we know the hard times crisis company can just stop the project even as big as Google but here I actually believe that unified approach is both in collecting the data in being framework agnostic language agnostic as well as on the side of the developers who are not from one company, but from a huge community. This is a good signal to bet on using exactly this approach. Then I already mentioned about interoperability, I guess, that there is a wide range of backends and tools that allow to adopt open telemetry fairly easy. And then finally, it's about rich context. I wrote it here this way because the tool is a result of many, many tries. Even this open census and open tracing, it was a good try. It was good by itself. And now with this experience, they learned how to combine these pieces, traces, metrics, and logs to guarantee that you have all the data needed when you want to analyze a particular aspect of your system and find some resolutions. Now we want to say a few words about how we are we still have time 15 minutes okay so how it works finally these are the steps I will go over it really quick here yeah so it starts with instrumentations these are your applications these are your Django's flasks or micro services or just a Python script somewhere out there running on the cron job doesn't matter that generates some data that you want collect and analyze. Generate data, we call it telemetry data, this over there in number two. And then this is where OpenTelemetry connects. We need a collector, they're actually called collectors, not receivers. Collectors is... I don't like this word. So they are collecting this data from you. There are many ways to collect the data, I will be showing on my demo the simplest one. After it's collected there is some transformation, there is or there is no transformation this is really like up to us to decide if you have a huge huge huge system that generates way too much data you probably want to use some filtering you do not want to track really everything while if you use if you're developing a prototype then you do need to track everything so that that's some decisions you need to take and open telemetry lets you take this fairly easy and then there are exporters that export the data collected in various formats like here I put an example the first one trace system is jagger jagger is very very deeply integrated with yeah not so deeply but the same author so the guy contributing to open telemetry is one developed jagger so that's my bet for now and of course classic tools like prometheus datadog which I already showed a fancy big dashboard if you wanted and so on now let's go closer to the code of the different ways how we can use it the first way is the easiest way you can use auto instrumentation. The same thing like you have in Eurelic and Datadog, there is a script that can do everything for you. At this advantage, of course, you get very, very basic outcome. So it will try to analyze what's happening there. If it's one of the frameworks that it knows, it will even try to give you some meaningful insights like request timing. But this is not a good way to to track production systems, obviously. It's fine if you already have something and you just want to see what data can it collect. Good. If you are up to some real stuff, then at least try to use the framework instrumentation. So all of the popular frameworks like Flask, FastAPI, in this case Django, obviously, have support of OpenTelemetry agents that would collect request data for you automatically. By this I mean they hook into the request-response process, they can measure how long it took, of course, as well as what went wrong, and the typical stuff that you do in a request-response cycle like in Django that would be analyze what happened in the database. So it could do this for you, and this is already something. This is already way better than just logging, so if you do not have anything, it's a no-brainer. Just use the framework agent, nothing custom, and this will already bring your data quality much, much, much higher. And this is the code, how to do it, omitting all imports and setting things up, still it's fairly easy. Now we go to something more custom here. If you really want to track a particular piece of your code, it's a good, in Python, a good way of doing it will be using a with statement, like here. So by using this with tracer start current span, you are specifying to OpenTelemetry like this is a span of something that is happening. It is important. So do tracking in the code that is five spaces, not tabs to the right below this point. It is good for many things. It is good for, well, easiest is timing. It will always time how fast these things under the with statement is happening, but it will also help to structure things so you'll probably have not one but many blocks like this it helps you to structure your traces your spans so that when you use some visual software to analyze a bigger system it will be shown in the correct order like this block is under this block under this block and here the error happens something like this now a more advanced case here is when you need custom data processing and here I will only go over two basic concepts which you usually do not need but yeah it can filter and batch the data to enhance telemetry before exporting so here in this case I'm showing a simple class batch span processor processor yeah yeah it collects traces and it just sends them in batches that does what it's supposed to do which is configurable for time, so you can specify time and span count there. This will just minimize your network calls and improve performance, so it's a good thing to know if you want this on production. Now a bit more complicated is with filtering, because with filtering you need to specify what actually you want to filter. So for the sake of example, I wrote a basic class here about a filtered span processor and here specify what to do on the start what to do on the end I think all this self-explanatory because in case we want custom filtering we need to specify which spans we want to send and which we don't in this case I'm just getting attributes important in the last second last line from below if it's important then we do send it this span if we if it's not important then we omit it yeah useful in production as well and it is important because we have gdpr and some sensitive data that we actually do not want to export finally integrations just made this huge screenshot because i saw no point in really specifying everything that is integrated with open telemetry out of the box so you go on the website you see if the tool framework on both sides, like on one side from the application point of view, on another side the tool you use to visualize the data or to have some automatic triggers for your metrics. On both sides there are a lot of integration, so you basically just need to check out the website and see if there is something for you. Now finally, demo time. I'm not sure we really have time for demo time, but let's just try. In this demo you don't see my console of course. I will turn on mirroring and I will rush over it. Or can I move terminal to here? Yeah, I can. Do you know a shortcut to make text bigger? Plus? Great. Works. Thanks. So, what do I have in the demo? I have three versions of a really, really basic Flask application. So let's go over the first one. Initial, you have a context, there is just one endpoint, get weather, and everything is fake, of course. Just weather data is sunny in Berlin with the comment, ha-ha, because that's never true. And we return JSON to the consumer. Now, I will not do it live because we do not have time. I will just open the version of this with integrating OpenTelemetry, which is just app. So okay, I'm not allowed to move, so I need to somehow do this. Don't be afraid a lot of imports. It's annoying, but your auto-completion does it for you. Main things happen here. We need to set up the tracer, stop the tracing. We set up a span processor here, and we say that for the sake of demo, we just want to write things in the console. You of course pick your exporter based on where you want your data to land. You can also pick many exporters. Then you use automatic instrumentation for Flask, which I showed on slide some time ago. Flask instrumentator, instrument app, you give the app as an argument, and your request response data will be automatically traced. And now here I also have this custom span with a with statement over here because for me this block is really, really important, so I want extra tracing of this particular block. Usually it would be some, I don't know, complicated algorithm query to a service where you want dedicated metrics collected just for that block. Okay. This is all I need there to get some basics. So automatic instrumentation and this manual custom span collection. I will quickly launch it. Now I need my browser. So I made a couple of requests. I need my terminal again. Okay, so yeah, this is what OpenTelemetry would send otherwise to your back-end to collect the tracing data. This is all the information out there and before I end this demo, because I promised to at least show you some custom use, which I don't have time for, I will just show you this scenario where I put a little bit more, I set attributes manually here, span set attribute, this is the reason why I put this custom span in the first place. When you want to log, like this is a key value basically, so I want to log the status out there, if the order was successful or not successful, this helps when in your backend you later on want to filter based on that, like for all failed orders for all orders that took more than n seconds, and so on. And you can also just set not just the status, but you can set any attributes out there. It's like a Python dictionary. Whatever you set will be sent. Okay, I think I have just below one minute, so I will go back to the presentation. Sorry for the short demo. I will say that there is a tool that I recommended. This is not Jagger, this is Jäger, like a hunter. We are in Germany, for God's sake, we need to learn this, okay? Yeah, done by the guy who contributes a lot to OpenTelemetry, so most of the things work. It's free, it's open source, use it. I had it in my demo to visualize things, but you might notice I'm not a big fan of visualizing things that can be easily read from console anyway, so I will not do that. All right, then I'm out of time. Thank you for your attention, and let's see if we have time for questions.

Speaker 2 [25:59]

Thank you, Anton, for the great talk. We have lots of questions. The first one is, Sentry comes with error tracking and performance tracing. There are also alerts, example, alerting on high response time. Does it make sense to have both tools?

Speaker 1 [26:16]

Could you repeat, please? Long one. Yes.

Speaker 2 [26:20]

Sentry comes with error tracking and performance tracing. There are also alerts, in example alerting on high response times. Does it make sense to have both tools?

Speaker 1 [26:32]

Sentry is custom and I think it's very very specific to your Python, to our Python world. The disadvantage is that you have a lot of tools in your stack which are not running on Python. You have a database, you have some key value store, you have some cache, they will be most likely integrated into Open Telemetry automatically. I mean there are agents that will collect data from them automatically. Not just collect the data but automatically connect with a trace ID, they will connect the request coming in, like your user buying something, they will have the same trace ID everywhere so by searching with what we have in Jagger for instance, Jaeger, not Jagger Jaeger, you will get not just your Python insights but you will have insights of the whole system so I think it does make sense to have still both, even if your sentry is tuned to perfection, in case you want to get benefit of a broader view, not just a view inside your code.

Speaker 2 [27:36]

Next question, can I use OpenTelemetry to monitor Spark applications?

Speaker 1 [27:42]

Spark? Yes. I think I saw that there is support for that. I mean, it's a guess, but I saw it.

Speaker 2 [27:50]

Next question, advantages compared to only logging?

Speaker 1 [27:55]

I love logging, so this will be really, really, really subjective. I like logging because everybody can read logs. Every developer did it. So I do not think it's a good way to say about advantages. In logs, you can only check logs that you send yourself manually. You need to really specify everything. If you want to analyze logs, you need to send them to some tool like Kibana that will parse logs for you, because if you have just visual logs, either you have one Hello World application or something really small running just in one place, or you need at least a system that will collect multiple logs together so that you can do some analysis over many logs. And if you're collecting logs from many destinations, then you are already sort of doing a bit of telemetry. Yeah, I would say it's much faster to analyze with logs prototyping, testing, logs. Production systems on the big distributed system scale certainly you need some more sophisticated telemetry collecting mechanism.

Speaker 2 [29:05]

Thank you. There are several more questions. I think you will be at the conference later on if someone has any questions. Thank you and give Anton a warm applause.