Instrumenting Python Applications with OpenTelemetry

Speaker 1 [00:09]

Cool, so yeah, just for some introductions. Yeah. Hi, I'm Mika Naylor. I work at Confluent mostly on distributed data processing with Apache Kafka and Flink And because of this I both have a love and also a deep hatred for distributed systems I'm a Nix and a Nix OS enthusiast and my favorite Dungeons & Dragons class probably the cleric

Speaker 2 [00:38]

Hi, I'm Emily, I'm a software engineer at CircleCI. We actually don't do any Python at CircleCI, it's mostly Go and Clojure, but I used to work a lot in Python, and I always found the community to be really friendly and welcoming, so that's kind of what brought me along here today. It's also my first PyCon, which is pretty exciting. I don't do a whole lot of programming in my free time these days, but when I do, I try to do it in Gleamlang, and I also do some pottery from time to time. So for the structure of this agenda today, we're going to start with an introduction to OpenTelemetry, just a little bit about what it is, why you might want to use it, and why you should consider using it for observability. And then we're going to take a tour of the tooling and the application that we'll be working with as part of this. And then we'll get on to the more hands-on side of things. Yeah, so we'll start with setting up an open telemetry resource in a service, and then go through adding metrics, logging, and tracing to that service and then we'll round it out with some distributed tracing and auto instrumentation. At its core, OpenTelemetry is an open source observability framework that provides a so-called golden path for instrumentation and observability. It provides a standardized way to collect and process telemetry data. It categorizes telemetry data in terms of signals, so metrics, logs, and traces are all examples of what kind of signals they have. These signals can be exported from applications and then collected and processed and then eventually explored in different ways. Most importantly, it provides a unified framework for observability, which is both vendor and language agnostic. And as a result, there are OpenTelemetry SDKs and APIs for many common languages and frameworks, such as Python and Go and Rust. And as a result, as part of this tutorial, we'll be making use of OpenTelemetry libraries and tooling, which offer specific instrumentation for Python itself and also the Flask framework and the request library. OpenTelemetry uses the OpenTelemetry protocol, or OTLP, to export data and can be integrated with many different observability platforms such as Prometheus, Jaeger, Grafana, essentially any which support it. And yeah, as a result, it provides a unified approach to observability without being tied to a specific tool or provider. So if you change your telemetry back end, you don't need to make any code changes. In a nutshell, OpenTelemetry provides tooling to help engineers better understand their systems, not just how individual services work, but how those services work as part of the system in which they exist. Distributed systems are very complex, they can be confusing, so you do need observability in place to have confidence in the services that you operate and are potentially on call for, or just simply have users which expect a certain level of reliability. And to reiterate, OpenTelemetry does this through providing a set of tools, such as libraries for you to use to instrument code and applications, and then also infrastructure tools, which you can then run to collect and export telemetry data. And it allows you as the owner and operator of software to generate insights about things such as usage, request flows, performance, errors, and much more. And of course then you can also communicate this information to the people that you cooperate with.

Speaker 1 [04:52]

Yeah, so as part of this workshop, we're going to be running a telemetry platform to support our ingesting and visualization of our telemetry data. We have opted, in this case, for a set of fairly popular open source telemetry backends. But the nice thing about OTEL is that any one of these changes could be swapped out, like for the tracing storage layer, you could maybe use a different tooling, logging metrics, et cetera. So, for metrics, we're using Prometheus to store and process those metrics. For logging, we're using a service from Grafana called Loki. And also for tracing, we're using a service called Tempo, also from Grafana. And all three of these telemetry back-end services will be set up as data sources to Grafana that we can then use for initially visualization, but then, you know, like operational dashboards, alerting, that sort of thing. But the thing at the bottom is the thing that really makes this whole setup work. It's the OpenTelemetryCollector. And so the OpenTelemetryCollector is a piece of software that essentially acts as a sync for exported telemetry data from your applications, does some optional processing on it. For example, if you're running in a Kubernetes cluster, for example, in the OTEL collector you might add processing to pull Kubernetes context, such as like a pod name or a namespace name or a node name, and then you'd add that to the metrics that you've just ingested. So you can add contextual information, and then it forwards it on to these services. So our applications, when we instrument them, don't know anything about Prometheus, Loki, They just know OpenTelemetry, and then they just know to throw their metrics at the OTEL collector. And just by way of example, there we go, cool. So in the repository, there is a OpenTelemetry collector config, which sets up essentially the behavior of the OpenTelemetry collector. So at the top, we're setting up a set of receivers. We're going to set up receivers for both GRPC and HTTP endpoints, although just through the workshop we'll just be focusing on the HTTP endpoints, and some extra core stuff because working with this stuff on localhost is a bit of a hassle sometimes. We're not setting up any processors, we're just going to export the metrics as they come in. But here we set up a series of exporters. So we're going to set up an exporter that uses OTLP called Tempo. It's going to target the Tempo service that we'll set up in a minute in our Docker Compose. Similarly for Loki, it's going to set up an endpoint where it should throw to. And then the Prometheus one is a bit of a kind of edge case in that the OpenTelemetry collector in this case is not just collecting and then exporting the metrics, but it's transforming them into a Prometheus readable endpoint, which Prometheus then scrapes. So Tempo and Loki in this configuration just both support ingesting OpenTelemetry metrics, but Prometheus we're doing it in a slightly different way, one that's more native to Prometheus. And we just manage this in the configuration for the OpenTelemetry collector, not in our application. We're also setting up a debug exporter, just so we can see some stuff on the console as it comes in. And then finally we set up our pipelines. So for traces, these are the various types of OpenTelemetry signals, we'll cover them in a bit more in a minute, but for traces, we receive it from this OTLP receiver that we defined above, and we export it to both tempo and also to debug. Same for metrics, which go to Prometheus, and logs, which go to Loki. If you're running the code along with us, we can bring up the telemetry platform just by doing a docker compose, so there should be a docker compose.yaml file, which defines essentially the telemetry back end, we're going to use docker for this. So if we just do docker compose up and then set it to D to make it a daemon, it's going to start those services in the background. Cool. Yeah, so that's the command that you should run if you'd like to bring those services up. Cool. Once those services are up, only two of them have frontends that you can access, Prometheus and Grafana. So if you wanted to check that out, you would want to go to localhost 90 for Prometheus and localhost 5,000 for Grafana. So we can just quickly check that that is not working. Cool. It's 3,000. 3,000. Thank you. Not 300. Okay. Sorry. Yeah. Thank you. 3,000. So Grafana is running, let's just check on Prometheus, so we don't necessarily need to. Cool and Prometheus is also running.

Speaker 2 [10:11]

Has everyone run those commands? Could you raise your hand if you haven't yet got that running yet? Okay. Were you able to get that running okay? Cool. Thanks. So now might be a good point to take a look at the backend services section of the README. So in order to explore how to use OpenTelemetry, we'll be instrumenting an example application. The purpose of this application is essentially to just provide a website where you can input a place name and it calculates the distance that that place is from the current coordinates of the International Space Station. of the hood, this service or this application is comprised of four microservices. The first is a frontend with a simple UI that you'll interact with via the browser. The second is a gateway service which handles the incoming requests from the frontend and then coordinates responses from the geolocator and ISS distance service. And then we have the geolocator service which gets the place name as a string and finds the geocoordinates for that location and returns them to the gateway. And finally, there's the ISS distance service, which receives those geocoordinates. It also makes a request to an external API to retrieve the coordinates of the ISS, and then it calculates the distance between the two and returns it. So we'll spend the first couple parts of the workshop just instrumenting the ISS distance service, and then the final part, when we do distributed tracing, we will instrument these three services.

Speaker 1 [12:18]

So yeah, so if you have the backend services running, this is the command you'll want to run to bring up this application. So what this basically does, we're doing Docker Compose. We're going to use the specific Docker Compose services YAML file. And the two flags at the end tell it to build the Docker images because we're going to run them locally from a local code. And also, dash dash watch means that whenever we change the application code, it will automatically rebuild the Docker image and run the service for us. So once we have this running, we won't need to bother with Docker Compose again. So if I just bring this up, Docker Compose. Yeah, so Docker Compose F, Docker Compose services up, dash dash build, dash dash watch. This command should also be in the repo, as well, if you need to copy and paste it. And once you do it, you should probably mostly just see a lot of logs from Grafana. Oh wait, no, Grafana's on the other one. So yeah, so once this is up, you'll see a bunch of debug logs from Flask, just telling us that we've set up a frontend, a gateway, a geolocator service, and an ISS distance service. And then, once you run that, the front end should be available at localhost 5000, which we can just check. Cool. And yeah, so this is our application, well, the front end to our application. And so if we search for, no, let's make it local, Darmstadt. So if we search for Darmstadt, click Submit, no, cool, what is going on? Live coding, folks, everyone loves it.

Speaker 2 [14:24]

I love that.

Speaker 1 [14:27]

Max retries exceeded Bollocks, okay I was throwing a wrench in my plan Okay, hadn't expected this let's see if we can just quickly fix this so that's

Speaker 2 [15:01]

Oh, could we change the...

Speaker 1 [15:02]

Yeah, so if you get this.

Speaker 2 [15:07]

Thank you. Oh, yeah.

Speaker 1 [15:13]

Maybe change the works maybe changing the user agent. Yeah

Speaker 2 [15:26]

in the geolocator service. Would you want to change it in the ISS?

Speaker 1 [15:31]

I think we're getting way limited on the geolocator. Ah, okay.

Speaker 2 [15:34]

Okay

Speaker 1 [15:34]

Oh, okay. Okay, cool. That works. Yeah, so if you run into this, it might be because you're getting rate limited. To fix this, just change the user agent that we're passing into the geolocator library here to something unique. I guess we're getting rate limited here. Yeah, so currently the ISS is at these coordinates. It's about 4,000, 5,400 kilometers from Darmstadt's coordinates. So this is our sample application. So again, just to reiterate the flow, a request comes in from the front end, goes to the gateway, the gateway calls out to the geolocator service, which tried to foil us, but luckily we bested it, returns the coordinates for the given location string, and then we grab the location of the ISS from the ISS distance service. That calculates the distance between Darmstadt's coordinates and the ISS, and then returns as the final distance.

Speaker 2 [16:42]

We have a question.

Speaker 1 [16:45]

Oh, yeah. So the fix, if you go to the geolocator service, in get coordinates for location, we're using a library called Noantim from the GeoPy Geocoders library. Here, it requires a user agent, which we set to PyCon 2025 Hotel Workshop. And I think we're getting rate limited because it's too many requests from the same user agent. So you probably just want to change this to something like unique, your name, date of birth, favorite sandwich, that sort of thing.

Speaker 2 [17:28]

Dockery limited Microphone work Attendees can also

Speaker 3 [17:52]

more of a tech issue I'm running into docker rate limiting as well I just heard in a room that switching to mobile network is probably the solution because a room full of people pulling docker slim

Speaker 1 [18:03]

It's probably not the ideal.

Speaker 3 [18:05]

Thank you. Set up.

Speaker 1 [18:06]

I forgot that they started rate limiting people.

Speaker 2 [18:09]

Okay.

Speaker 1 [18:09]

Okay.

Speaker 2 [18:10]

Yes.

Speaker 1 [18:10]

Yeah, switching to a mobile network and doing a Docker Compose poll might be a way around that.

Speaker 3 [18:17]

Another thing that worked for me is log into your personal docker account. It's free

Speaker 1 [18:23]

Was that too? Yeah. OK.

Speaker 3 [18:24]

register using Google account or Apple or whatever

Speaker 1 [18:25]

Just. Cool. Thank you. Thanks.

Speaker 2 [18:45]

So, the first part of the instrumentation that we'll be doing is adding a resource. Resources used to just represent the application that's producing telemetry data, and so you instantiate it with any information that you want to be associated with that telemetry data to identify the resource. So things like the service name, the service version, and maybe the environment in which it's running. It is immutable, so it isn't expected to change over the course of the application's lifespan.

Speaker 1 [19:27]

Yeah, so if you go to Section 0 in the repository, there's some instructions for how to follow along in setting up the OpenTelemetry resource. But we can also do some more live coding because everyone at all is live coding. So, yeah, so we'll be instrumenting the ISS distance service. So in here, it's essentially a very basic flask application. We have a single route we're exposing with a get, which does the getting of the ISS coordinates and then the calculating of distance. And so to set up a OpenTelemetry resource, we need to do some importing. So at the top, we want to import import from the OpenTelemetry SDK.resources. We're going to import some static values, there's deployment, environment, and service name, and also the resource object. And then we can just create a global one because we'll be reusing it. Yeah. So we're just going to create a resource object. It takes, among other arguments, just an attributes map. Here we're saying our service name is just going to be the ISS distance service, and the development environment is going to be extremely nerve-wracking live coding. And we're using the static values that we've imported rather than just arbitrary string values, which you can do. You can add basically any key value map into the attributes that you want, and these will get propagated to your metrics. We're using these static ones because OpenTelemetry also, as part of its API, has defined a set of, let's say, semantic conventions for certain attributes. So it's generally advisable that your service name's key should always be service.name. But instead of trying to enforce this everywhere, you can just pull in the static string value from the library. And I think there's a few more in here, but we're just going to use deployment environment and service.name.

Speaker 2 [22:16]

Could I get a show of hands for everyone who's currently completed that? Okay, let's do it. I'm sorry, I didn't catch that.

Speaker 1 [22:29]

It should also be in the readme as well.

Speaker 2 [22:33]

Yes, and it's also worth mentioning that we, for each section that we have, we also have a completed branch that we link to at the top of the section of the README, just in case at any point you just want to check out that branch and start from there.

Speaker 1 [22:48]

Yeah. So, for example, in the read me at the top, if you just wanted to start from the section we just completed, you could go to the section zero, resources branch. And in here, we have a commit, which adds the necessary import and code and stuff. And we have a branch for each one of the sections that we're going to do. So if you feel a bit lost, please feel free to, like, check out one of those branches and use that as inspiration or as a jumping off point.

Speaker 2 [24:26]

Okay, so just to keep it moving with the time, so the next form of instrumentation, or I guess the first thing we're going to be instrumenting, or the first signal type we're going to be instrumenting, are metrics. And so metrics are measurements of attributes of a service during runtime. the values of metrics change in response to events such as incoming requests, outgoing requests, and periodically these metrics are read and then reading them is associated with a timestamp which allows us to view how the measurement changes over time. Because of this metrics are often use for monitoring and alerting. There are different forms of metrics. So there's counters, which are measurements that monotonically increase from zero. An example of a counter metric could be like number of requests received. There are also up-down counters, which start from zero but monotonically increase or decrease. And you might use this for tracking the number of active users. A gauge is a sample of a value that can vary over time, and you might use this for tracking memory usage. And then there's also histograms, which are similar to gauges, but in the way that they are samples of an underlying value, but the samples are aggregated into buckets, which gives you a statistical view of the samples. So an example could be request latencies. Histogram allows you to answer questions like what's the 95th percentile of our request latency.

Speaker 1 [26:22]

Yes, also something that I neglected to mention earlier is that during the course of this workshop, we're going to be doing quite a lot of importing from OpenTelemetry libraries. In case you're wondering what libraries were imported from, you can check out the requirements.txt of the various microservices. So for example, in the, let's check out the, yeah, geolocator service. So the geolocator service, we're using Flask and GeoPy. But for the OpenTelemetry side of things, we're importing the API library, which exposes the kind of high-level API objects that we define, like metrics, resources, that sort of thing. There's the SDK side, which has a bunch of, let's say, language-specific implementation to work with the API objects. We're also importing the OTLP exporter because we're going to be exporting our generated metrics via the OpenTelemetry protocol, but there are also libraries that let you export it in other formats if you need to. And then these final two libraries are order instrumentation libraries for flask and requests. We'll get to these at the end, like towards the end of the workshop. So they're not quite so relevant right now, but yeah, mostly the API and the SDK and the exporter are what we're going to be using for the time being. So let's create some metrics. As I alluded to just before, we're going to be doing a lot of importing. So in the Section 1 section of the README, there's an import block which you can grab, which imports a bunch of stuff from both the OpenTelemetry Explorer and the Metrics API and the Metrics SDK. We can quickly talk about what these represent once we have set up a metrics set up function. This is not strictly necessary, but it's nice to kind of encapsulate this logic in a function just to keep it separated from the other stuff. So here we're going to just create a function called set up metrics. It takes in a resource, an OpenTelemetry resource as an argument, and creates a bunch of objects. So let's quickly just like go over what we're doing here. So we're going to create a metric exporter. We're going to set insecure equals true on the exporter because we're going to be using local host so we just kind of want to enforce it that it doesn't expect like any SSL termination or anything like that. Next we're going to create a metric reader. This is the thing that essentially reads the metrics on a given interval. There's other kinds of readers. For example, there's an in-memory metrics reader, which is useful if you're doing unit tests and you want to see that certain co-paths produce certain metrics. But here, yeah, we're going to set up an exporting metric reader, and we're going to set its It's export interval to 1,000 milliseconds, so every second it's going to export our measured metrics. And then finally, we're going to set up a meter provider. So in the context of OpenTelemetry, meters are the things that are doing the measuring, like gauges and counters. We have to initialize the meter provider with a resource that we've passed into the function, And we have to give it a list of metric readers that it should essentially, you know, export its values to. So we're only going to give it the metric reader that we defined here, but you could give it, you know, an arbitrary number of readers depending on, like, your application's needs. And then finally, we're going to set meter provider, which basically sets this configured provider to be globally available to anywhere in your application. So in your application, you can call getMeterProvider, and it will return the provider that we defined in here. So now that we've defined this function, and also we've defined the resource that we're going to use, let's call the setupMetrics function with this resource. And yeah, then we get a meter. So we get the meter provider that we assigned further up, and we get a meter, and we have to give it a string name. It's easy in this case just to give it the underscore, underscore name, underscore, underscore. But depending on where you're on your application, you might want to give it a more useful name. And so once we have this meter, we can start creating our various metrics from it. So here, we're just going to create two counters, one called an incoming request counter and one called a ISS request counter. The first argument is just the name of the metric that we're going to be measuring, and then we can give it an optional description, which will, I think, depending on the telemetry back end, will then be exported to the telemetry back end, so you can kind of see what each metric is. And once we have these, we can start instrumenting bits of our code. So given one of them is an incoming request counter, we probably just want to instrument it here in our base API function. So we could do something like incoming requests counter, and then we're going to maybe add one to it. So because it's the counter, it monotonically increases upwards, and yeah, so every time a request comes in, we increase this metric counter one at a time. And then we can also instrument the getting the ISS coordinates bit, and we can also do something a little more interesting here. So once we make a request to this ISS now URL, we can increase this counter by one, but we can also give it some extra attributes. So in this case, we're going to say that we're going to add to the metric that we're exporting a response status, and we're just going to grab the status code from the request here. What this will do is that when it's exported to your telemetry backend, it will contain whatever the value of the status code is. So not only would you have a metric for the number of ISS requests that you've made, you'd have metrics for the number of 200 responses, or the number of 404 responses, or 400 responses, or something like this. Then you could use that in your observability kind of things. And once we've added this, the Docker Composer should rebuild the service and start it again. And so these should already be available if we go to Prometheus. I'm just going to refresh. Good luck. So if we go to our Prometheus instance at localhost 1990, hopefully. Oh, okay. Okay. Why are you not working? Oh, maybe we need to make a request for us. That might be it, yeah. Yeah, I think that's correct. But we can also, if we just make a request to London. I can make a request to counter.add. Yeah. Exception. Oh, typos. See, this is what I get for not just copying and pasting everything. Oh. Oops. Okay, let's try that again. Okay. Yes. Yeah, we're getting grade limited on GeoPy. Yeah, let's do that for now. If you're on a Wi-Fi with less users, perhaps at home, it should hopefully still work, but here we're going to just... You can connect to Eduroam, really?

Speaker 2 [35:49]

Location is not defined. Oh, I guess he commented that out

Speaker 1 [35:53]

Oh yeah. Oopsies.

Speaker 2 [35:59]

Oh, we can just, yeah, I guess remove that. Yeah.

Speaker 1 [36:24]

Okay, so as this request is going we can maybe go back to Prometheus and take a look So yeah, so we have some incoming requests. You'll notice that Prometheus has changed the name of our metric. It's no longer incoming dot requests, but it's now incoming underscore request underscore total This is just something that's particular to Prometheus and if you swap out your backend provider OpenTelemetry will take care of this conversion for you But we can execute it and we see that we have one request We can go again and we can see in Prometheus that's now incremented to two. We should also be able to see our ISS. No. Oh, because maybe it commented it. Now if we head back to Grafana localhost 3000, we can also explore this. So if you go to the explore tab in Grafana and choose the Prometheus data source, we can select a metric. So maybe ISS requests total, let's see what we have. And so Grafana starts graphing it. We have two requests to the ISS service, and the response status is 200 because we set it as an attribute on that metric, so different requests will have different response code or response status attributes, and then you can start doing, like, visualization and alerting on those metric attributes.

Speaker 2 [38:12]

Give people like five minutes to catch up

Speaker 1 [38:30]

Yeah, so in the repo, we've basically just commented out this section here, and we're just going to return just a fixed coordinate of just 10, 10. So whatever's in here, just return a fixed value.

Speaker 3 [38:47]

Thank you, and I was just asked to...

Speaker 1 [38:48]

asked to repeat the question.

Speaker 3 [38:51]

the question anyway you know

Speaker 1 [38:51]

Anyway, even after the answer.

Speaker 3 [38:52]

after the answer. So the question was, what changes did

Speaker 1 [38:54]

changes that we have now

Speaker 3 [38:55]

that we have now as compared to the

Speaker 1 [38:57]

compared to the version in the repository? Yeah, so in the repository, we are actually doing real-for-real geolocation based on the string, but we're getting rate limited by the service, unfortunately. So if you want to just remove this code or comment it out and just return a raw set of coordinates. For those at home, hopefully you shouldn't be getting rate limited, but in the building, what can we do? Thank you. Yeah, that's enough time. Does anyone need a little bit more time to add metrics? Please raise your hand if you do.

Speaker 2 [41:02]

There's a question Over here

Speaker 1 [41:21]

Thank you. I was wondering, I have everything up and running, and I want to query now.

Speaker 3 [41:25]

to query now from permission.

Speaker 1 [41:26]

from Prometheus the metric but I missed I think the name of the metric. So when you're creating the counters this first argument will be the name of the metric. Prometheus transforms it a little bit by changing the underscore the full stop to an underscore and also adds total at the end to signify that it's a counter but hopefully your Prometheus should have auto completion. So if you type ISS or incoming it should hopefully show that these two counters are available for you to query.

Speaker 2 [42:10]

Cool. Okay. I think we can move on to the next section. Sure. So I imagine many, most of you are possibly familiar with logging already. So logs are timestamp messages which are emitted from a service. They can be unstructured, such as raw strings, structured, as in JSON payloads with keys and values. For this we're only going to focus on unstructured logs, but OpenTelemetry also supports structured logging, and logging is probably the most well-known and probably one of the most used telemetry signals.

Speaker 1 [42:57]

Yeah, totally. So again, this involves a new fresh round of imports. One thing to say is that although libraries and SDKs exist for a variety of languages, some sections are more stable than others. So the logging in the Python OpenTelemetry is, I think, currently in, it's not stable, but like the one before stable, I don't think beta, but it's being stabilized. So when we import it, you might notice that we're importing from underscore logs rather than logs because it's still a work in progress. And alongside the OpenTelemetry stuff that we're importing here, we're just going to import the logging library from the standard libraries. I believe OpenTelemetry also supports integrations with other popular logging libraries, but we're just going to use logging here for the sake of simplicity. And then we're going to create a setup logging function similar to the setup metrics function with a slight difference. So we're going to create a setup logging function, which takes, again, the resource that we've defined, and also a logger. And what this function does is that essentially it sets up the logging provider and assigns the handler that it creates to the logger that we pass in. So I'll just explain briefly what this means. So here we're going to create a logger provider, nothing more to add in here except for the resource, and then we're going to set that logger provider to be globally available in in case any other bits in our application need to pull it. Again, we're going to just create a simple log exporter and secure equals true because we don't want to bother with HTTPS here. And then we're going to create a logging handler. We're going to set a level. Well, we're going to unset the level just to make things easier in the workshop. And we're going to give it the provider that we specified up here. And then to the logging provider, have to add a log record processor. So here we're creating a simple log record processor from OpenTelemetry. And what this log record processor does is that basically as soon as a log is emitted, it's processed and then exported to the exporter that we define up here. In production, if you have a service that's making lots and lots of logs, you probably don't want to use this simple log record processor because it would have to, like, export it one at a time. It's quite insufficient. So there are, I think it's called, like, batch log record processor, which then does this in, like, batches with batch sizes that you can configure both in the code and also via environment variables. And once we've added the processor, we can use this logging handler that we created here and add it to the logger that we pass in. And we're just going to set the logging level on the logger to debug, just so we get everything. And so down in here, we're just going to create a globally available logger that we can use wherever in our application. Logging.get logger. And we have to give the logger a name. We'll just And then we can call the setupLogging function with the resource that we've defined above and also just pass in the logger. So we could, you know, you could have several loggers for different purposes, one that just like goes to debug, one that's instrumented with OpenTelemetry, just to explain what kind of why we chose this structure. And once we've added these logs, once we've created this logger and added the handler and set up the OTL stuff, then we can just start logging as you would do in any other application. So maybe down at the bottom we might want to add something like logger.info with started the application. I'm just going to quickly save this and then check that it's being built and started. Yeah. Okay. Cool. We might also want to add in logging at various error points. For example, we might want to do logger.error with no latitude or longitude. And then maybe when we get a request in to the API, we can maybe add logger.info, receive a request from the IP address, and then just substitute it into the string. So basically kind of like as you would normally use the logger library in a Python application, but we've just tacked on the OTL stuff. Once we add these in, because our application is being rebuilt, let's try another request. Okay, so request succeeds. If we go back to Grafana now, and again go to this explore tab, we can change our data source from Prometheus to Loki, which is the backend service we're using to handle logs. So let's maybe select a label, service name, and what is the deployment environment? Extremely nerve-wracking live coding, okay. So let's look for all logs in the extremely nerve-wracking live coding filter, and we see that we get some log volumes, and we also get some dotted application, but not the info I'm not quite sure why the info one is not coming through.

Speaker 2 [49:48]

Now can you switch back to the code?

Speaker 1 [49:51]

I should just do logo.info received request from IP address.

Speaker 2 [49:58]

And you did make a request, right? Yeah. You think? Yeah, I think so.

Speaker 1 [50:09]

It's amazing the stuff that works in the hotel room and not in the conference.

Speaker 2 [50:19]

Oh

Speaker 1 [50:20]

Oh. OK, let's try this. Let's resend the request. OK, go back to Grafana.

Speaker 2 [50:38]

Thank you so much.

Speaker 1 [50:40]

So, we started getting logs ingested into Grafana.

Speaker 2 [50:40]

Thank you.

Speaker 1 [50:46]

And we can expand these logs. And they'll have some fields associated with them, like the code file path. And notice that we have the deployment environment that we set up on the OTEL resource. This is being passed into the open telemetry version of the log as we pull it from the logger. have time stamps, scope names, service names, and the severity number, as well as the initial log when we start the application. Yeah, so whereas with the metrics, we were using pretty OpenTelemetry-specific language for describing them, here we have an example of hooking into something existing, in this case the logging core library. I think we have a question, yeah?

Speaker 2 [51:44]

On my way So in general there's also slido

Speaker 1 [51:51]

In case you have a question.

Speaker 2 [51:53]

of a question prepared, but here's one of the side ones.

Speaker 3 [52:00]

I didn't see in the logging setup a formatter being specified after the handler, but the message itself is formatted in the Grafana. Like it had the timestamp and everything. If you get that line that says like 2.25, blah, blah, blah.

Speaker 1 [52:19]

Um, it.

Speaker 3 [52:20]

Is that just a default or would you not format your log messages yourself?

Speaker 1 [52:28]

I think this is the default. Yeah, I think you can set up a login format to yourself, but I haven't drilled into it too much in this example. Yeah, I think this might just be the default that OpenTelemetry does.

Speaker 2 [53:04]

We have another question down the front.

Speaker 3 [53:14]

So, what is the advantage of doing it this way? So, another approach would be to collect the logs on the Docker level that use Promptail or something like that and pipe that directly into Loki.

Speaker 1 [53:28]

Yeah, so I guess an advantage of this would be that, I mean, I guess also Promtail, you can configure it to go to Loki as well. Yeah, I'm not quite sure off the top of my head what the benefit versus like a kind of like a log file scrape method, which I think is what Promtail uses. other than yeah you can define also like post-processing on the OpenTelemetry collector as the logs come in yeah I think this is also kind of one of the reasons why the logging part of the SDK is sort of not quite stable because it doesn't have quite a clear answer to this model as the metrics and traces kind of do yeah it's kind of like up to your preference Thank you.

Speaker 2 [54:36]

So for the next part of the workshop, we're going to be taking a look at tracing. So traces are possibly the most maybe complex, but also maybe the most expensive form of telemetry. But they're also extremely valuable. They describe the path that a request takes through an application. They're made up of spans, which are the kind of like essentially units of work, and provide a breakdown of what actions are happening and where they're happening and how long they take to happen across a particular request.

Speaker 1 [55:19]

So if we go back to our ISS distance service application, there's a couple of obvious kind of like blocks of work that we could carve up into spans in here. So first let's again import a bunch of stuff. So it's kind of pretty much similar to other import sections. We're importing an exporter, importing some functions for getting and setting the providers. We'll get into these two extra things that we're adding here because they're kind of quite interesting. And then we're creating a tracer provider and a simple span processor. Again we're going to set up a setup tracing function to kind of follow the pattern that we've been doing all along. So this takes in a resource, just like the metrics one. Again we're going to create an insecure exporter, we're going to create a simple span processor. Again spans are expensive and you might have thousands of requests per second and you probably don't want to send each span one by one to the open telemetry collector, it can be kind of expensive. So you would in production maybe use a batch span processor, but for here we're just going use a simple one which does it one as soon as the spans come in one at a time and then we also set up a tracer provider and the tracer is the object that lets us essentially create spans and then yeah we can add the span processor that we defined a bit further to the trace provider and then we can just set that trace provider globally so it's accessible to other parts of our application that need to create traces Again, we can call our setup tracing function and pass in the resource that we defined previously. And similar to how we created a kind of global meter for the metrics, we're going to create a global tracer using basically the same pattern as the metrics, which is we do get tracer provider and then we get a tracer with a particular string name. In this case, we're just going to default to the underscore, underscore name, underscore, underscore. And once this tracer is available, we can then use it throughout our application to create spans. So yeah, so once again to kind of reiterate, spans are units of work, they're like blocks of stuff that happens in your application, might be, you know, calling out to a service, doing a calculation, and when you, and also these spans can be nested. So a span can have within it other subspans, so a unit of work can have within it units of subwork and they can have subunits inside themselves. And altogether, from beginning to end, this collection of these spans is what forms a trace. And so with the OpenTelemetry Python SDK, there are a couple ways of creating a span. One of the easier ways is to use the function decorator. So start as current span is also available as a function decorator, and we have to give it a name for our span, so we can call it something like getting ISS coordinates, and what happens is that when this function is called, it already wraps whatever the function is doing inside of a span. Another way of starting a span is like manually, not with decorating. So we could maybe, let's think, yeah, maybe in here, we can use the, like, the with start Tracer's current span, we can give it a useful name, like getting ISS distance service. We'll need to indent all this stuff to be within the scope of this with. So if you've ever done stuff with opening files in Python, you've probably seen this pattern before where we call this function with width, and it returns a variable called a span, and this span is available in this scope here, and once this scope is exited, the span is then closed, it's marked as finished, there's been no errors, it's all good, it's fine. But let's say we've wrapped this function in the starter's current span, but we still need to access the span itself to maybe add some extra information to it. And that's where the getCurrentSpan function that we imported works. So we can say something like span equals getCurrentSpan. And what this will do is that if it's within a current span, it will pull it out and give it to you so that you can start adding stuff to it. And so in this context of making the request out to the ISS URL, we might want to add some attributes to the span. So let me just... So maybe after we get the coordinates, we can maybe add to the span and say, okay, once we have the coordinates, we're going to set an attribute on the span called iss.position with the string representation of the coordinates that we got returned. Otherwise, at the bottom here, if we're reaching this bit of code where we're just returning zero zero, it means that something has gone terribly wrong. We haven't got the ISS position, we're not returning it correctly, so we should fail the span. So we have to do this manually in this case, because we're not returning an exception. So we can do span, and then we can set the status on that span. And we imported the status code earlier, and we can set this to be error. So when a, excuse me, when we reach this bit of the code, we know that we haven't gotten the ISS position, something's gone wrong, so we're going to manually set the span to be error. There are two other types of status codes. There are okay, when you want to explicitly mark a span as okay, as finished, as all good. And then there's unset, which I think is the default state of a span, which basically just means it's finished. It has no bearing on whether it finished correctly or whether it finished with an error, just that whatever this block of code that the span was wrapping, it has terminated. Let's see, what else do we have? And while we're adding some attributes to spans, we could probably add some in our base API. And like I mentioned before, OpenTelemetry has agreed on a set of semantic conventions for certain attributes. We saw this a little earlier when we used one of these semantic conventions to set the service name and the deployment environment in our resource. But there is also semantic conversions for attributes for other OpenTelemetry API objects. So from the OpenTelemetry semantic conventions library, we can import span attributes. So these are essentially a set of semantic conventions for attributing, for putting spans in attributes. So in our API function, if we just scroll down, we might want to add some, oh, maybe in the ISS service, actually. That makes more sense because we're doing a request. So once we've got the current span, We can start adding some general attributes to it. I'm going to copy and paste this because I don't trust myself. So yeah, let's add some stuff. Just if you're following in the repo, I think HTTP status is wrong. I think it should be HTTP status code. I think this changed.

Speaker 2 [64:05]

The URL is all is underscored instead of capitalized

Speaker 1 [64:11]

I'm not sure why my order completion isn't working, it would be very helpful. So once we've got the current span when we're getting the ISS coordinates, we're going to set a bunch of attributes on that span. So we're going to set the HTTP method to get, because we're just calling a get request. We're going to set a span attribute HTTP status code, and we're going to just set it to whatever the status code is that the ISS now URL returned. And finally, we're just going to assign the URL that we're actually calling to the attributes of the span. What else can we do? Maybe we want to add an error case down here as well. So we have the span available to us, so in here, if we're down here, we can also just set the span, set status to be status code dot error, something like this. sure this is building. Cool. So now that we've added some spans across some of the units of work in our application, we can make another request. Cool. And then if we go back to Grafana, again to the Explore tab, we can switch our data source to Tempo, and here we can write a TraceQL query, which I think is sort of like a language that Grafana uses to search for traces, but we can just simply click on Search, and we can maybe set our service name that we're looking for to be the ISS distance service to see all the traces for that service, and we have one here. So it's got a trace ID, which is like a unique ID for each trace. We get the service, the name of the trace, and also the duration of the trace. And if we look into the trace, we can see we have essentially two blocks, although they're basically the same length because the major unit of work is this getting ISS coordinates block. We can examine it a bit further. We can see the start time, the status was unset because we didn't explicitly mark it as okay, but it did an error, so that's fine. And we can also see that we have some attributes assigned to the span. So we can see that we have the HTTP method get, it returned a 200, and the URL that we were calling was the Open Notify API for the current location of the ISS. And we can see that the coordinates that we got returned were negative 43 and a longitude of 113. And then if we look at the resource attributes associated with the span, we can see that the development environment and the service name that we defined have been percolated down to the trace. And so we can use these attributes both on the span attributes and the resource attributes for searching, for observability, for visualization, stuff like that. Am I missing anything on traces?

Speaker 2 [67:17]

No, I think that covers it.

Speaker 3 [67:32]

So the span object you get either from the context manager or directly is it also something you like Handled into different functions. Do you pass it around to do something with it like best practice to keep it on that level? Yeah

Speaker 1 [67:46]

Yeah, I think the way that I would write this is that I wouldn't pass this into, well, I suppose you could pass it into a function if you didn't want to mark that function as like a discrete unit of work. You could pass it in if you wanted to like set some attributes on it within that function. But I guess generally in this example and also generally I have often hewed to marking functions as spans in and of themselves, because then you also get timing information. So when the request comes through, if you have created a span for the functions that you're calling, you can see discreetly, like, ah, this function took 80% of the time. This took less time, and things like that. So it's kind of like, yeah, what you would like to get out of the trace information. Yeah, I hew to sticking with a span as a function almost, just to kind of see the timing of of the stuff that I'm calling.

Speaker 2 [68:43]

Yeah, I'd agree with that.

Speaker 3 [68:55]

So in the other function before, where we get the span out of the context, like a couple of lines above, if you want to show. Right there on 112. So does it happen that at some point you don't need to span, but still need the result of the function? So you mix like for two different purposes, the use of get ISS coordinates, or would you rather have a null handler? for example if there's no current span you just want to skip the function or throw an error so what's what's your ideal handling in this situation like one case where it's null and another where you don't want to use this function as a span or include the span in this function because now we assume there's always a span that's the that's the thing

Speaker 1 [69:50]

Yeah, I'm trying to think, do you know what it does in this context if you haven't wrapped the function in a span already? Does it just return like a no-op?

Speaker 2 [70:02]

Um, I'm not actually sure.

Speaker 1 [70:04]

y'all

Speaker 2 [70:05]

Yeah. I haven't tried.

Speaker 1 [70:10]

Let's find out What happens here and rebuild this let's see if we get an exception Resend quest Okay Yeah, so I think in in the context where the function Doesn't actually have a span I think it does actually this get current And span returns like a no-up one, which just like swallows all the operations that you do to it. So I guess maybe in this context, I would code it as if there always is going to be a span available in that function, and then let the consumers of that function decide whether they want to include it in a span or not, I suppose.

Speaker 2 [70:56]

Yeah, I think so. I think that makes sense. So I think maybe in the interest of time, should we move on to distributed traces?

Speaker 3 [71:08]

Thanks. I wanted to ask if you go to production, what you should maybe change about this approach or what considerations you have to take. You mentioned like batching spans already, but is it go ham, instrument everything and then kind of leave it to DevOps to figure out or are there some best practices?

Speaker 1 [71:28]

Yeah, so I'm always a big fan of going ham and letting DevOps clean up after me, but the thing about OpenTelemetry is that, I mean, I don't think we actually showed this, but if we go to the Docker Compose services, we can see when we're setting up the run environment for the services, We're exporting an environment variable called LTLP exporter, LTLP endpoint, which points to the OTEL collector. So this basically just says every exporter should, it pulls it from this environment variable. And what I mean by this is that a lot of OpenTelemetry's behavior on the operational side is also settable via environment variables. I think Emily will talk in a little bit about trace sampling, where you can define criteria for which traces that you actually export and which traces that you ditch. But I think the nice thing about OpenTelemetry as an application developer is that, yeah, you can kind of go ham. You can instrument as much as what you think is valuable, and operationally, you can tweak the behavior of what is sampled, how often it's sampled, stuff like that, through environment variables. So you can also have, depending on your environment and things like that, tweak it without necessarily having to dive into the Python code itself. So for example, this periodic exporting metric reader, where we set this export interval to be 1,000 milliseconds, this is also settable via an environment variable that will, I think, override this static value so that you can actually change the interval of this on the fly as you're deploying your application. So yeah, I would maybe hew to being extravagant with your application instrumentation and then letting DevOps be conservative in the YAMLs as they're deploying it to Docker or Kubernetes or whatever.

Speaker 2 [73:29]

Yeah, I suppose you can also just like do that temporarily if there's something that you're trying to like debug, like add tracing in temporarily and then scale it back once you've found the cause of your issue or been able to identify it and resolve it.

Speaker 1 [73:29]

Yeah. Do you think we should...

Speaker 2 [73:53]

Yeah, I suppose so. Okay. So, yeah, the last section is distributed tracing. So distributed tracing is a method of tracking a request as it flows through a distributed system, making its way through services, interacting with things like databases, caches, queues. And it gives insight into that request by capturing timing information and contextual info. It also provides a wide view of the system, so while logs tend to provide information about a specific event and metrics provide more of an aggregated view, distributed tracing shows you more of a wider view of the system and how the components within it are connected. OpenTelemetry makes distributed tracing much easier by providing standardized APIs and SDKs that automatically capture spans and tie these together to form a trace using their trace ideas. And then traces are exported using the OpenTelemetry protocol to the back ends. And distributed tracing is quite a powerful tool for understanding systems and their behavior and being able to drill down to issues as they appear. And once you're familiar with distributed tracing, you might find that you actually use other forms of instrumentation or rely on other forms of observability a little bit less such as logging because the traces tend to have a lot of the similar kind of information but also has this additional context of things around it. And yes, as Mika mentioned, there's also the concept of sampling in tracing, because tracing can be so expensive since you are capturing a lot of high cardinality data, and you don't always need 100% of that data. It is kind of common to sample and only, I suppose, collect a percentage of it. We won't really get into that today with this example, but you can configure sampling so that you don't lose important information such as requests that have errors or have a high latency or just have like specific important attributes

Speaker 1 [76:21]

Yeah, so like Emily mentioned, the key with distributed tracing is essentially the propagation of the trace context across service boundaries. And the way that you could do it manually, if you wanted to, would be to get the trace ID of whatever span you're currently in. And then, if you're making a request to a service, put the trace ID in the header of the HTTP request, and then the downstream service can then extract the trace ID from the HTTP request and then initiate a span with that trace ID, and then all the traces are correlated to the trace ID and have their own kind of like spans, and then you can see over time how the spans compose this one long trace. However, this is a real pain. It is not advisable to do manually, which is where something called auto instrumentation comes in. So auto instrumentation is a tool that OpenTelemetry provides for different libraries, which essentially wrap their existing functionality in OpenTelemetry. So it instrumentizes it for you. So if we go back to the requirements.txt of the gateway, we can see we're importing these two libraries, OpenTelemetryInstrumentationFlask and OpenTelemetryInstrumentationRequest. And what these two libraries do is essentially instrument flask and instrument requests for free. So instead of writing these spans yourself, you could use the flask or the instrumentation library, or to instrument your Flask application, and then you get the traces. You don't have to define the spans. Each API call is its own span. Similarly with requests, you can instrument the requests library, and it will then add the trace ID and all the other context that it needs to propagate to the request yourself. So you don't need to bother with manually handling stuff from HTTP headers. So, for a quick example of this in the ISS Distance Service, going to import something again, but massively only one thing. So, yeah, so from OpenTelemetry.Instrumentation.Flask, we're going to import the Flask Instrumentor. And when we're defining our Flask application here, like we're just creating a Flask app, we can just do flask instrumenter dot instrument app, and then we pass in the application, and then we're kind of done. Like, all this manual instrumentation that we've done has been for specific things we're interested in, but we could have just removed all this, just added the flask instrumenter, and then we would have spans on our API routes for free. It might handle logs if you're using like Unicorn, maybe, I'm not sure, but then also it adds a set of metrics as well to your Flask API endpoints, so like request latency and things like that. And with this in mind, we can then go to our other backend services, so like the geolocator service. We still need to import some stuff for tracing. So again, it's going to be a big block of imports. And of course, in a production environment, instead of doing this import, creating the resource for our application like this, and then setting up the tracing function like this, you would maybe abstract it into a common library that you could then give to developers that would do the setup for you. So you can kind of be quite minimal with it. But for now, we're going to do this manually, because it's just much more fun that way. So what we're doing here is we're setting up the tracing function that we defined previously in the ISS Distance Service, sets up the exporter, the processor, and all that stuff. We're creating a resource. We're going to call it gateway. Deployment environment, we're going to call it dev. We're going to set up the tracing with that resource, and then we're going to get the trace provider. And then finally, we can set up the auto-instrumentation. So here, we're going to do flask-instrumenter.instrument-app with the Flask application. But with the requests one, we don't have anything to pass in. We just call request-instrumenter.instrument, and then it instruments requests. Like, it wraps every request that you make in OpenTelemetry instrumentation. So for example, then down here in the API route, we could maybe do something like this. We would do like span is equal to get current span. And we could, let's see, like, yeah, we could, in the case that no coordinates are found for the location, we can maybe set the span's status to an error, and maybe also here as well, like if no location's been given, we can also just set this to an error. And then we can also, yeah, do the same thing, wait, are we even using requests in this? No, we're not actually. Oh, I got them mixed up, sorry. So we're not actually using requests in this particular service, so we can just ditch the request instrumenter, sorry. But we are using it in the backend, which is essentially the thing that's coordinating the responses from the two even more micro services.

Speaker 2 [82:36]

Yes, I guess so.

Speaker 1 [82:38]

It's just looking out for me. So yeah, so in the gateway service, again, we're going to import all the stuff that we need for telemetry for the tracing, the flask instrumenter, and also the request instrumenter. We're going to set up the resource called gateway. I guess in the geolocator service, I called it gateway as well. Sorry. Sorry, back into the gateway service, we're going to set up our resource, call it gateway, we're again going to set up the tracing setup encapsulation, create, set up the tracing, the tracer, and then once we've created our Flask application, we're going to instrument our Flask application and then implement the requests library. So when we make this request.get function down here, it will already propagate the trace context for us, and the trace will already be in a trace because the Flask instrumenter has wrapped this app.root in a span for us. So we can also do something on, yeah. So with all this in mind and making sure things are running, which I believe they are, cool, we can make another request. So let's say we want to do London. If we go back to Grafana and go back to this explore dashboard. We have our tracers. Let's do a search. And now we can see in our tracers we're no longer getting ISS distance service, we're getting gateway. And the name of the trace is get slash because that comes from the Flask order instrumenter. And hopefully if we open this trace, we can see the course a request has taken throughout our entire distributed system. So we can see that a request has come in on get. The whole thing takes 430 milliseconds. A get is then made to the geolocator service, which we can then see. We can have a look at some span attributes for this particular get. HTTP server, da-da-da-da-da. The resource attributes associated with the geolocator service. This takes 1.34 milliseconds, I guess because we hard-coded the coordinates, but it would take a little longer. And then The gateway, the trace context then returns to the gateway, a get request is made to the ISS distance service, which then enters the span that we created before, but because we used Flask-Older instrumentation, we have kind of like two spans that represent the same unit of work, basically, and then finally getting ISS coordinates. So if we look at this kind of, the full trace of our system, we can see that when a query is slow, the bulk of the time is in getting the ISS coordinates, specifically from the ISS coordinates API. So armed with this information, we could then be like, okay, we need to add some light caching here. We don't need to call out to the ISS every single time to get the coordinates. We could maybe have a one-minute cache or a 30-second cache or something like that in a high-request environment, let's say. So with a distributed trace, you can see the flow of a request through all of our systems. So we have five minutes left in the workshop. It was a lot of stuff to get through. Thank you for holding on with us. So just to quickly wrap up, yeah, that was OpenTelemetry. It's a vendor agnostic framework for expressing and exporting telemetry signals, particularly metrics, logs, and traces. We saw those signals and how we use them, how we instantiate them, and how we might like use them within our application to discover insights about our application's behavior. We saw how to export, collect, store, and visualize the signals in a set of open source telemetry backends, and we also saw how auto instrumentation lets you get a lot of the OpenTelemetry goodness for free with popular libraries and frameworks. And finally at the end we saw how to use those signals to examine the behavior of a request across a distributed system's service boundaries. So yeah, so what next? Hopefully if you have found this interesting and want to explore OpenTelemetry a bit more, we can really recommend having a look at the documentation. There's lots of very high-level documentation at the API layer where they discuss very, very high-level details of what makes a metric a metric, what is a log, how is this handled. There's also lots of stuff there that we didn't have time to explore here, like baggage, which essentially is where you bundle attributes for a particular request in with the metrics, so maybe like a customer ID or something like that. as well as, like, exemplars, which let you coordinate metrics with a particular trace ID. So you could get a trace, have a look at it, and also see the metrics for that particular trace. You could maybe try instrumenting or order instrumenting an existing Python application. There's probably, if you're using, like, a framework like FastAPI or Flask or things like this, there's probably an order instrument out there you could just instantiate and see what happens. You could maybe try swapping out one of the telemetry backends for something, like replacing Tempo with Jaeger, and you'll see how it requires no application code changes, just a configuration change in another service, which you can let the DevOps team do, you don't need to worry about it. And then finally, yeah, maybe explore the SDKs for other languages, the SDKs for Go, Java, Ruby, Rust, and a few others. They're all in various states of stability, so it might be worth checking out how stable the one you're using is. the Rust one is not stable yet, but I think it's getting there. And yeah, thank you for joining.