Observe all your applications

Speaker 1 [00:03]

Thank you very much. So, hi, I'm Christoph. I'm working with an SAP, and this talk is about how we are doing observability, monitoring, yeah, basically find out, does everything work well in our infrastructure? And therefore, I would like to give some background what my day-to-day work is, actually. What I'm actually doing is quality insurance. So most of the time you would now think, okay, quality assurance, what is that actually about? Is this software related? And we are actually doing quality assurance for one of the major SAP products since multiple years. It's SAP HANA, which is an in-memory database. You can also connect with Python to the database, as you will probably learn in the next talk. And what we are doing is we basically fully automate the quality process to ensure the highest quality for our customers. How we do that? We basically run a lot of automated tests. We have over 15 million lines of Python code in our source repository just for automated testing of our database. And we are running that on an infrastructure which is basically powered by a huge number of physical hardware. So we are currently using more than 1,500 nodes. And overall, we have something like 560 terabytes of memory over the full infrastructure. So that's already a bit you can play with. We basically have various sizes of nodes, different configurations. we start at something like 256 gigabyte of memory and then we go up to 8 terabyte of memory in a single server so just you have an idea about what dimensions we are currently talking about beside that beside keeping all this infrastructure up and ready for our developers we are developing special tools for the various developers who are working on sap hana all over the globe basically around the clock because we have in so many locations developers. So, and the first thing what you actually learn in such an infrastructure is that Murphy Laws is totally true. Anything that you can imagine which could go wrong will also go wrong. And we see that basically every day that something goes wrong and you can do as much as possible, you will always have some kind of a problem. but actually that's totally fine as long you have a strategy how you can actually work against that and this strategy basically contains three elements the first thing is you have to identify that something is wrong and that's basically the point how you actually identified it so is a user coming into office and is complaining my application you normally provide to us is not working anymore could you please take a look or does 500 mails coming in application something is not working why are my tests not running my build is currently not running what's going on and you have actually also to find out okay what is the impact is it just a single user who is now in the office or does it actually affect all developers around the globe and they are currently blocked from actually doing their work done so now you have identified the problem now the main question is how you can actually analyze the problem because if you cannot analyze it you can also not fix it so if you then have access to my to enough data to actually analyze the problem then you can go and actually start with the problem solving process so you can start building and fix what a problem and the best case if you actually follow this process of analyzing you can do that again and can prove that your fix which you created also works which is a very powerful technique because otherwise you maybe push out something and then you even make the problem bigger as before so what is actually observability so many people are actually saying observability is a buzzword for monitoring I hope I can change this a bit but let's take a look on a quite simple example and basically how our infrastructure works a bit we have a user and this user is interacting with some kind of system for example our infrastructure for the user itself the system is basically a black box I mean the user is pushing code to the code repository using our tools but the user doesn't know anything what's going on in the back end normally and the problem is if you don't instrument your tools if you don't instrument your infrastructure also for you as the developer of this infrastructure, the infrastructure behaves like a black box. You see actually nothing. And now, if you would like to change that, would like to know more about your infrastructure so that you can actually also improve it, you would like to give some light inside of the system. And therefore, you can apply various techniques. For example, we can apply logging so that you can have actually the log output of your application so the application is telling you what I'm doing at the moment and so on. And if you then put it in some kind of observability tool, then the developer can actually analyze the log messages, for example, can take a look on them. And with that, you're getting a better insight in the application itself, and then the developer can actually improve that. And if you now put more and more techniques, for example, error reporting, metrics collection, or distributed tracing to that system, you will get a much better understanding how your system actually behaves under a production environment. So, therefore, I would actually like to start with the first, but probably most important, and probably also most obvious observability tool, which is logging itself. And the first thing we definitely learned in the past years is log messages are for humans, especially for humans who actually doesn't know the source code so in detail as a normal developer knows it. Which means you should actually think a bit how you're writing log messages. Be a bit precise about what the application is currently doing and how it actually currently behaves. And that's actually a hard problem. I mean, if you can take a look in our productive coding itself, you will find some bad examples. I have just two of them right now. So, for example, we have acquire log. Okay, interesting. But a much better approach would be to say, okay, blocking attempt to acquire log, and you also have a name for that log. And what you now know is, okay, this thread, who just print out this log message, will probably know block. and I know which log the thread is currently waiting for and I could, for example, now go up in the log and find out, okay, which other thread currently holds that log, for example. Another thing which is also quite obvious, but if you think of it's very powerful, is think about log messages which containing numbers, especially large numbers. There we have some local caching mechanism and the log message is quite straightforward written. I wrote it myself, so I can say that's actually quite a stupid log message because can you now see if the cache size is actually below the target size? I cannot actually see that, and especially I cannot see that in a big log file with hundreds or millions of lines. Much better and much easier if I can actually write it down with nice units. And if I just add small things like you can actually configure the target size of a cache, these are some insights which are very valuable for an operator or for the developer who operates your application at that moment so that the person knows, ah, I could actually change that. That could be beneficial to work around my problem at the moment. The next thing which probably not many people actually think of is how you format your logs. I mean, if you just start with Python and using the Python logging framework, this is actually a great start because the Python logging framework is actually a quite good logging framework and especially it's in the standard library, so no extra dependencies at all. My personal main problem with the logging library itself, the default formatting is not so valuable. I mean, yes, you see now the log level of the message, you see the logger name, so you probably know where in your code this message occurred, but that's it. And to be honest, we could also just print it out. It doesn't make so much sense. And actually, a good log format is actually quite a personal thing. So please don't take now the following example as the best formatting ever because it highly depends on your use case. If you don't use multiprocessing, then it's probably not worth to include the process ID or something. One big learning we took is if you are working in an international team, please include the time zone information. It is so valuable if you know in which time zone this message actually occurred. another thing which is at first think maybe not that easy to implement but actually it's very easy to implement is the way how you look at logs i mean you cannot look on this wall of text and probably you won't see anything and i could now say yes there's an error included do you see the error probably not but I'm pretty sure now you can see the error quite easily still the resolution is not perfect and it's still a wall of text by at least you see okay there's one line red this is probably not a good sign and also the yellow lines included which are probably warnings so that's also not a good sign okay these are the simple and easy things to make actually your lock very powerful in a local development environment let's talk a bit how we can actually improve that in a productive environment so as I said we are actually operating a huge distributed system over a thousand of nodes that means we have also distributed locks and that's main a main problem of SSH and crap these are powerful tools there's a very valuable tools for a single server totally fine for two servers yes okay but if you ever try to open an SSH connection to more than 100 servers you know that is not working well and especially if you then try to grab across a log file and so on that also doesn't look so good the typical answer for that is using some centralized logging collection logging aggregation tools there are various out there personally I cannot really recommend something because you will see that in the next slide but the common architecture for that is that your application is writing into a log file and then you have some kind of a processor which reads the log messages from that file process that and then store it in a structured way in a database and provides the developer the possibility to query that data store that database over a nice web interface that's a huge benefit because you have all logs on one single place and you can aggregate and you can run queries on them it's very powerful the main problem about that is that the processor needs to understand the formatted log and that means actually you have to write some regex magic thing that you can actually pass your log file and then you are losing a lot of benefit because log formats are often quite complex and even this complex regular expression doesn't support multi-line and so on it's it's not that easy actually and therefore i'm actually a big believer that formatted log files are actually for humans so please don't feed your machine with a log file especially not if it's intended for human so um our approach for that is actually to build an in-house solution for that um that was also some years ago, at which point in time, things like Elasticsearch and Kibana and so on was not that big thing. And we're actually doing the approach like, okay, let's integrate our system as best as possible in the application logging framework itself, because that provides a lot of benefit. So we are integrating our central logging place just as a handler into the logging framework and we are receiving from the logging framework the raw log record data this is basically the log message with all the metadata around timestamp information with timestamp information as obvious all the messages the arguments for the message the line where the message was emitted and so on and so on and what we are doing we are actually taking that data putting them in a single event data structure and put it out and store them then in our favorite own database. Therefore, I don't recommend that if you don't sell a database, probably don't build that by yourself. We are actually doing that also as a showcase for our own database. And it works very well, actually. And how does it actually look like is we have some kind of web interface. We can query for that. And as you can see in the red box, we actually see various logs from the various hosts in our infrastructure. That's actually quite easy and actually quite powerful to get an insight about your applications, which are probably spread over multiple hosts. So what we can also use is to take a deeper look in this, as I mentioned, in this log record. So we can actually take a look into all the metadata. And you can see that such a log record actually includes a lot of data. And especially in an error case, as we can see it right now, it also includes a traceback. So you have some information about a line and so on. And that's already quite powerful. Which means we have now a system, we have now a central logging system which unveils exceptions. And that's actually totally fine. And very valuable, because now we know, okay, there's a problem. But now the problem is, is this enough data that you can actually fix the problem? I mean, do you, especially now, only this simple formatted log line, you actually don't know, okay, where was the data? And even then, you wouldn't see, okay, where is the exception coming from? You only see that at this point, you handled this exception and logged out, yeah there was an error but I don't know anything else about that and basically if you are operating such as a big system as we are currently doing you actually want to know as much as possible about errors because the main problem about errors is that they are often rare and you often don't see them multiple times so if they happen you would like to have as much information as possible And our solution, and I'm actually quite happy that this is something you can easily use, is Sentry. Because Sentry is an open source software, it's also a software as a service offering, which you can easily integrate into it. And as Sentry is written in Python itself, you have also very good quality how the integration works. And it's specialized for error cases, so there are a lot of thoughts how we can actually collect as much data as possible if such an error happens so that we can actually support the developer to fix the problem as best as possible. So, therefore, if we now take a look on the same error I just showed before, and you can still see the log message on the screen, in Sentry, then you will actually see a lot more details, and with that, we are actually quite able to identify the problem. I mean, we now see, okay, it's an attribute error. we have some none object and we try to access a not existing attribute okay and we also even have the snippet of the source code so we can easily see okay in that line there was the error okay there we access status code okay that means the response object is probably none okay perfect with that i can actually now go forward and create a test to reproduce the problem and then i can and also fix the problem. And also in Sentry, you have this idea you can mark an issue as resolved. In case it would recur again, you would also notice that. So we also proved that our solution is actually a suitable solution. The famous last thing about Sentry is actually that it also integrates very well in a logging infrastructure. So even if you actually cannot currently afford the investment in Sentry logging, sentry makes it quite easy because sentry will include as much logs as possible before the error occurs which is quite powerful if you try to understand okay how actually um um how you actually got into the situation of that error okay so we have so many errors that i can actually now show the next one which is an out of space error which means that we have a file system and the file system just run out of space that's a BGFS so that happens quite often and now we see okay at this single point in time there was a problem with that file system one major problem of sentry and all other event based tools is I don't see now the information okay is this quite common or what was the values before is this now a rare case or something and the one with that thinking of okay this is now an event at a single point in time we start about thinking how we would actually like to handle metrics and metrics collection and what we are actually doing is we are treating that error as a single event at a single point in time like we do with a log message and if we would now like to collect metrics we are just emitting multiple events over a certain interval and that also provides us the flexibility to put multiple data points inside of that events so that we have actually multiple timelines in parallel so what we never actually created is an own demon which is running on our servers today we would probably use something like collectee but also there were collective was not yet available so with that we basically emitting all over these events and we have an Python API which is actually quite nice to use and easy to use with was one of the main design goals so that also our developers are motivated to actually implement metric and event emission in the application itself so that we can actually emit certain data out of the application for example the queue length or something or the health itself this This is actually quite powerful. On the other side, that was also one disadvantage of building your own system. We focused a lot on how we actually can collect the data. How do we get the data out of the application? How do we store them? But what we learned is actually that if you don't visualize that, you actually don't have data. So you cannot see them. You don't get a real feeling about the data itself. And for that, we are also using existing technology. We are using Grafana for that. So that means we can easily create dashboards which are creating the data from our central database. And the nice thing is it actually enables developers to build their own dashboards. And that's actually a very powerful feature because in our team, mainly developers are responsible for one or multiple services. And they start to build their own dashboards for the services. And they know best what is a good value, Where should I set a threshold, for example? How should I configure an alerting rule and so on? That's a very powerful thing if you then enable the developer to actually develop their own dashboards. What is also quite nice is that you can actually use that to run ad hoc queries on your datas. So, for example, if you have now a problem, for example, a performance problem, you can now visualize them and you can also prove that your hotfix which you just applied also works therefore I have also an example I think it's pretty obvious where we deployed the hotfix over there and now everything is again in a much better space so there we have a processing time of something like 20 seconds and over there we had some crazy spikes going up to three minutes and so on that was not that nice so the last thing we are currently thinking of to improve our observability stack and one thing what you also can take a look in is distributed tracing so what is the idea about distributed tracing and why you actually should probably consider distributed tracing as you said as i said um we are now moving more and more into a distributed system landscape More and more people are using patterns like microservices and so on, deploying more and more applications on Kubernetes, for example, which means we don't have this simple, typical, monolithic application where you have one application log where you can take a look in and the one expensive statement log of your database where you can actually see, okay, what type of database query takes so long so that my application is actually affected, that you now have so many components that you can actually not create for everything a metric. And especially it's not that easy actually to combine all the things together. Distributed tracing has therefore a quite easy idea. So at a point your user is going into your system, for example, at a load balancer tier or at a front-end server or something, you assign this request a unique identifier. And then you just pass around this unique identifier through your full system and attach this unique identifier to each event which you're currently emitting. That can be log message, error message, metric, all that together, and especially measurements, how long certain operation takes. What we are currently doing and we are currently evaluating is open tracing. Open tracing is a nice library to instrument your code with distributed tracing primitives like, for example, this time spans, how long an operation takes. And then you can choose from various vendors which then provide the data store and also the visualization so you can actually take a look on the data. and even just with OpenTracing 2.0, which is a very nice tool because it makes the passing of context much easier and so on, even if you just use the OpenTracing to instrument your core parts, for example, the part in which you're calling out to your database or calling out to another component in your distributed system, this already helps a lot to understand where you are actually losing time in processing a request and the most important thing we learned from our experiments with distributed tracing is visualization is very important because as a human we are probably more trimmed to thinking more about how long a certain thing takes if you are looking on a bar chart for example than if you're just looking at row numbers. And even then, you can easily understand it quite easily what's going on. So, for example, that's one screenshot of our current open tracing experiments. You now see, okay, there's something going on. But even that there is data missing in the places over there and there, you already noticed, okay, something which takes a while is actually currently not covered by my instrumentation, which also already raises this question and also lets you think about, okay, I should probably invest some time, find out where are the spots, and you already know where you can actually start to optimize your performance. So let me end with a short conclusion. The first thing I would like to mention, as I said before, observability is not a buzzword for monitoring. It's more about to make our applications and your applications much easier to instrument and get an idea what is going on so that you emit more data, better data out of your system so that you can actually improve your monitoring. So that you can, for example, also if you have an incident, you can decide based on data, what should I do, what workaround should I apply. And also, if you have, for example, such system like a distributed tracing system in place, it also points out quite directly where you should actually invest time and effort to improve the performance over the overall application. One main thing we learned is that it's actually quite important that you include developers in such decisions. So if you're still splitting operation and development into two different teams, please include developers in these decisions which tools we should use because the developers have to instrument the applications and there are various quality levels how good the instrumentations are actually and how good the frameworks are working. So therefore, you should definitely include them and also allow developers to create dashboards because then you're also getting much better insights in your applications. and with that I would like to say thank you and if you are interested in some such space playing around with a lot of hardware with a lot of strange interesting scalability problems then you maybe would also like to check out these two job offerings we currently have thank you very much so we have time for one question anybody Okay. No questions. Thank you, Christophe.