Monitoring infrastructure and application using Django, Sensu and Celery.

Speaker 1 [00:02]

welcome everyone and thanks for showing interest in this topic so as introduced my name is Harikisha Srivela I work as a senior site reliability engineer or DevOps engineer at OpenTable Inc so I don't know if you guys have heard about OpenTable it's predominantly a US based company trying to set its base in Europe we do have some restaurants tied up with Berlin so what our mission is is to experience the world via dining. So we are an online restaurant table reservation company where we provide inventory management and guest center for different restaurants. So today my topic is monitoring apps and infrastructures using Python tools like Django and Celery. So before we get into the talk I would like to ask a question to the audience. How many of you have kids that's okay I don't know why I have picked up that analogy I'm not married and I'm single and I'm open to mingle but I can't advertise myself or market myself because I'm being monitored by the PICONDE code of conduct and at the same time when I ask you questions that if you have kids do you monitor them I think so most of you do or let me refresh it to uh up his wider audience do you monitor your girlfriends or boyfriends phones or personal lives so so the reason why i'm asking you is that we always monitor things and at the same time we are being monitored by the society governments and every stuff so why do we do the monitoring so monitoring is something that you do on your maybe allowed ones or whatever other things to make sure that it's performing as it is supposed to be so you are making sure that it's on the right path it's not deviating from the path that you intend to go and at the same time to have an early warning before it's too damaged like people say prevention is better than cure so that's the reason we do monitoring so let me come back to the analogy of parents monitoring kits and stuff so so when i write a code i see it as my kid so and this kid is going to create billions of dollars revenue and it's going to be a livelihood for many other customers and at the same time it's reliability and at the same time a reputation for your company so it's very essential for us to monitor our own apps and infrastructure to make sure that it's crucial it's on track it's not deviating from what you want to it's working as expected and so that even if there are any issues we would be indicated by monitoring so that we can take action before further damage is being done to your applications so that's all the talk is about and i'm going to give you the agenda is that we do have an existing platform set up using monitoring how we are doing the monitoring but there is a problem with it so we try to solve that problem using Python tools like Django salary keeping sensor intact so before I go into the Python tools how we used to used them for monitoring I want to give a short introduction to monitoring that's what I was doing till now and I'll give it short introduction about how the the existing setup was at our organization and what are the tools that we are using for monitoring. So people who are familiar with DevOps or site reliability engineer, Puppet, Ansible, Chef, these are all the configuration management and I want to give an introduction to this because Puppet plays a very major role in our existing monitoring setup. So Puppet is a configuration management tool where we get to set up the infrastructures we get to code the infrastructures it's a configuration management tool like the Wikipedia definition says it's for establishing and maintaining consistency of a product's performance functional and physical attributes with its requirements design and operational information threats limits blah blah lights it's it's what Wikipedia gives it so so what it basically does is that like you want to make sure that your servers are having a consistent configuration so that it runs perfectly in production and there are no deviations from your testing environment that you have done it in your local missions or whatsoever so it just basically make sure that it's consistent across your organization so for examples it's like there are other alternatives for Puppet those are Ansible, Chef, Solstack and stuff so Puppet is a pull mechanism tool i want to talk a bit more about here because that's where the problem of the existing setup comes in so puppet when i say it's a pull mechanism tool it's a client server architecture so puppet has a server where you send all your manifest manifest is nothing but a set of code that's written to perform some installations of files that you want to see it on a server so this is a called a manifest and server has it and the clients basically have puppet agent running on them and they pull this manifest every 5 or 10 minutes as you like it like you can set up the cron schedule for it and the client basically pulls the manifest every 5 or 10 minutes from puppet server and runs it on the agent or the clients and your server would be configured to that configuration so this is a sample code i hope you guys see it so i just will go through the puppet code how it's written so the keywords exec package service file these are all the resources that puppet provides so let's let's go through the exec is apt update so the command we are running on this server is that you just apt get update you're doing the apt get update and i'm making sure that apache 2 package is being installed that's what ensure does it's just making sure that the server has apache 2 installed on it so before installing it's just making sure that apt update is being required or it's being done so that the same applies for mysql server php5 and the php file configuration so puppet when it comes to puppet code puppet follows its own code like you see unlike ansible and stuff where ansible uses yml which everyone is familiar with so here when it comes to puppet code there is a bit of learning curve so puppet basically will be dealt with most of site reliability engineers or devop engineers and basically developers might not be aware of the puppet code but the thing is developers are gonna write checks for their applications right this monitoring checks that they want to perform on their applications to see if their application is working fine or not so they have to write it in the puppet which is a painful thing for them so I'm going to talk about sensor checks a bit more moving further so sensor is what the monitoring tool that we are using right now and we are going to keep it intact without peeling it off when we are just using Python tools upon the sensor monitoring tool so that we can get the whole thing and remove the puppet in the picture so as per sensors definition sensor is a telemetry and service cell checking solution for multicloud monitoring scale like I said whatever the monitoring that I have mentioned in the earlier that's what sensor does it runs some checks on the applications or on the socket or on the even some measurements that it collects and it sends to graphite and stuff so you can be aware of how your application is being performed so even sensu has the client server architecture I'll go through the diagram quickly so it follows it has a RabbitMQ message bus so the server and client never communicates they only communicate with the message bus though so server communicates to the data store and the data store here is the Redis that we are using and using data store makes us a stateless application so basically the checks that are written and are managed via puppet are stored in the Redis and server talks to the Redis and get the checks and send it to the RabbitMQ and clients subscribe to these checks if they want to run these checks so when the client run these checks it sends the results back to the RabbitMQ message bus and server basically grabs that results and put it in Redis and APA can leverage those results from datastore and you can build dashboards and everything out of the data that we app from the redis data store so this is a quick architecture diagram that we are following right now so i want to i want to i have given you this info because i'm going to tell you what the problem is or how we are doing it and how we are leveraging uh census server api to do the monitoring okay so let's come to sensor checks so this is what i was talking about the checks these are the crucial thing for monitoring your applications so checks are nothing but the commands that are run on your clients or on your applications or even on your graphite data so that we get to know idea of how it's performing like if there are more number of 500s that's coming out of your application you would be triggered an alarm saying that hey there are like more than 500 or thousand 500s your application is throwing on so there is some really problem with your application so you have to get onto your application and look at what's happening so when a sensor check is check or commands are executed it basically the output would be std out and std error and the output basically the exit status code will have 0 1 2 or unknown 0 means which is the check is working fine you are fine with the check the application or whatever you are monitoring is working as expected there are no anomalies everything is going good and 1 indicates warning which means there is a problem you might need to attend or you may not it depends on you and you know your application better so and this warning can be set by you it's not something that sensor does the warning threshold and critical threshold are set by you when you are writing the check definition so you are aware of how much it has to be and how much it should not be so two indicates critical which is a critical thing for your application you have to get onto the laptop and look at what your application problem is and try to fix it and other than 012 there is an unknown status which is there might be a problem running in the check or there might be a connection issue whatever it could be or even if you want you can make it as a customized message for that status so this is a brief introduction of checks and i want to talk a bit about nagios plugin so Nagios is an alternative monitoring tool for Censu but fortunately the plugins that are written in Nagios are compatible with Censu so there are lots of plugins that's already written like there are checking and HTTP check or be it checks that are working on your graphite data or lots of mathematical calculation plugins everything are available in Nagios already so we are not trying to reinvent the wheel we are just trying to use whatever is applicable and make sure that application is running so here is a sample check HTTP Nagios plugin this is how a check basically looks like so here what the check is doing is that so like I said we can set up the warnings and criticals these are set by the developers itself so here I am setting the warning threshold as 5 hyphens is the critical threshold is 10 and you are asking the check to connect to host open table via SSL which is 443 port and if it returns a status code within 5 seconds your application is running fine your website is running fine so if it goes more than 5 seconds it takes more than 5 seconds to give a response code then your website is responding slow so you have to get on to see why it's taking 10 and critical if it's more than 10 seconds so what are sensor events once these checks are run we get outputs which have the status code 0 1 2 so depending on the output events would be generated basically when there is a warning or a critical or an unknown status since who creates an event so base when an event is created there are Concept called handlers so you can write your own handlers or since it has a default Handler which handles these events. So what basically handlers does is that whenever there is an event triggered Based on the handler that you have written it performs that action. So say suppose you can remediate an automated You can make an automated action like say suppose if a disk is full and you know that these are the files which always fills up your disk and you get an alert saying that hey your disk is full like out of 100 GB your disk is already full up to 90 GB and you know the cost already so you can write an automated action saying that just clean up the disk and sensu handler make it as an handler and sensu whenever there is a alert triggered based on this check it's gonna automatically delete the files from that so that the check will be resolved yeah so that's mostly about handlers this is a key part for our application because this is where we are going to use sensu handlers using salary and Django to run the whole monitoring system so this is the existing setup we maintain all the checks in puppet like most of the tech industry is doing we basically have all our checks in configuration management so that we know that every server gets these checks to be run like infrastructure checks like what's the CPU of system going on what's the memory of the system how is it spiking up so these are all being managed by puppet and there is a puppet learning curve like i said the code puppet has its own language so there is a slight learning curve for developers to learn it and write their own checks and puppet code is maintained by ops guys or site related to devops engineers guys so we review the code and basically developers has to wait for our review and let it merge into production so even the prod mergers take some days hours of time and sometimes even days of time that's not the case ideal case every time like there are if there is an holiday season where we basically dine two million diners on a single day and if a new service is added into the production and they want to write checks which is a painful aspect for us so and at the same time in the puppet code like i said puppet has its own language in older versions at least still puppet 3 which we were using at that time doesn't have any iteration function so whenever we need to write a similar kind of code we used to write the whole code again there is no reusability done and which was a painful thing we have we need to literally write the same code again and again for the same checks so and at the same time sensor does give us a dashboard of number of checks and the results which is called which is not that great to view and stuff so that's another painful thing so what did we do to resolve this problem and how did Python played a major role for solving this or at least Python tools like Django and cellrated so we came up a solution that puppet should be removed at least for managing and maintaining these checks it has to be moved to a newer system so that it can be added quickly by the developers if they want and who doesn't love UI so we are gonna make it as a UI one and At the same time we are gonna give them role based access control So every team can check their own checks they can delete edit and stuff They need not wait for the ops guys or SRS to review their code and stuff At the same time there would be no minimal learning curve for engineers because it's a UI and it's self-explanatory. There are all informations provided. You need not learn puppet code. So we are eliminating that learning curve. and you can create multiple checks at one go it is basically ui so as soon as you create the check you get an option saying that clone the check and it clones the check with a different name it's as simple as that i can't show you the ui or demo due to security reasons under the same time restriction of the time but i can give you an explanation like how it's going to be so this is another big aspect that we try to solve which is testing the check result instantly because like i said puppet has its own language and in order to test those checks that are written by the developers if they are working fine or not they have to set up puppet in their local machine in vagrant and run these checks to see if it's working or fine and no one does that literally we tested our own checks in production which is a very bad way to do so so what we did in this solution to the problem is that we have run the checks whenever they created we have run those checks in the back end and i've shown them output and when they're satisfied with the output it's being created and stored in the database that's how we know it's being tested way before it's going into production and we also created apis for creation reading updating and deleting these checks and we do have bots in our slack messaging like i have written a bot called airbot which basically leverages these apis even to create and stuff so you can automate this checks and everything so this is the solution we tried for solving the problem that we had previously so how did we solve this what is the tech stack we used to solve this so we didn't remove the sensu we keep we kept the sensu intact because sensu has this nagios plugins which is working very fine we don't want to interfere with that at the same time we don't want to rewrite all those plugins again from ourself so we basically kept sensu as it is we just removed the functionality of running checks from sensu and we gave that function to celery so celery is a distributed management system it's an asynchronous task or job queue running system which runs tasks for you so what we did is that we remove that function of running checks from sensu like probe notes and stuff and we have given it to salary and let's already run for itself so sense is being only used to handle the results of it and to use the existing check plugins that we are already using and here comes the main thing which is Django we used Django for the web application which is a UI for check management and storing the checks everywhere in the database and for creating visualize their dashboards everything so Django played a very key role for us and it was a learning curve even for our team since we never worked with Django and we were very happy to do that so this is a view of how our UI looks like this is a old one this is not how it looks right now like I said I can't show it to you so this is something old one so you get like home Check services team. So once you log in I get to see your own checks What services you own what teams you are in and what's their performance how it looks? So when it comes to checks, there are multiple different types of shares It's not like every check does the same functionality, right? So there are like three major types of checks that we were doing one is HTTP or HTTPS checks basically the example that i've gave it just pings a host like www.opentable.com to see if it's up and running if it's responding to the ping and mesos is apache mesos if you know it's an orchestration tool which we are using for running all our tasks in productions so we are just we are wrote checks to check if the services are running fine in mesos if they are responding fine in mesos and graphite checks are the ones like i said sensu can do and have this plugins which basically collects all the measurements collects all the data and sends it to graphite and once you have this data in graphite you can run checks on this data so basically if say in our scenario we do have a reservation check basically like we know that every weekend we do have we do get some uh hundred thousand reservations uh on a weekend and if we see we can write a check on that to see that if it's hitting 100 000 if it's not going there is there a problem with the application that we are dealing with is there a problem with the booking flow that's happening so basically you can run checks on the graphite uh so the best example is that you know your application sometimes give five hundreds and stuff it could be because of multiple reasons but like say one in 10 500 is not a big thing but there if there is like 500 in a thousand number of 500s then it's a painful thing so you can run graphite checks on that to see how many 500s are being generated and if it's a crossing a threshold it you should be warned for it so that's the types of checks and so this is the ability to test and check before committing the check into production that's what we try to solve using Django and stuff you can ability to clone check and edit and use bots by leveraging APS so here is another big chunk of performance by salary with respect to running the checks so like i already mentioned salary is an asynchronous task or queue job queue distributed system so it's used for running these checks as tasks and what happened is that like i said we have different types of checks like http mesos and graphite checks so what we did is we created different queues one is for http one is for mesos and one is for graphite cube so what happens is that salary has a concept called salary beat which can be used to schedule this task like that it could be a crontab schedule or any type of schedule if you want it could be a customizable schedule everything so what this beat configuration does is that say suppose I want to run all the HTTP checks every one minute to see if my website is up and running and it's responding properly so salary i i have configured that it in salary bit and it's gonna run the task and the beat will be looking upon these queues so as i mentioned that i want to run these http checks every one minute so all the checks all the http checks are going to be into the http queue and beat is gonna run the http queue every one minute as per the schedule and salary workers run this task based on this beat schedule so salary has its own cluster where it has workers running for running for this task to run and it can be scalable easily so i'll give a an idea of how salary beat schedule looks like so this is the beat schedule that i was talking about so if i say http check is my beat schedule it's a schedule one and the task is written in task.py in django so which is my function which is master http which has all the http checks in our organization so i have given a schedule like run every one minute this master underscore http function which basically collects all http checks in the database and run it every one minute once the same applicable for Mesos check yeah so celery flower and this is a picture from internet it's not from our application so flower is a very good tool that celery provides to again monitor your own celery cluster so there's lots of monitoring going on so you're more you are using celery for monitoring and then your celery cluster is being monitored by a tool called flower and it gives very good details like what's the worker doing so the name salary1.py.local these are all the workers which are scalable and how many tasks are being running how many completed tasks what's the concurrency of it and what are the queues running on those tasks and everything so sensor results api so like i said we have removed puppet from our existing setup we have introduced django we have introduced salary but we kept sensu intact like i was mentioning but we like i also mentioned we removed the function of running checks from sensu and we are leveraging only sensu results api so what happens here is that when salary runs these checks it's going to create a output for you and what we did is we send this output to sensu results api so that sensu has all these handlers and based on this results events are created and sensu is going to deal with handling these events based on whatever handler it is being assigned to so we are not going to remove that functionality because it's working fine and there are existing plugins and it's it's very good we don't want to interfere in that so it it celerity runs the task sends the output to Sensu API and Sensu API depending on the output creates events and if there are events it's going to be handled like paging you or sending an email or pinging you in slack saying that your check is being failed you have to get on and see what's happening so this is a sample code that we have written to send output so if you see the command it's a message check that we are trying to run um so discovery is a service that uh we developed in those to know which services are up and running so that they can announce themselves each other saying that hey if there are like a and b services dependent on it hey service will announce i am up and running and b service also announce i am up and running so they can communicate each other so that's a in those application that we have developed which is called discovery and fns is name and the end point that you are looking for the timeout that we're looking for and what is the one threshold critical threshold and stuff and once it has been run we are using normal sub process command which we get the output of the out of it and we are sending this output to the send super arms what is the name of the check the output the status the handlers that the event that check was assigned to if there is a event based on the output of that check how should it sense handle it and what's the team that it has to uh send inform or does it need to be paged to that team if the check is very critical do you need to inform the team then you make sure it's true then notify email and stuff and here we are sending it to url request.post so how did we deploy this we deployed it in docker containers because it's everyone uses containers and it's very easy to scale and easy to monitor and stuff so we opted for two different docker files one for web app that is Django based one and another for celery the reason we did that is that if web app goes down we don't want checks not to be running check should be running always because that's very critical for our organization and for our application because that's how you will be informed what's wrong with the system so we separated two separately in the docker files so the docker file with salary will be having the salary workers running based on the beat schedule that I have mentioned previously and web app would be running for UI so that people can add checks and tell it or do whatever or visualize the checks based on the results and it's running in Apache mesos environment yeah that's that's pretty much about it that's how we solved using Django and salary keeping sense intact and that's how we monitor our applications and infra thank you guys

Speaker 2 [30:56]

So, a quick note from my side for those who popped in during the talk. Apologies that I started the talk ahead of schedule. I realized while Harry was already talking. So, sorry for that, but we do have more time for questions now. So, there is one.

Speaker 1 [31:22]

Hi, thank you for talking. My question is...

Speaker 2 [31:25]

Have you considered promise?

Speaker 1 [31:26]

prometheus for solving this issue we did but prometheus is more of the way we pursue prometheus is more of an alternative to graphite more into visualizing stuff and data but there are some checks like i was mentioning like apache mesos checks or beat http checks and all this has to be run and maintained somewhere and we thought it's better to have it a separate way and not too much with the data stuff that we are having already uh so and we did even face some bottlenecks with prometheus uh to be honest uh but i haven't dealt with it personally so i can't comment more on it uh but we did see some bottlenecks with prometheus

Speaker 2 [32:15]

Yeah, so I have no idea what your company does, actually, because I missed the start of the talk. But it seems to me that you guys invested a lot of time in creating this. You created a whole monitoring tool and the UI and everything else. Wouldn't it have been easier to just go with a solution that already exists?

Speaker 1 [32:31]

exists.

Speaker 2 [32:32]

Like just learn it and implement it and be done.

Speaker 1 [32:36]

Yeah, so are you talking about any paid tools?

Speaker 2 [32:41]

Possibly paid, possibly free.

Speaker 1 [32:42]

possibly free yeah so we are more into open source we always wanted to contribute it's obviously we can save money too and at the same time we did not depend on someone else so that's our main goal we have to solve our own issues and like i said we did have an existing setup like sensu and puppet it's widely used if you see their repo like thousands of organizations using it and it's the same format but like i said there were lots of pain points with that like developers have to wait for it and they there is a learning curve for it there are no iterations with the puppet version that we are running so it was a very painful thing it was going well for two through two to three years and then everyone started shouting at organization no we can't do this like i said during holiday times like christmas or new years our site goes up like anything everyone wants to book reservations at a restaurant and if there is a new service that we want to add in a production just adding checks will take lots of time and because it's a holiday time people would be on leave there would be no one to review the code there are lots of bottlenecks so we thought let's do it a new one because it's going to stay forever monitoring is not going anywhere so we will be monitoring as long as there are products in the production environment so we thought of going the whole way

Speaker 2 [34:00]

Thank you.

Speaker 1 [34:01]

Welcome.

Speaker 2 [34:10]

I have a question. How does your monitoring system support calculating how long your uptime of the system is? So if you are 90% or 99% or 99.9%?

Speaker 1 [34:25]

So so we do have run all these checks and send this data to graphite Like I said say suppose check HTTP and stuff this run or this run checks and based on these outputs are sent to the graphite and You can get the data out of graphite. So we are basically calculating that in graphite and grafana to

Speaker 2 [34:46]

Okay, thank you. Other question? I have yet another one. Sure. If I were to monitor a web application that is several orders of magnitude simpler than yours, like where I don't depend on the thing being 99.99% up at Christmas, for instance, what would you recommend to a beginner? Where would you start if I were to build my first monitoring system?

Speaker 1 [35:23]

that's great so that's a good question actually so i i would say like since we are used to sensu uh even nagios is good nagios is very easy to install and compared to sensu and even sensu is coming up with great um features in their newest product which is sensu go uh so i would say as a beginner into monitoring i would say nagios is the best way to go because there are lots of plugins already available and it's easy to install and you can run it on your local machine very easily without high consumption of resources and stuff yeah nagios would be your best

Speaker 2 [36:02]

Okay. If there's no more questions, then let's thank the speaker again.