Bonobo, Airflow and Grafana to visualize your business

Speaker 1 [00:04]

Thank you very much. Hi, everybody. I'm really happy to be here for a second time in Karlsruhe. I'm here to talk about how we built the business dashboard using a few different tools. Some I wrote and I write and contribute to, some I don't. But the idea is really to show how with really few code and few software, you can now, at least in 2018, build a really amazing thing without a lot of energy. So I'm Romain Dorgueil, I'm founder of a company called Maker Squad, which is a very recent company where we build software really quick with entrepreneurs. Our goal is to release software on the market the fastest possible, usually under one month, to allow them to learn from real people buying things instead of just having a plan which which will never come true because one year later when they release, they realized it wasn't working as expected. So we were trying to educate and work with entrepreneurs to build, release, operate software really quick. The content for today is, so first I will present the product I'm using to show all this demo. It's based on a real product we have, it's a product we built internally so it's not a client product then I will show how we planned this metrics we will measure how we we had the reflection about what we wanted to do and then we will have three section about technical implementation using first Bonobo for the ETL side then Grafana to visualize things to make things accessible to everybody and then how we run it in the cloud using Airflow to monitor the workloads we have. A few disclaimers, I'm the creator and main developer of Bonobo, so of course, and of course also for the other tools, all things I say here, I recommend nothing, I'm not saying you should use that, I just want to show what we did and if it's useful to you, feel free to use anything, it's here for that. So the product we built this on is a simple thing, really simple thing, which is an API that takes size and a URL, opens a browser, take a capture, and just save the image to the user. So people can directly use image tags with captures without having to prepare the captures before. have it yet, we will serve an in-progress image. If we have it, we just save it. It's not a new service, it's something we bought at the beginning of the year because the old owner wanted to close the service, so it's about 10 years old. Yeah, so nine years after, we changed everything. It was an old PHP site, hard to maintain, and we killed every bit of code and we relaunched it last April. It's now mostly Python. There is a few things not in Python, but mostly it's Python. What we did not expect about that is that we realized that the traffic on the existing service was around a bit less than one million images served per day, which was really not expected. In fact, the first release was not this, it was a 5.0 I don't know what, because we could not handle this traffic with the first version we released. But really quick we adapted things, so internally I won't detail everything, but internally we mostly have this API server which serves the image, users query the API server, says here I want the PyCon DE website, it looks If Redis says it exists, if it exists, it will either save it from local cache or download it from the object storage and return the image. If it doesn't exist, it will send a miss message to event message queue. This message will be picked up by a service called Janitor, written in AsyncIO, which will check if it's not an abuse, if it's not someone trying to do a bad thing, if it's not spamming a domain, if it's not... that there is a lot of checks here. If it decides it's legit, it will send a crawl message to another message queue. A spider will launch a Chrome browser somewhere, load the page, maybe without some ads and some tracking pixels and things like this, make a capture, store the image to the object storage and says to the event message queue, yeah, I created the image. And then the janitor will pick it again and update both the database and both Redis. So next time the user will come get the image well it just query the ready server and downloads it and saves it and as of today we can save image on average around 100 milliseconds per query sometimes it's better sometimes it's worse but average is that there is a lot of services behind also that generates a lot of data without going into many details. We have the web server, we have the database, we have also external services like Google Analytics, Stripe, Slack, etc. And all those services generate a lot of data. So the goal today is to pick whatever we need in those different data producers to quickly build something that we can use to change how we manage the business on day-to-day. So, before we start, we already had a Grafana instance at the beginning. You could see it in this slide before. But before we start, there is only technical metrics here. We just got the CPU memory, maybe from RabbitMQ, we got some information about the number of messages passing through the queues. That won't help on the business side. that will help for incidents, that will help to investigate whenever something bad happens. But there is a lot of data and it's really useless to know how to grow the service or how to do something on the market or business side. But that's already an infrastructure we have, really quick nowadays to set it up with containers and things like this. One thing to note is that we only keep this data for 15 days because we consider that after 15 days, if we did not need that, it's not useful. On the business side, it's different. We want to keep data the longest possible, maybe not every second because we don't care about the conversion rate at a given second, but we want to keep the data for a long time because we want to compare month on month or year on year. So it's a different kind of data scale or data resolution, maybe. So what was our plan when we started this? Basically it's summarized by this. We wanted to be able to improve things, and we considered that we can't improve something we can't measure. So the goal was to decide what to measure, so we could decide what to improve. It's hard to choose metrics, in fact, and I'm not even sure we choose the right one, we iterate on that. There is metrics that are just effect metrics, for example even revenue, if I just focus on revenue I can change artificially the revenue by maybe buying ads or things like this, but as soon as I stop changing how things are done and I go back to the old way, the revenue will come back. So it's just an effect of other things. We decided to focus on very few metrics, I think if you focus on everything it's not a focus. Of course there is not one answer to this question, there is not one answer to this question for our business, but for each business there is different possible answers that are not necessarily, there is not one truth about that. But having only one thing is probably the most important part. One point I want to say is, sooner I said, yeah, we realized that there was one million images served per day. That's a perfect example of what we call a vanity metric, which is a metric that is nice to tell to a media, to a blog post or anything, because you say, hey, there is quite a number. I don't know if it's a big or small number, but there is this number, but we can focus on that for the business. If we increase the amount of pitchers we serve every day, we just increase the charges, the costs, sorry, and nothing else. It doesn't change anything. So to plan this, we decided to use a framework because as in software for business metrics, for KPIs, people started to build tools that we can reuse. Maybe we will build some of our own later, but probably not. because there is a lot of smart people working on that. The one we choose to use is called Pirate Metrics. Who have heard about that here? Okay, not so much. So the goal is to say, okay, my business is, well, usually software as a service business is managing users, and users follow a journey through your product, which goes from acquisition, when the user discovers your service, activation, where you convince the user that he should act and give you something about him because he's interested about your service. So, for example, he gives you an email address or he downloads your mobile application or you create an account, so you turn it from anonymous to someone you know about. There is this retention phase where you not only have an onboarded user, but it's recurringly coming to your site because it's still interested and has some stickiness with your product. You have the revenue phase where at some point you want to get something from the user in exchange of the great service you're providing. It can be money, it can be something else in some different business. And there is this referral phase where he loves your product so much that he's willing to risk his social existence to recommend your product to his friends. That's one way to say it. Of course, not the only one, but you can't ask people to recommend something or you can say my product will go viral. It doesn't exist. In fact, people recommend things because they think that by recommending they will look more cool or more smart. So you can't make people share something they don't care about. There is this much more complicated version of the exact same thing. If you look, you will find the acquisition phase here. You will find the activation phase here, the retention phase around here. you will have the revenue phase here, you will have the referral phase here. It's just when your business grows and you need more fine metrics and more fine things, maybe you need to have a more detailed workflow about the users. This is coming from a book, by the way, called Lean Analytics, hence the title. But we are small. Our plan A is about, for now, acquisition to activation and a bit of retention. it's how are we getting this service really interesting to users and really sufficiently good so that user may pay something or may give something at one point. And for that, we decided to measure two things, which is the rate from acquisition to activation, how many users that I can convert from anonymous to the created in account, and some metrics about the quality of service, both because we want to display it as a proof of we have the best service out there. That's the only thing we found to differentiate from competing services. And also because we want to have actually the best service, so we use it to measure it internally and try to improve the technical metrics about the service. So, that's for the plan. Now we implemented it. I should have an introduction slide about that, but we implemented it in three steps. We have the first step where we will extract data from somewhere with Bonobo and store it in a database that we can use directly with the graphing software, a.k.a. Grafana. to extract data and format it for the graphing software with Bonobo. Then we will use Grafana to visualize it, and then we will show how to run it in the cloud in a way that you can sleep at night and not being paged every few minutes. So, the idea is very simple, we will use our first one, question, who did already hear about Bonobo before? Cool. So, the idea is very simple, we have different data sources, we've seen it before, we want to read them, aggregate them in some simple way, dimensions to metrics, and store it in a database which is specially built just for that. This database is, I probably had 10 minutes to build this, it's very simple, we just store a list of metrics and we will store two tables of values, hourly values and daily values. That's not the best way to do it, if you want to do something with a lot of data and a lot of related data, complex dimensions and things like this, you should look in Google for star and snowflake schemas if you don't know about that. That's pretty much the state of the art. But to start, it's much sufficient to do it like this. And we will store one data point per metric per hour or one data point per metric per day, so it's not a lot of data. It's really small data and the indexes here will be sufficient for that. So to present Bonobo real quick, so it's a library I started a few years ago, a rewrite from a Python 2.7 library that I had before. Bonobo is only Python 3.5 and more, and the goal is to build data pipelines, streaming data pipelines to transform data from one point to another. So, an example which is purely theoretical would be, I select something from a database, for each row I will call qualify, for each result of qualify I will call join, which will maybe yield more than one result per input row, and then it will be sent to report that will send an email per row. The big difference with other tools you may know is that it's really streaming data. It means that every row, as soon as it's ready, for example, this select could select 10,000 rows or 100,000 rows. As soon as one row is ready, it will yield it and start running Qualifier. Each node in the graph will run in parallel, so you don't have to worry about the blocking aspect of select as soon as the data is here. it will start to be qualified. Here it's a linear example because I want to try to explain it simply, but we'll see it supports any kind of graphs and we'll see an example in the data processor we'll write just now. If you want to try it, you can just pip install it, pip init sumjob.py, and you can just run the job, it will run on your laptop directly, and basically if you have you have a good network, in a few seconds it's done. Two aspects I want to emphasize is that it uses standard Python constructs, its callables and iterators. You can also use more evolved structures, but you can just use standard Python so you're not tying to Bonobo. Every node is taking things first in first out, so you keep the order. Let's write some jobs. I will run quite quickly in this, because there is a lot of content after. If I'm going too fast, we can speak about it afterwards, no problem. So here I wrote a reader that's based on the Bonobo SQL Alchemy extension and the select things we have here. We are defining, we're connecting to something we call website.engine, we'll define this later, the transformations are not tied to a specific database connection, it's the executor that will know what to connect to. There is a simple query that counts the number of rows in a given table, and we just format that as a tuple of two elements, a dictionary of dimensions, Here is date and hour, and a dictionary of metrics to value. So here it's objects.tableName or someName.count, and we pass the number of words found here. That's a simple one, but we'll use it to count users, profiles, and API keys to know how many people there are at a given moment. So we will store the amount of each of those per hour to see the evolution. So we can also compute afterward the difference between maybe the number of users yesterday and today, etc. And Bonobo will take lists of things to build the graphs later. So here we just say, OK, we will use the items, like the dict items built in of Python from the dictionary to generate some data. So here we have one row, two rows, three rows, and we'll pass that to the reader we defined just before. That's the first one. Then we use something provided by Bonobo, which is set fields, which just says I want two fields and I want to name them. Here I named them dims and metrics. We'll use that as what we call our normalizer, which is like a bottleneck we will use. You will see on the graph later. but that's a stupid one but it will be really cheap to change it to a more complex thing that will normalize really think validate that we have the good format etc but to start it's really sufficient and the last one which is probably not really complete complex but the most complicated one here is the insert of updates which will basically just take whatever gets an input and put it in a table as its output. And I added something that can filter things and ignore some rows, so we can use two different analytics writer instances in this graph, both to write hourly metrics and to write daily metrics. Now we can compose the graph, we can create a bonobo.graph instance that gets the list we define sooner, we have this normalized thing, we have an add chain both for analytics writer that will filter on whatever row has an hour and put it in hourly value table and both one that takes whatever not have no hour and put it in the daily value table, because it's better to understand like this. You can bonobo inspect this file and see that the graph is, whatever goes out of these items will go in object count reader, output will go to set fields, and output will go both to daily analytics and hourly analysis writer. Last thing, we need to configure the actual service implementations that we used before. So here we use two different SQLAlchemy engines, one for the website engine that contains users, API keys, etc., and one for what we call the events database, which contains both the simple schema we've seen before, to write things, and both some logs about the service. We can already run it, we see the live statistics while it runs, here there is only three rows, so it's really simple. But now that we built the skeleton here, we will add a lot of different inputs before the set fields thing. So I'll run quite quick into the code to keep some time for the next things. But for example, we wrote a Google Analytics reader that works the same as a database except that it uses the Google Analytics API, so here there is the query using the API. You've seen that the client is passed as parameter, and we use this to say, okay, it's a service that, once again, we'll inject at runtime, we'll pass it as a concrete implementation later, easier to make tests, for example. And then we iterate, and for each report, for each row, we yield two items, tuple with dimensioned metrics. This is already some BI software, so it's really easy because the data is already structured as dimensions to metrics. We wrote a Prometheus reader. Prometheus, for those who don't know, is a TSDB, a timestamp-oriented database, mostly used for monitoring activities. it's amongst other, it's what is getting the CPU values and memory values and amount of messages in RabbitMQ and we use it because it contains a lot of things and there is an API, an HTTP API that we can read from and the query language so here we will send a lot of queries to get some metrics and keep the data point hour-per-hour outside of Prometheus. We also have some spider status API that stores the status of our different browser every I don't know how many, but every maybe 30 seconds or minutes, so we select that. The big difference here is that we will have one row per spider, so for example if we have 200 spiders this will yield 200 rows and then we will have some kind of reducer a bit similar like similar to functools.reduce in Python that will aggregate those 200 rows into counts. We want to know the total number of spiders at a given moment, we want to know the number of active spiders actually getting a page and the number of idle spiders. So yes here Here we prepare the chain that we will give to Bonobo. We have the reader, we have the reducer in a reduce function, and we have a lambda here that will just format the two-element tuple with dictionaries, date-hour, and metrics to have the same format as before, etc. And now if we inspect again, we see that all the different chains we created are here, all going to set fields or normalize functions, and all still going to be written in the database using the same writers. If we run it, it's great, we have a lot of statistics, we have a linear display but it's nonlinear so for example here we see that that output comes here but for example this one doesn't come here it's got it comes probably here along with a lot of others so you can see the status real time there is a few problem with running everything at one time but we'll see after with airflow so I'm a bit biases. I told you about that, but I really like to be able to quickly prototype things and be able to substitute some crappy transformation I wrote at some point by some better one whenever I get the time or the energy, whenever I'm willing to do something better. And yeah, it just runs on my laptop. I don't have any fancy dependencies or any services to instantiate. It's just one process. Okay, so it's great to have a database, but it's not really usable, so who knows about Grafana? Yeah, okay. For all those who don't know, Grafana is a simple... I wouldn't say analytics and monitoring it's like they present it like this but it's it's a graphing solution that can read pretty much anything and present it in a human friendly way i've shown it before but the editor is a like a query editor within the browser directly so you can change and try to build your graph directly in the browser and here it's an example where we built a graph that just gets the amount of events of type created, crawl or banned. It's events that can happen in the system and just show it as bar graphs. I won't show all the queries because it's not really interesting and it will take time to understand them but I can share if you need some. It's not really complicated, it's a very simple SQL. So here is the graph once we exited the editor. We know here the number of images created per period of time, I don't know maybe it's hours, I don't know. We can display the spider, the output of what the spider reader that was reduced, etc. In yellow we see the active spiders. In blue, we see the total amount of spiders. It was at a period where we tested a lot of things, so there was a lot of idle spiders. But, yeah. We have much more interesting the spider timings, how many times to process a website in average, and both how many times since the message was accepted by the spider, and also how many times from the user order to the image creation so we use that for two things we use that internally and once again we have alerts set up whenever some metrics goes over some threshold and we also have public dashboards so we provide our users live statistics or 30 minutes lagging live statistics because we need to catch that not to hammer databases but we provide live statistic to users as proof of yeah yeah, we're maybe not perfect, but we're transparent about what we're doing, so you can know the real numbers. And so I started reverse based on the plan I exposed before, but we also can compute the acquisition rates, because when you have user counts and you have new sessions from analytics, you can, like in SQL, you can directly make computations and the simple computation we made is that we computed the amount of new users per day, the amount of new visits per day and the transformation rate in orange which is just a ratio between two. So we have lots more to do, we did a few more since all those captures which were this summer, but yeah, there is one problem with all this, is that you run it, we run it in Kubernetes, but that's an implementation detail, you run it and then you want to sleep at night, you want to focus on other projects, and things go wrong. The first implementation we did, it was a cron job running every 30 minutes, but maybe at some point you will lock your database for any reason and so the task will start to stack up and and everything not working anymore it's whenever something fails it's really hard to introspect because if it failed like two hours ago you don't necessarily have the the log anymore it's so maybe there is some tasks you don't want to run every 30 minutes etc., and it's not easy to run manually. So I had a proposal from my cat, but I suspect he was more interested in the head from the laptop than really monitoring the thing. So I installed Airflow in the Kubernetes cluster. Who knows about Airflow? Okay. So Airflow, according to the doc, is a platform to programmatically author, schedule programmatically author, schedule and monitor workflows. Two other ways to say it is that it's a GNU make for clusters and it's also a crontab for clusters. It's a bit under saying what it does but it's a really powerful thing that can schedule basically anything that can run on a computer in a really flexible manner by managing dependencies and retries and things like this and it's very flexible it can run workloads using a lot of different strategies at the time it was only providing salary or desk or local but salary or desk and there was a kubernetes one in progress i don't know what is the state today so we decided to use the salary one um i would pass quickly on this but the workers uh from airflow are depending the executor strategy and you have a web server that displays everything and a scheduler that just reads whatever you wrote from the web server in the metadata database, reads that and just schedules things to the workers using the strategy. It looks like this from the end user point of view. It's basically a list of tasks that you can disable or enable. You can run it manually here, you can get the logs, you can get the dependencies. You have a lot of information and most importantly you have the last runs status really quickly accessible so whenever something fails you can really quickly introspect what happens. If one job goes crazy it's really easy to disable it while you investigate. So it's really one tool to help us sleep at night. On the code side, it's also using directed acyclic graphs, DAGs, but not for the same thing as Bonobo. Bonobo is streaming data. It's using DAGs to define dependencies between what they call operator, which is one thing that runs. We don't define really dependencies for this example, but you can, for example, say, handle this data, then upload to S3, then when it's done, run a bunch of 10 tasks that will analyze this data and produce a report. When the 10 tasks are finished, then you can send a mail report. You can do really complex things in terms of dependencies like this. Here we just define different bash operators to run the Bonobo jobs. We will use a simple loop to generate different DAGs for all that. It matches the different things you see here. And we generate a special one that we will run only once a day. It's scheduled daily to clean up all the objects from the database because we don't need to clean up every 10 minutes or so. or every hour. It also allows to define connections, so we defined the two databases here, and what we did is, maybe it's probably not the best way, but it works really well, we just got this connection from Airflow and passed it as environment variables to the processes, so Bonobo will auto-configure its databases from this thing. So, it was less trivial than expected to install, mostly because I had to learn a lot about some internals of Airflow, mostly for the connections, I had to read the source a lot. For the Docker and Kubernetes users, there is the company called Astronomer that made a really good Helm distribution of Airflow. The community charts I found were not really great, so I would advise to look what they did. They open-sourced a lot of things. This slide is interesting, I didn't remember that. So, to summarize that, Bonobo helps you build assembly lines, may help you build assembly lines, but it doesn't really care about the whole system. We use the Airflow to manage the whole system, and really we don't see anymore the job contents. In fact, I guess, I'm not sure, but I think it's always also in containers, so even the code is not in the Airflow image. and Grafana lets you make the human accessible to humans and it's really quick to write and iterate there is a few reference books that I would suggest if you want to learn more I must put this one which is our bible even if it's not directly related and it's available for free so you should look at this one Linalytics contains a lot of recipes to think about what kind of metrics you would want to see. And both running lean and scaling lean also talk about, well, I deleted a bit of content from the presentation, so there is less, but a bit of content about how to iterate and how to experiment around data and KPIs. I'm always very eager for feedback, so feel free to grab me anytime in the conference. also I will probably no, not probably, I will be here this weekend and will sprint on Bonobo if you want to join, you're welcome if you want to join just to discover Bonobo, you're welcome if you want to write a crazy new parallelization strategy based on whatever you're also welcome and yeah otherwise that's my contact information feel free to send me an email or contact me on Twitter if you have questions or if you want me to share some content of this presentation, and I think we have like six minutes and a half for questions, so Yeah, thank you Thanks are there any questions thanks for the talk I want to pose a question regarding the tools that you use and how they differentiate for example I quite frequently used Luigi so how what is your experience and how do you compare it in comparison to the tools you have already presented here? So I don't know very well, Luigi. I had a few discussions about it. As far as I understand, it's very similar to Airflow in some ways. It allows also to define dependencies very easily with Python code. So that may be a good replacement for the Airflow part. Also, as far as I know, Luigi can't stream data from one, I don't know the name for them, but from one node to another. It's also dependency management more than Dataflow. I chose Airflow mostly because it joined the Apache Foundation, an Apache incubation project. So I guess it's a quality metric that was important to me. But probably Luigi, and also Luigi is made by Spotify, and Spotify announced that they started to use Airflow for some projects. So I thought it was kind of a hint that I should maybe go to Airflow more than Luigi. But probably both tools can achieve basically the same thing. And they can definitely achieve the simple thing we did here. Thanks. Any other questions? Okay, then. Thanks for the talk.