Web based live visualisation of sensor data

Speaker 1 [00:03]

Hi, welcome everybody. So I'm working as a sensor developer at ROSEN. So what is ROSEN? It's a company who's inspecting pipelines. If you want to learn more about it, just visit our sponsor booth. It's that one with the chocolate eggs and you could also see a device which we are using to inspect the pipelines there on display. And these kind of devices are equipped with a lot of sensors and that is related to my work. So I'm a sensor developer and that includes assembling sensor prototypes which are testing in a lab. And that's why I'm using Python and I would like to give you a short insight into a typical procedure in the lab. So, we are hosting a web interface with FastAPI, there we will adjust our measurement parameters like setup of the sensor, what sample is lying below the sensor and so on, and at some point we push a start button. This will cause a to-do list to be created in a Redis database. This to-do list will be consumed by our control process. It will take one step from this to-do list, which may be a command just to measure. This command is published on a Redis channel. Our sensor process is reading this command, triggers the sensor to do a measurement. This process answers that the measurement is finished, which is again published on a Redis channel. Then it will be picked up by the control process again, which knows now to take the next command from the to-do list. This may be a move command, so the sensor measures and is then moving to the next position on the test sample, which maybe just move right two millimeters. Then when the movement is finished, there is again a finished message published on a Redis channel. The control process knows to release the next command, which could be again a command for the sensor to measure until the complete to-do list is fully processed. This data which is also published, the sensor data which is also published on Redis channels is consumed by an HDF5 file writer which is assembling large HDF files, one for each series of measurements. Today, as the name of the talk already mentions, it's about the web interface. So what we are doing with the web interface is we have a preview of the sensor data so one measurement could take about one hour and if we want to know if maybe there's just a little wire damaged which disturbs our measurement or even makes it unusable. It's nice to see it instantly and not after waiting for a long time. Then we have an indicator of the progress of our measurement so that's more or less the state how many items from the to-do list have been processed and finally when the measurement is finished we can review the results. Today I would focus on two parts of this. The first one is just how is the sensor data transported to Redis and the second one is how do we connect our web interface to Redis, and there I will focus on sensor data preview only. But now to focus on the key components of this, I will switch over to my hobby project. It's a flight data recorder. So it all started with a colleague who owns a home-built aircraft, this one. And this aircraft is in the process of certification. And that requires some performance data. And so the idea was born just to create a little flight data recorder based on a Raspberry Pi. This one. So as you see, the infrastructure looks somehow similar to our lab setup. However, it's more or less, in most cases, unidirectional communication. So, there is a GPS receiver, which is mainly determining the position, where we are, publishes this once a second to a redis channel, then there's a barometer, which has a slightly a higher sample rate and publishes the current pressure to the Redis channel. And finally there is an orientation sensor setup which consists of a combined sensor accelerometer, gyro and magnetometer to know the aircraft orientation. In the initial state of this project, the FAST API was just used to export and review data so that we could get such a GPS track as you can see here. It's a flight above Blingen, the city where I'm living and working. And here you could see a height profile. The device itself, you can see here, it's mounted inside the cockpit of the aircraft. This little device here. Now let's take a closer look at the device. So it's exactly this one. Here's a GPS unit and the magnetometer, the gyro and accelerometer and the barometer here. The item in between here is a power supply electronics. It's connected to a small LiPo battery. It's as small as this here. And it would last for about four to eight hours depending on the CPU you load. And finally, on the bottom there is a Raspberry Pi Zero W, which means it has Wi-Fi. And I brought today a mobile Wi-Fi hotspot, which is being used to connect to this device with my computer. And later I will publish the credentials for that, that you could try it by yourself. I didn't mention before, all the setup I'm using is published on GitHub. So the GPS tracker device is located at my GPS tracker repo, but that doesn't include the live data visualization. That was added just for this talk, as demonstration. So at first I removed the processing of GPS data, as here inside this building we don't have any GPS reception. We still will have the barometer and the orientation sensor. Now I modified my FastRP web interface a bit, so it's using WebSockets for streaming live data. But at first, let us have a look how we get the data from the sensor to Redis. So it's a bit simplified code, so we import just the Redis client, JSON library, time, and the implementation to control our barometer sensor. You could look up the details on my GitHub repo. Then we initialize Redis connection, our sensor, and then simply run continuously in this while loop, reading sensor data, publishing to a channel which is named barometer. It's using JSON dumps. It's make a JSON string of that, and the JSON string just looks like this in the box on the right. It contains a timestamp, the current pressure, the current temperature, and a name of the sensor. Now let's have a look how to consume that data. So again, we need the Python Redis client and JSON. So again, we create a Redis connection. We create a PubSub object, and we are subscribing to the channel barometer. run a for loop over the pubsub.listen and then in every iteration we could decode the JSON string to obtain our barometer data and decide how to do further processing of of that. Now let's come to our first demo. So we have the barometer which publishes the pressure data to the register channel. Then our web interface which is reading that data and now let us just try it. This is now pressure read from this sensor here. You see it is lying on top of this desk. Now I take it, put it to the ground, and put it up again. Now you could see it is about a bit more than a meter. So, just by measuring the ambient pressure, you could read... Ah, it was stuck for a moment. Just by measuring the ambient pressure, you could measure altitude. It's like it's done in an aircraft. And the precision is... Normally, it's not stuck. Maybe there's some interference with the Wi-Fi here. That's how altitude of aircrafts is measured. The plot is done using Plotly, which is available for Python directly, as well as a JavaScript library which is used in my case. Now let's continue with the talk. How to create the web interface. So again, I simplified the code a bit for the presentation, so exception handling and so on is not done here. We are using FastAPI, WebSocket from FastAPI and the static files. We are using the AsyncIO Redis client, and we create a Redis connection here. As discussed in another talk today, maybe it's not a good idea to do it on a global scale, but it's how I'm doing it currently. This code here is just to create the static file hosting so that just means there are some HTML and JavaScript files in a folder called static and they will just host it at the endpoint static of our our web API. The more important part is the WebSocket which is hosted at the endpoint WSIMU pressure so this is the one for the combined data of the position and pressure sensor so again we are creating a PubSub object we are subscribing to the IMU pressure channel and then there's simply running a while loop we are trying to get a message from the PubSub object. So it's a little bit different to the PubSub.listen in the example before. If there is a message available, we just send it through the WebSocket, otherwise we are doing a short timeout here and this would run continuously as long as the WebSocket is open. But now let's take a look how to consume it. So this is a bit JavaScript. The first line is just assembling the URL, where to connect to. Instead of http, it's just ws. At the beginning with our API endpoint, we have a WebSocket object. Now the important part is just to define the onMessage function. This function is run every time a new message arrives on the web socket. This is parsed as a JSON, so this is a JSON string, it's parsed, then it will be a Java object, maybe similar to a Python dict, and then this is just generalization for all the following visualization, I will show just calling some function to visualize the data with the content of our message. Now, as we talked a lot about airborne species, I brought a bird. And let's try another visualization. So this is done using 3.js. Now let's just try a few flight maneuvers. So if we're doing a steep climb and a strong descent. Now let's fly a curve. Or maybe just upside down, hang around a bit, and as you will see, it reacts nearly instantly, So it's a few milliseconds delay. Let's continue. Now let's take a look at the performance. So the bird is currently equipped with a single core Raspberry Pi. The FAST API is run as a single process and the sensor data rate is 12.5 Hz and I tested this at home so up to eight clients are possible and the delay is still extremely low and the data rate of nearly 12 Hz could still be maintained. As soon as I exceed eight clients with that setup, it will dramatically slow down. Even the commands may stay in the queue. In that case, it may happen you move the bird and wait about half a minute and then it reaches the message which makes the movement and then it becomes a really long delay here. But now let's come to the final presentation where we combine all the sensor data. So let's get to the pilot's view. We have the artificial horizon. Then we have the compass. altitude display is made for real aircraft altitude, so you will not see a movement here, but maybe you could see here the vertical speed indicator. I think now it's your turn. You could try to connect to one of these two devices with the Wi-Fi hotspot. If it doesn't work now, maybe you could visit us at our sponsor booth. Maybe we could try there again. And thank you for your attention.

Speaker 2 [19:34]

Thank you for your talk. So there's a question. Why did you choose Redis over RabbitKMQ or 0MQ?

Speaker 1 [19:45]

So mainly it's because I just started with Redis and in our lab setup it's also used as a database, so we are storing there some configuration data, we are storing this kind of to-do list and all this stuff and so we are just using or I'm just using that what I already know, but maybe for the unidirectional communication, maybe 0MQ could also be a solution. But Redis has shown to be really fast.

Speaker 2 [20:26]

Thank you, and like did you think about the scalability like if you are tracking 10,000 birds? How will you scale the product?

Speaker 1 [20:35]

Could you repeat the question?

Speaker 2 [20:38]

So it's about like scalability of the backend. So if you are tracking 10,000 birds, what will be the bottleneck?

Speaker 1 [20:48]

Oh, so what I played around, so it's directly run on the Raspberry Pi, and the backend is running there. So I found out the bottleneck is not, at first, it's not Redis, but was the single process FastAPI, so with the hosting of the WebSocket. So I did not analyze in detail if it's just the connection of the WebSocket to the browser which is slowing down, but maybe it's the processing in the single FastAPI process. So a solution could be to run multiple processes, so each of the processes could subscribe to the red is. This is a quad-core CPU on this Raspberry Pi, so maybe I could start a full process or even more on that. If it's for a much larger audience, maybe data even needs to be duplicated to other devices and then distributed from there.

Speaker 2 [22:16]

Thank you, and there's a final question like how do you save the historical historical data? How do you save the historical data?

Speaker 1 [22:28]

This one is the historical data in my GPS tracker, I think. Yes, so currently it's just stored also in Redis, so I think Redis is writing when you don't modify the configuration, it's once a minute, or if you collect more than a hundred or message or something like that. It is dumping the data to the SD card in that case. It is enough for doing several flights and so on and then later it will be exported and stored on some other device.

Speaker 2 [23:13]

Thank you. And there is one more question. Like, all these sensors are publicly available or not?

Speaker 1 [23:21]

all sensors of my hobby project are publicly available, so that's why I chose it. So there is this electronics here. It's called from a company called Ozmaker in Australia. I think it's about 40 to 50 euros with all the sensors on top of this. And this is a PiJuice use a power supply board it's also similar at a similar price. So the whole flight data recorder is about maybe 100 euros and depends on which kind of sensor you would like to spend a bit more or less. The code for the visualization is available here and you even could replace the bird with, I think, a pony and a flamingo and a parrot. So these models are also available in these three JS examples I used for this talk.

Speaker 2 [24:28]

Thank you. That's all. Please give a round of applause.