Offline Disaster Relief Coordination with OpenStreetMap and FastAPI

Speaker 1 [00:07]

Thank you for the introduction. In addition to my work as a sensor developer at ROSEN, I'm also one out of 88,000 volunteers in the German Federal Agency for Technical Relief, the THW. Internally we refer to ourselves as helpers, and we are distributed into 668 local sections all over Germany. Each local section houses at least one technical platoon. It consists of the command squad, the rescue squad, and the technical squad. So in total, one technical platoon is about 30 helpers. And the technical squad may be specialized in over 12 different areas, like electricity, infrastructure, water hazards, and more. I'm a helper in the command squad. Our primary role there is to support our platoon leader with managing the radio communication, drawing a situation map, and creating a personal overview. When we are deployed to disaster areas, often energy, due to the disaster effect, the energy grid may be interrupted and also mobile phone network may not work anymore or may be overloaded. So the only way we can communicate in that case is our radio communication system. It looks a bit like a mobile phone from the last century. And it cannot do much more. So spoken communication is maybe really challenging when it's a loud environment. And we are also training to work with paper maps and even extract coordinates from that maps and if we get a coordinate to locate it on that map. And in a recent training of that kind, I had the idea, as it was quite time consuming, why don't we do it in a digital way? So I created a small hobby project I will show today. So what are the requirements for such a map? It needs to cover the whole of Germany. It should work offline and off grid, meaning without external electricity and without connection to the internet. We need to draw on the map, which is our, yeah, creating our situation map. And we need a search option for places and streets as, yeah, we want to find the location where our mission is. It should be accessible over a Wi-Fi hotspot and it should support the UTM ref coordinates. That's the coordinate system where we are usually trained on to work with. Now let us take a look at OpenStreetMap data. So for Germany, the OpenStreetMap protocol buffer, how it's called, protocol buffer binary format PBF is about 4.2 gigabytes large and it consists mainly of three different types of objects. There are nodes that are more or less just a location and are equipped with tags describing that node, then there are ways which are a collection of nodes and finally there are relations which could consists of several nodes and ways. This OpenStreetMap file of 4 gigabytes is much too large to serve it to the user. So to handle this, I'm using the MBTiles format. based on SQLite and there the OpenStreetMap data is clustered using a plane tiler into small tiles depending on the zoom level so yeah I provide that data finally via fast API so I'm created a fast API endpoint looking like this where you have set as parameters the zoom level and the X&Y location for the tiles. Yeah, and that's how to obtain the data. As hardware, I selected a Raspberry Pi 02W. It's low cost, energy efficient and can easily be powered by mobile power sources. To create a Wi-Fi access point with a Raspberry Pi, I used a script from raspberrypi.com rather than reinventing the wheel. If no known Wi-Fi network is detected, it creates a hotspot. When a known Wi-Fi network is detected, it connects to that and that makes it easy to update the system at the home base. To display the map, I'm using LeafletJS for rendering the map. I'm using a MapLibre plugin to add vector map support. This allows to use data which is quite small, like this several gigabytes for whole of Germany. Otherwise if I would take render pixel graphics for zoom levels up to 18, then it would be about nearly a terabyte. Then I'm using the geoman framework to add editing capabilities and the leaflet measure path plug-in to measure distances and areas on the map. Using vector maps, as mentioned, the map is rendered in the browser. The style file defines how and whether features from the OpenStreetMap file are displayed. For the offline version, I had to modify the style files that they consider the local path of the tiles, the fonts and the sprites, rather than referring to somewhere in the Internet. And what are the sprites? That are these icons you can see here on the right side image. To prepare for the location search, I downloaded shapefiles describing administrative boundaries from the federal agency for cartography in Germany. I converted these shapefiles to GeoJson using GeoPandas. And I also downloaded data for the postcode regions. In this case, the data was already available as GeoJson, let me just check here. And it shows a download with reduced precision, which allows a smaller footprint of the data itself and verified in my local community that the postcodes are still quite reliable, so matching the right houses and streets, yeah, with that kind of precision. But still doing searches on that GeoJson is quite computationally demanding for the Raspberry To address this, I created a Python module in Rust using PyO3 to accelerate this search. The main challenge for me was, as I'm not a native Rust programmer, that most tutorials did not contain an example where you create an object once and then reuse it several times to call some operations on that. And now as I implemented it here, I'm opening the quite large file, it's about 30 megabytes. Then I could apply multiple times either a function to request the post code for a given latitude-longitude coordinate or in the other way I could also ask for a postcode or even a segment of a postcode to find the corresponding shapes from the postcode areas. Now let's return to the OpenStreetMap data to extract places. used Pyosmium. If you would like to know a bit more about Pyosmium, maybe you've seen today the talk from Katie, otherwise I would recommend to review it. So with Pyosmium I'm selecting certain nodes to extract the places, so they must to at least have a name tag and they must have a place tag which is one of the values shown here which correspond to a location like city, village, suburb and so on. I did the same with streets. For this I extracted the ways from Open Street Map including their nodes they consist of. Again they must have a name tag, so only named streets are relevant for a search and there the highway tag is important, I didn't list all of them here, yeah it was more or less the same as with the nodes and I'm storing them in an SQLite database for further processing. In the next step I clustered these nodes and places I extracted before by the postcodes and administrative regions where I have taken the shapefile before. One of the reasons is that, yeah, streets and locations occur multiple times. So the place Altstadt, meaning historical city center in Germany, occurs about 79 times. And Hauptstrasse, meaning main street, even more than 6,000 times. So it's quite useful to have some way to cluster it. For the streets, I look for a way to simplify the street to a single point. But this point should still be on the street. What I did here, I used the Shapely library. Then a street like this here is an example of the M's own alley, a street where I'm living. And it consists of two OpenStreetMap ways, and they have different OpenStreetMap IDs, so it's not easy to cluster them directly. And then for each of them, I looked at first for a centroid. That's this one. And then I looked for a point on the way itself, which is closest to the centroid. That would be this one for the green way. I did the same for the red way here. Then from these simplified points, I took again the centroid. That would be this one. And the closest point on the street would, in that case, this one. So, this would now represent the whole street for my search. Then I have also had to integrate the military reference system, also called UTMREF. You may be familiar with latitude and longitude coordinates describing a location on the globe. In the UTM-REF, the world is projected onto 100 kilometer squares with a two-letter code. I would think for the region here it might be MB, I'm not sure, I didn't check. The precision within this square depends on the number of digits after a coordinate like in this format. So for typical applications in case of a disaster response, a resolution of 10 or 100 meters would be totally fine. However as I'm using PyGU-DZ for the coordinate transformation, I need to do some zero padding as it expects float values which are meters. So I had to do some zero padding. After all that theory, I will show a live demo hosted on this Raspberry Pi like the talk itself. And I hope it works. As I mentioned earlier, we are usually limited to spoken radio communication in the event of a disaster. So let's receive some incoming messages. So we have an unexploded World War II bomb found during construction. That happens multiple times a year in my hometown. And we should provide coordinates for the Stadtwerke parking lot entrance in Luisenstrasse 49074 Osnabrück. So let's start with the postcode, 49074. I typed a bit too fast. So now all the requests. So when I'm doing this, in the background, requests to fast API are performed. And acquiring the database, yeah, and then finally I receive the shape of the target region. Yeah, now I have to find Luisenstrasse, yeah, it is here. Now I don't know where the parking is, so maybe I switch the layout, and I see here Here it is. If I click here, normally if I click here, ah, here it is. This level staff would appear and directly on the staff I see the postcode and the administrative region. But here on the bottom I can read the coordinate of that place. And all the different coordinate system depending which you need. Now let's continue. We require an evacuation around this, in a thousand meter radius around this coordinate. It's 32 UMC 36339139. So let's take a look where it is. We enter the coordinate. So here it is. Now we place a marker here. As the bump is quite dangerous, we mark it red. And now when we move the mouse here, we see there's the bump. Now we need to draw one kilometer circle. It's maybe a bit small on the map, but you might read that this is one kilometer. So we don't want to fill it, and we take another color. Yeah. Now we have to handle the operational area A, southwest of the railway tracks. So that means we need to draw some lines here to mark our region. And we could also put labels here. Then we get a receiving radio message, Detmarsstrasse is cleared. I don't know where it is, but maybe we can search for it. There it is. As this is cleared, we take a green color. Okay, next one. Oh, person at Collegio Invalid 11 refuses evacuation, police required. That's typical, it happens several times a year. Now we see it this year. Okay, Schlagforderstraße is cleared, I can see it, I don't need to search for it. And finally police arrived at Collegian Valley 11 and removed the person and Collegian Valley is cleared. Okay, then I would draw another line here and remove the marker. So this is how it would typically work. But after some times, when the units have cleared the streets, they become idle and request new missions. In such a case, it would be helpful to know which unit is closest to a specific street that needs to be cleared. This is why I am considering integrating LoRaWAN into the system. LoRaWAN operates at 868 MHz in Europe. It's designed for long range, low power and low data rate applications. Its low energy consumption means that the battery of a sensor node measuring temperature or water level may last for several years. However, a GPS tracker may only last for a few days, depending on the upgrade rate we have. Typically, the infrastructure consists of gateways that receive data from sensor nodes and forwards them to a cloud server. To be honest, I'm a bit behind my personal schedule compared to my announcement in the abstract, which I wrote under the impression of several flood events. But as there were so many bomb evacuations on the time, in the meantime I shifted the focus a bit. However, let's take a look at my experiments. So I did a few tests with a gateway and a temperature sensor node. So the temperature sensor node was much easier to handle than water level measurements. And yeah, I took a short, started with a short walk around my home, where I have a gateway on my balcony. It's about five meters above ground. And then it followed by a bicycle ride, as I was a bit surprised by the range. So when I started with my first LoRaWAN experiments, I got hardly a signal from outside my apartment. But as you can see here, in the densely populated area here in the north, I had a range of up to 400 or a range of at least about 400 meters, while to the south, where there is more clear line of sight, it was even more than two kilometers. The next steps beyond this presentation will be to create a mobile version of that low-revan setup so that it will work without cloud, which means having an internal cloud which works entirely without the internet. And then extend it to other devices like GPS trackers for the units or as initially promised in my abstract, yeah, at water level sensors, which is based on pressure measurement, yeah, to integrate things like that in the situation map. And yeah, maybe I would also ask my neighbors on two floors above me if I could use their balcony for a test to improve the range by increasing the altitude. And yeah, thanks for the attention, and if you have any questions, feel free to ask.

Speaker 2 [23:30]

Thanks, Yanis, for your wonderful presentation and live demonstration. I would like to first read some appreciation from some anonymous comments. It looks cool. How often was this already used in production? What was the feedback?

Speaker 1 [23:48]

So it's more or less a prototype, and to be honest, we have in production a system supplied by our Geodata department of the city, which was optimized by the firefighters and so on. So we have for our home city, we have a system like that or even more advanced, but that relies on a VPN connection to the Geodata department of the city. Yeah, so we are more or less already working with a system like this, and this is starting as a hobby project. But it could be especially of interest if we go to a region where we don't have the local knowledge. And then, yeah, we may try within our unit to test it. So we have in total three of these command squads in my home location in Osnabrück. So it has not been in full production yet, but we will try.

Speaker 2 [24:53]

Okay, thank you. So let's come into the questions. So what devices are people using to access the service?

Speaker 1 [25:01]

The idea is that it runs best if you have a tablet or a laptop computer. It works also with a cell phone, but the screen is quite small and it's difficult to type or draw lines on it. But in principle you could use every Wi-Fi device which has a browser.

Speaker 2 [25:26]

So the next question is, since THW is a federal agency, why do you not use the official map data provided by the state serving agencies?

Speaker 1 [25:39]

Yeah, so the idea here was to use offline data, so the official way would be that we get some paper map data or that we also have access to official map data in the internet, but then we need the internet.

Speaker 2 [25:58]

Okay, coming to the next question, where open source solution like QGIS field not sufficient for your use case?

Speaker 1 [26:08]

Which name?

Speaker 2 [26:09]

the name is QGIS field

Speaker 1 [26:14]

I don't know this exactly, but so the solution was a bit derived from another application I developed where I needed OpenStreetMap data on a Raspberry Pi and so I extended it for that use case as I already started somehow with this framework. So I did not do analysis which other frameworks are available of that kind. So it was, except for the street search, it was mostly available to me already.

Speaker 2 [26:53]

Okay, thank you. Next question is did you automate the data manipulation? So it's easy to get fresh POI or street data, for example

Speaker 1 [27:03]

I created some scripts that could more or less, if you put a new OSM file into it, it could refresh the database. But that's something I would not do on the Raspberry Pi itself. But if you download the OSM data, you can run the script on it. So it's not fully automatic, but quite automatic.

Speaker 2 [27:30]

So coming to the next question Again appreciation first awesome demo. So a big round of applause again for this awesome demo The question is is it also possible to search for alternative spellings that is trusses truss a SDR Or do we need to use some spelling in OSM?

Speaker 1 [27:57]

I, in this case, I did not include alternative writings or alternative writings of the streets itself. And I know the problem that the name of a street may be abbreviated. So that's in the practical use I would not write the full name of street. So in German there could be like a double S or an SZ, it could be several ways to write it. So I just start with a name and then it will look for which string starts with that name and so I could match it even if I don't know exactly if it's written with SS or SZ or just abbreviated to str.

Speaker 2 [28:44]

Thank you. The next question is, maybe you could use a dark DB query to select and read only shape from the big JSON file instead of reading the whole file. It's a kind of a suggestion. Maybe you could use the dark DB query to select and read only the shape from the big JSON file instead of reading the whole file.

Speaker 1 [29:04]

Okay, I don't have experience yet with a DuckDB, but maybe I should look it up then

Speaker 2 [29:10]

Thank you, and there are two more questions Why did you choose the vector data over the raster tiles for the base map?

Speaker 1 [29:20]

Yeah, it was, I started with the, my first experiments were with the raster data from also the BKG organization. But then when I tried to include not just Germany, but also Switzerland, as I have another application where I need to include Switzerland, and yeah, then the problem was for, to have a certain zoom level it became much too large for the SD card on the Raspberry Pi and even if you may consider using a compute module which has the flash memory and that's even smaller than the SD card and so there was not just not the memory to put the pixel data inside. The implementation would be much easier but then yeah I tried out with the vector data and now it works.

Speaker 2 [30:15]

The last question is the annotation data also stored in the PI and Explore exportable it might be crucial that the data is recorded recoverable if the PI power off

Speaker 1 [30:28]

Yeah, so currently my approach is that I'm using the local storage of the browser. So when you draw the map, refresh the browser or even power off the device, even replace it by another one and then open it in the browser again, it will be stored in the browser. The next approach might be that I create a database on the Raspberry Pi which has some time information so that I can have a timeline and can go back in history of that mission to maybe analyze at which time a certain unit gave a message out of something and so on. But then I would also require at least a GPS to get time information so as the Raspberry Pi itself has no internal clock which could keep its time.

Speaker 2 [31:25]

There are no more questions, then we thank you very much for the presentation and demonstration.