Maps with Django

Speaker 1 [00:03]

Thank you very much. Hello everyone, I'm very happy to be here with you all. In this talk we'll see together how to build web maps from scratch using Django. If you're asking yourself what type of maps we can build with Django, let's see an example. This map shows mountain peaks all around the world. We all use maps like this every day in web or mobile applications, and I built that one specifically with Django and Postgres. In this talk, we'll see together how to build a map like that, but first, I'll present a bit myself. As I said, I'm Paolo Macchiore, I'm a software engineer and long-time Python developer. After using Django for a few years, I became a contributor to the project. I also really like hiking in the mountains in Italy, we have a lot of them, so in this talk I decided to build with you a map where I can put all mountain peaks I've reached. I took this photo from the starting point on one of my last hikes. I was in the Italian Amphinaeans, and the sun has not yet risen, but in the distance there is the Adriatic Sea. The making of this map will be a bit like this hike. We'll start from easy scratch and then we'll go up in altitude. But let's start from the basics about web maps. A web map has many features. It can be static, dynamic, you can interact with it or you can only view it. The map can use raster or vector tiles to represent the surface. the data is stored in a special database and a web map will use a JavaScript library to show data on your map. On Wikipedia, regarding web maps, we can read that web mapping is the process of using web maps delivered by geographic information system on the internet. But implementing a geographic information system from scratch is beyond the scope of this tool. So to do this, as you can imagine, we are going to use Django, my preferred Python-based web framework. The requirements to create our map with Django are a stable and supported version of Python and the latest stable version of Django itself. In my example, I've installed all of it in a virtual environment. To create the map project, our project, I switched to my project directory and then use the start project Django command and the basic files of our project will be created for us after switching in the new created my map directory we create our markers app with the Django startup command again all the necessary file will be created for us. Now we have to activate our markers application by inserting its name in the list of installed apps in our setting file. At this point we can proceed to insert in the markers view file a new template view for the page of our map. simple one in the markers template directory we can now create a template file for our map for now we had only usual board plates with a title but without anybody content and in the marker URL file we must now add the path to the viewer map using its template view. As a last step we include in turn the URL file of the marker app in that of the project. We just made a first view in Django but it will only show a blank page for now so we can move on something more challenging. I took this other photo after half an hour of walking in the dark and you can see the sunrise in the distance and we are about to start the high-altitude path. As for our hike, something will be to be seen in our project as well. In fact, we'll add a blank map page using the leaflet library. Leaflet is one of the most used JavaScript libraries for web maps. It's a free software and it's desktop and mobile friendly and is very light and it has a very good documentation that you can read directly online. To use leaflet we need to link in the head of the page its JavaScript and CSS module in our template. In addition using the Django static template tag will also link our our custom CSS file. In the body we need also a dfv tag with a specific ID and our custom JavaScript file which we are going to create now. We add our CSS file in the static directory and inside it we only had the basic rule to show a full screen map. And now, in our JavaScript file, we add the code to view our map. Using the defined variables, we initialize an OpenStreetMap layer, we hook the newly defined layer to our map, and in the last statement, sets a map view that mostly contains the whole world with the maximum zoom level possible that's it we can now start our Django project with the run server command and we can finally visit our map page in the browser so we just created an empty map with Django and the result is pretty much what you see now a map without markers showing the whole world. This photo shows a crossroads at the end of the first part of my hike, just before a very challenging uphill stretch. This is a reason for why the landscape is visible around. Likewise, having visualized our map with Leaflet, we'll now start a bit more elaborate part, writing more code to create our markers and display them. So it's time to get to know and activate GeoDjango, the Django geographic module. Django added geographic function a few years ago in the GIS module with specific fields, database backends, special queries and also admin integration. Since then many new useful feature have been added every year until the latest version. Before activating it, we need to install some requirements. A mandatory GeoDjango requirement is GDHL. It's a OS Geolibrary for reading and writing raster and vector geospatial data formats. It has a very useful command for data translation and processing. To easily install GDAL package on a Debian based system you can use apt-get package manager in this simple way and we can now activate GeoDjango by adding the contrib GIS module to the installed apps in our project settings. But to use GeoDjango correctly, we need to change also our database engine and use one of the compatible database back-ends. In this talk, for a more complex example, we decided to start using Specialites, the specialized extension to SQLite, which is the default Django database back-end. provides vector geodatabase functionality and it's released with a free software license. It has a simple architecture, a complete database in an ordinary file. That's it. As before, on Debian base system, we can install the Specialite package using the apt package manager in that way. And now let's add Specialite as the back end in the setting, default engine and in our project we leave the rest of our code unchanged. We only have to change the engine definition. So now we have everything to define our marker model to store a location and a name. Our two fields are both mandatory. The location is a simple point field and we'll use the name to represent the model. We can now generate a new database migration and then apply it to our database using this command. To easily insert new markers in the map, we use the Django admin interface. we define a marker admin class by inheriting the GIS model admin class, which uses the OpenStreetMap layer in its widget. We'll see in a bit. We can now create a super user to access our admin interface after starting the project locally. And finally, we can start our Django project with run server command and visit our admin section in the browser and at the end here is the page for inserting the markers in our admin as you can see we have a text field to enter the name of the location and GIS widget we have to manually navigate the map and manually define the point in the space. But it works. It's ready to use. Okay. After having added more of the markers in the admin, we can finally show them in our map. We can do that by adding the information in our view. Here we are retrieving all the markers from the database and converting them to GeoJSON before adding them to the context of our view. The value of the markers key in the context dictionary will be something like that, GeoJSON, standard GeoJSON documents, in this case only one point. Okay, now, in our template, we use the JSON script, the template tag to add our markers to our page in a secure way. And now, let's set it, our JavaScript file, and store the JSON in a variable. Starting from this variable, we build a layer for our map, and we extract also the name of the single marker. Finally, we add the layer in our map by setting the view to contain all the data, so all the points in the map. We can finally start again our project with the run server command and see our marker in our map in this URL. In this map we see a few markers I uploaded through the admin. They are inside the page code, but the loading is still fluid and fast. You can also see the pop-up in the marker of the peak I'm heading toward in the Zyke. But if you add a lot more markers to show, our map loading will be much slower and Leaflet will have a harder time rendering all of them. So we need a better, more complex solution here. This photo shows a beautiful landscape that came to my sight at the end of a challenging climb. The Itis Peak begins to be seen, but there are still challenging passages before reaching the summit of our hike. So we then continue implementing the final version of our map. To do that, first let's start using PostGIS as a new backend engine. PostGIS is a Postgres extension, and it's also the best database backend for GeoDjango. It internally integrates spatial data and has spatial data types, indexes, and functions. In this chart, I've synthesized the compatibility table of the open source geographic backend supported by GeoDjango. In the GeoDjango documentation, there are three of these tables. As you can see, the PostGIS is the only geographic backend that supports 100% of the feature. So it's a better choice for us. In order to use PostGIS as a database backend, we need to install the PostgreSQL client library. As before, you can do it easily with apt-get package manager. The Python requirements of our project are increasing and therefore is a good practice to create a requirement file with the package list. In addition to Django, we are going to use PsychoPG, Django filter, Django REST framework and its geographic add-on. We install all the Python requirements using the Python package installer module. We modified the project database settings, adding the PostGIS engine and the connection parameters of our project for our Postgres database, which you can have locally or remotely or in Docker. We have now to apply again all the integration to the new configured Postgres database. The package that we'll use directly in the code of our project are Django REST framework and its geographic add-on, which we then insert in the list of the styled apps of our project settings. So let's create a serializer for our marker class. In everything from REST framework GIS serializer, we only have to define the optional field to be shown in the additional properties, geographic field location and the model marker. Our intention is to expose our market via RESTful API and to do so we define a read-only viewset. We set the location as a field to filter our markers and then a filter based on a bound box. We also return all markers instance without any filter. In the Marker application, we define the URL of our new endpoint using the Django REST framework default router to create our path. And finally, we add to the definition of the URL of our project a new path for the API that includes the path just specified for our Marker app. So, after finishing our RESTful API, we move on to update our JavaScript file. As we no longer have the data preloaded in our page, we no longer have a way to position the map so that it contains all the markers. So we try to locate users through the browser that will ask them to share their location. In the positive case, we use the shared location to center the map, and that's it. In the negative case, we locate them on an arbitrary point in the map with a low zoom level. So we no longer load the marker directly on the page. So we ask our endpoint to return only marker on the specific displayed area, pass it as a bounding box string as you can see on the URL. To build the marker layer, we ask our endpoint the data for data asynchronously and extract the properties we want to show in the pop-ups. We invoke this flow every time the user stops moving on the map, not every second. this way our query will be retrieved only a minimum number of data from the database and the response to the browser will be lighter and faster to render for leaflet okay finally here is our complete map in this example we can see how market specific map are look but let's small let's do a small demo of how dynamically the map load if it work okay the loading takes place in very fluid way because the number of calls occurs only when the movement of the map stop and therefore the data traffic is reduced to the essential as well as the rendering of the marker carried out by leaflets okay Okay. Okay. Our map project, as our map project, the hike also reached its final destination and unfortunately the view in this spot is a bit covered, but we can finally add a new marker in our map built with Django. It was one of the longest hike I ever made and I hope to be able to do new ones soon. But there are also many other features that can be added in the future in the map. For example, marker customization and pop-ups to show more information, marker filtering based on relational data, clustering of the markers both in front-end and back-end part, or using geocoding services to add marker locations starting from an address and not manually, and so on. At the end of the talk, before saying goodbye, I want to share with you some tips based on my experience as a map developer with Django. The first one is read the documentation in the Django website, because it's full information about the GeoDjango. Read also details about geographic queries in the post-JS website, because it helps you to understand how things work at a lower level. Read also the source code of both projects on GitHub because there is something you can learn only from the code. And I also recommend to search for questions on Stack Exchange by trying to answer them by yourself and not reading the answer. It's a good way to improve. Last but not least, you can also study this presentation because it's released with Creative Commons license. And I'll share the link in a bit. So in 20.tab, we have developed many maps with Django. You can find more about our open source project and map project of our platonic works using this context. And finally, to find out more of my personal works with PostGIS, Django, and maps, you can use all my contacts. With this QR code, you can download directly this presentation from my website. So thanks again for having me, and enjoy the other talks in the conference.

Speaker 2 [21:48]

Thank you so much Paulo I'm going to jump into the questions on Slido

Speaker 1 [21:53]

I have one request, if you don't mind. If I can take a picture with you all. It's a thing I do every time.

Speaker 2 [22:10]

Great, okay ready for the questions. Yeah, okay Can we extend the map widget in Django admin page for example adding stuff like scale bar map legend and North Arrow?

Speaker 1 [22:24]

Yes, we can do that and the widgets in the admin, it's ready to use but it's built using another JavaScript library, open layers, it's an historic choice because it started from Django 1.0. It worked very well but you have to learn another way to do things if you are using leaflet for your front-end page but it work we with it

Speaker 2 [23:02]

Any use case where a SQLite with spatial extension would not be enough and need to use advanced databases like PostGIS.

Speaker 1 [23:13]

Yeah, I used SQLite and Specialite to have an example. If you have a small map with few points, you don't need a distributed database with very good performance. It's the best choice you can do. But in the second part, I show you how to use PostGIS because the performance are better and also it's more complete from the point of view of function expression you can use directly from GeoDjango. So there is some queries you can do using only Specialized. They are very specific but if you start doing maps more in deep maybe you need something very specific at a certain point. If you need only to show some point in the map, Specialized is enough.

Speaker 2 [24:11]

So Christian wants to know do you have any experience how useful GeoDjango would be considering map campus maps or even building or floor plans

Speaker 1 [24:22]

campus maps

Speaker 2 [24:23]

Yeah, so maybe indoor maps.

Speaker 1 [24:27]

never thought about that what we we are doing a project with layer under the level of the ground so layer a different altitude to is an geocoding and is a project that studied the layer of the terrains and under the under heat so it's not in indoor but it's different from the ground

Speaker 2 [25:00]

Another question, how can you deploy the app using Django for around 200 users in a company?

Speaker 1 [25:09]

To enter the user? Yeah.

Speaker 2 [25:10]

Yeah, so how do you, I guess, scale the app?

Speaker 1 [25:14]

And I don't think 200 users are a great number.

Speaker 2 [25:19]

No, so the app should be robust to that. Yes. OK. Are there any other questions in the room? If not, we can.

Speaker 1 [25:36]

There was so much markers on the maps that you're

Speaker 2 [25:38]

maps that was very impressive.

Speaker 1 [25:39]

was very impressive to see yeah I like traveling and some some hikes but also when I go abroad I search for highest or lowest points I put also a point in the Death Valley in the US. It is the lowest point in the US. I enjoy working with maps and also going physically in the point.

Speaker 2 [26:17]

I like hiking, too, so that was a great personal project, I suppose. Great, thank you so much, Paolo. Thank you, everyone who participated.