Vector tile serving: Some introductory thoughts & hints for a fast way to serve vector tiles

I. Introduction

In the last few weeks I played around and tested some approaches to  self hosted serve (and style) vector tiles. In this blog entry I share some of my observations to you. Don’t expect a fully designed tutorial, it’s more a collection of starting points for your own tests to serve vector tiles. There are several approaches how to serve vector tiles. Mapbox [1] can be used, also GeoServer  has a vector tile extension [2] In my experiments I used tileserver-gl, an open source tile server developed by Klokantech [3] . Tileserver-gl is a great way to quickly serve vector tiles on an own web server, escaping typically pay per view scenarios. Mapbox for example, which is today’s big player in serving web maps, uses a pay per map-view payment model, charging 0.50 US$ per 1000 map views [4]. With tileserver-gl it is possible to self host you vector map tiles and later embed them in your web map using e.g. Leaflet.js or OpenLayers and tileserver.gl has the great advantage, that it allows using GL styles, a JSON style specification, basically telling your map how it should look [5]

For those of you that never heard about vector tiles, I recommend reading this entry of the OpenStreetMap Wiki: [6]. In its nutshell, vector tiles are a vector based representation of geo-objects. The idea is rather old and appeared shortly after the appearance of GIS systems and first was used by the US Wildfire Service in 1975, in their software  Wetlands Analytical Mapping System (WAMS) [7]. Vector tiles started to “revolutionize” the way how map data is served in the WWW, when Google introduced vector tiles to serve Google maps in 2010  [7].

II. Prerequisites

If you plan to serve some vector tiles and would like to follow the suggestions of this blog post, you need a running LINUX server with docker installed on the server. For my tries I use a LINUX V-Server from Strato [8], a Berlin based company belonging to 1&1 (as far as I know). They offer some interesting and cheap virtual server packages. I recently  switched from UnitedHoster to STRATO . Of course, you can use your own server, or Ubuntu from the Amazon AWS market place [9].  As you can see, there are plenty of other hosting companies out there. For some testing, a starter package normally is sufficient, if bandwidth and speed matters, you have to spend more bucks for the fun. 

III. Use OpenMapTiles to get OpenStreetMap(OSM) vector tiles

OpenMaptTiles [10 ] allows downloading vector tiled OSM data in the MBtiles format free of charge with smaller regions; for customized and bigger regions, the service charges a fee, e.g. to download the whole planet.osm in MBtiles format costs 2048 US$ for business users (1024 for individual users, as free lancers).

IV. How can you create vector tiles of your own map data?

The makers of OpenMaptiles published a nice introduction how to create vector tiles using the command line tool Tippecanoe developed by Mapbox [11] What isn’t answered is, how to install Tippecanoe on a LINUX system. First you have to clone the Git repository, Next change to the folder where the repository was cloned to, compile the software and install it.

git clone mapbox/tippecanoe.git 
cd tippecanoe
make -j
make install

After the installation you need files in the GeoJSON format as an input of Tippecanoe.

Ogr2ogr easily allows to convert file formats as the still widely popular ESRI Shapefiles to GeoJSON:

ogr2ogr -f GeoJSON your_data_in_4326.json -t_srs EPSG:4326 your_data.shp

Next Tippecanoe converts the GeoJSON in the MBtiles format in 14 different zoom levels, the MBtiles later can be served using a tileserver.

tippecanoe -o your_data.mbtiles your_data_in_4326.json

If you just need raster tiles for you web map?

In some cases vector tiles aren’t necessary and raster tiles are all you need to quickly visualize something on a slippy map. A quick way to get raster tiles, without zoom-level (scale) dependent rendering of objects would be:  1. Style your data in QGIS . 2. Export your styled map as high resolution png and 3. Tile your date with gdal2tiles:  

gdal2tiles.py -s EPSG:4326  -z 10-16 yourdata.png yourdata_xyz=4

Gdal2tiles creates a folder structure, with subfolders for each zoom level, depending on how much zoom levels you have defined. For quite a while I used the great software Tilemill [12]  to style geodata using carto.css and to create raster tiles. Unfortunately active development of Tilemill stopped a few years ago.

V. How to install and use tileserver.gl?

First the good news, tileserver.gl is available as docker container. In case you want to fight through your way  to install the full stack and not to use docker, you can install the server this way:

  1. Install node.js on Ubuntu Server: DigitalOcean [13]  offers a nice introduction on how to install node.js. To use tileserver.gl, at least the version 6 of node.js is necessary.
  2. Install Tileserver.gl: For the installation this tutorial found on “Ralphs blog” (interesting blog, by the way) was used [14]

Use Klokantechs Docker container

As mentioned above, it is not necessary to install the full stack. You can also use Klokantechs Docker container. If you haven’t installed Docker, Digitalocean also offers great tutorials on how to install Docker on your Ubuntu server [15]Start the container in the same folder your MBTiles are located.  After installing docker, start the docker container from the directory, where you host your MBtiles with the command:

 sudo docker run --rm -it -v $(pwd):/data -p 8080:80 klokantech/tileserver-gl -c config.json

The map servers frontpage, with access to the styles and the data,  can be opened via your servers IP and the port number 8080 by default. Here you can see the frontpage of my test server: http://h2800220.stratoserver.net:8080/ The behavior of the map, e.g. the map style can be controlled in the style.json file. In the configuration file, the style.json file has to be referenced.  

VI. Style your map with Maputnik

Styles are controlled by changing the style.json file and have to be referenced in the config.json. How can you change this file, to get an own style for a custom web map? Styles can be changed using Maputnik [16], an open source style editor for Mapbox styles. Maputnik can be used in the browser as online editor, but does not allow adding vector tiles served with http, just https. On my test server I did not activate https, but the offline version of Maputnik also accepts http as data source. Maputnik just hast to be downloaded from: https://github.com/maputnik/editor/releases/tag/v1.5.0 and extracted, than it can be used locally in a browser. Edit the style as you want and then export the style.json and upload it to your server. I recommend to start with an existing style and to adapt it to your needs. After modifying the style, upload the style.json file to your server, don’t forget to reference the config.json and restart the docker container, so that the changes take effect.  Just an observation, Maputnik looks and feels a lot like the Mapbox Studio [17] , the usage is quite intuitive.

And finally I got  a self hosted vector map (of Managua, Nicaragua) with some edits of the nice dark matter style (which is just a product of playing around a little with the style):  http://h2800220.stratoserver.net:8080/styles/test/#15/12.14186/-86.28059

In my next blog post I plan to write a little more about the GL style definition.

 

References

[1] https://www.mapbox.com/vector-tiles/

[2] https://docs.geoserver.org/stable/en/user/extensions/vectortiles/

[3] https://github.com/klokantech/tileserver-gl

[4] https://www.mapbox.com/pricing/?utm_source=chko&utm_medium=search&utm_content=MapboxPricePlan&utm_campaign=CHKO-PR01-BR-Mapbox-INT-Exact&gclid=CjwKCAiAlvnfBRA1EiwAVOEgfMMmTtJem8_gOUFz8gE5S3RshxEqdPOON_Z8OiCGzV6Ht1O2BmzI6xoCb5EQAvD_BwE

[5]  https://www.mapbox.com/mapbox-gl-js/style-spec/

[6] https://wiki.openstreetmap.org/wiki/Vector_tiles

[7] https://en.wikipedia.org/wiki/Vector_tiles#History

[8] https://www.strato.de/server/linux-vserver/

[10  ]https://openmaptiles.com/

[11] https://openmaptiles.org/docs/generate/custom-vector-from-shapefile-geojson/

[12] https://github.com/tilemill-project/tilemill

[13] https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04

[14] https://golb.hplar.ch/2018/07/self-hosted-tile-server.html

[15]https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04

[16] https://maputnik.github.io/

[17] https://www.mapbox.com/studio/

 

 

Turf.js for Inverse distance weighted interpolation (IDW) of Potsdam´s base rents. Web mapping playground part IV

Introduction

In this blog post I’ll show how an inverse distance weighted interpolator (IDW) can be applied with Turf.js. My blog post can serve as a tutorial for you, to try yourself  to interpolate own data sets with Turf.js  (e.g. temperature data).

The example uses a set of geocoded rental data of 3-room, apartments located in the city of Potsdam (fetched on the 19th of March).

From an analytical viewpoint, the resulting map isn’t that useful and just shows a spatial trend of the rental offers rental costs in € without service costs. Absolute prices had to be interpolated, as the apartment sizes in the fetched data were missing.

IDW actually is one of the least applicable spatial interpolators for rental data, due to known effects as bull eyes;  but Turf.js´s client side processing is in heavy development and for sure other interpolation algorithms will be integrated soon. So far Turf.js is capable to interpolate isolines, plane points, triangulated irregular networks and IDW. If you are interested how IDW works, I can recommend the interpolation tutorial on Gitta Info: http://www.gitta.info/ContiSpatVar/de/html/Interpolatio_learningObject2.xhtml

The map

Click on the map to get a price estimation per grid cell.

A full screen map can be accessed via:

http://hatschit.alkaid.uberspace.de/IDW_TURF_Choropletes/Potsdam_example.html

Turf.js IDW example

The resulting map is styled as choropleth map, using the leaflet.js choroplet plugin. The example geoprocesses the GeoJson data with the turf.js´s IDW plugin by Michele Ferreti. His turf-inverse-distance-weighting module was accepted as part of the turf.js library. He also hosts an example of a Leaflet.js, Turf.js IDW vizualisation, which served as basis for my tutorial.

Relevant code blocks

In the body of the document, several JavaScript libraries have to be integrated. That’s first leaflet.js for the map canvas and the functions of the slippy map, second turf.js,for the interpolation and choropleth.js, that styles the resulting map as choropleth map and creates the map legend.

 <script src="leaflet/leaflet.js"></script>
  <script src="choropleth.js"></script>
  <script src='https://npmcdn.com/@turf/turf@3.13.2/turf.min.js'></script>

After setting the parameters for the leaflet.js map canvas and after integrating the TileMapService for the base map,
the GeoJson file with the points for the interpolation has to be integrated to the script. Unlikely external referencing
via Ajax can’t pass the points to the interpolator. Therefore, the script soon can get quite long, which can be confusing.

<script>// <![CDATA[
var map = L.map('map').setView([52.39, 13.06], 12)
 
    // Add basemap
    L.tileLayer('http://{s}.tiles.wmflabs.org/bw-mapnik/{z}/{x}/{y}.png', {
      maxZoom: 18,
      attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>'
    }).addTo(map)
 
    // Add Sampled Point with known 'value'
    var sampledPoints = {
      "type": "FeatureCollection",
      "crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
      "features": [
      { "type": "Feature", "properties": { "field_1": 3, "latitude": 52.395768, "longitude": 13.065812, "price_high": 1480.0, "room_numbe": "3" }, "geometry": { "type": "Point", "coordinates": [ 13.065812, 52.395768 ] } },
      { "type": "Feature", "properties": { "field_1": 5, "latitude": 52.42388, "longitude": 13.057468, "price_high": 1145.0, "room_numbe": "3" }, "geometry": { "type": "Point", "coordinates": [ 13.057468, 52.42388 ] } },

In the next section, the actual interpolation is realized by the turf.idw function. The attribute of the GeoJson to be interpolated, the power of the interpolation, the cell size of the resulting grid and the measurement units have to be defined.

 
var Grid = turf.idw(sampledPoints,'price_high', 0.3, 0.5,'kilometers');
 
    L.geoJson(sampledPoints.features, {
      onEachFeature: function(feature, layer) {
        layer.bindPopup("Estimated base rent for a three room apartment " + feature.properties.value)
      }
    })

Finally, the settings of the choropleth map have to be defined. The legend automatically is created from the interpolated values. As in my last example, the legend has to be improved.

var legend = L.control({
      position: 'bottomright'
    })
    legend.onAdd = function(map) {
      var div = L.DomUtil.create('div', 'info legend')
      var limits = choroplethLayer.options.limits
      var colors = choroplethLayer.options.colors
      var labels = []
      // Add min & max
      div.innerHTML = '<div class="labels"><div class="min">' + limits[0] + '</div> \
			<div class="max">' + limits[limits.length - 1] + '</div></div>'
      limits.forEach(function(limit, index) {
        labels.push('<li style="background-color: ' + colors[index] + '"></li>')
      })
      div.innerHTML += '<ul>' + labels.join('') + '</ul>'
      return div
    }
    legend.addTo(map)

To try IDW with turf.js with own data, the script and the data packed to a zip file can be downloaded here:
hatschit.alkaid.uberspace.de/IDW_TURF_Choropletes.zip

GeoJSON Live feed of earthquakes with marker clusters: Web mapping playground part III

Full screen web map:
http://hatschit.alkaid.uberspace.de/Earthquake_Live_Mapbox.html

About the map

In this blog post I´ll show a live updating web map of several thousend marker-clustered live streamed seismic activities.

Just click on one of the numbered signatures showing the number of activities within the cluster and the map zooms and the marker spans.
The seismic activities are provided by the United States Geological Survey (USGS) in the geojson format. The USGS provides several live geojson feeds of earthquakes (live, daily and monthly feeds updated incremental): http://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php

My web map marker-clusters all earthquake activities from magnitude 1 to >4.5 in Richter magnitude scale, in the last month, events are updated every 15 minutes. The web map basically was created by using Mapbox´s JavaScript library: https://www.mapbox.com/ and by using a Leaflet.js plugin. Mapbox.js and Leaflet.js are interoperable, as Leaflet.js genial main developer Vladimir Agafonkin (@mourner) works for Mapbox. To use Mapbox, you need a Mapbox access token. Mapbox has a price per user payment model, similar to the Google
maps API. Up to 50k users per map is for free, for more useres mapbox charges according to their pricing plan: https://www.mapbox.com/pricing/

Making your own Mapbox.js or Leaflet.js  web map

If you want starting to develop web maps as well, you should know basics of HTML and JavaScript. As starting point, I can recommend the online courses at Codeacademy: https://www.codecademy.com/learn/javascript
Codeacademy should enable you to start using web mapping libraries. I also recommend Nurma Gremlins
course material for beginners, recently in our February MaptimeBER sessions he did a basic leaflet.js intro . Here you can find his slides: http://geosysnet.de/en/downloads.html If you are interested in further readings about marker clusters, I can recommend Seb Meiers paper “The marker cluster”:
http://www.sebastianmeier.eu/2016/01/30/the-marker-cluster/

Relevant code blocks

Next I comment some relevant pieces of the code, just in case you plan to do a similar web map as the one shown above. In the first code block, two JavaScript libraries are included into the HTML document. In the head of the HTML document I included the Mapbox.js library for providing a base map, and the Leaflet Marker cluster plugin (https://github.com/Leaflet/Leaflet.markercluster), the plugin that cares for the clustering of the clusters. This plugin actually is just one possibility for marker clustering, several alternatives are listed at the Leaflet.js website: http://leafletjs.com/plugins.html#clusteringdecluttering

<script src='https://api.mapbox.com/mapbox.js/v3.0.1/mapbox.js'></script>
<script src='https://api.mapbox.com/mapbox.js/plugins/leaflet-markercluster/ v1.0.0/leaflet.markercluster.js'></script>

In the next section of the code, the body of the HTML document, the JavaScript part starts with the script tag and hidden (the x) part of my Mapbox access token is shown. In the .set view part the zoom level and the center coordinate of the web map is shown.

<style>
body { margin:0; padding:0; }<br /> #map { position:absolute; top:0; bottom:0; width:100%; }<br /></style>
<style>
<div id="map"></div>
<script>
L.mapbox.accessToken = 'xxxxxxxxxx';
var map = L.mapbox.map('map', 'mapbox.streets')
   .setView([38.1089, 13.3545], 2);

The next line provides a nice dynamic scale bar.

L.control.scale().addTo(map);

Here the Geojson file, provided by the USGS is fetched.

var featureLayer = L.mapbox.featureLayer()
   .loadURL('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson')
   .on("ready", function()

Finally the markers of the seismic events (several thousand events) are clustered Leaflet.js´s marker cluster plugin and and the html document ends.

 
{
     var markers = L.markerClusterGroup();
     markers.addLayer(featureLayer);
     markers.addTo(map);
       });
 
</script>

I´ll take a two weeks break from blogging, as I will go for a Erasmus+ stay to the Czech Republic, I´ll be back with a trip report :-).

Visualizing the third dimension: Overview and QGIS2threejs example. Web mapping playground part II

About ten years ago, in the boom years of 3D geodata visualization, expensive expert systems appeared. Systems as the costly LandeXplorer, developed by 3DGeo in Potsdam, later acquired by Autocad, were developed to visualize the z-dimension of geodata. Proprietary Arc Scene could be used, though Arc Scene was not as powerful as LandXplorer, but  quite OK, unlikely users needed a license from ESRI.

Open source alternatives

Only a few open source alternatives existed. Paraview should be mentioned . Paraview, which I used to map soil contamination, improved a lot and nowadays it´s worth a look. Nviz, a GRASS GIS module for 3D could be used and Fledermaus, a system for the 4-dimensional rendering of geodata were alternatives.

All in all visualizing the third dimension was not affordable or a pain in the ass (sorry for the hard words). 

Three.js revolution

Visualizing the third, or better the two and a half dimension in a browser was not imaginable. Later WebGL appeared, but the learning curve for WebGL, just for visualizing some 3D data, was really high. Fortunately Three.js was developed. Three.js according to Wikipedia is “a cross-browser JavaScript library/API used to create and display animated 3D computer graphics in a web browser.” Three.js really facilitated 3D mapping and unexpectedly a new tool was out there, to do web based three-dimensional mapping of geodata.
A plugin for QGIS was developed, to use three.js within QGIS: http://qgis2threejs.readthedocs.io/
QGIS2three.js provides a powerful, extremely versatile tool for a  visualization of z-dimensional data within web browsers.

QGIS2Three.js example

The following example was done with QGIS2three.js. The plugin leads the user and pre-writes the three.js code, ready to upload as a web map. The JavaScript code later can be customized e.g. using an editor as ATOM. My QGIS2Threejs example shows landslides in Rio Blanco-Nicaragua mapped in a landslide inventory, in which I took part during my master thesis project in 2005. The landslide polygons are mapped on a ASTER Global DEM, which was exaggerated by the scale of 3.

Even quadtrees can be built for data compression purposes. So the plugin gives the user a powerful open source tool, for developing all kind of 3D (geo-)data visualization usable within web browsers.

Bildschirmfoto 2017-02-19 um 14.17.44

If you plan to do some 3D mapping of geodata for the web, my recommendation is, take a look on QGIS2Three.js.

 

 

 

 

Thematic isolines with turf.js: Webmapping Playground part I

In the next weeks I plan to publish some past and recently made leaflet.js, mapbox.js and carto webmap examples, hosted on uberspace or my Vserver. The maps will show prototypes of different development stages made for test purposes.

As kick off, I show a map created with turf.js, a JavaScript library for “advanced geospatial analysis”.

The map shows isolines of equal rental prices (without service costs) in Berlin. The isolines are generated by a client side processing of approximately 3000 data points (from November 2016). The map legend is still work in progress, in my opinion the making of custom legends in Mapbox.js should be improved (adaptive legends would be great). Watch out, loading the map takes a while, as the size is 5.6 MB. The isolines are rendered from the data points.

The Isolineshave been generated by using the turf.js isoline function: http://turfjs.org/docs.html#isolines

I encountered two major problems in making this web map:

    1. The alignment of the legend was quite troublesome
    2. Processing a huge number of data points with JavaScript on a client can take quite a while

Consider this map as a prototype of what can be done with Turf.js; which in my eyes is a very interesting JavaScript library.

The full screen webmap also can be opened via: http://hatschit.alkaid.uberspace.de/Turf_tin_3000_isolines_colors.html

 

Introducing to Leaflet.js at Maptime BER

I did a basic introduction to Leaflet.js at Maptime BER. The goal was to give a short introduction on how to make a webmap with Leaflet.js. About 40 mappers, from novices to experts, attented the event at the IXDS Berlin. we talked anput Map API´s in general and I should code examples and explained how to integrate different Tilemap Services into leaflet and how to make basic markers and popups with some content :-). 90 minutes hardly have been enough to demonstrate the basic ideas of leaflet. Beside fighting with technical problems, that made me even switch to a Mac, the tutorial was fun and I hope the participants will profit from it. The IXDS in Kreuzberg was a great location! At the end we had a very basic webmap with a marker of the location of our event, here you can see the result: https://schernthanner.de/Demo.html

The examples (scripts) kann be downloaded at:  https://schernthanner.de/Maptime_Leaflet_Intro.zip

Here you can see the slides of the tutorial:

 

Webmapping Seminar an der Universität Potsdam

Vom 17.9 bis 19.9 ging das erste von mir organisierte Webmapping Seminar an der Uni Potsdam über die Bühne. In ca. 21 Stunden gab es für 15 Teilnehmer eine Einführung in die Bereitstellung von OGC Services mit dem Geoserver, in die Installaton des Geoserver und eine Einführung in die Leaflet Map API. Wir stellten einen WMTS, einen WFS und einen WCS bereit und besprachen die Potentiale von Webprocessing Services. Wir sahen uns Styled Layer Descriptor an, als Möglichkeit Geodaten zu stylen und lernten als Alternative Carto CSSkennen. Maptiles wurden mit Tilemill erstellt. Mit Leaflet haben wir Webmaps anhand von Beispielcode erstellt. Geojson in Leaflet eingebunden und unsere Karten für das Smartphone angepasst, Photon als Geocoder genutzt und einige tolle Plugins kennengelernt. Christian Kuntsch zeigte uns wie wir aus Twitterfeeds GeoJSON ableiten und Patrick Voland gab eine fundierte CartoDB Einführung, zeigte uns torque.js und wie man den Photon selbst installieren könnte. Ich freue mich auf die Abschlussprojekte der Studenten.