Beiträge von hatschito

My names is Harald Schernthanner and I am a research associate at the Geoinformatics Research Group at the University of Potsdam. I am working in the fields of spatial analysis and visualization of real estate prices, applied remote sensing ,GIS and GIS technologies on the web. I have several years of teaching experience and I hold seminaries at universities masters level. I'm am holding a M.Sc. in the field of applied Geoinformation from the university of Salzburg and a Ph.D. in the field of Geoinformatics from the university of Potsdam. The research topic I followed in my PhD. thesis was the spatial analyis and visualization of rental data for real estate portals.

Vector tile serving: Some introductory thoughts & hints for a fast way to serve vector tiles

I. Introduction

In the last few weeks I played around and tested some approaches to  self hosted serve (and style) vector tiles. In this blog entry I share some of my observations to you. Don’t expect a fully designed tutorial, it’s more a collection of starting points for your own tests to serve vector tiles. There are several approaches how to serve vector tiles. Mapbox [1] can be used, also GeoServer  has a vector tile extension [2] In my experiments I used tileserver-gl, an open source tile server developed by Klokantech [3] . Tileserver-gl is a great way to quickly serve vector tiles on an own web server, escaping typically pay per view scenarios. Mapbox for example, which is today’s big player in serving web maps, uses a pay per map-view payment model, charging 0.50 US$ per 1000 map views [4]. With tileserver-gl it is possible to self host you vector map tiles and later embed them in your web map using e.g. Leaflet.js or OpenLayers and tileserver.gl has the great advantage, that it allows using GL styles, a JSON style specification, basically telling your map how it should look [5]

For those of you that never heard about vector tiles, I recommend reading this entry of the OpenStreetMap Wiki: [6]. In its nutshell, vector tiles are a vector based representation of geo-objects. The idea is rather old and appeared shortly after the appearance of GIS systems and first was used by the US Wildfire Service in 1975, in their software  Wetlands Analytical Mapping System (WAMS) [7]. Vector tiles started to “revolutionize” the way how map data is served in the WWW, when Google introduced vector tiles to serve Google maps in 2010  [7].

II. Prerequisites

If you plan to serve some vector tiles and would like to follow the suggestions of this blog post, you need a running LINUX server with docker installed on the server. For my tries I use a LINUX V-Server from Strato [8], a Berlin based company belonging to 1&1 (as far as I know). They offer some interesting and cheap virtual server packages. I recently  switched from UnitedHoster to STRATO . Of course, you can use your own server, or Ubuntu from the Amazon AWS market place [9].  As you can see, there are plenty of other hosting companies out there. For some testing, a starter package normally is sufficient, if bandwidth and speed matters, you have to spend more bucks for the fun. 

III. Use OpenMapTiles to get OpenStreetMap(OSM) vector tiles

OpenMaptTiles [10 ] allows downloading vector tiled OSM data in the MBtiles format free of charge with smaller regions; for customized and bigger regions, the service charges a fee, e.g. to download the whole planet.osm in MBtiles format costs 2048 US$ for business users (1024 for individual users, as free lancers).

IV. How can you create vector tiles of your own map data?

The makers of OpenMaptiles published a nice introduction how to create vector tiles using the command line tool Tippecanoe developed by Mapbox [11] What isn’t answered is, how to install Tippecanoe on a LINUX system. First you have to clone the Git repository, Next change to the folder where the repository was cloned to, compile the software and install it.

git clone mapbox/tippecanoe.git 
cd tippecanoe
make -j
make install

After the installation you need files in the GeoJSON format as an input of Tippecanoe.

Ogr2ogr easily allows to convert file formats as the still widely popular ESRI Shapefiles to GeoJSON:

ogr2ogr -f GeoJSON your_data_in_4326.json -t_srs EPSG:4326 your_data.shp

Next Tippecanoe converts the GeoJSON in the MBtiles format in 14 different zoom levels, the MBtiles later can be served using a tileserver.

tippecanoe -o your_data.mbtiles your_data_in_4326.json

If you just need raster tiles for you web map?

In some cases vector tiles aren’t necessary and raster tiles are all you need to quickly visualize something on a slippy map. A quick way to get raster tiles, without zoom-level (scale) dependent rendering of objects would be:  1. Style your data in QGIS . 2. Export your styled map as high resolution png and 3. Tile your date with gdal2tiles:  

gdal2tiles.py -s EPSG:4326  -z 10-16 yourdata.png yourdata_xyz=4

Gdal2tiles creates a folder structure, with subfolders for each zoom level, depending on how much zoom levels you have defined. For quite a while I used the great software Tilemill [12]  to style geodata using carto.css and to create raster tiles. Unfortunately active development of Tilemill stopped a few years ago.

V. How to install and use tileserver.gl?

First the good news, tileserver.gl is available as docker container. In case you want to fight through your way  to install the full stack and not to use docker, you can install the server this way:

  1. Install node.js on Ubuntu Server: DigitalOcean [13]  offers a nice introduction on how to install node.js. To use tileserver.gl, at least the version 6 of node.js is necessary.
  2. Install Tileserver.gl: For the installation this tutorial found on “Ralphs blog” (interesting blog, by the way) was used [14]

Use Klokantechs Docker container

As mentioned above, it is not necessary to install the full stack. You can also use Klokantechs Docker container. If you haven’t installed Docker, Digitalocean also offers great tutorials on how to install Docker on your Ubuntu server [15]Start the container in the same folder your MBTiles are located.  After installing docker, start the docker container from the directory, where you host your MBtiles with the command:

 sudo docker run --rm -it -v $(pwd):/data -p 8080:80 klokantech/tileserver-gl -c config.json

The map servers frontpage, with access to the styles and the data,  can be opened via your servers IP and the port number 8080 by default. Here you can see the frontpage of my test server: http://h2800220.stratoserver.net:8080/ The behavior of the map, e.g. the map style can be controlled in the style.json file. In the configuration file, the style.json file has to be referenced.  

VI. Style your map with Maputnik

Styles are controlled by changing the style.json file and have to be referenced in the config.json. How can you change this file, to get an own style for a custom web map? Styles can be changed using Maputnik [16], an open source style editor for Mapbox styles. Maputnik can be used in the browser as online editor, but does not allow adding vector tiles served with http, just https. On my test server I did not activate https, but the offline version of Maputnik also accepts http as data source. Maputnik just hast to be downloaded from: https://github.com/maputnik/editor/releases/tag/v1.5.0 and extracted, than it can be used locally in a browser. Edit the style as you want and then export the style.json and upload it to your server. I recommend to start with an existing style and to adapt it to your needs. After modifying the style, upload the style.json file to your server, don’t forget to reference the config.json and restart the docker container, so that the changes take effect.  Just an observation, Maputnik looks and feels a lot like the Mapbox Studio [17] , the usage is quite intuitive.

And finally I got  a self hosted vector map (of Managua, Nicaragua) with some edits of the nice dark matter style (which is just a product of playing around a little with the style):  http://h2800220.stratoserver.net:8080/styles/test/#15/12.14186/-86.28059

In my next blog post I plan to write a little more about the GL style definition.

 

References

[1] https://www.mapbox.com/vector-tiles/

[2] https://docs.geoserver.org/stable/en/user/extensions/vectortiles/

[3] https://github.com/klokantech/tileserver-gl

[4] https://www.mapbox.com/pricing/?utm_source=chko&utm_medium=search&utm_content=MapboxPricePlan&utm_campaign=CHKO-PR01-BR-Mapbox-INT-Exact&gclid=CjwKCAiAlvnfBRA1EiwAVOEgfMMmTtJem8_gOUFz8gE5S3RshxEqdPOON_Z8OiCGzV6Ht1O2BmzI6xoCb5EQAvD_BwE

[5]  https://www.mapbox.com/mapbox-gl-js/style-spec/

[6] https://wiki.openstreetmap.org/wiki/Vector_tiles

[7] https://en.wikipedia.org/wiki/Vector_tiles#History

[8] https://www.strato.de/server/linux-vserver/

[10  ]https://openmaptiles.com/

[11] https://openmaptiles.org/docs/generate/custom-vector-from-shapefile-geojson/

[12] https://github.com/tilemill-project/tilemill

[13] https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04

[14] https://golb.hplar.ch/2018/07/self-hosted-tile-server.html

[15]https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04

[16] https://maputnik.github.io/

[17] https://www.mapbox.com/studio/

 

 

Learn to geocode addresses in 5 minutes

Geocode a list of addresses in 5 Minutes: a beginners guide

Actually I first planned to show this little 5 minutes guide in our last Maptime Berlin session (September 2018), unfortunately I could not make it, so I thought: Let‘s write a little blog post about geocoding addresses. I will quickly explain the concepts of what geocoding means, demonstrate how to geocode a list of addresses with the great MMQGIS plugin and I also  give hints as starting point for some more efficient address-geocoding.

What is geocoding?

So what is geocoding? Imagine you have a list of addresses and you want to locate them on a map. This conversion into a geographic coordinate is called “geocoding”. The other way round is named reverse geocding. Revers geocoding converts a coordinate to a street address. Many geocoding services are out there: ESRI has their own, Google, HERE, Bing,TomTom and also a great  open one,, Nominatim, is part of the OpenStreetMap project. Some of them are really performant, but also cost a lot.

If you are interested in the principles behind geocoding and the underlying algorithms, I recommend the article by Goldberg et al.:

Goldberg, D. W., Wilson, J. P., & Knoblock, C. A. (2007). From text to geographic coordinates: the current state of geocoding. URISA-WASHINGTON DC-, 19(1), 33, available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.3589&rep=rep1&type=pdf#page=34

Very simplified the main core of geocoding algorithms is the fuzzy matching of two strings, with some distance calculations in between them.

What are applications for geocoding addresses? Literally you can connect any street address with a geographic coordinate. Applications can be found e.g. in geo-marketing to locate costumers.

Where to get the addresses from? Enterprises as Schober sell address data. The city you live normally trades your address data and sells your registration to the German Post. Beside the mentioned enterprises a great open address service exists: Open addresses, an open data address collection: http://openaddresses.io/ The service covers around ~500 Mioaddresses. For Berlin the service covers good data with around 375 000 entries.

Geocode using the MMQGIS QGIS plugin

So let’s dive in into the practical part. To demonstrate you how to geocode a short list of addresses, I downloaded the list of addresses of all polling stations for Potsdam’s mayors election, that take place tomorrow (21st of September). The data was published on Potsdam’s open data portal: https://opendata.potsdam.de/explore/dataset/wahlbezirke_wahllokale/

As download format I have chosen “csv” and what I get, is a semicolon separated csv table. To use the data with the MMQGIS plugin, some data cleaning had to be done. You can use an editor (as Atom, Gedit, or Notepad++) and convert the semicolons to colons (needed by the plugin). Just find and and replace them. Street-name and house-numbers are separated and  have to be concatenated (e.g. in LibreOffice). If not, MMQGIS does not allow to geocode down to the house number. Further a city field is recommended, otherwise the geocoder will search world wide and match addresses with the same name. So finally an extract of the list would look like this:

 

 

Next you have to start QGIS and install the MMQGIS plugin. MMQGIS is a really great plugin for vector data manipulation developed by Michael Minn: http://michaelminn.com/linux/mmqgis/

Once you installed the plugin, you can start the GUI with QGIS -> MMQGIS -> Geocode

Two geocoding services can be chosen, the proprietary google service and Nominatim. The google service requires an API key. According to their latest price plan, google charges nothing up to 1000 requests, than they charge 0,50 US cent per 1000 requests up to 100 000 daily.

Nominatim is open source, so for our demo I chose Nominatim from the pull down menu. Watch out: To geocode via a GUI is very slow. So you have to be patient and wait a little while but finally Nominatim manages to find 103 of 130 poll stations in Potsdam. Below you can see a screenshot of the result.

Some final remarks: Make it more efficient

Geocoing with MMQGIS is recommended, when it should be really quick and when you just have a few addresses. For bigger lists use a script based approach. Suitable libraries are for example geopy: https://github.com/geopy/geopy For R I found this nice blog post describing how to geocode with R and providing a nice script: http://www.storybench.org/geocode-csv-addresses-r/    I used the script already several times and it works just fine. I am sure there are more libraries out there, I am not aware of.

For constant maybe automatized geocoding of mass data, I recommend a quick and robust service as google or an own Nominatim instance on a server. But how to set up a Nominatim instance? A tutorial can be found here: http://nominatim.org/release-docs/latest/admin/Installation/ Also Photon comes to my mind, to speed up geocoding-tasks: Komoot, a Potsdam based company, that offers navigation for bikers and hikers provided Photon, an alternative to Nominatim (or built on top of Nominatim): https://github.com/komoot/photon

 

Playing around with open addresses

For fun I played around with open addresses, the service I mentioned above, and downloaded the address list of Berlin (~8.7 MB). The list is accessible from the open addresses website: https://s3.amazonaws.com/data.openaddresses.io/runs/491996/de/berlin.zip The csv contains around 375k addresses of Berlin and the data already has coordinates (this is way too much for MMQGIS). Just as a thought experiment: when I would send a flyer to each of the addresses this would cost (according to the Deutsche Post) 99€ for 1000 flyer. So for 37500 € I could send a flyer to almost each household in Berlin. OK, long live spam for advertising :-).

 

 

 

 

 

 

 

OSM für Anfänger am Potsdamer Workshop am Tag der Wissenschaften 18´

Am Tag der Wissenschaften gaben meine Kollegin Lucia Tyrallová und ich eine an Anfänger gerichtete, praktische Einführung in OpenStreetMap. Unser Workshop wendete sich an Interessierte der freien Weltkarte OpenStreetMap (OSM). Die Teilnehmer lernten, wie Sie mit Ihrem eigenen Smartphone und mit einem PC Geodaten aufnehmen, diese aufbereiten und dem OSM-Projekt zur Verfügung stellen. Neben diesem ‘hands on tutorial’ wurde demonstriert, wie OSM-Daten in der Wissenschaft zum Beispiel für die Fußgängernavigation genutzt wird.

Link zur Ankündigung: https://www.potsdamertagderwissenschaften.de/programm/openstreetmap-osm-mapping-workshop-fuer-beginner/

Unsere Präsentation ist hier zu finden: https://bit.ly/2rlJvSx

 

 

 

 

Meine Buchveröffentlichung: Räumliche Analyse und Visualisierung von Mietpreisdaten, Geoinformatische Studie zur räumlichen Optimierung von Immobilienportalen

Meine Dissertation wurde von Springer Spektrum als Buch veröffentlicht. Unter dem folgenden Link findet man die Veröffentlichung: http://www.springer.com/de/book/9783658177737#springer

Das Inhaltsverzeichnis kann man hier herunterladen: http://bit.ly/2p0Aj6F

Einen Produktflyer findet man hier: http://bit.ly/2ovHeAM

 

Über das Buch

Harald Schernthanner verfolgt das Ziel, aus geoinformatischer Sicht eine konzeptionelle Grundlage zur räumlichen Optimierung von Immobilienportalen zu schaffen. Dabei geht der Autor davon aus, dass Verfahren der räumlichen Statistik und des maschinellen Lernens zur Mietpreisschätzung sich besser als die bisher eingesetzten Verfahren der hedonischen Regression zur räumlichen Optimierung von Immobilienportalen eignen. Er zeigt, dass die von Immobilienportalen publizierten webbasierten Mietpreiskarten nicht die tatsächlichen räumlichen Verhältnisse auf Immobilienmärkten wiedergeben. Alternative webbasierte Darstellungsformen, wie beispielsweise „Gridmaps“, sind dem Status quo der Immobilienpreiskarten von Immobilienportalen überlegen und visualisieren die tatsächlichen räumlichen Verhältnisse von Immobilienpreisen zweckmäßiger.

Web Feature Service: Rental offers from Berlin and it’s surroundings

Since September 2017 I fetch rental data from Berlin and it’s surroundings (e.g. Potsdam, Ludwigsfelde, Oranienburg). I hook myself to the Nestoria portal and parse their data to the shapefile format ; shapefile is used for the convenience of several R-libraries I use.

So I thought, why not to publish some of the data via my Geoserver instance as WFS. I chose the following daily snapshots: 7.11.17, 22.12.17, 23.12.17, 19.1.18 From time to time I´ll add more layers. Using this url you can add the datasets as WFS layer to your favorite GIS: http://31.172.93.180:8080/geoserver/Rental_data/wfs?version=1.1.0&layers=Rental_data

Have fun with the data :-). If you need more or all data I collected till September, just contact me.

Each data set covers around 7000 geocoded rental offers, coming from different providers. I’ll go on collecting, as I want to observe quarterly changes, the standard interval in rental market observation; let´s see maybe I can discover some interesting spatial patterns after longer observation period. Maybe someone of you has ideas, what else can be done with the data. Just as eye catcher I did the heat map below, unfortunately I can’t figure out how to create a legend of the heat map in QGIS (2.18) – so basically red means a high base rent per square metre. It seems that heat maps in QGIS don´t have an option to add a legend. The grey points represent the rental offers locations.

Refer to this older blog post, if you are interested in the data-collection method script: https://schernthanner.de/shell-script-to-automate-downloading-of-rental-data-using-wget-jq-a-cronjob

Playing around with the Google Earth Engine: Canny edge detection realized within just a few minutes

Yesterday, after a few months of not  paying attention to the Google Earth Engine (in March I did some NDVI time series with MODIS data), by accident I stumbled over this remote sensing image processing platform in the “cloud” and used Saturdays rainy November weather to try a few things.

Some preliminary thoughts about Google´s Earth EnginE

The Google Earth Engine is a seductive tool for remote sensing enthusiasts. I remember a workshop organized by the German Aerospace Service (DLR) I attended in spring, where the organizers several times asked themselves in front of the participants, how Google “does it” and how they can achieve this user-friendliness combined with this strong analyzing capabilities, claiming that they are far away to reach Google regarding their efforts bringing image processing to the cloud. Just imagine that you can filter and analyze thousands (petabytes) of image products just with a few lines of code in a few minutes. Googles earth engine makes this possible, it’s still for free — the question is how long, or to we pay with our code? In my opinion google earths engine is worth a look, but don’t invest too much time in coding in their engine — in the end all your code and ideas belongs to them. Better choose a coding language as Python, Matlab or R and process your data in your environment. Is it frightening or good that the world food organization (FAO) in 2015 partnered Google? See: http://www.fao.org/news/story/en/item/350761/icode/

But isn’t google´s slogan: „Don’t be evil“? We will see, I have no answer for it, to you have?

…. But for quick prototyping, image searching and testing of methods the Google Earth Engine without doubt is really interesting, so if you are into remote sensing and know some JavaScript I can recommend giving it a try. @Webkid for example did an interesting project with the Google Earth Engine about the “Greenness of German cities” and documented their workflow on their blog: https://blog.webkid.io/analysing-satellite-images-with-google-earth-engine/ That´s a really good resource for getting started with the Google Earth Engine. If you want to try the Earth Engine, you have to sign up at: https://earthengine.google.com/ After signing up you can explore it’s computation capabilities and data catalogs at: https://explorer.earthengine.google.com/#workspace

The Engine covers a wide range of classical image processing tools (maximum likelihood classification etc.) , machine learning algorithm’s as e.g.support vector machines, convolutions and different edge detectors, just to name a few of possible algorithms. For all algorithms code examples exist. When you are done with exploring the environment, you can start coding using the online code editor: https://code.earthengine.google.com/

Demonstration of google earth engine´s analyzing capabilities

My goal was quickly finding the least cloudy Sentinel 2 image covering Berlin between it’s launch in June 2015 and today and applying a canny edge detector on the least cloudy image.

I shared my repo via: I shared my repos at: https://code.earthengine.google.com/?accept_repo=users/haraldschernthanner/default, so feel free to explore it, but don’t await the most polished code and in depth analysis, I used the engine just for testing some stuff. The code covered in this blog entry has the name Sentinel2_Canny.

To limit my area to play with a region of interest had to be defined ROI has to be defined, otherwise Google calculates globally and this also within minutes. To get the right coordinates of an area of interest http://geojson.io is a very helpful webtool  You can upload e.g. kmls or shapefiles to get the coordinate values of their vertices in order to define your ROI, and you can convert your data to geojson. The Google Earth Engine uses their fusion tables to define geometries, I didn´t follow this approach in this try. Here you can see the code snippet to limit a ROI:

//Limit the ROI to a rectangle around Berlin
var roi = ee.Geometry.Polygon( [[[12.9364013671875, 52.70218656416109],
[12.937774658203125,  52.70218656416109],
[12.937774658203125, 52.70218656416109],
[12.9364013671875, 52.70218656416109], [12.9364013671875, 52.70218656416109]]]);

Google seems to mirror all existing Sentinel-2 images and within seconds you can filter the least cloudy image in a given time span, using this code snippet:

// Get the least cloudy image between 2015 and 11/2017 with the defined ROI
var image = ee.Image(
  sentinel_2_date.filterBounds(roi)
    .filterDate('2015-01-01', '2017-11-23')
    .sort('CLOUD_COVERAGE_ASSESSMENT')
    .first()
);

By printing out a defined image variable using print(image) as command, the metadata of the least cloudy image, dating the 27th of January 2017, can be printed in Google Earths console.

False color composite of Sentinel 2 image: 20170127T102301_20170127T102258_T32UQD

The filtered image can also be downloaded, if you which to change to another processing environment for example to the GIS of your choice. For downloading use:

//Export Sentinel-2 image, Creates a tab named task with a menu to download the image
Export.image.toDrive({
  image: image,
  description: 'S2_leastCloudy',
  scale: 30,
});

If you want to use different images, an overview over google earths image collections can be found here: https://code.earthengine.google.com/datasets For example products from MODIS, ASTER, Landsat or NOAA AVHRR are provided.

So I tried a Canny edge detector, a detector developed by John Canny in 1986. He describes his filter in his paper: Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6), 679-698[Online: http://ieeexplore.ieee.org/abstract/document/4767851] Canny´s edge detector identify s diagonal, vertical and horizontal edges and serves very well to extract coastlines, stream or road networks, building footprints – but please don’t expect that I extract Berlin´s road network in my half an hour trial 🙂 The Canny edge detection uses two parameters, 1. The Sigma parameter is the standard deviation of a Gaussian pre-filter to remove high-frequency noise — Earth engine uses a Hough transform for this purpose and 2. a threshold value defining the minimum gradient magnitude. With this code snippet the Canny filter in it’s most simple form can be applied with the Google Earth Engine. Of course in a real world example much more thoughts about how to chose the parameter values would have to be done.

// Canny edge detection
var canny = ee.Algorithms.CannyEdgeDetector({
  image: image, threshold: 10, sigma: 1
});

After applying the algorithm, the results just have to be visualized, by defining some visualization parameters and by displaying the result on a map – of course google maps in their universe.

// Define the visualization parameters to get a false color
var vizParams = {
  bands: ['B8', 'B3', 'B2'],
  min: -1,
  max: 5000,
  gamma: [0.95, 1.1, 1]
};
// Display the result on map
Map.centerObject(roi, 7);
Map.addLayer(canny, {min: 0, max: 1}, 'Canny Edges');
Map.addLayer(image, vizParams, 'false color composite');

The results looked really promising; so as mentioned before, for quick prototyping this cloud platform is really interesting. Just compare the satellite imagery showing the runway of Schönefeld airport, with the detected edges. That does not look that bad for a “quick and dirty” analysis, right? 🙂

Schoenefeld airfield

Detected edges

ICCSA 2017: Short recap & slides of my presentation as teaser to my forthcoming article

From 3-6 of July I took part in the 17th International Conference on Computational Science and Applications (ICCSA 2017) in Trieste, Italy. ICCSA 2017 is over and as conference participant I had a great time listening to outstanding presentations in the field of computer science, remote sensing and geoinformatics. There have been some very interesting sessions. Just one detail has to criticized in my opinion: In almost every session I was attending, there was at least one speaker missing, so several talks I was curious to listen to, didn’t take place. Nevertheless, all in all the conference and the time with many interesting discussions has been great. I got good feedback on my talks on automated web based geoprocessing of rental data and on the TET-1 fire experiment of the German aerospace service (DLR).

I look forward to Springer’s LNCS book (Lecture notes of computational science and it’s applications), which should be published at the 10th of August.

Meanwhile, for everybody interested and as a little teaser to my forthcoming article, here you can find the slides of my presentation titled: “Automated web-based geoprocessing of rental prices”. Basically this project was a proof of concept of realising the workflow from autmatically fetching, parsing, filtering and after tests of autocorrelation Kriging + mapping rental prices.

Here you can download the slides:  2017_Automated_web_based_processing_Real_estate_data_28_06_17_new_design.pdf (254 Downloads)

Here two impressions from the beautiful venue of Trieste, home of Illy Esspresso:

 

 

Shell script to automate the download of rental data using WGET, JQ & Cron

Until recently I fetched rental data via a script based on R´s Jsonlite library and by accessing Nestoria´s API  (a vertical search engine, that bundles the data of several real estate portals). My first script had to be executed manually; in the next attempt, I started to automate the data downloading, unlikely Nestoria blocked my IP.  I admit, I excessively downloaded data and did the download using a static IP and a  cronjob that has been executed always on a daily basis, on the same daytime. This resulted in a 403 error (IP forbidden, I used a static IP). So together with Nico (@bellackn) an alternative was figured out. Instead of Jsonlite, our shell script uses WGET and makes use of the great JQ tool (CSV to JSON Parser). Thank´s Nico, for the input and ideas.

Next a few of the most relevant lines of code are explained. The entire code can be seen and downloaded from Github: https://github.com/hatschito/Rental_data_download_shell_script_WGET_JQ

We use the  w 60 and –random-wait flag, this tells WGET to either wait 0, 60 or 120 secs to download. This behavior tricks the server. Within WGET also the area of interest is defined. The API allows LAT/LONG in decimal degrees or place names.

wget -w 60 --random-wait -qO- "http://api.nestoria.de/api?country=de&pretty=1&encoding=json&action=search_listings&
place_name=$place&listing_type=rent&page=1";

After that, the first page is downloaded. The first page has to be altered with the sed command (UNIX command to parse text). A while loop does the downloading of the remaining pages, the page number to be downloaded can be modified. We receive JSON files, that have to be parsed to a geodata format.

While Loop:

echo -e "\nOrt: $place\nSeite 1 wird heruntergeladen."
sed '/application/ d' ./rentalprice-$datum.json > clr-rentalprice-$datum.json
 
i=1
while [ $i -le 25 ]
#insert the number of pages you want to download, here: 2 to 28
#(find out how much pages you need/Nestoria offers - the json with "locs" in the file name should have just one comma at the end
#of the file - lower the number according to the odd commas - e.g. for Potsdam, it's 28)
# -----> (you'll also have to comment the deletion of the locs-file way down the script in order to do so...) > ./rentalprice-locs-$datum.json
  printf "," >> ./rentalprice-locs-$datum.json
  i=$[$i+1]
done

Parse JQ to CSV:

JQ, a command line JSON interpreter, parses the JSON to CSV

jq .response.listings < rental_prices-$place-$datum.json | in2csv -f json > CSV-rental_prices.csv

In the following step the data is loaded to a short R script. R´s SP library converts the CSV to a shapefile (actually we will skip this part and in the next version – GDAL should manage the file conversion).
Back in the shell script a timestamp with the current date and time is appended to the shapefile. After some cleaning at the end of the Shell script, finally a Cronjob is created in order to schedule the daily data download. The Cronjob also can be done via a GUI: https://wiki.ubuntuusers.de/GNOME_Schedule/

Still the resulting shapefiles are stored file based, but I plan to hook the script to a PostgreSQL database, that was already installed on my Linux VServer.

Feel free to use our script, if you are interested in downloading geocoded rental data for your analysis and your area of interest. Any feedback or comment is appreciated.

How to get GRASS GIS 7.0.5 working on Mac OS Sierra?

For some image processing tasks (especially I needed  the GRASS GIS r.report command) and inspired by Peter Löwes recent blog post about the origins of GRASS GIS,  I installed GRASS GIS 7.0.5 on my MAC with MAC OS Sierra. The plan was to use GRASS GIS within QGIS. The installation can be tricky 😃.

That´s why I thought, I’ll provide a short tutorial, where I gather all necessary installation steps, collected from different websites. This should help you, to have a smooth installation and to support you in avoiding certain possible pitfalls. The most important parts of the installation are well documented on http://grassmac.wikidot.com/. GRASS GIS for Mac can be downloaded here: http://grassmac.wikidot.com/downloads

Several changes and installations have to be made on your Mac with macOS Sierra to run GRASS GIS:

I. The following frameworks have to be installed:

The frameworks are compiled by William Kyngesburye and can be downloaded from his website: http://www.kyngchaos.com/software:frameworks  Big thanks to William, for porting and compiling QGIS and GRASS GIS for all GIS users! The frameworks also can be downloaded from the sources below  and  should be installed in the order given:

1. GDAL Complete 2.1 (2016-9-17 – 32bit) download
2. GDAL Complete 1.11 (2015-10-26 – 32bit) download
3. FreeType 2.4.12-1 download
4. cairo 1.12.2-1 (Install AFTER GDAL and FreeType) download
5. Numpy 1.8.0-1 download
6. MatPlotLib 1.3.0-3 (32bit) download
7. pandoc 1.13.1 download
8. PIL 1.1.7-4 download
10. Active States TclTk 8.5.7 (only for TclTk NVIZ in GRASS 6) download

II.MAC´s System Integrity Protection feature has to be disabled:

The steps to disable the system integrity protection were taken from http://grassmac.wikidot.com/:

1. Restart your Mac in Recovery Mode. To do this, choose Restart from the Apple menu, and as soon as the screen turns black hold down Command + R on the keyboard until the Apple logo appears on your screen.

2. Select Terminal from the Utilities menu.

3. In the Terminal Window that opens type: csrutil disable
– Press the Return key.
– Choose Restart from the Apple menu.

III. Install Python 2.7 to avoid the well-known error: “Bad CPU type in executable”

This error is well-known and documented: http://bit.ly/2n6BFcl

As workaround, I (re-)installed the latest version of Python 2.7, as documented by Sylvain Poulain. Also for me, this worked.

IV. The Bash profile has to be altered

The following two lines have to be added to the bash profile, because GRASS GIS doesn´t seem to get along with 64-bit architectures:

  • export GRASS_PYTHON=/usr/bin/python2.7 export
  • GRASS_PYTHONWX=/usr/bin/pythonw2.7
  1. Start up the terminal
  2. Type “cd ~/” to go to your home folder
  3. Type “touch .bash_profile” to create a  new file.
  4. Edit .bash_profile with your favorite editor (or you can just type “open -e .bash_profile” to open it in TextEdit.
  5. Add the following to lines: export GRASS_PYTHON=/usr/bin/python2.7 export -GRASS_PYTHONWX=/usr/bin/pythonw2.7
  6. Type “. .bash_profile” to reload .bash_profile and update any functions you add.

If everything works, GRASS GIS should start up and can be used by the command line, by it’s  GUI. 

V Configure GRASS applications folder in QGIS

The way I prefer to use GRASS is within QGIS´s processing toolbox. To run GRASS GIS tools within QGIS, the GRASS GIS application folder has to be defined. This will avoid the error below.

In my opinion is´s very convenient to run GRASS algorithms within QGIS. Just define the path of your GRASS GIS installation via QGIS -> processing ->providers -> GRASS7 folder . In my case the path is: /Applications/QGIS.app/Contents/MacOS/grass7. 

After pointing to the GRASS GIS folder, I finally can use GRASS GIS the way I want to and I get access to more than 300 image processing algorithms within QGIS and more than 400 within the GRASS GUI!

 

 

Turf.js for Inverse distance weighted interpolation (IDW) of Potsdam´s base rents. Web mapping playground part IV

Introduction

In this blog post I’ll show how an inverse distance weighted interpolator (IDW) can be applied with Turf.js. My blog post can serve as a tutorial for you, to try yourself  to interpolate own data sets with Turf.js  (e.g. temperature data).

The example uses a set of geocoded rental data of 3-room, apartments located in the city of Potsdam (fetched on the 19th of March).

From an analytical viewpoint, the resulting map isn’t that useful and just shows a spatial trend of the rental offers rental costs in € without service costs. Absolute prices had to be interpolated, as the apartment sizes in the fetched data were missing.

IDW actually is one of the least applicable spatial interpolators for rental data, due to known effects as bull eyes;  but Turf.js´s client side processing is in heavy development and for sure other interpolation algorithms will be integrated soon. So far Turf.js is capable to interpolate isolines, plane points, triangulated irregular networks and IDW. If you are interested how IDW works, I can recommend the interpolation tutorial on Gitta Info: http://www.gitta.info/ContiSpatVar/de/html/Interpolatio_learningObject2.xhtml

The map

Click on the map to get a price estimation per grid cell.

A full screen map can be accessed via:

http://hatschit.alkaid.uberspace.de/IDW_TURF_Choropletes/Potsdam_example.html

Turf.js IDW example

The resulting map is styled as choropleth map, using the leaflet.js choroplet plugin. The example geoprocesses the GeoJson data with the turf.js´s IDW plugin by Michele Ferreti. His turf-inverse-distance-weighting module was accepted as part of the turf.js library. He also hosts an example of a Leaflet.js, Turf.js IDW vizualisation, which served as basis for my tutorial.

Relevant code blocks

In the body of the document, several JavaScript libraries have to be integrated. That’s first leaflet.js for the map canvas and the functions of the slippy map, second turf.js,for the interpolation and choropleth.js, that styles the resulting map as choropleth map and creates the map legend.

 <script src="leaflet/leaflet.js"></script>
  <script src="choropleth.js"></script>
  <script src='https://npmcdn.com/@turf/turf@3.13.2/turf.min.js'></script>

After setting the parameters for the leaflet.js map canvas and after integrating the TileMapService for the base map,
the GeoJson file with the points for the interpolation has to be integrated to the script. Unlikely external referencing
via Ajax can’t pass the points to the interpolator. Therefore, the script soon can get quite long, which can be confusing.

<script>// <![CDATA[
var map = L.map('map').setView([52.39, 13.06], 12)
 
    // Add basemap
    L.tileLayer('http://{s}.tiles.wmflabs.org/bw-mapnik/{z}/{x}/{y}.png', {
      maxZoom: 18,
      attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>'
    }).addTo(map)
 
    // Add Sampled Point with known 'value'
    var sampledPoints = {
      "type": "FeatureCollection",
      "crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
      "features": [
      { "type": "Feature", "properties": { "field_1": 3, "latitude": 52.395768, "longitude": 13.065812, "price_high": 1480.0, "room_numbe": "3" }, "geometry": { "type": "Point", "coordinates": [ 13.065812, 52.395768 ] } },
      { "type": "Feature", "properties": { "field_1": 5, "latitude": 52.42388, "longitude": 13.057468, "price_high": 1145.0, "room_numbe": "3" }, "geometry": { "type": "Point", "coordinates": [ 13.057468, 52.42388 ] } },

In the next section, the actual interpolation is realized by the turf.idw function. The attribute of the GeoJson to be interpolated, the power of the interpolation, the cell size of the resulting grid and the measurement units have to be defined.

 
var Grid = turf.idw(sampledPoints,'price_high', 0.3, 0.5,'kilometers');
 
    L.geoJson(sampledPoints.features, {
      onEachFeature: function(feature, layer) {
        layer.bindPopup("Estimated base rent for a three room apartment " + feature.properties.value)
      }
    })

Finally, the settings of the choropleth map have to be defined. The legend automatically is created from the interpolated values. As in my last example, the legend has to be improved.

var legend = L.control({
      position: 'bottomright'
    })
    legend.onAdd = function(map) {
      var div = L.DomUtil.create('div', 'info legend')
      var limits = choroplethLayer.options.limits
      var colors = choroplethLayer.options.colors
      var labels = []
      // Add min & max
      div.innerHTML = '<div class="labels"><div class="min">' + limits[0] + '</div> \
			<div class="max">' + limits[limits.length - 1] + '</div></div>'
      limits.forEach(function(limit, index) {
        labels.push('<li style="background-color: ' + colors[index] + '"></li>')
      })
      div.innerHTML += '<ul>' + labels.join('') + '</ul>'
      return div
    }
    legend.addTo(map)

To try IDW with turf.js with own data, the script and the data packed to a zip file can be downloaded here:
hatschit.alkaid.uberspace.de/IDW_TURF_Choropletes.zip

GeoJSON Live feed of earthquakes with marker clusters: Web mapping playground part III

Full screen web map:
http://hatschit.alkaid.uberspace.de/Earthquake_Live_Mapbox.html

About the map

In this blog post I´ll show a live updating web map of several thousend marker-clustered live streamed seismic activities.

Just click on one of the numbered signatures showing the number of activities within the cluster and the map zooms and the marker spans.
The seismic activities are provided by the United States Geological Survey (USGS) in the geojson format. The USGS provides several live geojson feeds of earthquakes (live, daily and monthly feeds updated incremental): http://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php

My web map marker-clusters all earthquake activities from magnitude 1 to >4.5 in Richter magnitude scale, in the last month, events are updated every 15 minutes. The web map basically was created by using Mapbox´s JavaScript library: https://www.mapbox.com/ and by using a Leaflet.js plugin. Mapbox.js and Leaflet.js are interoperable, as Leaflet.js genial main developer Vladimir Agafonkin (@mourner) works for Mapbox. To use Mapbox, you need a Mapbox access token. Mapbox has a price per user payment model, similar to the Google
maps API. Up to 50k users per map is for free, for more useres mapbox charges according to their pricing plan: https://www.mapbox.com/pricing/

Making your own Mapbox.js or Leaflet.js  web map

If you want starting to develop web maps as well, you should know basics of HTML and JavaScript. As starting point, I can recommend the online courses at Codeacademy: https://www.codecademy.com/learn/javascript
Codeacademy should enable you to start using web mapping libraries. I also recommend Nurma Gremlins
course material for beginners, recently in our February MaptimeBER sessions he did a basic leaflet.js intro . Here you can find his slides: http://geosysnet.de/en/downloads.html If you are interested in further readings about marker clusters, I can recommend Seb Meiers paper “The marker cluster”:
http://www.sebastianmeier.eu/2016/01/30/the-marker-cluster/

Relevant code blocks

Next I comment some relevant pieces of the code, just in case you plan to do a similar web map as the one shown above. In the first code block, two JavaScript libraries are included into the HTML document. In the head of the HTML document I included the Mapbox.js library for providing a base map, and the Leaflet Marker cluster plugin (https://github.com/Leaflet/Leaflet.markercluster), the plugin that cares for the clustering of the clusters. This plugin actually is just one possibility for marker clustering, several alternatives are listed at the Leaflet.js website: http://leafletjs.com/plugins.html#clusteringdecluttering

<script src='https://api.mapbox.com/mapbox.js/v3.0.1/mapbox.js'></script>
<script src='https://api.mapbox.com/mapbox.js/plugins/leaflet-markercluster/ v1.0.0/leaflet.markercluster.js'></script>

In the next section of the code, the body of the HTML document, the JavaScript part starts with the script tag and hidden (the x) part of my Mapbox access token is shown. In the .set view part the zoom level and the center coordinate of the web map is shown.

<style>
body { margin:0; padding:0; }<br /> #map { position:absolute; top:0; bottom:0; width:100%; }<br /></style>
<style>
<div id="map"></div>
<script>
L.mapbox.accessToken = 'xxxxxxxxxx';
var map = L.mapbox.map('map', 'mapbox.streets')
   .setView([38.1089, 13.3545], 2);

The next line provides a nice dynamic scale bar.

L.control.scale().addTo(map);

Here the Geojson file, provided by the USGS is fetched.

var featureLayer = L.mapbox.featureLayer()
   .loadURL('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson')
   .on("ready", function()

Finally the markers of the seismic events (several thousand events) are clustered Leaflet.js´s marker cluster plugin and and the html document ends.

 
{
     var markers = L.markerClusterGroup();
     markers.addLayer(featureLayer);
     markers.addTo(map);
       });
 
</script>

I´ll take a two weeks break from blogging, as I will go for a Erasmus+ stay to the Czech Republic, I´ll be back with a trip report :-).

Visualizing the third dimension: Overview and QGIS2threejs example. Web mapping playground part II

About ten years ago, in the boom years of 3D geodata visualization, expensive expert systems appeared. Systems as the costly LandeXplorer, developed by 3DGeo in Potsdam, later acquired by Autocad, were developed to visualize the z-dimension of geodata. Proprietary Arc Scene could be used, though Arc Scene was not as powerful as LandXplorer, but  quite OK, unlikely users needed a license from ESRI.

Open source alternatives

Only a few open source alternatives existed. Paraview should be mentioned . Paraview, which I used to map soil contamination, improved a lot and nowadays it´s worth a look. Nviz, a GRASS GIS module for 3D could be used and Fledermaus, a system for the 4-dimensional rendering of geodata were alternatives.

All in all visualizing the third dimension was not affordable or a pain in the ass (sorry for the hard words). 

Three.js revolution

Visualizing the third, or better the two and a half dimension in a browser was not imaginable. Later WebGL appeared, but the learning curve for WebGL, just for visualizing some 3D data, was really high. Fortunately Three.js was developed. Three.js according to Wikipedia is “a cross-browser JavaScript library/API used to create and display animated 3D computer graphics in a web browser.” Three.js really facilitated 3D mapping and unexpectedly a new tool was out there, to do web based three-dimensional mapping of geodata.
A plugin for QGIS was developed, to use three.js within QGIS: http://qgis2threejs.readthedocs.io/
QGIS2three.js provides a powerful, extremely versatile tool for a  visualization of z-dimensional data within web browsers.

QGIS2Three.js example

The following example was done with QGIS2three.js. The plugin leads the user and pre-writes the three.js code, ready to upload as a web map. The JavaScript code later can be customized e.g. using an editor as ATOM. My QGIS2Threejs example shows landslides in Rio Blanco-Nicaragua mapped in a landslide inventory, in which I took part during my master thesis project in 2005. The landslide polygons are mapped on a ASTER Global DEM, which was exaggerated by the scale of 3.

Even quadtrees can be built for data compression purposes. So the plugin gives the user a powerful open source tool, for developing all kind of 3D (geo-)data visualization usable within web browsers.

Bildschirmfoto 2017-02-19 um 14.17.44

If you plan to do some 3D mapping of geodata for the web, my recommendation is, take a look on QGIS2Three.js.

 

 

 

 

Thematic isolines with turf.js: Webmapping Playground part I

In the next weeks I plan to publish some past and recently made leaflet.js, mapbox.js and carto webmap examples, hosted on uberspace or my Vserver. The maps will show prototypes of different development stages made for test purposes.

As kick off, I show a map created with turf.js, a JavaScript library for “advanced geospatial analysis”.

The map shows isolines of equal rental prices (without service costs) in Berlin. The isolines are generated by a client side processing of approximately 3000 data points (from November 2016). The map legend is still work in progress, in my opinion the making of custom legends in Mapbox.js should be improved (adaptive legends would be great). Watch out, loading the map takes a while, as the size is 5.6 MB. The isolines are rendered from the data points.

The Isolineshave been generated by using the turf.js isoline function: http://turfjs.org/docs.html#isolines

I encountered two major problems in making this web map:

    1. The alignment of the legend was quite troublesome
    2. Processing a huge number of data points with JavaScript on a client can take quite a while

Consider this map as a prototype of what can be done with Turf.js; which in my eyes is a very interesting JavaScript library.

The full screen webmap also can be opened via: http://hatschit.alkaid.uberspace.de/Turf_tin_3000_isolines_colors.html

 

Satellite eyes observe our world: An intro to remote sensing

The original announcement from our MaptimeBER Meetup website: http://www.meetup.com/de-DE/Maptime-BER/events/228702652/

In our next session we look into the field of remote sensing. Remote sensing technologies are a wide field and earth observation serves as basis for a multitude of mapping applications (e.g. topographical maps). Andreas Fricke, Christian Kuntzsch and Harald Schernthanner from Potsdam university’s geoinformation research group give us a beginner tutorial in processing remote sensing data of Copernicus’ Sentinel programme in QGIS; of course the data is from Berlin!

Questions will be answered, as how to find proper satellite data and how to get the data into QGIS. The tutorial serves as a starting point for your own discoveries in processing satellite imagery. To take part in the workshop, please install QGIS on your notebook. Installers and instructions on how to install QGIS in a Windows, Linux or Mac environment can be found on the QGIS project website: https://www.qgis.org/en/site/forusers/alldownloads.html

Information on where to download the satellite data before the Maptime session follows.

Andreas tells us about his expedition to Siberia organized by the Alfred Wegener Institute, Helmholtz Centre for Polar and Marine Research (AWI). He gives us some insights on how he maps

To you need geocoded rental data for German cities? This small R script helps

The following short R script uses the Json parser Jsonlite and fetches gecoded real estate offers via the Nestoria REST API. The Nestoria API does not require any kind of authentification (as OAuth or an API key) and delivers current real estate date from all big and also smaller German real estate portals. Downloads are limited to 20 or 50 data points per page and to 1000 offers per request. So if you need a larger area, just adapt the bounding box or by filtering using the request paramaters documented on the Nestoria developers website: http://www.nestoria.co.uk/help/api-search-listings
The script loops through the pages provided by the API and puts the result into a R dataframe. The R dataframe easily can be exported as CSV for further use in QGIS or any other GIS software (e.g. Arc GIS).
If you have any questions, feel free to ask. The comments in the script are partly German and partly English, sorry for that mess :-). I tested the script fetching Berlin and Potsdam real estate data.

 


#Author Harald Schernthanner, basierend auf: JSOn Data über API mit meheren Pages: https://cran.r-project.org/web/packages/jsonlite/vignettes/json-paging.html
#Schleife zum fetchen der Daten

install.packages("jsonlite")
install.packages('curl')
library(jsonlite)
  
baseurl <- "http://api.nestoria.de/api?country=de&pretty=1&encoding=json&action=search_listings&place_name=Potsdam&listing_type=rent"
#Anzahl der "Seiten entscheidet wie oft durch den Datensatz geloopt werden soll
#eine for Schleife loopt durch die Seiten und schreibt die Ergebnisse in den Dataframe "pages"

pages <- list()
for(i in 0:24){
  mydata <- fromJSON(paste0(baseurl, "&page=", i))
  message("Retrieving page ", i)
  pages[[i+1]] <- mydata$response$listings
}

#Kombiniert die einzelnen Abfragen
angebote_potsdam <- rbind.pages(pages)

#check output
nrow(angebote_potsdam)


#Exportieren der Daten

write.csv2(angebote_potsdam, "potsdam.csv")



Housing prices with service costs in Potsdam inner city

Housing prices with service costs in Potsdam inner city

Untersuchungen zur räumlichen Analyse und Visualisierung von Mietpreisdaten für Immobilienportale (Dissertation, Schernthanner 2015)

Schernthanner, H. (2015). Untersuchungen zur räumlichen Analyse und Visualisierung von Mietpreisdaten für Immobilienportale. XII, 134 Bl. : graph. Darst. Potsdam, Univ., Diss., 2015: https://publishup.uni-potsdam.de/opus4-ubp/frontdoor/index/index/docId/8949

Betreuung: Prof. Hartmut Asche. Begutachtung: Prof. Andreas Koch (Universität Salzburg) & Prof. Julia Siemer (University of Regina), Note: Magna cum laude

Wissenschaftliche Zusammenfassung:

Die Arbeit verfolgte das Ziel, aus geoinformatischer Sicht eine konzeptionelle Grundlage zur räumlichen Optimierung von Immobilienportalen zu schaffen. Die Arbeit ging dabei von zwei Hypothesen aus:

1. Verfahren der räumlichen Statistik und des Maschinellen Lernens zur Mietpreisschätzung sind den bisher eingesetzten Verfahren der hedonischen Regression überlegen und eignen sich zur räumlichen Optimierung von Immobilienportalen.

2. Die von Immobilienportalen publizierten webbasierten Mietpreiskarten geben nicht die tatsächlichen räumlichen Verhältnisse auf Immobilienmärkten wieder. Alternative webbasierte Darstellungsformen, wie z.B. Gridmaps, sind dem Status Quo der Immobilienpreiskarten von Immobilienportalen überlegen und visualisieren die tatsächlichen räumlichen Verhältnisse von Immobilienpreisen zweckmäßiger.

Beide Thesen konnten bewiesen werden.

Es erfolgte zunächst eine umfangreiche Erhebung des Forschungsbedarfs mittels Literaturstudien und technologischer Recherche. Zur Beantwortung der Forschungsfragen wurde als quantitative Datenbasis ein 74.098 Mietangebote umfassender Datensatz (von Januar 2007 bis September 2013) eines Immobilienportals akquiriert. Dieser reichte jedoch nicht in vollem Umfang zur Beantwortung der Fragestellungen aus. Deshalb führte der Autor Experteninterviews zur Erhebung einer qualitativen Datenbasis. Deren Analyse ergibt in Kombination mit der Literaturstudie und der technologischen Recherche ein umfassendes, bisher so nicht verfügbares Bild. Es stellt den Status Quo der räumlichen Sicht sowie der raumanalytischen und geovisuellen Defizite von Immobilienportalen dar.

Zur Optimierung der raumanalytischen und geovisuellen Defizite wurden forschungsbasierte Lösungsansätze herausgearbeitet und teilimplementiert. Methoden des Maschinellen Lernens und räumliche Schätzverfahren wurden als Alternativen zu den von Immobilienportalen bisher genutzten „nicht räumlichen“ Analyseverfahren zur Preismodellierung untersucht. Auf Grundlage eines hierfür konzipierten Validierungsrahmens wurden diese Methoden für die Nutzung im Kontext von Immobilienportalen adaptiert. Die prototypische Teilimplementierung zeigte die programmiertechnische Umsetzung des Konzeptes auf.
Eine umfassende Analyse geeigneter Sekundärvariablensets zur Mietpreisschätzung lieferte als methodisches Resultat, dass Interpolatoren, die Sekundärvariablen benötigen (Kriging with external drift, Ordinary Cokriging), kaum zu valideren Mietpreisschätzergebnissen gelangen als die Methode des Ordinary Kriging, die keine Sekundärvariablen benötigt.
Die Methoden Random Forest aus dem Maschinellen Lernen und die Geographisch Gewichtete Regression hingegen bergen großes Potential zur Nutzung der räumlichen Mietpreisschätzung im Kontext von Immobilien-portalen. Die Forschungsergebnisse der räumlichen Preismodellierung wurden in die räumliche Visualisierung von Mietpreisen transferiert.

Für die webbasierte Mietpreisdarstellung wurde ein Set alternativer Darstellungsmethoden entwickelt, um Mietpreiskarten-Prototypen abzuleiten. Ein methodisches Ergebnis der Entwicklung der Mietpreiskarten-Prototypen war die Entwicklung eines geeigneten Ansatzes der Loslösung des Preisbezugs von fachfremd verwendeten Bezugsgeometrien. Hierfür wurde vom Autor der Begriff der zonenlosen Preiskarte geprägt. Diese wurden mit Methoden des Gridmapping erstellt. Es wurden optimale Rasterauflösungen zur Darstellung interpolierter Rastergrößen ermittelt. Zonenlose Preiskarten mit Methoden des Gridmapping, gepaart mit einer optionalen gebäudescharfen Darstellung in größeren Maßstäben, sind als Resultate der Forschung die bestmögliche, sich an realen Verhältnissen orientierende, räumliche Mietpreisdarstellung. Die entstandenen Prototypen sind eine Annäherung der wahren Verteilung des Mietpreises im Raum und um einiges schärfer, als die auf der hedonischen Regression basierenden Darstellungen. Somit kann die wahre „Topographie“ der Mietpreislandschaft abgebildet werden. Ein Einsatz der Karten für Nutzergruppen wie Makler, Investoren oder Kommunen zur Analyse städtischer Mietmärkte ist denkbar. Alle entstandenen Prototypen sind unter der Nutzung von Map APIs umgesetzt. Ein Ergebnis dessen ist, dass Map APIs noch an diversen „Kinderkrankheiten“ leiden und derart umgesetzte Mietpreiskarten noch einen weiten Weg vor sich haben, bis sie das Niveau thematischer Karten von Immhof oder Arnberger erreichen.
Die konzeptionellen Überlegungen und Teilimplementierungen mündeten in drei Prozessketten, die Umsetzungsoptionen für eine räumliche Optimierung von Immobilienportalen darstellen. Dabei wurden zwei Szenarien für eine räumlich optimierte Mietpreisschätzung und ein Szenario für eine räumlich optimierte Mietpreisdarstellung herausgearbeitet.

 

 

 

GIS Feeds News Aggregator

Gisfeeds  ist ein von mir angelegter News Aggregators der RSS Nachrichtenfeeds der GI Wirtschaft,Wissenschaft, GIS Jobangebote einsammelt und übersichtlich auf einer Seite darstellt.Technisch vermascht Gisfeed RSS 1.0, 2.0 und Atom feeds basierend auf WordPress mit mehreren Plugins, Yahoo Pipe, RSSforme und RSS Graffiti.

Anregungen, Kritik und Links zu fehlenden RSS Feeds werden gerne 
entgegengenommen.

Introducing to Leaflet.js at Maptime BER

I did a basic introduction to Leaflet.js at Maptime BER. The goal was to give a short introduction on how to make a webmap with Leaflet.js. About 40 mappers, from novices to experts, attented the event at the IXDS Berlin. we talked anput Map API´s in general and I should code examples and explained how to integrate different Tilemap Services into leaflet and how to make basic markers and popups with some content :-). 90 minutes hardly have been enough to demonstrate the basic ideas of leaflet. Beside fighting with technical problems, that made me even switch to a Mac, the tutorial was fun and I hope the participants will profit from it. The IXDS in Kreuzberg was a great location! At the end we had a very basic webmap with a marker of the location of our event, here you can see the result: https://schernthanner.de/Demo.html

The examples (scripts) kann be downloaded at:  https://schernthanner.de/Maptime_Leaflet_Intro.zip

Here you can see the slides of the tutorial:

 

Webmapping Seminar an der Universität Potsdam

Vom 17.9 bis 19.9 ging das erste von mir organisierte Webmapping Seminar an der Uni Potsdam über die Bühne. In ca. 21 Stunden gab es für 15 Teilnehmer eine Einführung in die Bereitstellung von OGC Services mit dem Geoserver, in die Installaton des Geoserver und eine Einführung in die Leaflet Map API. Wir stellten einen WMTS, einen WFS und einen WCS bereit und besprachen die Potentiale von Webprocessing Services. Wir sahen uns Styled Layer Descriptor an, als Möglichkeit Geodaten zu stylen und lernten als Alternative Carto CSSkennen. Maptiles wurden mit Tilemill erstellt. Mit Leaflet haben wir Webmaps anhand von Beispielcode erstellt. Geojson in Leaflet eingebunden und unsere Karten für das Smartphone angepasst, Photon als Geocoder genutzt und einige tolle Plugins kennengelernt. Christian Kuntsch zeigte uns wie wir aus Twitterfeeds GeoJSON ableiten und Patrick Voland gab eine fundierte CartoDB Einführung, zeigte uns torque.js und wie man den Photon selbst installieren könnte. Ich freue mich auf die Abschlussprojekte der Studenten.

Mapping im Norden Nicaraguas: Ein kleines Urlaubsprojekt

Derzeit befinde ich mich auf einer einmonatigen Reise im Norden Nicaraguas in Zentrakamerika, genau in der Kleinstadt El Jicaro wo ich zum Schreiben dieses Blogeintrags in einem kleinen Internetcafe sitze. Zuerst sende ich der OSM Community liebe Urlaubsgrüsse und danke allen Mappern die es ermöglichen, das man in Nicaragua mittels der OpenStreetMap sehr gut mit dem Auto und in grossen Staeten auch zu Fuss navigieren kann. Garmin und co. funktionieren nämlich zur Strassennavigation sehr eingeschraenkt in Nicaragua. Mapper wie der Mapper “Centralamerica” haben zum Beispiel bereits 9000 Aenderungen zur OpenStreetMap beigetragen. Vor zwei Wochen war die Kleinstadt “El Jicaro” ein weisser Fleck in der OpenStreetMap, also machte ich mich als kleines Urlaubsprojekt daran POIs mit dem Handy aufzunehmen und zu taggen. Einzig einige Strassen, die Kirche und der Friedhof waren in der OpenStreetMap vorhanden.

Wie in den meisten Nikaraguanischen Städten gibt es in El Jicaro die Besonderheit, dass es keine Strassennamen und Hausnummern gibt und die Adressen aus bekannten Referenzpunkten bestehen. Man orientiert sich zum Beispiel anhand des Rathauses und einer vorhandenen Mobilfunkantenne und da alles rechteckig in Cuadras (Blöcken) angeordenet ist, orientiert man sich zum Beispiel wie folgt: von der Mobilfunkantenne 2 Blöcke nach Süden oder vom Kreisverkehr 4 Blöcke nach Norden und 3 nach Osten. Wie die Post ankommt, ist mir immer noch ein Raetsel.

Primäres Ziel meines Mappings war also herauszufinden, was die bekannten Referenzpunkte sind und vor allem diese der OSM hinzuzufügen um die OpenStreetMap in El Jicaro fuer grobe Orientierungs- und Navigationszwecke nutzbar zu machen. Aufgrund des Fehlens einer mobilen Internetverbindung nutzte ich als Werkzeug die bekannte Android App OsmAND und in Deutschland heruntergelade Offline Daten.Die aufgenommen Nodes habe ich dann mittels eines öffentlich zugänglichen Wlans zur OSM hochgeladen und dann in einem Internetcafe weiter bearbeitet.

Auf Dauer war mir das Mappen mit OsmAND zu unsystematisch und das Herumlaufen im Ort und die Aufnahme mit OsmAND zu zeitaufwendig und ich suchte eine systematischere Herangehensweise die wie folgt aussah. Ich suchte mir ein Internetcafé, wo ich ich mir das von Bing bereitgestellte GeoEye 1 Satellitenbild als Kartiergrundlage ausgedruckt habe. Damit spazierte ich herum und befragte Leute über diverse Gebäude und wichtige Referenzpunkte und zeichnete POIs analog ein.

Danach ging ich in dasselbe Internetcafé und editierte und taggte die POIS mit dem iD Editor im Browser, Ausserdem korrigierte ich den Verlauf vorhandener Strassen, taggte den Strassenbelag und lud einige Tracks hoch. Grössere Flächen von Bedeutung , wie ein Krankenhausgelände und ein lokaler Busterminal digitalisierte ich in iD auf der GeoEye- 1 Grundlage. Es machte Spass im Urlaub zur OpenStreetMap beizutragen, vielleicht animiert der Beitrag Reisende unter den Mappern ebenfalls “weisse Flecken” auf der OpenStreetMap zu entfernen.

In diesem Sinne, Happy Mapping oder Feliz Mapeando und liebe Grüße aus Zentralamerika.