Latest revision as of 19:38, 8 January 2017

OSM Vector Tiles

Creating tiles from a pbf file

As an example we'd like to render tiles for tile level 13, 15 and 17 of San Francisco. First you need the osm.pbf file for that region. In our case we get it from:

http://download.geofabrik.de/north-america/us/california.html

where we locate the link to the pbf file as http://download.geofabrik.de/north-america/us/california-latest.osm.pbf

Creating the tools - osmconvert and marble-vectorosm-tilecreator

Now we need the tools for conversion and filtering the pbf file into tiles. In order to compile the tools you should have zlib installed on your system (e.g. via sudo apt-get install zlib1g-dev). Then run:

cd /tmp
wget http://m.m.i24.cc/osmconvert.c
gcc -O3 -o osmconvert osmconvert.c -lz
sudo mv osmconvert /usr/local/bin

if that works, running

osmconvert -h

should output usage instructions.

Please compile Marble from source code as well as described in https://marble.kde.org/sources.php and make sure to compile the development version (master branch). Pass the option -DBUILD_MARBLE_TOOLS=TRUE to cmake in order to compile all Marble tools.

Running the VectorTileCreator

Now our tool is needed. In a shell, go to the marble build directory and then

cd tools/vectorosm-tilecreator

now running

./marble-vectorosm-tilecreator -h

should give usage instructions.

The tile cutter tool works as follows: You call it with a link to an osm .pbf file as parameter.

The tool will generate tiles for zoom levels 11, 13, 15 and 17 by default. You can overwrite that using the -z switch, e.g. -z 13 15.

Full Example

If you follow the steps above, the final tile creation run after installing the required tools will look like this:

cd ~/marble/build/
cd tools/vectorosm-tilecreator
./marble-vectorosm-tilecreator http://download.geofabrik.de/north-america/us/california-latest.osm.pbf
 [========================================>]  100%  734.4/734.4 MB  Downloading california-latest.osm.pbf
 [========================================>]  100%  0.0/0.0 MB  Downloading california.poly
 [========================================>]  100%  osm cache tiles complete.
 [========================================>]  100%  landmass cache tiles complete.                    
 [========================================>]  100%  Vector OSM tiles complete.

Smaller Regions

For large countries the tile creation can take several hours or even days. If you're just interested in a certain region (say, a city), you can preprocess the pbf and extract the region you're interested in. Use osmconvert for that like this:

osmconvert --complex-ways --complete-ways -b=8.3735,49.0016,8.4409,49.0261 -o=smallregion.pbf largeregion.pbf
./marble-vectorosm-tilecreator smallregion.pbf

The four numbers of the bounding box passed with the -b=... parameter can be obtained from the OSM export tool in the order left, bottom, right, top.

Resolving errors

If you get an error message after the tile creation downloaded the cache files, a likely cause is that you are missing the shapelib (libshp) dependency. It's an optional dependency of Marble, but a required dependency for vector tile creation.

To check whether shapelib (libshp) was found, examine the first few lines of cmake. They should contain a list of dependencies needed, with a note if it was found or not. You might get a warning that shapelib (libshp) is missing. Installing it and recompiling Marble will then fix the error.

In the example cmake output below, the optional packages libgps and libwlocate are missing, but shapelib (libshp) is found after installing it using the package manager of the Linux distribution:

$ cmake -DBUILD_MARBLE_TOOLS=TRUE ../marble
-- Could NOT find libgps (missing:  LIBGPS_INCLUDE_DIR LIBGPS_LIBRARIES) 
-- Could NOT find libwlocate (missing:  LIBWLOCATE_INCLUDE_DIR LIBWLOCATE_LIBRARIES) 
-- 
-- The following OPTIONAL packages have been found:
[...]
 * libshp , reading and writing of ESRI Shapefiles (.shp) , <http://shapelib.maptools.org/>
   reading and displaying .shp files
[...]

@@ Line 4: / Line 4: @@
 As an example we'd like to render tiles for tile level 13, 15 and 17 of San Francisco.
-First you need the osm.pbf file for that region. In our case we download that from:
+First you need the osm.pbf file for that region. In our case we get it from:
 http://download.geofabrik.de/north-america/us/california.html
-or to be more specifically: http://download.geofabrik.de/north-america/us/california-latest.osm.pbf
+where we locate the link to the pbf file as http://download.geofabrik.de/north-america/us/california-latest.osm.pbf
-=== Creating the tools - osmconvert and osmfilter ===
+=== Creating the tools - osmconvert and marble-vectorosm-tilecreator ===
 Now we need the tools for conversion and filtering the pbf file into tiles. In order to compile the tools you should have zlib installed on your system (e.g. via sudo apt-get install zlib1g-dev). Then run:
@@ Line 19: / Line 19: @@
 gcc -O3 -o osmconvert osmconvert.c -lz
 sudo mv osmconvert /usr/local/bin
-and pretty much the same for osmfilter
-wget http://m.m.i24.cc/osmfilter.c
-gcc -O3 -o osmfilter osmfilter.c
-sudo mv osmfilter /usr/local/bin
 </source>
@@ Line 29: / Line 25: @@
 <source lang="bash">
 osmconvert -h
-</source>
-and also
-<source lang="bash">
-osmfilter -h
 </source>
 should output usage instructions.
+Please compile Marble from source code as well as described in https://marble.kde.org/sources.php and make sure to compile the development version (master branch). Pass the option -DBUILD_MARBLE_TOOLS=TRUE to cmake in order to compile all Marble tools.
 === Running the VectorTileCreator ===
-Now our tool is needed. You need python version 3 to be installed. In a shell, go to the marble sources directory and then
+Now our tool is needed. In a shell, go to the marble build directory and then
 <source lang="bash">
-cd tools/vectortilecreator
+cd tools/vectorosm-tilecreator
 </source>
 now running
 <source lang="bash">
-./vectortilecreator.py -h
+./marble-vectorosm-tilecreator -h
 </source>
 should give usage instructions.
-The tile cutter tool works as follows: You call it with a region file as parameter (or several). A region file is a comma separated text file that describes the input to process. You cannot pass a pbf file directly. Here is an example of a region file:
+The tile cutter tool works as follows: You call it with a link to an osm .pbf file as parameter.
-<tt>http://download.geofabrik.de/europe/germany/baden-wuerttemberg-latest.osm.pbf;Karlsruhe;8.30831;48.93133;8.62496;49.17019</tt>
+The tool will generate tiles for zoom levels 11, 13, 15 and 17 by default. You can overwrite that using the -z switch, e.g.
+-z 13 15.
-The first entry is the pbf file to process. You can use a URL here and vectortilecreator.py will download that file to the cache directory if it does not exist yet.
+=== Full Example ===
-The second entry is an arbitrary name that helps you remember which region this is.
+If you follow the steps above, the final tile creation run after installing the required tools will look like this:
-The remaining four entries are a rectangular region (bounding box) where data is extracted from. Make sure that the region you specify here is actually part of the .pbf file! An easy way to get it right is to use openstreetmap.org and its Export feature. Refer to [http://nienhueser.de/marble/marble-vectortilecreator-order.jpg this image] to get the order of the coordinates right.
-The script will generate tiles in the current directory unless you use the -d option: For example -d ~/.local/share/marble/maps/earth/vectorosm is what you want for local tiles. By default it will generate tiles for zoom levels 13, 15, 17. You can overwrite that using the -z switch
+<source lang="bash">
--z 13 15. This would allow to exclude e.g. level 17 (which can take between < 1 sec for small tiles but > 10 secs for large tiles depending on the input size). Once called, the script does the following
+cd ~/marble/build/
-* It parses the region file to read which file to work on and which boundaries it has
+cd tools/vectorosm-tilecreator
-* To speed things up, it then creates another file with only the given boundaries inside (e.g. if you pass california-latest.osm.pbf and work on san francisco it will create a temporary file that has only san francisco inside)
+./marble-vectorosm-tilecreator http://download.geofabrik.de/north-america/us/california-latest.osm.pbf
-* Afterwards it iterates over the zoom levels passed. For each level it calculates the tiles that are inside the boundaries.
+ [========================================>]  100%  734.4/734.4 MB  Downloading california-latest.osm.pbf
-Then it calls osmconvert for each tile to generate the tile
+ [========================================>]  100%  0.0/0.0 MB  Downloading california.poly
-* If it also finds a file levels/$ZOOM.level it will call osmfilter afterwards
+ [========================================>]  100%  osm cache tiles complete.
-* And finally the result is zipped
+ [========================================>]  100%  landmass cache tiles complete.
+ [========================================>]  100%  Vector OSM tiles complete.
+</source>
-===  What should I do if this doesn't seem to work for my own data? ===
+=== Smaller Regions ===
-Now you might have created your own tiles already. But what are possible pitfalls if your own tile data doesn't show up?
+For large countries the tile creation can take several hours or even days. If you're just interested in a certain region (say, a city), you can preprocess the pbf and extract the region you're interested in. Use osmconvert for that like this:
+<source lang="bash">
+osmconvert --complex-ways --complete-ways -b=8.3735,49.0016,8.4409,49.0261 -o=smallregion.pbf largeregion.pbf
+./marble-vectorosm-tilecreator smallregion.pbf
+</source>
+The four numbers of the bounding box passed with the -b=... parameter can be obtained from the [https://www.openstreetmap.org/export#map=15/49.0139/8.4072 OSM export tool] in the order left, bottom, right, top.
-* If you get an error something like this "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 22: invalid start byte" from the vectortilecreator.py script, then this means that you are passing it a .pbf file directly. That is wrong: It wants a .csv file which itself refers to one or more pbf files and which rectangular regions and zoom levels it should create.
+=== Resolving errors ===
-* Check the directory ~/.local/share/marble/maps/earth/vectorosm . It should have subdirectories of the levels that you created. Since the vector tiles are created as .osm.zip files you can zgrep their content in order to look for features that should be inside the data you wanted to create
+If you get an error message after the tile creation downloaded the cache files, a likely cause is that you are missing the shapelib (libshp) dependency. It's an optional dependency of Marble, but a required dependency for vector tile creation.
-* Check the bounding rect definition inside the text file. It should specify the bottom left and top right corner of your area.
+To check whether shapelib (libshp) was found, examine the first few lines of cmake. They should contain a list of dependencies needed, with a note if it was found or not. You might get a warning that shapelib (libshp) is missing. Installing it and recompiling Marble will then fix the error.
+In the example cmake output below, the optional packages libgps and libwlocate are missing, but shapelib (libshp) is found after installing it using the package manager of the Linux distribution:
+<source lang="bash">
+$ cmake -DBUILD_MARBLE_TOOLS=TRUE ../marble
+-- Could NOT find libgps (missing:  LIBGPS_INCLUDE_DIR LIBGPS_LIBRARIES)
+-- Could NOT find libwlocate (missing:  LIBWLOCATE_INCLUDE_DIR LIBWLOCATE_LIBRARIES)
+--
+-- The following OPTIONAL packages have been found:
+[...]
+ * libshp , reading and writing of ESRI Shapefiles (.shp) , <http://shapelib.maptools.org/>
+   reading and displaying .shp files
+[...]
+</source>