Marble currently uses the very old and outdated MWDBII dataset for vector map outlines and we really need to replace it with more up-to-date data. However, MWDBII has two key advantages, it is very compact in size enabling Marble to ship it by default, and the individual nodes have a zoom level value which speeds up drawing.
The Natural Earth data set is a "public domain map dataset available at 1:10m, 1:50m, and 1:110 million scales. Featuring tightly integrated vector and raster data, with Natural Earth you can make a variety of visually pleasing, well-crafted maps with cartography or GIS software." This data set seems ideal as a replacement for the MWDBII.
Advantages:
Disadvantages:
The 1:10m dataset seems ideal as the base map in Marble as it provides a higher level of detail than the current MWDBII. The 1:110m dataset seems ideal for use in a country selector widget in kdelibs. The 1:50m dataset being less detailed than the MWDBII is probably less useful to Marble.
Using the data in the default shapefile format is not considered desirable however:
The ideal solution would therefore be to convert the Natural Earth data into a more efficient file format that includes a zoom level attribute
As you say, there's the two approaches that could work for the lightweight
default layer:
a) Convert just the required NE datasets to pnt format, either merging the 3
scale levels into a single file with just 3 detail levels, or use Douglas-
Peucker on the 1:10m files to create the required detail levels
b) Implement an internal lightweight shapefile parser without dbf support and
ship only the required NE datasets.
Some pros/cons to consider:
The 1:10m country file is 6.55MB and contains 533,202 points = 12.28
bytes/point compared to the PNT which is 745KB and contains 127,246 points =
5.85 bytes/point, which would suggest the NE data in PNT format would be half
the size, so 6 MB in total. This could probably be further reduced by a light
application of Douglas-Peucker.
The NE shapefiles have been carefully processed so shared borders and overlapping features like rivers match exactly and other such niceties, applying the Douglas-Peucker algorithm might affect that.
A lightweight shapefile parser would allow users/apps to load other shapefiles.
We would have to reconvert and check the data every time there's a new NE release which could be a lot of effort, but an automated shp2pnt script could prove useful to allow apps/users to display their own shapefiles in a simple way.
Overall, it seems the best approach for the updating the lightweight layer is indeed to convert the shapefiles to PNT format, provided the D-H algorithm can be deployed in a way to mark each point with a detail level rather than just throwing the points away.
Using GeoPainter and GeoDataLineString ("libgeodata"):
Potential new Marble file format based on PNT:
Applying this to the 1:10m dataset:
Metadata file:
Rather than the Geonames ID, we could just use the Natural Earth object ID, then a look-up file/table that matches the NE ID to the ISO / FIPS / whatever code (NE provides this in the metadata) and Geonames ID (which we would have to provide). This would allow look-ups via whatever code or ID is available, and we wouldn't be reliant on Geonames IDs staying constant.
So required work is: 1) Fix GeoPainter LinearRings which contain a pole not rendered correctly
- Torsten knows, problem will fill in flat map needs to create polygon if closed and crosses dateline once only
2) Implement Douglas-Peuker reduction in GeoDataLineString
- big investigation
3) New PNT file format definition (with a different name, MBL?)
- existing serial format from geodata???
4) Metadata file format definition (could just be csv or xml? or sqlite?) 5) New GeoData PNT2 file loading code (convert olf data). 6) shp2mbl script to convert shp to new formats (using Perl::shp? there's shp2xxx scripts out there we could copy?), including matching to Geonames ID 7) split files into 'ship with', 'download asap', 'ghns'@
Later add simple shapefile loading to GeoData, maybe with attibute layer?
GHNS:
KDE TechBase
Key Natural Earth data files from v1.2, recent updates to 1.3 not included.
1:110m 1:50m 1:10m
------ ----- -----
Admin level 0 countries 172 KB 1.36 MB 6.55 MB
Admin level 0 land borders 39 KB 301 KB 896 KB
Admin level 0 sea borders 12 KB 40 KB 79 KB
Admin level 0 disputed 40 KB 157 KB
Admin level 1 regions 39 KB 339 MB 13.9 MB *
Admin level 1 land borders 16 KB 60 KB 4.82 MB
Coastlines 79 KB 883 KB 2.15 MB
Rivers 19 KB 420 KB 3.29 MB
Lakes 10 KB 286 KB 786 MB
Glaciers 13 KB 208 KB 1.23 MB
Dateline 18 KB 18 KB 18 KB
Playas 18 KB 106 KB
Ice Shelves 105 KB 211 KB
Minor Islands 449 KB
Reefs 171 KB
------- ------- ---------
417 KB 4.08 MB 34.03 MB
* level 1 regions are USA/Canada only at 110m and 50m, but whole world at 10m,
perfect for KGeography use :-)
Other useful files:
Physical Features Land 146 KB 1.50 MB 692 KB
Physical Features Sea 348 KB 836 KB 836 MB
Populated Places 347 KB 1.48 MB
Urban Areas 439 KB 3.48 MB
Bathmetry 11.64 MB
Datasets:
sovereign states. Includes dependencies (French Polynesia), map units (U.S. Pacific Island Territories) and sub-national map subunits (Corsica versus mainland Metropolitan France).
