The Natural Earth data set is a "public domain map dataset available at 1:10m, 1:50m, and 1:110 million scales. Featuring tightly integrated vector and raster data, with Natural Earth you can make a variety of visually pleasing, well-crafted maps with cartography or GIS software."
The main benefit for the lightweight requirement switching to Natural Earth (NE) would be updated borders, a separate internal border dataset, and a few more physical features, but this should be worth it.
The main benefit for the second case is the embedded data attributes such as country code, groupings, relative magnitude, etc, which will allow easy linking and manipulation. We could do something similar using PNT, but we would have to create our own attribute storage to link things together, and then maintain the attribute data, so sticking with the shapefiles would seem a better solution there.
As you say, there's the two approaches that could work for the lightweight default layer: a) Convert just the required NE datasets to pnt format, either merging the 3 scale levels into a single file with just 3 detail levels, or use Douglas- Peucker on the 1:10m files to create the required detail levels b) Implement an internal lightweight shapefile parser without dbf support and ship only the required NE datasets.
Some pros/cons to consider:
Shapefiles don't support node attributes at all, and only support feature and file attributes via dbf files, so different detail levels require separate files, or multiple features in the same file with the dbf attributes to identify the detail level. Either way this would obviously be slower and take more disk space and memory than the PNT files.
Visually comparing it's clear that the 1:50m data is less detailed than the MWDBII, but the 1:10m data is more detailed. This would imply to me that we would have to use the 1:10m data, but to match the features currently displayed from MWDBII using NE shapefiles would take 14 MB which is clearly unacceptable (see table of sizes below). By comparison the PNT files only take 2.6 MB.
The 1:10m country file is 6.55MB and contains 533,202 points = 12.28 bytes/point compared to the PNT which is 745KB and contains 127246 points = 5.85 bytes/point, which would suggest the NE data in PNT format would be half the size, so 6 MB in total. This could probably be further reduced by a light application of Douglas-Peucker.
Visually comparing it's clear the 110, 50, and 10 datasets do not match each other or share common vertices so can't be merged together as the detail levels, they would have to be used separately.
The NE shapefiles have been carefully processed so shared borders and overlapping features like rivers match exactly and other such niceties, applying the Douglas-Peucker algorithm might affect that.
A lightweight shapefile parser would allow users/apps to load other shapefiles.
We would have to reconvert and check the data every time there's a new NE release which could be a lot of effort, but an automated shp2pnt script could prove useful to allow apps/users to display their own shapefiles in a simple way.
Overall, it seems the best approach for the updating the lightweight layer is indeed to convert the shapefiles to PNT format, provided the D-H algorithm can be deployed in a way to mark each point with a detail level rather than just throwing the points away.
For the heavier vector layer with attributes, rather than just implementing a shapefile importer based on shapelib (a C library), it might be better to implement an GDAL/OGR importer which would import any vector format and is a C++ library. That would depend on packaging weight and difficulty, so needs more investigation, but that's for another day.
A lot of the NE data is duplicated / re-mixed, the core shapefiles sizes are:
1:110m 1:50m 1:10m ------ ----- -----
Admin level 0 countries 172 KB 1.36 MB 6.55 MB Admin level 0 land borders 39 KB 301 KB 896 KB Admin level 0 sea borders 12 KB 40 KB 79 KB Admin level 0 disputed 40 KB 157 KB Admin level 1 regions 39 KB 339 MB 13.9 MB * Admin level 1 land borders 16 KB 60 KB 4.82 MB Coastlines 79 KB 883 KB 2.15 MB Rivers 19 KB 420 KB 3.29 MB Lakes 10 KB 286 KB 786 MB Glaciers 13 KB 208 KB 1.23 MB Dateline 18 KB 18 KB 18 KB Playas 18 KB 106 KB Ice Shelves 105 KB 211 KB Minor Islands 449 KB Reefs 171 KB
------- ------- --------- 417 KB 4.08 MB 34.03 MB
perfect for KGeography use :-)
Other useful files: Physical Features Land 146 KB 1.50 MB 692 KB Physical Features Sea 348 KB 836 KB 836 MB Populated Places 347 KB 1.48 MB Urban Areas 439 KB 3.48 MB Bathmetry 11.64 MB
– matched boundary lines and polygons with names attributes for countries and
sovereign states. Includes dependencies (French Polynesia), map units (U.S. Pacific Island Territories) and sub-national map subunits (Corsica versus mainland Metropolitan France).
Disputed areas and breakaway regions
From Kashmir to the Elemi Triangle, Northern Cyprus to Western Sahara.
First order admin (provinces, departments, states, etc.)
– internal boundaries and polygons for all but a few tiny island nations. Includes names attributes and some statistical groupings of the same for smaller countries.
– point symbols with name attributes. Includes capitals, major cities and towns, plus significant smaller towns in sparsely inhabited regions. We favor regional significance over population census in determining rankings.
– derived from 2002-2003 MODIS satellite data.
Parks and protected areas
Pacific nation groupings
– boxes for keeping these far-flung islands tidy.
Water boundary indicators
– partial selection of key 200-mile nautical limits, plus some disputed, treaty, and median lines.
– ocean coastline, including major islands. Coastline is matched to land and water polygons.
– Land polygons including major islands
– Ocean polygon split into contiguous pieces.
– additional small ocean islands ranked to two levels of relative importance.
– major coral reefs from WDB2.
Physical region features
– polygon and point labels of major physical features.
Rivers and Lake Centerlines
– ranked by relative importance. Includes name and line width attributes. Don’t want minor lakes? Turn on their centerlines to avoid unseemly data gaps.
– ranked by relative importance, coordinating with river ranking. Includes name attributes.
– polygons derived from DCW, except for Antarctica derived from MOA. Includes name attributes for major polar glaciers.
Antarctic ice shelves
– derived from 2003-2004 MOA. Reflects recent ice shelf collapses.
– nested polygons at 0, -200, -1,000, -2,000, -3,000, -4,000, -5,000, -6,000, -7,000, -8,000, -9,000,and -10,000 meters. Created from SRTM Plus.
– Polar circles, tropical circles, equator, and International Date Line.
– 1-, 5-, 10-, 15-, 20-, and 30-degree increments. Includes WGS84 bounding box.
Using GeoPainter and GeoDataLineString ("libgeodata"):
New PNT format:
highest bit indicates new polygon starts: header information has to be read from 3rd integer
highest bit feature geometry (line or ring).
1:10 dataset in PNT2 format:
Rather than the Geonames ID, we could just use the Natural Earth object ID, then a look-up file/table that matches the NE ID to the ISO / FIPS / whatever code (NE provides this in the metadata) and Geonames ID (which we would have to provide). This would allow look-ups via whatever code or ID is available, and we wouldn't be reliant on Geonames IDs staying constant.
So required work is: 1) Fix GeoPainter LinearRings which contain a pole not rendered correctly
- Torsten knows, problem will fill in flat map needs to create polygon if closed and crosses dateline once only
2) Implement Douglas-Peuker reduction in GeoDataLineString
- big investigation
3) New PNT file format definition (with a different name, MBL?)
- existing serial format from geodata???
4) Metadata file format definition (could just be csv or xml? or sqlite?) 5) New GeoData PNT2 file loading code (convert olf data). 6) shp2mbl script to convert shp to new formats (using Perl::shp? there's shp2xxx scripts out there we could copy?), including matching to Geonames ID 7) split files into 'ship with', 'download asap', 'ghns'@
Later add simple shapefile loading to GeoData, maybe with attibute layer?