Marble/Sentinel2MapTheme: Difference between revisions

From KDE TechBase
(47 intermediate revisions by the same user not shown)
Line 3: Line 3:
== Introduction/Motivation ==
== Introduction/Motivation ==


TBD
This page is about the process that allows to create a satellite map theme for Marble based on the Sentinel data.


== Sentinels Scientific Data Hub ==
== Sentinels Scientific Data Hub ==
Line 9: Line 9:
=== Registering at the SciHub ===
=== Registering at the SciHub ===


Go to the
Go to the [https://scihub.copernicus.eu/dhus/#/self-registration Registration website of the SciHub]  and enter your name, username, password, e-Mail and country. As a domain you can choose "Land" and "Research" for the usage.


[https://scihub.copernicus.eu/dhus/#/self-registration Registration website of the SciHub]  and enter your name, username, password, e-Mail and country. As a domain you can choose "Land" and "Research" for the usage.
Once you press the '''Register''' button you will receive an e-Mail which allows you to finalize your registration in the SciHub system by pressing the included link.


Once you press the "Register" button you will receive an e-Mail which allows you to finalize your registration in the SciHub system by pressing the included link.
As an alternative to creating your own login you can use "guest/guest" as a username/password when logging in.


=== Looking for data ===
=== Looking for data ===


Go to the [https://scihub.copernicus.eu/dhus/#/home SciHub website] and press the Login Button to enter it.
Go to the [https://scihub.copernicus.eu/dhus/#/home SciHub website] and press the '''Login Button''' to enter it.
On the right side you can press the layers button to choose between the Hybrid, Road (OpenStreetMap) and Satellite map (Blue Marble/LandSat) for orientation.
On the right side you can press the layers button to choose between the Hybrid, Road (OpenStreetMap) and Satellite map (Blue Marble/LandSat) for orientation.


Now we are ready to query some actual Sentinel data. In order to limit our query to useful entries we recommend these adjustments: Press the menu button on the left of the query line edit to enter the Advanced Search
Now we are ready to query some actual Sentinel data. In order to limit our query to useful entries we recommend these adjustments: Press the menu button on the left of the query line edit to enter the Advanced Search


* Check "Mission: Sentinel-2"
[[File:ss1.png]]
* Enter "[0 TO 10]" in the cloud coverage Section


Press the menu button again. The button will turn into a filter icon. Then enter a country name (e.g. Germany). As a result of the query you will get a list of entries. You can browse through the pages. The thumb nails give you an idea about the quality of the photographic images. You can press the "eye" button to get an enlarged version of the thumbnail and to get more information.
* Check '''Mission: Sentinel-2'''
* Enter '''[0 TO 10]''' in the cloud coverage Section


* images that contain an area that hasn't been included in the map theme yet
Press the menu button again. The button will turn into a filter icon. Then enter a '''country name''' (e.g. Germany). As a result of the query you will get a list of entries.
* images that ideally have no cloud cover
 
* images that were taken in bright light.
[[File:ss2.png]]
 
You can browse through the pages. The '''thumb nails''' give you an idea about the quality of the photographic images. You can press the "eye" button to get an enlarged version of the thumbnail and to get more information.
 
* images that contain an '''area that hasn't been included in the map theme''' yet
* images that ideally have '''no cloud cover'''
* images that were taken in '''bright light'''.
* images that includes a maximum amount of complete tiles (with a minimum of tilted cut off black portions).


Once you've made a choice we suggest that you write down:
Once you've made a choice we suggest that you write down:


* the id of the entry (which you can retrieve by pressing the "eye" icon for the entry and copy the id from the title line.
* the '''id''' of the entry (which you can retrieve by pressing the "eye" icon for the entry and copy the id from the title line.
* and the download url
* and the '''download url'''


Good examples are:
in a text file. Good examples are:


id:    S2A_OPER_PRD_MSIL1C_PDMC_20160508T220612_R008_V20160508T104027_20160508T104027
id:    S2A_OPER_PRD_MSIL1C_PDMC_20160508T220612_R008_V20160508T104027_20160508T104027
Line 44: Line 51:
url:  https://scihub.copernicus.eu/dhus/odata/v1/Products('0ded3149-d971-4524-ab80-536ffc1cbb55')/$value
url:  https://scihub.copernicus.eu/dhus/odata/v1/Products('0ded3149-d971-4524-ab80-536ffc1cbb55')/$value


Now click the download link to download the tile. The download will amount to 4-7GB on average. Even if the download gets aborted in the browser it should be possible to continue the download later on.
Now click the download link to download the tile. The download will amount to 4-7GB on average. Even if the download gets aborted in the browser it should be possible to continue the download later on. Depending on the connection the download might take a few hours. At times the SciHub server has temporary maintenance down times.
 
Once the zip file is download '''unzip''' it in a directory of your choice. Make sure that your hard disc still has enough space left (approx. 10 GB).
 
''Note:'' Once QGIS is installed you could also download further tiles using the SCP plugin, however this approach lacks the nice thumbnails for introspection:
 
* [https://www.youtube.com/watch?v=fVS2Ls2bUbk Search and Download Sentinel-2 Images Using the Semi-Automatic Classification Plugin ]


== Installing QGIS and plugins ==
== Using QGIS to edit the Sentinel 2 data ==


=== QGIS installation ===
=== QGIS installation ===
Line 57: Line 70:
Type in a terminal:
Type in a terminal:


sudo apt-get install qgis  
'''sudo apt-get install qgis'''


=== Installing the required plugins ===
=== Installing the required plugins ===
Line 71: Line 84:
Once installed please right-click the toolbar area and enable the '''SCP Toolbar''' .
Once installed please right-click the toolbar area and enable the '''SCP Toolbar''' .


=== Opening the Sentinel data in QGIS using the SCP plugin ===
Launch QGIS and choose '''Layer -> Add Layer -> Add Raster Layer'''  (or the associated toolbar '''checkboard''' button).
Then navigate into the '''GRANULE''' directory. There choose the directory for the correct sub tile. The order of the tiles follows columns.
Inside the sub tile directory use the Ctrl button to '''choose multiple files at once''' - namly those which end on b2-b8, b8a, b11, b12:
'''b2.jp2, b3.jp2, b4.jp2, b5.jp2, b6.jp2, b7.jp2, b8.jp2, b8a.jp2, b11.jp2, b12.jp2'''
Press '''Open''' to open them and wait for them to be loaded which might take about 10 secs.
For convenience you might want to press the '''Collapse All''' button in the Layers panel.
[[File:ss3.png]]
Now press the rainbow colored "Band set" button in the SCP toolbar. All files should appear in the upper section. Now apply the following steps:
* Press '''Select All'''
* Press '''Add rasters to set'''
* In the middle to the right press the '''Sort by Name''' button
* In the '''Quick wavelength settings''' press the drop down list combobox, scroll down and choose '''Sentinel-2 [bands 2, 3, 4, ...]''' (if it is Sentinel-2 right from the start you might need to change to a different entry and back to Sentinel-2 to apply the correct wave length band sets).
* Press '''Create virtual raster of band set'''
* Enter a '''band_set.vrt''' as a file name.
* Press '''Close'''
* Then in the SCP Toolbar choose RGB first to be 4-3-2 '''then change it to be 3-2-1''' from the combobox (only after changing the correct color will be applied).
You should see a colorized version of your loaded tiles now. We suggest that you tweak the colors by right clicking on the vrt file entry on the right and press "Properties". In the Band rendering section we suggest that you make the following adjustments:
[[File:ss3b.png]]
Red Band: Band 03 - 398 2310
Green Band: Band 02 - 630 1962
Blue Band: Band 01 - 851 1903
In the Resampling section choose
* Zoomed: in '''Bilinear''' out '''Average''' Oversampling 2,00
* Press '''Apply'''
As a result you should now see a nice relatively realistic looking rendering of the sentinel data.
[[File:ss4.png]]
You can zoom into it using the mouse scroll wheel down to street level (10m per pixel).
[[File:ss5.png]]
''Note:''
For further illustration we also recommend this [https://www.youtube.com/watch?v=HbxbbGyHjNs video].
For more information please consult the [https://media.readthedocs.org/pdf/semiautomaticclassificationmanual-v4/latest/semiautomaticclassificationmanual-v4.pdf Documentation]
''Questions: ''
''1. are there other ways how we could achieve more realistic and better colors?''
=== Tile Creation ===
With the new sentinel layer loaded and proper band sets in place we can now start to create our own tiles:
Press '''Plugins -> QTiles -> QTiles'''  then


* In the ''Output'' section choose '''Directory''' and browse to your directory of choice where the tiles should be stored
* Enter '''sentinel2''' as a Tileset name
* In the ''Extent'' section choose '''Layer Extent'''
* In the ''Zoom'' section choose '''14''' as a maximum zoom level
* In the ''Parameters'' section enable '''Make lines appear less jagged''' and '''Render tiles outside of layers extents (within combined extent)'''.




[[File:ss6.png]]


''Questions: ''


''1. Are there any other alternatives to using QTiles? Any better QGIS plugins than this?''


''2. How can we merge multiple datasets?''


=== Conversion / Reprojection===
== Creating as custom DGML file for the Sentinel 2 data set ==


TBD
Creation of the DGML file should mostly be compatible with


=== Tiling ===
[https://techbase.kde.org/Marble/OSMSlippyMaps How to integrate OSM "slippy" maps into Marble]


TBD
== Deployment on the server ==


=== Deployment on the server ===
For deploying the tiles on the server follow


TBD
[https://techbase.kde.org/Marble/ServerUpload Upload to our KDE server]

Revision as of 23:20, 24 June 2016

Satellite Map Theme based on the Sentinel 2 Mission

Introduction/Motivation

This page is about the process that allows to create a satellite map theme for Marble based on the Sentinel data.

Sentinels Scientific Data Hub

Registering at the SciHub

Go to the Registration website of the SciHub and enter your name, username, password, e-Mail and country. As a domain you can choose "Land" and "Research" for the usage.

Once you press the Register button you will receive an e-Mail which allows you to finalize your registration in the SciHub system by pressing the included link.

As an alternative to creating your own login you can use "guest/guest" as a username/password when logging in.

Looking for data

Go to the SciHub website and press the Login Button to enter it. On the right side you can press the layers button to choose between the Hybrid, Road (OpenStreetMap) and Satellite map (Blue Marble/LandSat) for orientation.

Now we are ready to query some actual Sentinel data. In order to limit our query to useful entries we recommend these adjustments: Press the menu button on the left of the query line edit to enter the Advanced Search

  • Check Mission: Sentinel-2
  • Enter [0 TO 10] in the cloud coverage Section

Press the menu button again. The button will turn into a filter icon. Then enter a country name (e.g. Germany). As a result of the query you will get a list of entries.

You can browse through the pages. The thumb nails give you an idea about the quality of the photographic images. You can press the "eye" button to get an enlarged version of the thumbnail and to get more information.

  • images that contain an area that hasn't been included in the map theme yet
  • images that ideally have no cloud cover
  • images that were taken in bright light.
  • images that includes a maximum amount of complete tiles (with a minimum of tilted cut off black portions).

Once you've made a choice we suggest that you write down:

  • the id of the entry (which you can retrieve by pressing the "eye" icon for the entry and copy the id from the title line.
  • and the download url

in a text file. Good examples are:

id: S2A_OPER_PRD_MSIL1C_PDMC_20160508T220612_R008_V20160508T104027_20160508T104027 url: https://scihub.copernicus.eu/dhus/odata/v1/Products('4658007a-76a0-455c-bb9a-d9e789b5e1e3')/$value

id: S2A_OPER_PRD_MSIL1C_PDMC_20160508T221741_R008_V20160508T104027_20160508T104027 url: https://scihub.copernicus.eu/dhus/odata/v1/Products('0ded3149-d971-4524-ab80-536ffc1cbb55')/$value

Now click the download link to download the tile. The download will amount to 4-7GB on average. Even if the download gets aborted in the browser it should be possible to continue the download later on. Depending on the connection the download might take a few hours. At times the SciHub server has temporary maintenance down times.

Once the zip file is download unzip it in a directory of your choice. Make sure that your hard disc still has enough space left (approx. 10 GB).

Note: Once QGIS is installed you could also download further tiles using the SCP plugin, however this approach lacks the nice thumbnails for introspection:

Using QGIS to edit the Sentinel 2 data

QGIS installation

In order to process the data you need to install QGIS. On Ubuntu/Debian you can do this e.g. by opening a terminal and type:

sudo apt-get update

Type in a terminal:

sudo apt-get install qgis

Installing the required plugins

Launch QGIS and enter the Plugins menu. There choose Manage and Install Plugins. Select Not Installed and choose the following plugins for installation in the list:

  • Semi-Automatic Classification Plugin
  • QTiles
  • OpenLayers Plugin


Once installed please right-click the toolbar area and enable the SCP Toolbar .

Opening the Sentinel data in QGIS using the SCP plugin

Launch QGIS and choose Layer -> Add Layer -> Add Raster Layer (or the associated toolbar checkboard button). Then navigate into the GRANULE directory. There choose the directory for the correct sub tile. The order of the tiles follows columns. Inside the sub tile directory use the Ctrl button to choose multiple files at once - namly those which end on b2-b8, b8a, b11, b12:

b2.jp2, b3.jp2, b4.jp2, b5.jp2, b6.jp2, b7.jp2, b8.jp2, b8a.jp2, b11.jp2, b12.jp2

Press Open to open them and wait for them to be loaded which might take about 10 secs.

For convenience you might want to press the Collapse All button in the Layers panel.


Now press the rainbow colored "Band set" button in the SCP toolbar. All files should appear in the upper section. Now apply the following steps:

  • Press Select All
  • Press Add rasters to set
  • In the middle to the right press the Sort by Name button
  • In the Quick wavelength settings press the drop down list combobox, scroll down and choose Sentinel-2 [bands 2, 3, 4, ...] (if it is Sentinel-2 right from the start you might need to change to a different entry and back to Sentinel-2 to apply the correct wave length band sets).
  • Press Create virtual raster of band set
  • Enter a band_set.vrt as a file name.
  • Press Close
  • Then in the SCP Toolbar choose RGB first to be 4-3-2 then change it to be 3-2-1 from the combobox (only after changing the correct color will be applied).

You should see a colorized version of your loaded tiles now. We suggest that you tweak the colors by right clicking on the vrt file entry on the right and press "Properties". In the Band rendering section we suggest that you make the following adjustments:


Red Band: Band 03 - 398 2310

Green Band: Band 02 - 630 1962

Blue Band: Band 01 - 851 1903


In the Resampling section choose

  • Zoomed: in Bilinear out Average Oversampling 2,00
  • Press Apply

As a result you should now see a nice relatively realistic looking rendering of the sentinel data.



You can zoom into it using the mouse scroll wheel down to street level (10m per pixel).



Note:

For further illustration we also recommend this video.

For more information please consult the Documentation

Questions:

1. are there other ways how we could achieve more realistic and better colors?

Tile Creation

With the new sentinel layer loaded and proper band sets in place we can now start to create our own tiles:

Press Plugins -> QTiles -> QTiles then

  • In the Output section choose Directory and browse to your directory of choice where the tiles should be stored
  • Enter sentinel2 as a Tileset name
  • In the Extent section choose Layer Extent
  • In the Zoom section choose 14 as a maximum zoom level
  • In the Parameters section enable Make lines appear less jagged and Render tiles outside of layers extents (within combined extent).


Questions:

1. Are there any other alternatives to using QTiles? Any better QGIS plugins than this?

2. How can we merge multiple datasets?

Creating as custom DGML file for the Sentinel 2 data set

Creation of the DGML file should mostly be compatible with

How to integrate OSM "slippy" maps into Marble

Deployment on the server

For deploying the tiles on the server follow

Upload to our KDE server