diff --git a/Dockerfile b/Dockerfile index fc332c5..0cca693 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,7 +11,8 @@ ENV JUPYTER_ENABLE_LAB=yes RUN conda install -c conda-forge geopandas==0.8.1 && \ pip install \ descartes==1.1.0 sentinelsat==0.14 rasterio==1.2.0 folium==0.11.0 \ - jupyterlab-spellchecker \ - jupyter-book 0.10.0 && \ + jupyterlab-spellchecker nbdime \ + jupyter-book==0.10.0 && \ fix-permissions $CONDA_DIR && \ - fix-permissions /home/$NB_USER + fix-permissions /home/$NB_USER && \ + nbdime extensions --enable diff --git a/README.md b/README.md index 78db5ce..e39bdc4 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Remote sensing for journalism +# Remote Sensing for Journalism This repository contains a series of notebooks describing interaction with the Copernicus Open Acces Hub in order to obtain and manipulate earth observation data. The aim is to document common tasks that might make the data from the Copernicus Sentinel missions attractive for usage in data journalism reporting. @@ -11,7 +11,7 @@ Copernicus Open Access Hub is the platform which is openly distributing the Terr A (free) Scihub account is needed in order to follow this documentation interactively. The registration form can be found at https://scihub.copernicus.eu/dhus/. -## Obtaining and running the code +## Obtaining and Running the Code The source code lives at https://github.com/heyarne/remote-sensing-for-journalism. @@ -39,6 +39,12 @@ This starts up a `JupyterLab` environment which allows you to interactively exec The Docker image is based on the [jupyter/scipy-notebook](https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook). Follow the link for more information on installed packages or other configuration details. +## Hardware Requirements + +Note that working with this kind of data is resource intensive. +These notebooks download or create roughly 50GB of data, most of which is occupied by compressed GeoTIFF files. +They have been executed and tested on a virtual server with 4 CPU cores at 2.6 GHz each and 32 GB of RAM. + ## Building the Jupyter Book The `jupyter-book` dependency is included in the `Dockerfile`. diff --git a/_config.yml b/_config.yml index e60d01a..4262d27 100755 --- a/_config.yml +++ b/_config.yml @@ -8,7 +8,6 @@ title: Remote Sensing for Journalism # The title of the book. Will be placed in the left navbar. author: Arne Schlüter # The author of the book copyright: "CC-BY-SA 4.0 2021" # Copyright year to be placed in the footer +logo: great-forty-foot-telescope-wikipedia.png # A path to the book logo execute: execute_notebooks: "off" - -# logo : logo.png # A path to the book logo diff --git a/_static/custom.css b/_static/custom.css new file mode 100644 index 0000000..e49cdc4 --- /dev/null +++ b/_static/custom.css @@ -0,0 +1,4 @@ +div.navbar-brand-box a.navbar-brand img { + max-height: unset !important; + max-width: 80% +} diff --git a/_toc.yml b/_toc.yml index dd64b98..56d60f9 100644 --- a/_toc.yml +++ b/_toc.yml @@ -3,7 +3,8 @@ - file: README - part: "Data retrieval and processing" chapters: - - file: "remote-sensing-research/01a Download from Scihub.ipynb" - - file: "remote-sensing-research/01b Plot downloaded products.ipynb" - - file: "remote-sensing-research/01c Coverage analysis.ipynb" - - file: "remote-sensing-research/01 True-color pipeline.ipynb" + - file: "sources/01 Data retrieval and processing.md" + - file: "sources/01a Download from Scihub.ipynb" + - file: "sources/01b Plot downloaded products.ipynb" + - file: "sources/01c Coverage analysis.ipynb" + - file: "sources/01 True-color pipeline.ipynb" diff --git a/great-forty-foot-telescope-wikipedia.png b/great-forty-foot-telescope-wikipedia.png new file mode 100644 index 0000000..ee54950 Binary files /dev/null and b/great-forty-foot-telescope-wikipedia.png differ diff --git a/sources/01 Data retrieval and processing.md b/sources/01 Data retrieval and processing.md new file mode 100644 index 0000000..1ce7130 --- /dev/null +++ b/sources/01 Data retrieval and processing.md @@ -0,0 +1,7 @@ +# Data Retrieval and Processing + +The first chapter contains notebooks that document the data retrieval process from the Copernicus Open Access Hub. + +It explains how to use [`sentinelsat`](https://github.com/sentinelsat/sentinelsat) library to interact with the API: How to specify what kind of data we are interested in with the aide of open map data from [OpenStreetMap](https://www.openstreetmap.org/) and how to download it. It then goes on to show how to use [`rasterio`](https://rasterio.readthedocs.io/) and [`matplotlib`](https://matplotlib.org/stable/index.html) to work with the data. + +The Sentinel-2 satellite captures light in visible and invisible parts of the spectrum. This can be used to derive different kinds of useful information about ground-level phenomena. This chapter will detail how to use it to create true-color images for different moments in time. \ No newline at end of file diff --git a/remote-sensing-research/01 True-color pipeline.ipynb b/sources/01 True-color pipeline.ipynb similarity index 99% rename from remote-sensing-research/01 True-color pipeline.ipynb rename to sources/01 True-color pipeline.ipynb index bcf9f01..8f6d164 100644 --- a/remote-sensing-research/01 True-color pipeline.ipynb +++ b/sources/01 True-color pipeline.ipynb @@ -1,21 +1,5 @@ { "cells": [ - { - "cell_type": "code", - "execution_count": 32, - "metadata": {}, - "outputs": [], - "source": [ - "import rasterio" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, { "cell_type": "markdown", "metadata": {}, diff --git a/remote-sensing-research/01a Download from Scihub.ipynb b/sources/01a Download from Scihub.ipynb similarity index 100% rename from remote-sensing-research/01a Download from Scihub.ipynb rename to sources/01a Download from Scihub.ipynb diff --git a/remote-sensing-research/01b Plot downloaded products.ipynb b/sources/01b Plot downloaded products.ipynb similarity index 99% rename from remote-sensing-research/01b Plot downloaded products.ipynb rename to sources/01b Plot downloaded products.ipynb index 1cd2b66..564a17d 100644 --- a/remote-sensing-research/01b Plot downloaded products.ipynb +++ b/sources/01b Plot downloaded products.ipynb @@ -64,7 +64,7 @@ { "data": { "text/plain": [ - "PosixPath('input/raster/true_color_image/S2B_MSIL2A_20200602T100559_N0214_R022_T33UUU_20200602T135324.zip')" + "PosixPath('input/true_color_image/S2B_MSIL2A_20200602T100559_N0214_R022_T33UUU_20200602T135324.zip')" ] }, "execution_count": 2, @@ -577,6 +577,13 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "state": {}, + "version_major": 2, + "version_minor": 0 + } } }, "nbformat": 4, diff --git a/remote-sensing-research/01c Coverage analysis.ipynb b/sources/01c Coverage analysis.ipynb similarity index 100% rename from remote-sensing-research/01c Coverage analysis.ipynb rename to sources/01c Coverage analysis.ipynb diff --git a/remote-sensing-research/01d Brandenburg mosaic.ipynb b/sources/01d Brandenburg mosaic.ipynb similarity index 100% rename from remote-sensing-research/01d Brandenburg mosaic.ipynb rename to sources/01d Brandenburg mosaic.ipynb diff --git a/remote-sensing-research/02 Spectral Indices.ipynb b/sources/02 Spectral Indices.ipynb similarity index 100% rename from remote-sensing-research/02 Spectral Indices.ipynb rename to sources/02 Spectral Indices.ipynb diff --git a/remote-sensing-research/02a NDVI.ipynb b/sources/02a NDVI.ipynb similarity index 100% rename from remote-sensing-research/02a NDVI.ipynb rename to sources/02a NDVI.ipynb diff --git a/remote-sensing-research/02b Timeseries.ipynb b/sources/02b Timeseries.ipynb similarity index 100% rename from remote-sensing-research/02b Timeseries.ipynb rename to sources/02b Timeseries.ipynb diff --git a/remote-sensing-research/02c Corrupted Zip File.ipynb b/sources/02c Corrupted Zip File.ipynb similarity index 100% rename from remote-sensing-research/02c Corrupted Zip File.ipynb rename to sources/02c Corrupted Zip File.ipynb diff --git a/remote-sensing-research/02c Multithreading.ipynb b/sources/02c Multithreading.ipynb similarity index 100% rename from remote-sensing-research/02c Multithreading.ipynb rename to sources/02c Multithreading.ipynb diff --git a/remote-sensing-research/02d Cloud Masks.ipynb b/sources/02d Cloud Masks.ipynb similarity index 100% rename from remote-sensing-research/02d Cloud Masks.ipynb rename to sources/02d Cloud Masks.ipynb diff --git a/remote-sensing-research/03a Forest Fire Product Downloads.ipynb b/sources/03a Forest Fire Product Downloads.ipynb similarity index 100% rename from remote-sensing-research/03a Forest Fire Product Downloads.ipynb rename to sources/03a Forest Fire Product Downloads.ipynb diff --git a/remote-sensing-research/03b Example Initialized Downloads.ipynb b/sources/03b Example Initialized Downloads.ipynb similarity index 100% rename from remote-sensing-research/03b Example Initialized Downloads.ipynb rename to sources/03b Example Initialized Downloads.ipynb diff --git a/remote-sensing-research/03b Forest Fires 2018 and 2019.ipynb b/sources/03b Forest Fires 2018 and 2019.ipynb similarity index 100% rename from remote-sensing-research/03b Forest Fires 2018 and 2019.ipynb rename to sources/03b Forest Fires 2018 and 2019.ipynb diff --git a/remote-sensing-research/03c dNBR.ipynb b/sources/03c dNBR.ipynb similarity index 100% rename from remote-sensing-research/03c dNBR.ipynb rename to sources/03c dNBR.ipynb diff --git a/remote-sensing-research/input/berlin.geojson b/sources/input/berlin.geojson similarity index 100% rename from remote-sensing-research/input/berlin.geojson rename to sources/input/berlin.geojson diff --git a/remote-sensing-research/input/forest_fires/evacuated_2018.json b/sources/input/forest_fires/evacuated_2018.json similarity index 100% rename from remote-sensing-research/input/forest_fires/evacuated_2018.json rename to sources/input/forest_fires/evacuated_2018.json diff --git a/remote-sensing-research/input/forest_fires/jueterbog_2019.json b/sources/input/forest_fires/jueterbog_2019.json similarity index 100% rename from remote-sensing-research/input/forest_fires/jueterbog_2019.json rename to sources/input/forest_fires/jueterbog_2019.json diff --git a/remote-sensing-research/input/forest_fires/luebtheen_2019.json b/sources/input/forest_fires/luebtheen_2019.json similarity index 100% rename from remote-sensing-research/input/forest_fires/luebtheen_2019.json rename to sources/input/forest_fires/luebtheen_2019.json diff --git a/remote-sensing-research/input/tempelhofer_feld/tempelhofer_feld.geojson b/sources/input/tempelhofer_feld/tempelhofer_feld.geojson similarity index 100% rename from remote-sensing-research/input/tempelhofer_feld/tempelhofer_feld.geojson rename to sources/input/tempelhofer_feld/tempelhofer_feld.geojson diff --git a/remote-sensing-research/input/tempelhofer_feld_test/tempelhofer_feld.geojson b/sources/input/tempelhofer_feld_test/tempelhofer_feld.geojson similarity index 100% rename from remote-sensing-research/input/tempelhofer_feld_test/tempelhofer_feld.geojson rename to sources/input/tempelhofer_feld_test/tempelhofer_feld.geojson diff --git a/remote-sensing-research/output/2020-05-26.svg b/sources/output/2020-05-26.svg similarity index 100% rename from remote-sensing-research/output/2020-05-26.svg rename to sources/output/2020-05-26.svg diff --git a/remote-sensing-research/output/2020-05-28.svg b/sources/output/2020-05-28.svg similarity index 100% rename from remote-sensing-research/output/2020-05-28.svg rename to sources/output/2020-05-28.svg diff --git a/remote-sensing-research/output/2020-05-29.svg b/sources/output/2020-05-29.svg similarity index 100% rename from remote-sensing-research/output/2020-05-29.svg rename to sources/output/2020-05-29.svg diff --git a/remote-sensing-research/output/2020-05-30.svg b/sources/output/2020-05-30.svg similarity index 100% rename from remote-sensing-research/output/2020-05-30.svg rename to sources/output/2020-05-30.svg diff --git a/remote-sensing-research/output/2020-05-31.svg b/sources/output/2020-05-31.svg similarity index 100% rename from remote-sensing-research/output/2020-05-31.svg rename to sources/output/2020-05-31.svg diff --git a/remote-sensing-research/output/2020-06-02.svg b/sources/output/2020-06-02.svg similarity index 100% rename from remote-sensing-research/output/2020-06-02.svg rename to sources/output/2020-06-02.svg diff --git a/remote-sensing-research/output/2020-06-03.svg b/sources/output/2020-06-03.svg similarity index 100% rename from remote-sensing-research/output/2020-06-03.svg rename to sources/output/2020-06-03.svg diff --git a/remote-sensing-research/output/2020-06-12.svg b/sources/output/2020-06-12.svg similarity index 100% rename from remote-sensing-research/output/2020-06-12.svg rename to sources/output/2020-06-12.svg diff --git a/remote-sensing-research/output/2020-06-13.svg b/sources/output/2020-06-13.svg similarity index 100% rename from remote-sensing-research/output/2020-06-13.svg rename to sources/output/2020-06-13.svg diff --git a/remote-sensing-research/output/2020-06-16.svg b/sources/output/2020-06-16.svg similarity index 100% rename from remote-sensing-research/output/2020-06-16.svg rename to sources/output/2020-06-16.svg diff --git a/remote-sensing-research/output/2020-06-17.svg b/sources/output/2020-06-17.svg similarity index 100% rename from remote-sensing-research/output/2020-06-17.svg rename to sources/output/2020-06-17.svg diff --git a/remote-sensing-research/output/2020-06-22.svg b/sources/output/2020-06-22.svg similarity index 100% rename from remote-sensing-research/output/2020-06-22.svg rename to sources/output/2020-06-22.svg diff --git a/remote-sensing-research/output/2020-06-23.svg b/sources/output/2020-06-23.svg similarity index 100% rename from remote-sensing-research/output/2020-06-23.svg rename to sources/output/2020-06-23.svg diff --git a/remote-sensing-research/output/2020-06-25.svg b/sources/output/2020-06-25.svg similarity index 100% rename from remote-sensing-research/output/2020-06-25.svg rename to sources/output/2020-06-25.svg diff --git a/remote-sensing-research/output/2020-06-27.svg b/sources/output/2020-06-27.svg similarity index 100% rename from remote-sensing-research/output/2020-06-27.svg rename to sources/output/2020-06-27.svg diff --git a/remote-sensing-research/output/2020-06-30.svg b/sources/output/2020-06-30.svg similarity index 100% rename from remote-sensing-research/output/2020-06-30.svg rename to sources/output/2020-06-30.svg diff --git a/remote-sensing-research/output/2020-07-04.svg b/sources/output/2020-07-04.svg similarity index 100% rename from remote-sensing-research/output/2020-07-04.svg rename to sources/output/2020-07-04.svg diff --git a/remote-sensing-research/output/2020-07-12.svg b/sources/output/2020-07-12.svg similarity index 100% rename from remote-sensing-research/output/2020-07-12.svg rename to sources/output/2020-07-12.svg diff --git a/remote-sensing-research/output/2020-07-13.svg b/sources/output/2020-07-13.svg similarity index 100% rename from remote-sensing-research/output/2020-07-13.svg rename to sources/output/2020-07-13.svg diff --git a/remote-sensing-research/output/2020-07-15.svg b/sources/output/2020-07-15.svg similarity index 100% rename from remote-sensing-research/output/2020-07-15.svg rename to sources/output/2020-07-15.svg diff --git a/remote-sensing-research/output/2020-07-18.svg b/sources/output/2020-07-18.svg similarity index 100% rename from remote-sensing-research/output/2020-07-18.svg rename to sources/output/2020-07-18.svg diff --git a/remote-sensing-research/sentinel_helpers.py b/sources/sentinel_helpers.py similarity index 100% rename from remote-sensing-research/sentinel_helpers.py rename to sources/sentinel_helpers.py