Finish section 02

sources/02e Spectral Indices.ipynb

@@ -6,7 +6,10 @@
    "source": [
     "# Spectral Index Pipeline\n",
     "\n",
-    "Before running this notebook the files have to be downloaded. See [01a Download Process.ipynb](01a Download Process.ipynb) for further information.\n",
+    "Elaborating on the work done in previous sections, this section contains a complete implementation of the calculation of various spectral indicators.\n",
+    "\n",
+    "It does not contain code to download products from the Open Access Hub[^download_process].\n",
+    "It is rather a re-usable notebook that can be re-used for the calculation of indices only.\n",
     "\n",
     "The calculation in this notebook depends on three parameters:\n",
     "\n",
@@ -18,51 +21,21 @@
     "  - ndwi -- normalized difference in water\n",
     "- `fill_value`, the value which is used to represent invalid pixels to handle division by zero.\n",
     "\n",
-    "Change the values below and select _Kernel → Restart and Run All Cells_ to re-evaluate all cells in this notebook.\n",
-    "The path of the output file will be printed below the last cell."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[PosixPath('resources/forest_fires/S2A_MSIL2A_20180807T101021_N0208_R022_T33UUT_20180809T112302.zip'),\n",
-       " PosixPath('resources/forest_fires/S2A_MSIL2A_20180919T102021_N0208_R065_T33UUT_20180919T132226.zip'),\n",
-       " PosixPath('resources/forest_fires/S2A_MSIL2A_20190603T101031_N0212_R022_T33UUT_20190603T114652.zip'),\n",
-       " PosixPath('resources/forest_fires/S2A_MSIL2A_20190613T101031_N0212_R022_T33UUT_20190614T125329.zip'),\n",
-       " PosixPath('resources/forest_fires/S2A_MSIL2A_20190626T102031_N0212_R065_T33UUT_20190626T125319.zip'),\n",
-       " PosixPath('resources/forest_fires/S2A_MSIL2A_20190629T103031_N0212_R108_T32UPE_20190629T135351.zip'),\n",
-       " PosixPath('resources/forest_fires/S2A_MSIL2A_20190726T102031_N0213_R065_T32UPE_20190726T125507.zip'),\n",
-       " PosixPath('resources/forest_fires/S2B_MSIL2A_20180822T101019_N0208_R022_T33UUT_20180822T161243.zip'),\n",
-       " PosixPath('resources/forest_fires/S2B_MSIL2A_20190701T102029_N0212_R065_T32UPE_20190701T134657.zip')]"
-      ]
-     },
-     "execution_count": 1,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "#downloads = ['input/forest_fires/S2A_MSIL2A_20190726T102031_N0213_R065_T32UPE_20190726T125507.zip',\n",
-    "# 'input/forest_fires/S2B_MSIL2A_20190701T102029_N0212_R065_T32UPE_20190701T134657.zip',\n",
-    "# 'input/forest_fires/S2A_MSIL2A_20190629T103031_N0212_R108_T32UPE_20190629T135351.zip']\n",
+    "When running this notebook interactively, _Kernel → Restart and Run All Cells_ can be used to re-evaluate all cells in this notebook after configuring the pipeline.\n",
+    "The path of the output file containing the processed values will be printed below the last cell.\n",
     "\n",
-    "from pathlib import Path\n",
-    "sources = list(sorted(Path('resources/forest_fires').glob('*.zip')))\n",
-    "sources"
+    "[^download_process]: See [](01 Download Process.ipynb) for details"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [],
    "source": [
-    "product_path = Path(sources[0])\n",
+    "from pathlib import Path\n",
+    "\n",
+    "product_path = Path('resources/forest_fires/S2A_MSIL2A_20180807T101021_N0208_R022_T33UUT_20180809T112302.zip')\n",
     "index_to_calculate = 'nbr'"
    ]
   },
@@ -77,7 +50,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -95,8 +68,8 @@
    "source": [
     "## Define Formulas\n",
     "\n",
-    "We define the formulas as data so we can substitute the bands with actual values later on and execute the operations when needed.\n",
-    "We use a lisp-like language with prefix notation for this.\n",
+    "Formulas are defined as data so that the bands can be substituted with actual values later on. By declaratively  expressing the formula calculations computations can be executed lazily and only when needed. \n",
+    "This is done so the formulas can be defined independent from actual data, which is needed only much later.\n",
     "\n",
     "### Operators\n",
     "\n",
@@ -106,12 +79,12 @@
     "\\text{NDVI} = \\frac{\\text{B08} - \\text{B04}}{\\text{B08} + \\text{B04}}\n",
     "$$\n",
     "\n",
-    "To define the index calculation formulas in this way, first the the basic arithmetic operations `+`, `-`, `*` and `/` are wrapped in functions taking variadic arguments:"
+    "To define the index calculation formulas in this way, first the basic arithmetic operations `+`, `-`, `*` and `/` are wrapped in functions taking variadic arguments:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -141,21 +114,20 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "These function are used to define formulas for a selection of indices.\n",
-    "These definitions are not exhaustive - there are many spectral indices which are not implemented in this notebook - however the general shape of theses formulas allows for enough flexibility to implement other indices as well.\n",
+    "These function are used to define formulas for the selection of indices mentioned in the introduction.\n",
+    "These indices are not exhaustive - there are many spectral indices which are not implemented in this notebook. The general shape of theses formulas however allows for enough flexibility to implement other indices as well.\n",
     "\n",
     "The formulas are defined in a lisp-like prefix notation: `(add, 1, 2, 3)` translates to `1 + 2 + 3`.\n",
-    "Each element in a formula can be either a function, a string or a tuple.\n",
+    "Each element in a formula can be either a function, a string or a tuple. Tuples are delimited using `()`. The first element of these tuples is one of the operations defined above. It is followed by at least one other element, which can be any of the following:\n",
     "\n",
-    "This is done so the formulas can be defined independent from actual data.\n",
-    "The data is passed in much later.\n",
-    "\n",
-    "Instead they are defined using tuples, which are delimited using `()`. These tuples have as their first element one of the operations defined above and continue with a flexible amount of other tuples (allowing the recursive expression of formulas), strings, which encode band numbers, or constants:"
+    "- Tuples, allowing the recursive expression of formulas.\n",
+    "- Strings, which encode band numbers.\n",
+    "- Constants, i.e. integers or floats."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -176,12 +148,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "An error is thrown if `index_to_calculate` did not define an implemented index:"
+    "An error is thrown if `index_to_calculate` does not mach any of the implements indices above:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -201,7 +173,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -221,7 +193,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The resolving process needs a `band_map` in the form of `band_num` → `numpy.array`:"
+    "The resolving process needs a `band_map` in the form of `band_num` → `numpy.array`. By defining the arithmetic operations like above, it can be treated like any other python function - read from the formula and called using `op(args)`:"
    ]
   },
   {
@@ -250,7 +222,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Because the prefix-notation is not commonly used to define mathematical formula, a function is given that converts prefix a formula from above to infix notation. This should help to avoid errors when transcribing the index formulas, which are usually given in common infix notation:"
+    "Because the prefix-notation is not commonly used to define mathematical formula, a function is defined that converts prefix a formula from above to infix notation. This should help to avoid errors when transcribing the index formulas, which are usually given in common infix notation:"
    ]
   },
   {
@@ -294,7 +266,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Test Case"
+    "#### Test Cases"
    ]
   },
   {
@@ -372,7 +344,8 @@
    "source": [
     "## Extraction of Relevant Band File Paths\n",
     "\n",
-    "The index calculation starts with the list of bands are referenced by the index formula given by `index_to_calculate`:"
+    "Subsections from here on below contain the actual calculations.\n",
+    "They start with the list of bands are referenced by the index formula given by `index_to_calculate`:"
    ]
   },
   {
@@ -480,58 +453,6 @@
     "These are encoded as metadata in the raster file:"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "PosixPath('zip+file:/home/jovyan/sources/resources/forest_fires/S2A_MSIL2A_20180807T101021_N0208_R022_T33UUT_20180809T112302.zip!/S2A_MSIL2A_20180807T101021_N0208_R022_T33UUT_20180809T112302.SAFE/GRANULE/L2A_T33UUT_A016321_20180807T101024/IMG_DATA/R10m/T33UUT_20180807T101021_B08_10m.jp2')"
-      ]
-     },
-     "execution_count": 17,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from sentinel_helpers import scihub_cloud_mask\n",
-    "import rasterio as r\n",
-    "import matplotlib.pyplot as plt\n",
-    "\n",
-    "highres_raster_path = [band_path for band_path in highest_resolution_band_paths if resolution(band_path) == target_resolution][0]\n",
-    "highres_raster_path"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "((10980, 10980),\n",
-       " Affine(10.0, 0.0, 300000.0,\n",
-       "        0.0, -10.0, 5800020.0))"
-      ]
-     },
-     "execution_count": 18,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "with r.open(highres_raster_path) as src:\n",
-    "    target_transform = src.transform\n",
-    "    target_shape = src.shape\n",
-    "\n",
-    "# shape is height and width of a 2d-array, target_transform is an affine matrix\n",
-    "target_shape, target_transform"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 19,
@@ -562,11 +483,7 @@
    ],
    "source": [
     "# pixels with clouds are True, pixels without are False\n",
-    "raster_cloud_mask = scihub_cloud_mask(product_path,\n",
-    "                                      rasterize=True,\n",
-    "                                      target_shape=target_shape,\n",
-    "                                      # the affine matrix is used to calculate array indices from world coordinates\n",
-    "                                      target_transform=target_transform)\n",
+    "raster_cloud_mask = scihub_cloud_mask(product_path)\n",
     "\n",
     "plt.imshow(raster_cloud_mask)"
    ]