Start writing jupyter book

This commit is contained in:
heyarne 2021-03-01 11:47:21 +00:00
commit 58749556b8
19 changed files with 57 additions and 498 deletions

4
.gitignore vendored
View file

@ -1,3 +1,7 @@
# ignore jupyter-book build artefacts
/_build
.bash_history
# avoid exposing secrets
.netrc

View file

@ -1,17 +1,51 @@
# Earth Observation Data in a Journalistic Context
# Remote sensing for journalism
This repository contains a series of notebooks describing interaction with the Copernicus Open Acces Hub in order to obtain and manipulate earth observation data. A (free) Scihub account is needed in order to execute them, the registration form can be found at https://scihub.copernicus.eu/dhus/.
This repository contains a series of notebooks describing interaction with the Copernicus Open Acces Hub in order to obtain and manipulate earth observation data.
The aim is to document common tasks that might make the data from the Copernicus Sentinel missions attractive for usage in data journalism reporting.
The Docker image can be built using the following command:
The publication and research uses [Jupyter notebooks](https://jupyter.org) and is published using [jupyter-books](https://jupyter-book.org), an open-source python project that allows generating HTML pages from a collection of Jupyter notebooks.
## Copernicus Open Access Hub
Copernicus Open Access Hub is the platform which is openly distributing the Terrabytes of Sentinel-2 data which these notebooks rely on.
A (free) Scihub account is needed in order to follow this documentation interactively.
The registration form can be found at https://scihub.copernicus.eu/dhus/.
## Obtaining and running the code
The source code lives at https://github.com/heyarne/remote-sensing-for-journalism.
A `Dockerfile` is present at the root of the repository to help with reproducing the computing environment.
The image can be built by running the following command from the project root:
``` bash
docker build . -t eratosthenes:latest
```
The resulting notebook is based on the [jupyter/scipy-notebook](https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook). Follow the link for more information on installed packages or other configuration details.
You need to define your Scihub user credentials as environment variables in order to be able to execute the notebooks:
When running the docker image you need to define your Scihub user credentials as environment variables:
``` bash
docker run -it -e SCIHUB_USERNAME='<username>' -e SCIHUB_PASSWORD='<password>' eratosthenes:latest
docker run -it \
--name eratosthenes \
--net host \
--volume "$(pwd)":/home/jovyan \
-e SCIHUB_USERNAME='<username>' \
-e SCIHUB_PASSWORD='<password>' \
eratosthenes:latest
```
This starts up a `JupyterLab` environment which allows you to interactively execute all notebooks and modify them to suit your needs.
The Docker image is based on the [jupyter/scipy-notebook](https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook).
Follow the link for more information on installed packages or other configuration details.
## Building the Jupyter Book
The `jupyter-book` dependency is included in the `Dockerfile`.
You can build a book from a running container by executing the following command on the Docker host:
```
docker run -it eratosthenes jupyter-book build .
```
The resulting book can be found in the directory `_build/html/`.

View file

@ -5,7 +5,10 @@
#######################################################################################
# Book settings
title : Sentinel Data for Earth Observation # The title of the book. Will be placed in the left navbar.
title: Remote Sensing for Journalism # The title of the book. Will be placed in the left navbar.
author: Arne Schlüter # The author of the book
copyright : "2020" # Copyright year to be placed in the footer
logo : logo.png # A path to the book logo
copyright: "CC-BY-SA 4.0 2021" # Copyright year to be placed in the footer
execute:
execute_notebooks: "off"
# logo : logo.png # A path to the book logo

View file

@ -1,6 +1,9 @@
# Table of contents
# Learn more at https://jupyterbook.org/customize/toc.html
- file: README
- part: True Color Images
- part: "Data retrieval and processing"
chapters:
- file: "true-color-image/00 Download and plot single true-color image"
- file: "remote-sensing-research/01a Download from Scihub.ipynb"
- file: "remote-sensing-research/01b Plot downloaded products.ipynb"
- file: "remote-sensing-research/01c Coverage analysis.ipynb"
- file: "remote-sensing-research/01 True-color pipeline.ipynb"

View file

@ -1,44 +0,0 @@
# Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant, version 1.4](http://contributor-covenant.org/version/1/4).

View file

@ -1,56 +0,0 @@
# Contributing
Contributions are welcome, and they are greatly appreciated! Every little bit
helps, and credit will always be given. You can contribute in the ways listed below.
## Report Bugs
Report bugs using GitHub issues.
If you are reporting a bug, please include:
* Your operating system name and version.
* Any details about your local setup that might be helpful in troubleshooting.
* Detailed steps to reproduce the bug.
## Fix Bugs
Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
wanted" is open to whoever wants to implement it.
## Implement Features
Look through the GitHub issues for features. Anything tagged with "enhancement"
and "help wanted" is open to whoever wants to implement it.
## Write Documentation
Sentinel Data for Earth Observation could always use more documentation, whether as part of the
official Sentinel Data for Earth Observation docs, in docstrings, or even on the web in blog posts,
articles, and such.
## Submit Feedback
The best way to send feedback is to file an issue on GitHub.
If you are proposing a feature:
* Explain in detail how it would work.
* Keep the scope as narrow as possible, to make it easier to implement.
* Remember that this is a volunteer-driven project, and that contributions
are welcome :)
## Get Started
Ready to contribute? Here's how to set up `Sentinel Data for Earth Observation` for local development.
1. Fork the repo on GitHub.
2. Clone your fork locally.
3. Install your local copy into a virtualenv, e.g., using `conda`.
4. Create a branch for local development and make changes locally.
5. Commit your changes and push your branch to GitHub.
6. Submit a pull request through the GitHub website.
## Code of Conduct
Please note that the Sentinel Data for Earth Observation project is released with a [Contributor Code of Conduct](CONDUCT.md). By contributing to this project you agree to abide by its terms.

View file

@ -1,33 +0,0 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
This cookiecutter creates a simple boilerplate for a Jupyter Book.
Copyright (C) 2020 Arne Schlüter
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.

View file

@ -1,35 +0,0 @@
# Sentinel Data for Earth Observation
This cookiecutter creates a simple boilerplate for a Jupyter Book.
## Usage
### Building the book
If you'd like to develop on and build the Sentinel Data for Earth Observation book, you should:
- Clone this repository and run
- Run `pip install -r requirements.txt` (it is recommended you do this within a virtual environment)
- (Recommended) Remove the existing `Sentinel Data for Earth Observation/_build/` directory
- Run `jupyter-book build Sentinel Data for Earth Observation/`
A fully-rendered HTML version of the book will be built in `Sentinel Data for Earth Observation/_build/html/`.
### Hosting the book
The html version of the book is hosted on the `gh-pages` branch of this repo. A GitHub actions workflow has been created that automatically builds and pushes the book to this branch on a push or pull request to main.
If you wish to disable this automation, you may remove the GitHub actions workflow and build the book manually by:
- Navigating to your local build; and running,
- `ghp-import -n -p -f Sentinel Data for Earth Observation/_build/html`
This will automatically push your build to the `gh-pages` branch. More information on this hosting process can be found [here](https://jupyterbook.org/publish/gh-pages.html#manually-host-your-book-with-github-pages).
## Contributors
We welcome and recognize all contributions. You can see a list of current contributors in the [contributors tab](https://github.com/heyarne/sentinel_data_for_earth_observation/graphs/contributors).
## Credits
This project is created using the excellent open source [Jupyter Book project](https://jupyterbook.org/) and the [executablebooks/cookiecutter-jupyter-book template](https://github.com/executablebooks/cookiecutter-jupyter-book).

View file

@ -1,3 +0,0 @@
jupyter-book
matplotlib
numpy

View file

@ -1,5 +0,0 @@
Content in Jupyter Book
=======================
There are many ways to write content in Jupyter Book. This short section
covers a few tips for how to do so.

View file

@ -1,7 +0,0 @@
Welcome to your Jupyter Book
============================
This is a small sample book to give you a feel for how book content is
structured.
Check out the content pages bundled with this sample book to get started.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.6 KiB

View file

@ -1,125 +0,0 @@
# Markdown Files
Whether you write your book's content in Jupyter Notebooks (`.ipynb`) or
in regular markdown files (`.md`), you'll write in the same flavor of markdown
called **MyST Markdown**.
## What is MyST?
MyST stands for "Markedly Structured Text". It
is a slight variation on a flavor of markdown called "CommonMark" markdown,
with small syntax extensions to allow you to write **roles** and **directives**
in the Sphinx ecosystem.
## What are roles and directives?
Roles and directives are two of the most powerful tools in Jupyter Book. They
are kind of like functions, but written in a markup language. They both
serve a similar purpose, but **roles are written in one line**, whereas
**directives span many lines**. They both accept different kinds of inputs,
and what they do with those inputs depends on the specific role or directive
that is being called.
### Using a directive
At its simplest, you can insert a directive into your book's content like so:
````
```{mydirectivename}
My directive content
```
````
This will only work if a directive with name `mydirectivename` already exists
(which it doesn't). There are many pre-defined directives associated with
Jupyter Book. For example, to insert a note box into your content, you can
use the following directive:
````
```{note}
Here is a note
```
````
This results in:
```{note}
Here is a note
```
In your built book.
For more information on writing directives, see the
[MyST documentation](https://myst-parser.readthedocs.io/).
### Using a role
Roles are very similar to directives, but they are less-complex and written
entirely on one line. You can insert a role into your book's content with
this pattern:
```
Some content {rolename}`and here is my role's content!`
```
Again, roles will only work if `rolename` is a valid role's name. For example,
the `doc` role can be used to refer to another page in your book. You can
refer directly to another page by its relative path. For example, the
role syntax `` {doc}`intro` `` will result in: {doc}`intro`.
For more information on writing roles, see the
[MyST documentation](https://myst-parser.readthedocs.io/).
### Adding a citation
You can also cite references that are stored in a `bibtex` file. For example,
the following syntax: `` {cite}`holdgraf_evidence_2014` `` will render like
this: {cite}`holdgraf_evidence_2014`.
Moreover, you can insert a bibliography into your page with this syntax.
The `{bibliography}` directive must be used for all the `{cite}` roles to
render properly.
For example, if the references for your book are stored in `references.bib`,
then the bibliography is inserted with:
````
```{bibliography} references.bib
```
````
Resulting in a rendered bibliography that looks like:
```{bibliography} references.bib
```
### Executing code in your markdown files
If you'd like to include computational content inside these markdown files,
you can use MyST Markdown to define cells that will be executed when your
book is built. Jupyter Book uses *jupytext* to do this.
First, add Jupytext metadata to the file. For example, to add Jupytext metadata
to this markdown page, run this command:
```
jupyter-book myst init markdown.md
```
Once a markdown file has Jupytext metadata in it, you can add the following
directive to run the code at build time:
````
```{code-cell}
print("Here is some code to execute")
```
````
When your book is built, the contents of any `{code-cell}` blocks will be
executed with your default Jupyter kernel, and their outputs will be displayed
in-line with the rest of your content.
For more information about executing computational content with Jupyter Book,
see [The MyST-NB documentation](https://myst-nb.readthedocs.io/).

View file

@ -1,122 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Content with notebooks\n",
"\n",
"You can also create content with Jupyter Notebooks. This means that you can include\n",
"code blocks and their outputs in your book.\n",
"\n",
"## Markdown + notebooks\n",
"\n",
"As it is markdown, you can embed images, HTML, etc into your posts!\n",
"\n",
"![](https://myst-parser.readthedocs.io/en/latest/_static/logo.png)\n",
"\n",
"You an also $add_{math}$ and\n",
"\n",
"$$\n",
"math^{blocks}\n",
"$$\n",
"\n",
"or\n",
"\n",
"$$\n",
"\\begin{aligned}\n",
"\\mbox{mean} la_{tex} \\\\ \\\\\n",
"math blocks\n",
"\\end{aligned}\n",
"$$\n",
"\n",
"But make sure you \\$Escape \\$your \\$dollar signs \\$you want to keep!\n",
"\n",
"## MyST markdown\n",
"\n",
"MyST markdown works in Jupyter Notebooks as well. For more information about MyST markdown, check\n",
"out [the MyST guide in Jupyter Book](https://jupyterbook.org/content/myst.html),\n",
"or see [the MyST markdown documentation](https://myst-parser.readthedocs.io/en/latest/).\n",
"\n",
"## Code blocks and outputs\n",
"\n",
"Jupyter Book will also embed your code blocks and output in your book.\n",
"For example, here's some sample Matplotlib code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from matplotlib import rcParams, cycler\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"plt.ion()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Fixing random state for reproducibility\n",
"np.random.seed(19680801)\n",
"\n",
"N = 10\n",
"data = [np.logspace(0, 1, 100) + np.random.randn(100) + ii for ii in range(N)]\n",
"data = np.array(data).T\n",
"cmap = plt.cm.coolwarm\n",
"rcParams['axes.prop_cycle'] = cycler(color=cmap(np.linspace(0, 1, N)))\n",
"\n",
"\n",
"from matplotlib.lines import Line2D\n",
"custom_lines = [Line2D([0], [0], color=cmap(0.), lw=4),\n",
" Line2D([0], [0], color=cmap(.5), lw=4),\n",
" Line2D([0], [0], color=cmap(1.), lw=4)]\n",
"\n",
"fig, ax = plt.subplots(figsize=(10, 5))\n",
"lines = ax.plot(data)\n",
"ax.legend(custom_lines, ['Cold', 'Medium', 'Hot']);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is a lot more that you can do with outputs (such as including interactive outputs)\n",
"with your book. For more information about this, see [the Jupyter Book documentation](https://jupyterbook.org)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.0"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}

View file

@ -1,55 +0,0 @@
---
---
@inproceedings{holdgraf_evidence_2014,
address = {Brisbane, Australia, Australia},
title = {Evidence for {Predictive} {Coding} in {Human} {Auditory} {Cortex}},
booktitle = {International {Conference} on {Cognitive} {Neuroscience}},
publisher = {Frontiers in Neuroscience},
author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Knight, Robert T.},
year = {2014}
}
@article{holdgraf_rapid_2016,
title = {Rapid tuning shifts in human auditory cortex enhance speech intelligibility},
volume = {7},
issn = {2041-1723},
url = {http://www.nature.com/doifinder/10.1038/ncomms13654},
doi = {10.1038/ncomms13654},
number = {May},
journal = {Nature Communications},
author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Rieger, Jochem W. and Crone, Nathan and Lin, Jack J. and Knight, Robert T. and Theunissen, Frédéric E.},
year = {2016},
pages = {13654},
file = {Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:C\:\\Users\\chold\\Zotero\\storage\\MDQP3JWE\\Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:application/pdf}
}
@inproceedings{holdgraf_portable_2017,
title = {Portable learning environments for hands-on computational instruction using container-and cloud-based technology to teach data science},
volume = {Part F1287},
isbn = {978-1-4503-5272-7},
doi = {10.1145/3093338.3093370},
abstract = {© 2017 ACM. There is an increasing interest in learning outside of the traditional classroom setting. This is especially true for topics covering computational tools and data science, as both are challenging to incorporate in the standard curriculum. These atypical learning environments offer new opportunities for teaching, particularly when it comes to combining conceptual knowledge with hands-on experience/expertise with methods and skills. Advances in cloud computing and containerized environments provide an attractive opportunity to improve the effciency and ease with which students can learn. This manuscript details recent advances towards using commonly-Available cloud computing services and advanced cyberinfrastructure support for improving the learning experience in bootcamp-style events. We cover the benets (and challenges) of using a server hosted remotely instead of relying on student laptops, discuss the technology that was used in order to make this possible, and give suggestions for how others could implement and improve upon this model for pedagogy and reproducibility.},
author = {Holdgraf, Christopher Ramsay and Culich, A. and Rokem, A. and Deniz, F. and Alegro, M. and Ushizima, D.},
year = {2017},
keywords = {Teaching, Bootcamps, Cloud computing, Data science, Docker, Pedagogy}
}
@article{holdgraf_encoding_2017,
title = {Encoding and decoding models in cognitive electrophysiology},
volume = {11},
issn = {16625137},
doi = {10.3389/fnsys.2017.00061},
abstract = {© 2017 Holdgraf, Rieger, Micheli, Martin, Knight and Theunissen. Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of “Encoding” models, in which stimulus features are used to model brain activity, and “Decoding” models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aimis to provide a practical understanding of predictivemodeling of human brain data and to propose best-practices in conducting these analyses.},
journal = {Frontiers in Systems Neuroscience},
author = {Holdgraf, Christopher Ramsay and Rieger, J.W. and Micheli, C. and Martin, S. and Knight, R.T. and Theunissen, F.E.},
year = {2017},
keywords = {Decoding models, Encoding models, Electrocorticography (ECoG), Electrophysiology/evoked potentials, Machine learning applied to neuroscience, Natural stimuli, Predictive modeling, Tutorials}
}
@book{ruby,
title = {The Ruby Programming Language},
author = {Flanagan, David and Matsumoto, Yukihiro},
year = {2008},
publisher = {O'Reilly Media}
}