Create pipeline for spectral indices w/ correct cloud masking

sentinel_data_for_earth_observation/CONDUCT.md

@@ -0,0 +1,44 @@
+
+# Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant, version 1.4](http://contributor-covenant.org/version/1/4).

sentinel_data_for_earth_observation/CONTRIBUTING.md

@@ -0,0 +1,56 @@
+# Contributing
+
+Contributions are welcome, and they are greatly appreciated! Every little bit
+helps, and credit will always be given. You can contribute in the ways listed below.
+
+## Report Bugs
+
+Report bugs using GitHub issues.
+
+If you are reporting a bug, please include:
+
+* Your operating system name and version.
+* Any details about your local setup that might be helpful in troubleshooting.
+* Detailed steps to reproduce the bug.
+
+## Fix Bugs
+
+Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
+wanted" is open to whoever wants to implement it.
+
+## Implement Features
+
+Look through the GitHub issues for features. Anything tagged with "enhancement"
+and "help wanted" is open to whoever wants to implement it.
+
+## Write Documentation
+
+Sentinel Data for Earth Observation could always use more documentation, whether as part of the
+official Sentinel Data for Earth Observation docs, in docstrings, or even on the web in blog posts,
+articles, and such.
+
+## Submit Feedback
+
+The best way to send feedback is to file an issue on GitHub.
+
+If you are proposing a feature:
+
+* Explain in detail how it would work.
+* Keep the scope as narrow as possible, to make it easier to implement.
+* Remember that this is a volunteer-driven project, and that contributions
+  are welcome :)
+
+## Get Started
+
+Ready to contribute? Here's how to set up `Sentinel Data for Earth Observation` for local development.
+
+1. Fork the repo on GitHub.
+2. Clone your fork locally.
+3. Install your local copy into a virtualenv, e.g., using `conda`.
+4. Create a branch for local development and make changes locally.
+5. Commit your changes and push your branch to GitHub.
+6. Submit a pull request through the GitHub website.
+
+## Code of Conduct
+
+Please note that the Sentinel Data for Earth Observation project is released with a [Contributor Code of Conduct](CONDUCT.md). By contributing to this project you agree to abide by its terms.

sentinel_data_for_earth_observation/LICENSE

@@ -0,0 +1,33 @@
+GNU GENERAL PUBLIC LICENSE
+                      Version 3, 29 June 2007
+
+    This cookiecutter creates a simple boilerplate for a Jupyter Book.
+    Copyright (C) 2020  Arne Schlüter
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<http://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<http://www.gnu.org/philosophy/why-not-lgpl.html>.
+

sentinel_data_for_earth_observation/README.md

@@ -0,0 +1,35 @@
+# Sentinel Data for Earth Observation
+
+This cookiecutter creates a simple boilerplate for a Jupyter Book.
+
+## Usage
+
+### Building the book
+
+If you'd like to develop on and build the Sentinel Data for Earth Observation book, you should:
+
+- Clone this repository and run
+- Run `pip install -r requirements.txt` (it is recommended you do this within a virtual environment)
+- (Recommended) Remove the existing `Sentinel Data for Earth Observation/_build/` directory
+- Run `jupyter-book build Sentinel Data for Earth Observation/`
+
+A fully-rendered HTML version of the book will be built in `Sentinel Data for Earth Observation/_build/html/`.
+
+### Hosting the book
+
+The html version of the book is hosted on the `gh-pages` branch of this repo. A GitHub actions workflow has been created that automatically builds and pushes the book to this branch on a push or pull request to main.
+
+If you wish to disable this automation, you may remove the GitHub actions workflow and build the book manually by:
+
+- Navigating to your local build; and running,
+- `ghp-import -n -p -f Sentinel Data for Earth Observation/_build/html`
+
+This will automatically push your build to the `gh-pages` branch. More information on this hosting process can be found [here](https://jupyterbook.org/publish/gh-pages.html#manually-host-your-book-with-github-pages).
+
+## Contributors
+
+We welcome and recognize all contributions. You can see a list of current contributors in the [contributors tab](https://github.com/heyarne/sentinel_data_for_earth_observation/graphs/contributors).
+
+## Credits
+
+This project is created using the excellent open source [Jupyter Book project](https://jupyterbook.org/) and the [executablebooks/cookiecutter-jupyter-book template](https://github.com/executablebooks/cookiecutter-jupyter-book).

sentinel_data_for_earth_observation/requirements.txt

@@ -0,0 +1,3 @@
+jupyter-book
+matplotlib
+numpy

sentinel_data_for_earth_observation/sentinel_data_for_earth_observation/content.md

@@ -0,0 +1,5 @@
+Content in Jupyter Book
+=======================
+
+There are many ways to write content in Jupyter Book. This short section
+covers a few tips for how to do so.

sentinel_data_for_earth_observation/sentinel_data_for_earth_observation/intro.md

@@ -0,0 +1,7 @@
+Welcome to your Jupyter Book
+============================
+
+This is a small sample book to give you a feel for how book content is
+structured.
+
+Check out the content pages bundled with this sample book to get started.

sentinel_data_for_earth_observation/sentinel_data_for_earth_observation/markdown.md

@@ -0,0 +1,125 @@
+# Markdown Files
+
+Whether you write your book's content in Jupyter Notebooks (`.ipynb`) or
+in regular markdown files (`.md`), you'll write in the same flavor of markdown
+called **MyST Markdown**.
+
+## What is MyST?
+
+MyST stands for "Markedly Structured Text". It
+is a slight variation on a flavor of markdown called "CommonMark" markdown,
+with small syntax extensions to allow you to write **roles** and **directives**
+in the Sphinx ecosystem.
+
+## What are roles and directives?
+
+Roles and directives are two of the most powerful tools in Jupyter Book. They
+are kind of like functions, but written in a markup language. They both
+serve a similar purpose, but **roles are written in one line**, whereas
+**directives span many lines**. They both accept different kinds of inputs,
+and what they do with those inputs depends on the specific role or directive
+that is being called.
+
+### Using a directive
+
+At its simplest, you can insert a directive into your book's content like so:
+
+````
+```{mydirectivename}
+My directive content
+```
+````
+
+This will only work if a directive with name `mydirectivename` already exists
+(which it doesn't). There are many pre-defined directives associated with
+Jupyter Book. For example, to insert a note box into your content, you can
+use the following directive:
+
+````
+```{note}
+Here is a note
+```
+````
+
+This results in:
+
+```{note}
+Here is a note
+```
+
+In your built book.
+
+For more information on writing directives, see the
+[MyST documentation](https://myst-parser.readthedocs.io/).
+
+
+### Using a role
+
+Roles are very similar to directives, but they are less-complex and written
+entirely on one line. You can insert a role into your book's content with
+this pattern:
+
+```
+Some content {rolename}`and here is my role's content!`
+```
+
+Again, roles will only work if `rolename` is a valid role's name. For example,
+the `doc` role can be used to refer to another page in your book. You can
+refer directly to another page by its relative path. For example, the
+role syntax `` {doc}`intro` `` will result in: {doc}`intro`.
+
+For more information on writing roles, see the
+[MyST documentation](https://myst-parser.readthedocs.io/).
+
+
+### Adding a citation
+
+You can also cite references that are stored in a `bibtex` file. For example,
+the following syntax: `` {cite}`holdgraf_evidence_2014` `` will render like
+this: {cite}`holdgraf_evidence_2014`.
+
+Moreover, you can insert a bibliography into your page with this syntax. 
+The `{bibliography}` directive must be used for all the `{cite}` roles to
+render properly.
+For example, if the references for your book are stored in `references.bib`,
+then the bibliography is inserted with:
+
+````
+```{bibliography} references.bib
+```
+````
+
+Resulting in a rendered bibliography that looks like:
+
+```{bibliography} references.bib
+```
+
+
+### Executing code in your markdown files
+
+If you'd like to include computational content inside these markdown files,
+you can use MyST Markdown to define cells that will be executed when your
+book is built. Jupyter Book uses *jupytext* to do this.
+
+First, add Jupytext metadata to the file. For example, to add Jupytext metadata
+to this markdown page, run this command:
+
+```
+jupyter-book myst init markdown.md
+```
+
+Once a markdown file has Jupytext metadata in it, you can add the following
+directive to run the code at build time:
+
+````
+```{code-cell}
+print("Here is some code to execute")
+```
+````
+
+When your book is built, the contents of any `{code-cell}` blocks will be
+executed with your default Jupyter kernel, and their outputs will be displayed
+in-line with the rest of your content.
+
+For more information about executing computational content with Jupyter Book,
+see [The MyST-NB documentation](https://myst-nb.readthedocs.io/).

sentinel_data_for_earth_observation/sentinel_data_for_earth_observation/notebooks.ipynb

@@ -0,0 +1,122 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Content with notebooks\n",
+    "\n",
+    "You can also create content with Jupyter Notebooks. This means that you can include\n",
+    "code blocks and their outputs in your book.\n",
+    "\n",
+    "## Markdown + notebooks\n",
+    "\n",
+    "As it is markdown, you can embed images, HTML, etc into your posts!\n",
+    "\n",
+    "![](https://myst-parser.readthedocs.io/en/latest/_static/logo.png)\n",
+    "\n",
+    "You an also $add_{math}$ and\n",
+    "\n",
+    "$$\n",
+    "math^{blocks}\n",
+    "$$\n",
+    "\n",
+    "or\n",
+    "\n",
+    "$$\n",
+    "\\begin{aligned}\n",
+    "\\mbox{mean} la_{tex} \\\\ \\\\\n",
+    "math blocks\n",
+    "\\end{aligned}\n",
+    "$$\n",
+    "\n",
+    "But make sure you \\$Escape \\$your \\$dollar signs \\$you want to keep!\n",
+    "\n",
+    "## MyST markdown\n",
+    "\n",
+    "MyST markdown works in Jupyter Notebooks as well. For more information about MyST markdown, check\n",
+    "out [the MyST guide in Jupyter Book](https://jupyterbook.org/content/myst.html),\n",
+    "or see [the MyST markdown documentation](https://myst-parser.readthedocs.io/en/latest/).\n",
+    "\n",
+    "## Code blocks and outputs\n",
+    "\n",
+    "Jupyter Book will also embed your code blocks and output in your book.\n",
+    "For example, here's some sample Matplotlib code:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from matplotlib import rcParams, cycler\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "plt.ion()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Fixing random state for reproducibility\n",
+    "np.random.seed(19680801)\n",
+    "\n",
+    "N = 10\n",
+    "data = [np.logspace(0, 1, 100) + np.random.randn(100) + ii for ii in range(N)]\n",
+    "data = np.array(data).T\n",
+    "cmap = plt.cm.coolwarm\n",
+    "rcParams['axes.prop_cycle'] = cycler(color=cmap(np.linspace(0, 1, N)))\n",
+    "\n",
+    "\n",
+    "from matplotlib.lines import Line2D\n",
+    "custom_lines = [Line2D([0], [0], color=cmap(0.), lw=4),\n",
+    "                Line2D([0], [0], color=cmap(.5), lw=4),\n",
+    "                Line2D([0], [0], color=cmap(1.), lw=4)]\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(10, 5))\n",
+    "lines = ax.plot(data)\n",
+    "ax.legend(custom_lines, ['Cold', 'Medium', 'Hot']);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "There is a lot more that you can do with outputs (such as including interactive outputs)\n",
+    "with your book. For more information about this, see [the Jupyter Book documentation](https://jupyterbook.org)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.0"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {},
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

sentinel_data_for_earth_observation/sentinel_data_for_earth_observation/references.bib

@@ -0,0 +1,55 @@
+---
+---
+
+@inproceedings{holdgraf_evidence_2014,
+	address = {Brisbane, Australia, Australia},
+	title = {Evidence for {Predictive} {Coding} in {Human} {Auditory} {Cortex}},
+	booktitle = {International {Conference} on {Cognitive} {Neuroscience}},
+	publisher = {Frontiers in Neuroscience},
+	author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Knight, Robert T.},
+	year = {2014}
+}
+
+@article{holdgraf_rapid_2016,
+	title = {Rapid tuning shifts in human auditory cortex enhance speech intelligibility},
+	volume = {7},
+	issn = {2041-1723},
+	url = {http://www.nature.com/doifinder/10.1038/ncomms13654},
+	doi = {10.1038/ncomms13654},
+	number = {May},
+	journal = {Nature Communications},
+	author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Rieger, Jochem W. and Crone, Nathan and Lin, Jack J. and Knight, Robert T. and Theunissen, Frédéric E.},
+	year = {2016},
+	pages = {13654},
+	file = {Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:C\:\\Users\\chold\\Zotero\\storage\\MDQP3JWE\\Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:application/pdf}
+}
+
+@inproceedings{holdgraf_portable_2017,
+	title = {Portable learning environments for hands-on computational instruction using container-and cloud-based technology to teach data science},
+	volume = {Part F1287},
+	isbn = {978-1-4503-5272-7},
+	doi = {10.1145/3093338.3093370},
+	abstract = {© 2017 ACM. There is an increasing interest in learning outside of the traditional classroom setting. This is especially true for topics covering computational tools and data science, as both are challenging to incorporate in the standard curriculum. These atypical learning environments offer new opportunities for teaching, particularly when it comes to combining conceptual knowledge with hands-on experience/expertise with methods and skills. Advances in cloud computing and containerized environments provide an attractive opportunity to improve the effciency and ease with which students can learn. This manuscript details recent advances towards using commonly-Available cloud computing services and advanced cyberinfrastructure support for improving the learning experience in bootcamp-style events. We cover the benets (and challenges) of using a server hosted remotely instead of relying on student laptops, discuss the technology that was used in order to make this possible, and give suggestions for how others could implement and improve upon this model for pedagogy and reproducibility.},
+	author = {Holdgraf, Christopher Ramsay and Culich, A. and Rokem, A. and Deniz, F. and Alegro, M. and Ushizima, D.},
+	year = {2017},
+	keywords = {Teaching, Bootcamps, Cloud computing, Data science, Docker, Pedagogy}
+}
+
+@article{holdgraf_encoding_2017,
+	title = {Encoding and decoding models in cognitive electrophysiology},
+	volume = {11},
+	issn = {16625137},
+	doi = {10.3389/fnsys.2017.00061},
+	abstract = {© 2017 Holdgraf, Rieger, Micheli, Martin, Knight and Theunissen. Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of “Encoding” models, in which stimulus features are used to model brain activity, and “Decoding” models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aimis to provide a practical understanding of predictivemodeling of human brain data and to propose best-practices in conducting these analyses.},
+	journal = {Frontiers in Systems Neuroscience},
+	author = {Holdgraf, Christopher Ramsay and Rieger, J.W. and Micheli, C. and Martin, S. and Knight, R.T. and Theunissen, F.E.},
+	year = {2017},
+	keywords = {Decoding models, Encoding models, Electrocorticography (ECoG), Electrophysiology/evoked potentials, Machine learning applied to neuroscience, Natural stimuli, Predictive modeling, Tutorials}
+}
+
+@book{ruby,
+  title     = {The Ruby Programming Language},
+  author    = {Flanagan, David and Matsumoto, Yukihiro},
+  year      = {2008},
+  publisher = {O'Reilly Media}
+}