In this tutorial, you will learn how to efficiently read in hyperspectral surface directional reflectance hdf5 data and metadata, plot a single band and Red-Green-Blue (RGB) band combinations of a reflectance data tile using Python functions created for working with and visualizing NEON AOP hyperspectral data.
We can combine any three bands from the NEON reflectance data to make an RGB image that will depict different information about the Earth's surface. A natural color image, made with bands from the red, green, and blue wavelengths looks close to what we would see with the naked eye. We can also choose band combinations from other wavelenghts, and map them to the red, blue,
and green colors to highlight different features. A false color image is made with one or more bands from a non-visible portion of the electromagnetic spectrum that are mapped to red, green, and blue colors. These images can display other information about the landscape that is not easily seen with a natural color image.
The NASA Goddard Media Studio video "Peeling Back Landsat's Layers of Data" gives a good quick overview of natural and false color band combinations. Note that the Landsat multispectral sensor collects information from 11 bands, while NEON AOP hyperspectral data captures information spanning 426 bands!
First we can import the required packages and the neon_aop_hyperspectral module, which includes a number of functions which we will use to read in the hyperspectral hdf5 data as well as visualize the data.
import os
import sys
import time
import h5py
import requests
import numpy as np
import matplotlib.pyplot as plt
This next function is a handy way to download the Python module and data that we will be using for this lesson. This uses the requests package.
# function to download data stored on the internet in a public url to a local file
def download_url(url,download_dir):
if not os.path.isdir(download_dir):
os.makedirs(download_dir)
filename = url.split('/')[-1]
r = requests.get(url, allow_redirects=True)
file_object = open(os.path.join(download_dir,filename),'wb')
file_object.write(r.content)
Download the module from its location on GitHub, add the python_modules to the path and import the neon_aop_hyperspectral.py module.
module_url = "https://raw.githubusercontent.com/NEONScience/NEON-Data-Skills/main/tutorials/Python/AOP/aop_python_modules/neon_aop_hyperspectral.py"
download_url(module_url,'../python_modules')
# os.listdir('../python_modules') #optionally show the contents of this directory to confirm the file downloaded
sys.path.insert(0, '../python_modules')
# import the neon_aop_hyperspectral module, the semicolon supresses an empty plot from displaying
import neon_aop_hyperspectral as neon_hs;
The first function we will use is aop_h5refl2array. We encourage you to look through the code to understand what it is doing behind the scenes. This function automates the steps required to read AOP hdf5 reflectance files into a Python numpy array. This function also cleans the data: it sets any no data values within the reflectance tile to nan (not a number) and applies the reflectance scale factor so the final array that is returned represents unitless scaled reflectance, with values ranging between 0 and 1 (0-100%).
If you forget what this function does, or don't want to scroll up to read the docstrings, remember you can use help or ? to display the associated docstrings.
help(neon_hs.aop_h5refl2array)
# neon_hs.aop_h5refl2array? #uncomment for an alternate way to show the help
Help on function aop_h5refl2array in module neon_aop_hyperspectral:
aop_h5refl2array(h5_filename, raster_type_: Literal['Cast_Shadow', 'Data_Selection_Index', 'GLT_Data', 'Haze_Cloud_Water_Map', 'IGM_Data', 'Illumination_Factor', 'OBS_Data', 'Radiance', 'Reflectance', 'Sky_View_Factor', 'to-sensor_Azimuth_Angle', 'to-sensor_Zenith_Angle', 'Visibility_Index_Map', 'Weather_Quality_Indicator'], only_metadata=False)
read in NEON AOP reflectance hdf5 file and return the un-scaled
reflectance array, associated metadata, and wavelengths
Parameters
----------
h5_filename : string
reflectance hdf5 file name, including full or relative path
raster : string
name of raster value to read in; this will typically be the reflectance data,
but other data stored in the h5 file can be accessed as well
valid options:
Cast_Shadow (ATCOR input)
Data_Selection_Index
GLT_Data
Haze_Cloud_Water_Map (ATCOR output)
IGM_Data
Illumination_Factor (ATCOR input)
OBS_Data
Reflectance
Radiance
Sky_View_Factor (ATCOR input)
to-sensor_Azimuth_Angle
to-sensor_Zenith_Angle
Visibility_Index_Map: sea level values of visibility index / total optical thickeness
Weather_Quality_Indicator: estimated percentage of overhead cloud cover during acquisition
Returns
--------
raster_array : ndarray
array of reflectance values
metadata: dictionary
associated metadata containing
bad_band_window1 (tuple)
bad_band_window2 (tuple)
bands: # of bands (float)
data ignore value: value corresponding to no data (float)
epsg: coordinate system code (float)
map info: coordinate system, datum & ellipsoid, pixel dimensions, and origin coordinates (string)
reflectance scale factor: factor by which reflectance is scaled (float)
wavelengths: array
wavelength values, in nm
--------
Example Execution:
--------
refl, refl_metadata = aop_h5refl2array('NEON_D02_SERC_DP3_368000_4306000_reflectance.h5','Reflectance')
Now that we have an idea of how this function works, let's try it out. First, let's download a file. For this tutorial, we will use requests to download from the public link where the data is stored on the cloud (Google Cloud Storage). This downloads to a data folder in the working directory, but you can download it to a different location if you prefer.
# define the data_url to point to the cloud storage location of the the hyperspectral hdf5 data file
data_url = "https://storage.googleapis.com/neon-aop-products/2021/FullSite/D03/2021_DSNY_6/L3/Spectrometer/Reflectance/NEON_D03_DSNY_DP3_454000_3113000_reflectance.h5"
# download the h5 data and display how much time it took to download (uncomment 1st and 3rd lines)
# start_time = time.time()
download_url(data_url,'.\data')
# print("--- It took %s seconds to download the data ---" % round((time.time() - start_time),1))
# display the contents in the ./data folder to confirm the download completed
os.listdir('./data')
# read the h5 reflectance file (including the full path) to the variable h5_file_name
h5_file_name = data_url.split('/')[-1]
h5_tile = os.path.join(".\data",h5_file_name)
print(f'h5_tile: {h5_tile}')
Now that we've specified our reflectance tile, we can call aop_h5refl2array to read in the reflectance tile as a python array called refl , the metadata into a dictionary called refl_metadata, and the wavelengths into an array.
# read in the reflectance data using the aop_h5refl2array function, this may also take a bit of time
start_time = time.time()
refl, refl_metadata, wavelengths = neon_hs.aop_h5refl2array(h5_tile,'Reflectance')
print("--- It took %s seconds to read in the data ---" % round((time.time() - start_time),0))
Reading in .\data\NEON_D03_DSNY_DP3_454000_3113000_reflectance.h5
--- It took 7.0 seconds to read in the data ---
# display the reflectance metadata dictionary contents
refl_metadata
We can use the shape method to see the dimensions of the array we read in. Use this method to confirm that the size of the reflectance array makes sense given the hyperspectral data cube, which is 1000 meters x 1000 meters x 426 bands.
refl.shape
(1000, 1000, 426)
plot_aop_refl: plot a single band of the reflectance data
Next we'll use the function plot_aop_refl to plot a single band of reflectance data. You can use help to understand the required inputs and data types for each of these; only the band and spatial extent are required inputs, the rest are optional inputs. If specified, these optional inputs allow you to set the range color values, specify the axis, add a title, colorbar, colorbar title, and change the colormap (default is to plot in greyscale).
band56 = refl[:,:,55]
neon_hs.plot_aop_refl(band56/refl_metadata['scale_factor'],
refl_metadata['extent'],
colorlimit=(0,0.3),
title='DSNY Tile Band 56',
cmap_title='Reflectance',
colormap='gist_earth')
RGB Plots - Band Stacking
It is often useful to look at several bands together. We can extract and stack three reflectance bands in the red, green, and blue (RGB) spectrums to produce a color image that looks like what we see with our eyes; this is your typical camera image. In the next part of this tutorial, we will learn to stack multiple bands and make a geotif raster from the compilation of these bands. We can see that different combinations of bands allow for different visualizations of the remotely-sensed objects and also conveys useful information about the chemical makeup of the Earth's surface.
We will select bands that fall within the visible range of the electromagnetic spectrum (400-700 nm) and at specific points that correspond to what we see as red, green, and blue.
NEON Imaging Spectrometer bands and their respective wavelengths. Source: National Ecological Observatory Network (NEON)
For this exercise, we'll first use the function stack_rgb to extract the bands we want to stack. This function uses splicing to extract the nth band from the reflectance array, and then uses the numpy function stack to create a new 3D array (1000 x 1000 x 3) consisting of only the three bands we want.
# pull out the true-color band combinations
rgb_bands = (58,34,19) # set the red, green, and blue bands
# stack the 3-band combinations (rgb and cir) using stack_rgb function
rgb_unscaled = neon_hs.stack_rgb(refl,rgb_bands)
# apply the reflectance scale factor
rgb = rgb_unscaled/refl_metadata['scale_factor']
We can display the red, green, and blue band center wavelengths, whose indices were defined above. To confirm that these band indices correspond to wavelengths in the expected portion of the spectrum, we can print out the wavelength values in nanometers.
Center wavelengths:
Band 58: 669.3 nm
Band 33: 549.1 nm
Band 19: 474.0 nm
plot_aop_rgb: plot an RGB band combination
Next, we can use the function plot_aop_rgb to plot the band stack as follows:
# plot the true color image (rgb)
neon_hs.plot_aop_rgb(rgb,
refl_metadata['extent'],
plot_title='DSNY Reflectance RGB Image')
False Color Image - Color Infrared (CIR)
We can also create an image from bands outside of the visible spectrum. An image containing one or more bands outside of the visible range is called a false-color image. Here we'll use the green and blue bands as before, but we replace the red band with a near-infrared (NIR) band.
For more information about non-visible wavelengths, false color images, and some frequently used false-color band combinations, refer to NASA's Earth Observatory page.
cir_bands = (90,34,19)
print('Band 90 Center Wavelength = %.1f' %(wavelengths[89]),'nm')
print('Band 34 Center Wavelength = %.1f' %(wavelengths[33]),'nm')
print('Band 19 Center Wavelength = %.1f' %(wavelengths[18]),'nm')
cir = neon_hs.stack_rgb(refl,cir_bands)
neon_hs.plot_aop_rgb(cir,
refl_metadata['extent'],
ls_pct=20,
plot_title='DSNY Color Infrared Image')
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Band 90 Center Wavelength = 829.6 nm
Band 34 Center Wavelength = 549.1 nm
Band 19 Center Wavelength = 474.0 nm
In this tutorial you will learn how to open a .tiff file in Jupyter Notebook and learn about kernels.
The goal of the activity is simply to ensure that you have basic
familiarity with Jupyter Notebooks and that the environment, especially the
gdal package is correctly set up before you pursue more programming tutorials. If you already
are familiar with Jupyter Notebooks using Python, you may be able to complete the
assignment without working through the instructions.
This will be accomplished by:
*Create a new Jupyter kernel
*Download a GEOTIFF file
*Import file onto Jupyter Notebooks
*Check the raster size
Assignment: Open a Tiff File in Jupyter Notebook
Set up Environment
First, we will set up the environment as you would need for each of the live
coding sections of the Data Institute. The following directions are copied over
from the Data Institute Set up Materials.
In your terminal application, navigate to the directory (cd) that where you
want the Jupyter Notebooks to be saved (or where they already exist).
We need to create a new Jupyter kernel for the Python 3.8 conda environment
(py38) that Jupyter Notebooks will use.
In your Command Prompt/Terminal, navigate to the directory (cd) that you
created last week in the GitHub materials. This is where the Jupyter Notebook
will be saved and the easiest way to access existing notebooks.
###Open Jupyter Notebook
Open Jupyter Notebook by typing into a command terminal:
jupyter notebook
Once the notebook is open, check which version of Python you are in.
# Check what version of Python. Should be 3.8.
import sys
sys.version
To ensure that the correct kernel will operate, navigate to Kernel in the menu,
select Kernel/Restart Kernel And Clear All Outputs.
To ensure that the correct kernel will operate, navigate to
Kernel in the menu, select "Restart/Restart & Clear Output".
Source: National Ecological Observatory Network (NEON)
Once downloaded, navigate through the folder to C:NEON-DS-Airborne-Remote-Sensing.zip\NEON-DS-Airborne-Remote-Sensing\SJER\DTM and save this file onto your own personal working directory.
.
###Open GEOTIFF file in Jupyter Notebooks using gdal
The gdal package that occasionally has problems with some versions of Python.
Therefore test out loading it using:
import gdal.
If you have trouble, ensure that 'gdal' is installed on your current environment.
Establish your directory
Place the downloaded dtm file in a repository of your choice (or your current
working directory). Navigate to that directory.
wd= '/your-file-path-here' #Input the directory to where you saved the .tif file
Import the TIFF
Import the NEON GeoTiFF file of the digital terrain model (DTM)
from San Joaquin Experimental Range. Open the file using the gdal.Open command.Determine the size of the raster and (optional) plot the raster.
#Use GDAL to open GEOTIFF file stored in your directory
SJER_DTM = gdal.Open(wd + 'SJER_dtmCrop.tif')>
#Determine the raster size.
SJER_DTM.RasterXSize
Add in both code chunks and text (markdown) chunks to fully explain what is done. If you would like to also plot the file, feel free to do so.
You can set up your notebook in several ways. Here we present the Anaconda Python
distribution method so as to follow the
Data Institute set up instructions.
Browser
First, make sure you have an updated browser on which to run the app. Both Mozilla
Firefox and Google Chrome work well.
Installation
Data Institute participants should have already installed Jupyter Notebooks
through the Anaconda installation during the
Data Institute set up instructions.
If you install Python using pip you can install the Jupyter package with the
following code.
We need to set up the Python environment that we will be working in for the Notebook.
This allows us to have different Python environments for different projects. The
following directions pertain directly to the set up for the 2018 Data Institute
on Remote Sensing with Reproducible Workflows, however, you can adapt them to
the specific Python version and packages you wish to work with.
If you haven't yet created a Python 3.8 environment (released October 2019), you'll need to do
that now. You can use the single line provided below, or refer back to the Python section of the installation instructions,
for more details. To create this Python 3.8 environment, you must first install Anaconda Navigator onto your computer, then open the Anaconda Prompt application (or your terminal) and type the following into the prompt window:
conda create -n p38 python=3.8 anaconda
And activate the Python 3.8 environment:
On Mac:
source activate p38
On Windows:
activate p38
In the terminal application, navigate to the directory (cd) where you
want the Jupyter Notebooks to be saved (or where they already exist).
Once here, we want to create a new Jupyter kernel for the Python 3.8 conda environment
(p38) that we'll be using with Jupyter Notebooks.
With the p38 environment activated, in your Command Prompt/Terminal, type:
This command tells Python to create a new ipy (aka Jupyter Notebook) kernel using
the Python environment we set up and called "p38". Then we tell it to use the display
name for this new kernel as "Python 3.8 NEON-RSDI". You will use this
name to identify the specific kernel you want to work with in the Notebook space,
so name it descriptively, especially if you think you'll be using several different
kernels.
Using Jupyter Notebooks
Launching the Application
To launch the application either launch it from the Anaconda Navigator or by
typing jupyter notebook into your terminal or command window.
If everything launched correctly, you should be able to see a screen which looks
something like this. Note that the home directory will be whatever directory you
have navigated to in your terminal before launching Jupyter Notebooks.
To start a new Python notebook, click on the right-hand side of the application
window and select New (the expanded menu is shown in the screen shot above).
This will give you several options for new notebook kernels depending on what
is installed on your computer. In the above screenshot, there are two available
Python kernels and one Matlab kernel. When starting a notebook, you should choose
Python 3 if it is available or conda(root) .
Once you start a new notebook, you will be brought to the following screen.
There are many available buttons for you to click. The three most important
components of the notebook are highlighted in colored boxes.
In blue is the name of the notebook. By clicking this, you can rename
the notebook.
In red is the cell formatting assignment. By default, it is registered
as code, but it can also be set to markdown as described later.
In purple, is the code cell. In this cell, you can type an execute
Python code as well as text that will be formatted in a nicely readable format.
Selecting a Kernel
A kernel is a server that enables you to run commands within Jupyter Notebook. It is visible via a prompt window that logs all your actions in the notebook, making it helpful to refer to when encountering errors. You'll be prompted to select a kernel when you open a new notebook, however, if
you are opening an existing notebook you will want to ensure that you are
using the correct kernel. The commands for selecting and changing kernels are in
the Kernel menu.
When you select or switch a kernel, you may want to use the navigate to Kernel in the menu,
select Restart/ClearOutlook. The Restart/ClearOutlook option ensures that
the correct kernel will operate.
You can always check what version of Python you are running by typing the following
into a code cell.
# Check what version of Python. Should be 3.5.
import sys
sys.version
All code you write in the notebook will be in the code cell. You can write
single lines, to entire loops, to complete functions. As an example, we can
write and evaluate a print statement in a code cell, as is shown below.
If you would like to write several lines of code, hit Enter to continue entering code into another line. To execute the code, we can simply hit Shift + Enter while our cursor is in the
code cell.
# This is a comment and is not read by Python
print('Hello! This is the print function. Python will print this line below')
Hello! This is the print function. Python will print this line below
We can also write a 'for' loop as an example of executing multiple lines of code at once.
# Write a basic for loop. In this case a range of numbers 0-4.
for i in range(5):
# Multiply the value of i by two and assign it to a variable.
temp_variable = 2 * i
# Print the value of the temp variable.
print(temp_variable)
0
2
4
6
8
There are two other useful keyboard shortcuts for running code:
Alt + Enter runs the current cell and inserts a new one below
Ctrl + Enter run the current cell and enters command mode.
**Data Tip:** Code cells can be executed in any
order. This means that you can overwrite your current variables by running
things out of order. When coding in notebooks, be cautious of the order in
which you run cells.
If you would like more details on running code in Jupyter Notebooks, please go through the
following short tutorial by
Running Code
by contributors to the Jupyter project. This tutorial touches on start and stopping
the kernel and using multiple kernels (e.g., Python and R) in one notebook.
Arguably the most useful component of the Jupyter Notebook is the ability to
interweave code and explanatory text into a single, coherent document. Through
out the Data Institute (and one's everyday workflow), we encourage all code and
plots should be accompanied with explanatory text.
Each cell in a notebook can exist either as a code cell or as a text-formatting
cell called a markdown cell. Markdown is a mark-up language that very easily
converts to other type-setting formats such as HTML and PDF.
Whenever you make a new cell, its default assignment will be a code cell.
This means when you want to write text, you will need to specifically change it
to a markdown cell. You can do this by clicking on the drop-down menu that reads code'
(highlighted in red in the second figure of this page) and selecting 'Markdown'.
You can then type in the code cell and all Python syntax highlighting will be
removed.
Jupyter notebooks are set up to autosave your work every 15 or so minutes.
However, you should not rely on the autosave feature! Save your work frequently
by clicking on the floppy disk icon located in the upper left-hand corner of the
toolbar.
To navigate back to the root of your Jupyter notebook server, you can click on
the Jupyter logo at any time.
To quit your Jupyter notebook, you can simply close the browser window and the
Jupyter notebook server running in your terminal.
Converting to HTML and PDF
In addition to sharing notebooks in the.ipynb format, it may useful to convert
these notebooks to highly-portable formats such as HTML and PDF.
To convert, you can either use the dropdown menu option
File -> download as -> ...
or via the command line by using the following lines:
jupyter nbconvert --to pdf notebook_name.ipynb
Where "notebook_name.ipynb" matches the name of the notebook you want to convert. Prior to
converting the notebook, you must be in the same working directory as your notebook or use
the correct file path from your current working directory.
Converting to PDF requires both Pandoc and LaTeX to be installed. You can find
out more in the ReadTheDoc for nbconvert.
If you prefer to convert to a different format, like HTML, you simply change the
file type.
jupyter nbconvert --to html notebook_name.ipynb
Read more on what formats you can convert to and more about the
nbconvert package .
“The Jupyter Notebook is an open-source web application that allows you to
create and share documents that contain live code, equations, visualizations and
explanatory text. Uses include: data cleaning and transformation, numerical
simulation, statistical modeling, machine learning and much more."
-- Jupyter Notebook documentation.
We use markdown syntax in Notebook documents to document workflows and
to share data processing, analysis and visualization outputs. We can also use it
to create documents that combine code in your language of choice, output and text.
The Jupyter Notebooks grew out of iPython. Jupyter is a close acronym meaning
Julia, Python, and R, which were the first languages outside Python that the Jupyter
application was designed for. Jupyter Notebooks now supports over
40 coding languages. You may still find some references to iPython in materials
related to Jupyter Notebooks. This series will focus on using Jupyter Notebooks with Python,
but the information presented can apply to other languages as well.
The Jupyter Notebooks application is a browser-based application. Therefore, you
need an updated browser (the Jupyter programmers recommend Mozilla Firefox or Google
Chrome, but not Microsoft Explorer). When installed on your computer, you can
always access the app even without internet access. You can also use Jupyter
installed on a remote server. For example, Jupyter runs a
training (temporary) server based version.
Why Jupyter Notebooks?
There are many advantages to using Jupyter Notebooks in your work:
Human readable syntax.
Simple syntax - it can be learned quickly.
All components of your work are clearly documented. You don't have to remember
what steps, assumptions, tests were used.
You can easily extend or refine analyses by modifying existing or adding new
code blocks.
Analysis results can be disseminated in various formats including HTML, PDF,
slideshows and more.
Code and data can be shared with a colleague to replicate the workflow.
Explore Examples of Notebooks
Before we jump into how to work with notebooks, check out a few shared notebooks.
As you look at these different notebooks, what aspects of the layout do you like,
what don't you like? Is there a place in your current workflow that these
notebooks would be useful?
There are many different indices you might want in your research. NEON provides
several indices as data products that have already been calculated and can will
be available for download from the NEON data portal.
NEON Remote Sensing Vegetation Indices, Data Products, and Uncertainty
In this 20 minute video David Hulslander describes NEON Data Products including
several remote sensing vegetation indices.
Work with your small group to create a script to calculate this index from
the NEON data. Be sure to add comments so that the script is useful to others.
Add your script to the GitHub Repo: DI-NEON-participants to share with your
colleagues. Save scripts to the DI-NEON-participants/2018-RemoteSensing/rs-indices.
Be sure to provide a clear file name reflecting the contents. If you are
comfortable, we recommend you put you names in the script as others may want to
contact you about it.
This page outlines the tools and resources that you will need to install Git, Bash and Python applications onto your computer as the first step of our Python skills tutorial series.
Checklist
Detailed directions to accomplish each objective are below.
Adjusting your PATH environment:
Select "Use Git from the Windows Command Prompt" and click on "Next".
If you forgot to do this programs that you need for the event will not work properly.
If this happens rerun the installer and select the appropriate option.
Configuring the line ending conversions: Click on "Next".
Keep "Checkout Windows-style, commit Unix-style line endings" selected.
Configuring the terminal emulator to use with Git Bash:
Select "Use Windows' default console window" and click on "Next".
Configuring experimental performance tweaks: Click on "Next".
Completing the Git Setup Wizard: Click on "Finish".
This will provide you with both Git and Bash in the Git Bash program.
Install Bash for Mac OS X
The default shell in all versions of Mac OS X is bash, so no
need to install anything. You access bash from the Terminal
(found in
/Applications/Utilities). You may want to keep
Terminal in your dock for this workshop.
Install Bash for Linux
The default shell is usually Bash, but if your
machine is set up differently you can run it by opening a
terminal and typing bash. There is no need to
install anything.
Git Setup
Git is a version control system that lets you track who made changes to what
when and has options for easily updating a shared or public version of your code
on GitHub. You will need a
supported
web browser (current versions of Chrome, Firefox or Safari, or Internet Explorer
version 9 or above).
Git installation instructions borrowed and modified from
Software Carpentry.
Git for Windows
Git should be installed on your computer as part of your Bash install.
Install Git on Macs by downloading and running the most recent installer for
"mavericks" if you are using OS X 10.9 and higher -or- if using an
earlier OS X, choose the most recent "snow leopard" installer, from
this list.
After installing Git, there will not be anything in your
/Applications folder, as Git is a command line program.
**Data Tip:**
If you are running Mac OSX El Capitan, you might encounter errors when trying to
use git. Make sure you update XCODE.
Read more - a Stack Overflow Issue.
Git on Linux
If Git is not already available on your machine you can try to
install it via your distro's package manager. For Debian/Ubuntu run
sudo apt-get install git and for Fedora run
sudo yum install git.
Setting Up Python
Python is a popular language for
scientific computing and data science, as well as being a great for
general-purpose programming. Installing all of the scientific packages
individually can be a bit difficult, so we recommend using an all-in-one
installer, like Anaconda.
Regardless of how you choose to install it, **please make sure your environment
is set up with Python version 3.7 (at the time of writing, the gdal package did not work
with the newest Python version 3.6). Python 2.x is quite different from Python 3.x
so you do need to install 3.x and set up with the 3.7 environment.
We will teach using Python in the
Jupyter Notebook environment,
a programming environment that runs in a web browser. For this to work you will
need a reasonably up-to-date browser. The current versions of the Chrome, Safari
and Firefox browsers are all
supported
(some older browsers, including Internet Explorer version 9 and below, are not).
You can choose to not use notebooks in the course, however, we do
recommend you download and install the library so that you can explore this tool.
Windows
Download and install
Anaconda.
Download the default Python 3 installer (3.7). Use all of the defaults for
installation except make sure to check Make Anaconda the default Python.
Mac OS X
Download and install
Anaconda.
Download the Python 3.x installer, choosing either the graphical installer or the
command-line installer (3.7). For the graphical installer, use all of the defaults for
installation. For the command-line installer open Terminal, navigate to the
directory with the download then enter:
bash Anaconda3-2020.11-MacOSX-x86_64.sh (or whatever you file name is)
Linux
Download and install
Anaconda.
Download the installer that matches your operating system and save it in your
home folder. Download the default Python 3 installer.
Open a terminal window and navigate to your downloads folder. Type
bash Anaconda3-2020.11-Linux-ppc64le.sh
and then press tab. The name of the file you just downloaded should appear.
Press enter. You will follow the text-only prompts. When there is a colon at
the bottom of the screen press the down arrow to move down through the text.
Type yes and press enter to approve the license. Press enter to
approve the default location for the files. Type yes and press
enter to prepend Anaconda to your PATH (this makes the Anaconda
distribution the default Python).
Install Python packages
We need to install several packages to the Python environment to be able to work
with the remote sensing data
gdal
h5py
If you are new to working with command line you may wish to complete the next
setup instructions which provides and intro to command line (bash) prior to
completing these package installation instructions.
Windows
Create a new Python 3.7 environment by opening Windows Command Prompt and typing
conda create –n py37 python=3.7 anaconda
When prompted, activate the py37 environment in Command Prompt by typing
activate py37
You should see (py37) at the beginning of the command line. You can also test
that you are using the correct version by typing python --version.
Install Python package(s):
gdal: conda install gdal
h5py: conda install h5py
Note: You may need to only install gdal as the others may be included in the
default.
Mac OS X
Create a new Python 3.7 environment by opening Terminal and typing
conda create –n py37 python=3.7 anaconda
This may take a minute or two.
When prompted, activate the py37 environment in Command Prompt by typing
source activate py37
You should see (py37) at the beginning of the command line. You can also test
that you are using the correct version by typing python --version.
Install Python package(s):
gdal: conda install gdal
h5py: conda install h5py
Linux
Open default terminal application
(on Ubuntu that will be gnome-terminal).
Launch Python.
Install Python package(s):
gdal: conda install gdal
h5py: conda install h5py
Set up Jupyter Notebook Environment
In your terminal application, navigate to the directory (cd) that where you
want the Jupyter Notebooks to be saved (or where they already exist).
Open Jupyter Notebook with
jupyter notebook
Once the notebook is open, check which version of Python you are in by using the
prompts
# check what version of Python you are using.
import sys
sys.version
You should now be able to work in the notebook.
The gdal package that occasionally has problems with some versions of Python.
Therefore test out loading it using
During the NEON Data Institute, you will share the code that you create daily
with everyone on the NEONScience/DI-NEON-participants repo.
Through this week’s tutorials, you have learned the basic skills needed to
successfully share your work at the Institute including how to:
Create your own GitHub user account,
Set up Git on your computer (please do this on the computer you will be
bringing to the Institute), and
Create a Markdown file with a biography of yourself and the project you are
interested in working on at the Institute. This biography was shared with the
group via the Data Institute’s GitHub repo.
Checklist for this week’s Assignment:
You should have completed the following after Pre-institute week 2:
Fork & clone the NEON-DataSkills/DI-NEON-participants repo.
Create a .md file in the participants/2018-RemoteSensing/pre-institute2-git directory of the
repo. Name the document LastName-FirstName.md.
Write a biography that introduces yourself to the other participants. Please
provide basic information including:
name,
domain of interest,
one goal for the course,
an updated version of your Capstone Project idea,
and the list of data (NEON or other) to support the project that you created
during last week’s materials.
Push the document from your local computer to your GithHub repo.
Created a Pull Request to merge this document back into the
NEON-DataSkills/DI-NEON-participants repo.
NOTE: The Data Institute repository is a public repository, so all members of
the Institute, as well as anyone in the general public who stumbles on the repo,
can see the information. If you prefer not to share this information publicly,
please submit the same document but use a pseudonym (cartoon character names
would work well) and email us with the pseudonym so that we can connect the
submitted document to you.
Have questions? No problem. Leave your question in the comment box below.
It's likely some of your colleagues have the same question, too! And also
likely someone else knows the answer.
This tutorial covers how create and format Markdown files.
Learning Objectives
At the end of this activity, you will be able to:
Create a Markdown (.md) file using a text editor.
Use basic markdown syntax to format a document including: headers, bold and italics.
What is the .md Format?
Markdown is a human readable syntax for formatting text documents. Markdown can
be used to produce nicely formatted documents including pdfs, web pages and more.
In fact, this web page that you are reading right now is generated from a markdown document!
In this tutorial, we will create a markdown file that documents both who you are
and also the project that you might want to work on at the NEON Data Institute.
Markdown Formatting
Markdown is simple plain text, that is styled using symbols, including:
#: a header element
**: bold text
*: italic text
`: code blocks
Let's review some basic markdown syntax.
Plain Text
Plain text will appear as text in a Markdown document. You can format that
text in different ways.
For example, if we want to highlight a function or some code within a plain text
paragraph, we can use one backtick on each side of the text ( ), like this:
Here is some code. This is the backtick, or grave; not an apostrophe (on most
US keyboards it is on the same key as the tilde).
To add emphasis to other text you can use bold or italics.
Have a look at the markdown below:
The use of the highlight ( `text` ) will be reserved for denoting code.
To add emphasis to other text use **bold** or *italics*.
Notice that this sentence uses a code highlight "``", bold and italics.
As a rendered markdown chunk, it looks like this:
The use of the highlight ( text ) will be reserve for denoting code when
used in text. To add emphasis to other text use bold or italics.
Horizontal Lines (rules)
Create a rule:
***
Below is the rule rendered:
Section Headings
You can create a heading using the pound (#) sign. For the headers to render
properly there must be a space between the # and the header text.
Heading one is 1 pound sign, heading two is 2 pound signs, etc as follows:
Data Tip:
There are many free Markdown editors out there! The
atom.io
editor is a powerful text editor package by GitHub, that also has a Markdown
renderer allowing you to see what your Markdown looks like as you are working.
Activity: Create A Markdown Document
Now that you are familiar with the Markdown syntax, use it to create
a brief biography that:
Introduces yourself to the other participants.
Documents the project that you have in mind for the Data Institute.
Add Your Bio
First, create a .md file using the text editor of your preference. Name the
file with the naming convention:
LastName-FirstName.md
Save the file to the participants/2017-RemoteSensing/pre-institute2-git directory in your
local DI-NEON-participants repo (the copy on your computer).
Add a brief bio using headers, bold and italic formatting as makes sense.
In the bio, please provide basic information including:
Your Name
Domain of interest
One goal for the course
Add a Capstone Project Description
Next, add a revised Capstone Project idea to the Markdown document using the
heading ## Capstone Project. Be sure to specify in the document the types of
data that you think you may require to complete your project.
NOTE: The Data Institute repository is a public repository visible to anyone
with internet access. If you prefer to not share your bio information publicly,
please submit your Markdown document using a pseudonym for your name. You may also
want to use a pseudonym for your GitHub account. HINT: cartoon character names work well.
Please email us with the pseudonym so that we can connect the submitted document to you.
Got questions? No problem. Leave your question in the comment box below.
It's likely some of your colleagues have the same question, too! And also
likely someone else knows the answer.
We've forked (made an individual copy of) the NEONScience/DI-NEON-participants repo to
our github.com account.
We've cloned the forked repo - making a copy of it on our local computers.
We've added files and content to our local copy of the repo and committed
the changes.
We've pushed those changes back up to our forked repo on github.com.
Once you've forked and cloned a repo, you are all setup to work on your project.
You won't need to repeat those steps.
When you want to add materials from your repo to the central repo,
you will use a Pull Request. LEFT: Initial workflow after you fork and clone
a repo. RIGHT: Typical workflow once a repo is established (see Git 07 tutorial). Both use pull
requests.
Source: National Ecological Observatory Network (NEON)
In this tutorial, we will learn how to transfer changes from our forked
repo in our github.com account to the central NEON Data Institute repo. Adding
information from your forked repo to the central repo in GitHub is done using a
pull request.
LEFT: To sync changes made and committed to the repo from your
local computer, you will first push the changes from your
local repo to your fork on github.com. RIGHT: Then, you will submit a
Pull Request to update the central repository.
Source: National Ecological Observatory Network (NEON)
**Data Tip:**
A pull request to another repo is similar to a "push". However it allows
for a few things:
It allows you to contribute to another repo without needing administrative
privileges to make changes to the repo.
It allows others to review your changes and suggest corrections, additions,
edits, etc.
It allows repo administrators control over what gets added to
their project repo.
The ability to suggest changes to ANY (public) repo, without needing administrative
privileges is a powerful feature of GitHub. In our case, you do not have privileges
to actually make changes to the DI-NEON-participants repo. However you can
make as many changes
as you want in your fork, and then suggest that NEON add those changes to their
repo, using a pull request. Pretty cool!
Adding to a Repo Using Pull Requests
Pull Requests in GitHub
Step 1 - Start Pull Request
To start a pull request, click the pull request button on the main repo page.
Location of the Pull Request button on a fork of the NEON
Data Institute participants repo (Note, screenshot shows a previous version of
the repo, however, the button is in the same location). Source: National Ecological Observatory
Network (NEON)
Alternatively, you can click the Pull requests tab, then on this new page click the
"New pull request" button.
Step 2 - Choose Repos to Update
Select your fork to compare with NEON central repo. When you begin a pull
request, the head and base will auto-populate as follows:
base fork: NEONScience/DI-NEON-participants
head fork: YOUR-USER-NAME/DI-NEON-participants
The above pull request configuration tells Git to sync (or update) the NEON repo
with contents from your repo.
Head vs Base
Base: the repo that will be updated, the changes will be added to this repo.
Head: the repo from which the changes come.
One way to remember this is that the “head” is always ahead of the base, so
we must add from the head to the base.
Step 3 - Verify Changes
When you compare two repos in a pull request page, git will provide an overview
of the differences (diffs) between the files (if the file is a binary file, like
code. Non-binary files will just show up as a fully new file if it had any changes).
Look over the changes and make sure nothing looks surprising.
In this split view, shows the differences between the older (LEFT)
and newer (RIGHT) document. Deletions are highlighted in red and additions
are highlighted in green.
Pull request diffs view can be changed between unified and split (arrow).
Source: National Ecological Observatory Network (NEON)
Step 4 - Create Pull Request
Click the green Create Pull Request button to create the pull request.
Step 5 - Title Pull Request
Give your pull request a title and write a brief description of your changes.
When you’re done with your message, click Create pull request!
All pull requests titles should be concise and descriptive of
the content in the pull request. More detailed notes can be left in the comments
box.
Source: National Ecological Observatory Network (NEON)
Check out the repo name up at the top (in your repo and in screenshot above)
When creating the pull request you will be automatically transferred to the base
repo. Since the central repo was the base, github will automatically transfer
you to the central repo landing page.
Step 6 - Merge Pull Request
In this final step, it’s time to merge your changes in the
NEONScience/DI-NEON-participants repo.
NOTE 1: You are only able to merge a pull request in a repo that you have
permissions to!
NOTE 2: When collaborating, it is generally poor form to merge your own Pull Request,
better to tag (@username) a collaborator in the comments so they know you want
them to look at it. They can then review and, if acceptable, merge it.
To merge, your (or someone else's PR click the green "Merge Pull Request"
button to "accept" or merge the updated commits in the central repo into your
repo. Then click Confirm Merge.
We now synced our forked repo with the central NEON Repo. The next step in working
in a GitHub workflow is to transfer any changes in the central repository into
your local repo so you can work with them.
Data Institute Activity: Submit Pull Request for Week 2 Assignment
Submit a pull request containing the .md file that you created in this
tutorial-series series. Before you submit your PR, review the
Week 2 Assignment page.
To ensure you have all of the required elements in your .md file.
To submit your PR:
Repeat the pull request steps above, with the base and head switched. Your base
will be the NEON central repo and your HEAD will be YOUR forked repo:
base fork: NEONScience/DI-NEON-participants
head fork: YOUR-USER-NAME/DI-NEON-participants
When you get to Step 6 - Merge Pull Request (PR), are you able to merge the PR?
Finally, go to the NEON Central Repo page in github.com. Look for the Pull Requests
link at the top of the page. How many Pull Requests are there?
Click on the link - do you see your Pull Request?
You can only merge a PR if you have permissions in the base repo that you are
adding to. At this point you don’t have contributor permissions to the NEON repo.
Instead someone who is a contributor on the repository will need to review and
accept the request.
After completing the pull request to upload your bio markdown file, be sure
to continue on to Git 07: Updating Your Repo by Setting Up a Remote
to learn how to update your local fork and really begin
the cycle of working with Git & GitHub in a collaborative manner.
Workflow Summary
Add updates to Central Repo with Pull Request
On github.com
Button: Create New Pull Request
Set base: central Institute repo, set head: your Fork
Make sure changes are what you want to sync
Button: Create Pull Request
Add Pull Request title & comments
Button: Create Pull Request
Button: Merge Pull Request - if working collaboratively, poor style to merge
your own PR, and you only can if you have contributor permissions
Have questions? No problem. Leave your question in the comment box below.
It's likely some of your colleagues have the same question, too! And also
likely someone else knows the answer.
