NEON @ ESA: A Hands-On Primer for Working with Big Data in R – Introduction to Hierarchical Data Formats, Lidar Data & Efficient Data Visualization

August 09, 2015
Ecological Society of America Annual Meeting

Ecologists working across scales and integrating disparate datasets face new data management and analysis challenges that demand tools beyond the spreadsheet. This workshop will overview three key data formats: ASCII, HDF5 and las and several key data types including temperature data from a tower, vegetation structure data, hyperspectral imagery and lidar data, that are often encountered when working with ‘Big Data’. It will provide an introduction to available tools in R for working with these formats and types.

Things to Do Before the Workshop

To participant in this workshop, you will need a laptop with the most current version of R and, preferably, RStudio loaded on your computer. For details on setting up R & RStudio in Mac, PC, or Linux operating systems please see Additional Set up Resources below.

Install R Libraries

Please install or update each package prior to the start of the workshop.

  • rgeos: install.packages("rgeos")
  • maptools: install.packages("maptools")
  • raster: install.packages("raster")
  • rgdal: install.packages("rgdal")
  • ggplot2: install.packages("ggplot2")
  • rhdf5: source("http://bioconductor.org/biocLite.R") biocLite("rhdf5")
  • ggplot2: install.packages("ggplot2")
  • dpylr: install.packages("dplyr"): data manipulation at its finest!

Data to Download

Download NEON Teaching Data Subset: Field Site Spatial Data

These remote sensing data files provide information on the vegetation at the National Ecological Observatory Network's San Joaquin Experimental Range and Soaproot Saddle field sites. The entire dataset can be accessed by request from the NEON Data Portal.

Download NEON Teaching Data Subset: Imaging Spectrometer Data - HDF5

These hyperspectral remote sensing data provide information on the National Ecological Observatory Network's San Joaquin Exerimental Range field site. The data were collected over the San Joaquin field site located in California (Domain 17) and processed at NEON headquarters. The entire dataset can be accessed by request from the NEON Data Portal.

Download NEON Teaching Data Subset: Sample LiDAR Point Cloud Data (.las)

This .las file contains sample LiDAR point cloud data collected by National Ecological Observatory Network's Airborne Observation Platform group. The .las file format is a commonly used file format to store LIDAR point cloud data.

After downloading the data, please uncompress and set your R working directory to the parent directory containing these files. For additional instructions see the bottom of the page. We will be setting up an RStudio project within this working directory. Read more about RStudio projects here.

Background Reading

This workshop will be most useful to those with some basic understanding of spatial & hierarchical data. Please read the following materials if you would like additional background on these topics.

Workshop Instructors

  • Leah Wasser @leahawasser, Supervising Scientist, NEON, Inc
  • Natalie Robinson, Staff Scientist, NEON, Inc
Workshop Fearless Instruction Assistants
  • Claire Lunch @dr_lunch, Staff Scientist
  • Kate Thibault @fluby, Senior Staff Scientist
  • Christine Laney @cmlaney, Staff Scientist, NEON, Inc
  • Mike Smorul @msmorul, Associate Director of Cyberinfrastructure, SESYNC
  • Philippe Marchand, Scientific Support Specialist, SESYNC

Social Media

Please tweet using the hashtag #WorkWithData during this workshop! Also you can tweet at @NEON_Sci !

Schedule

Location: ESA Annual Meeting, Baltimore, Maryland

The schedule below is subject to change.

Time Topic Instructor
8:00 Welcome, Introductions, & Logistics
8:05 Getting Started with Rasters in R Natalie
9:30 Raster Resolution, Extent & CRS in R Natalie
10:15 ------- BREAK -------
10:30 LiDAR Data Derived Rasters in R Leah
11:45 - 1:00 PM Lunch on Your Own
1:00 Introduction to HDF5 in R Leah
2:30 ------- BREAK -------
2:45 Hyperspectral Imagery in R Leah
Create Raster Stacks & NDVI in R (if time allows) Leah
3:45 ------- BREAK -------
4:00 Application of Skills with Capstone Project

Additional Set Up Instructions

R & RStudio

Prior to the workshop you should have R and, preferably, RStudio installed on your computer.

Setting Up R & RStudio

Windows R/RStudio Setup

Once R and RStudio are installed, click to open RStudio. If you don't get any error messages you are set. If there is an error message, you will need to re-install the program.

Mac R/RStudio Setup

  • If your Mac is set up for UiO use, you can install R from Managed Software Center

  • Go to CRAN and click on Download R for (Mac) OS X

  • Select the .pkg file for the version of OS X that you have and the file will download.
  • Double click on the file that was downloaded and R will install
  • Go to the RStudio Download page
  • Under Installers select RStudio 0.98.1103 - Mac OS X 10.6+ (64-bit) to download it.
  • Once it's downloaded, double click the file to install it

Once R and RStudio are installed, click to open RStudio. If you don't get any error messages you are set. If there is an error message, you will need to re-install the program.

Linux R/RStudio Setup

  • R is available through most Linux package managers. You can download the binary files for your distribution from CRAN. Or you can use your package manager (e.g. for Debian/Ubuntu run sudo apt-get install r-base and for Fedora run sudo yum install R).
  • To install RStudi, go to the RStudio Download page
  • Under Installers select the version for your distribution.
  • Once it's downloaded, double click the file to install it

Once R and RStudio are installed, click to open RStudio. If you don't get any error messages you are set. If there is an error message, you will need to re-install the program.

<

h2 id="set-wd">Set Working Directory to Downloaded Data

1) Download Data

After clicking on the Download Data button, the data will automatically download to the computer.

2) Locate .zip file

Second, find the downloaded .zip file. Many browsers save downloaded files to your computer’s Downloads directory. If you have previously specified a specific directory (folder) for downloaded files, the .zip file will download there.

3) Move to **data** directory

Third, move the downloaded file to a directory called data within the Documents directory on your computer. You can choose to place the data in other locations, however, you will need to remember to set your R Working Directory to that location and not as we demonstrate in the workshop.

4) Unzip/uncompress

Fourth, we need to unzip/uncompress the file so that the data files can be accessed. Use your favorite tool that can unpackage/open .zip files (e.g., winzip, Archive Utility, etc). The files will now be accessible in three directories:

These directories contain all of the subdirectories and files that we will use in this workshop.

5) Set working directory

Fifth, we need to set the working directory in R to this data directory that is parent to the directories containing the data we want. For complete directions, on how to do that check out the Set A Working Directory in R tutorial.

Install HDFView

The free HDFView application allows you to explore the contents of an HDF5 file.

To install HDFView:

  1. Click to go to the download page.

  2. From the section titled HDF-Java 2.1x Pre-Built Binary Distributions select the HDFView download option that matches the operating system and computer setup (32 bit vs 64 bit) that you have. The download will start automatically.

  3. Open the downloaded file.

    • Mac - You may want to add the HDFView application to your Applications directory.
    • Windows - Unzip the file, open the folder, run the .exe file, and follow directions to complete installation.
  4. Open HDFView to ensure that the program installed correctly.

Data Tip: The HDFView application requires Java to be up to date. If you are having issues opening HDFView, try to update Java first!

QGIS (Optional)

QGIS is a cross-platform Open Source Geographic Information system.

Online LiDAR Data/las Viewer (Optional)

Plas.io is a open source LiDAR data viewer developed by Martin Isenberg of Las Tools and several of his colleagues.

Time Topic Instructor
8:00 Welcome, Introductions & Logistics
8:05 Getting Started with Rasters in R Natalie
9:30 Raster Resolution, Extent & CRS in R Natalie
10:15 ------- BREAK -------
10:30 LiDAR Derived Rasters in R Leah
11:45 - 1:00 PM Lunch on Your Own
1:00 Introduction to HDF5 in R Leah
2:30 ------- BREAK -------
2:45 Hyperspectral Imagery in R Leah
Create Raster Stacks & NDVI in R (If time permits) Leah
3:45 ------- BREAK -------
4:00 Application of Skills: Capstone Activity

Workshop Notes

Visit the Workshop Etherpad to review collaborative notes from the workshop.

Tutorial: Raster Data in R - The Basics

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Tutorial: The Relationship Between Raster Resolution, Spatial Extent & Number of Pixels

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Tutorial: Create a Canopy Height Model from LiDAR-derived Rasters in R

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Tutorial: Introduction to HDF5 Files in R

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Tutorial: Intro to Working with Hyperspectral Remote Sensing Data in HDF5 Format in R

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Tutorial: Creating a Raster Stack from Hyperspectral Imagery in HDF5 Format in R

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Tutorial: Spatial Data Tutorial Series Capstone Challenges

You can choose to do this optional data activity (opens in a new window), or continue to the next lesson plan.

Add new comment

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Find more workshops
Workshop Overview
Workshop Materials
Dialog content.