ESA 2015: A Hands-On Primer for Working with Big Data in R: Introduction to Hierarchical Data Formats, Lidar Data & Efficient Data Visualization

Ecologists working across scales and integrating disparate datasets face new data management and analysis challenges that demand tools beyond the spreadsheet. This workshop will overview three key data formats: ASCII, HDF5 and las and several key data types including temperature data from a tower, vegetation structure data, hyperspectral imagery and lidar data, that are often encountered when working with ‘Big Data’. It will provide an introduction to available tools in R for working with these formats and types.

Things to Do Before the Workshop

To participant in this workshop, you will need a laptop with the most current version of R and, preferably, RStudio loaded on your computer. For details on setting up R & RStudio in Mac, PC, or Linux operating systems please see Additional Set up Resources below.

Install R Libraries

Please install or update each package prior to the start of the workshop.

• rgeos: install.packages("rgeos")
• maptools: install.packages("maptools")
• raster: install.packages("raster")
• rgdal: install.packages("rgdal")
• ggplot2: install.packages("ggplot2")
• rhdf5: source("http://bioconductor.org/biocLite.R") biocLite("rhdf5")
• ggplot2: install.packages("ggplot2")
• dpylr: install.packages("dplyr"): data manipulation at its finest!

Data to Download

NEON Teaching Data Subset: Field Site Spatial Data

These remote sensing data files provide information on the vegetation at the National Ecological Observatory Network's San Joaquin Experimental Range and Soaproot Saddle field sites. The entire dataset can be accessed by request from the NEON Data Portal.

DOWNLOAD DATASET

Download NEON Teaching Data Subset: Imaging Spectrometer Data - HDF5

These hyperspectral remote sensing data provide information on the National Ecological Observatory Network's San Joaquin Experimental Range field site. The data were collected over the San Joaquin field site located in California (Domain 17) and processed at NEON headquarters. The entire dataset can be accessed by request from the NEON Data Portal.

DOWNLOAD DATASET

NEON Teaching Data Subset: Sample LiDAR Point Cloud Data (.las)

This .las file contains sample LiDAR point cloud data collected by National Ecological Observatory Network's Airborne Observation Platform group. The .las file format is a commonly used file format to store LIDAR point cloud data. NEON data are available on the NEON data portal.

DOWNLOAD NEON TEACHING DATA SUBSET: SAMPLE LIDAR POINT CLOUD DATA (.LAS)

After downloading the data, please uncompress and set your R working directory to the parent directory containing these files. For additional instructions see the bottom of the page. We will be setting up an RStudio project within this working directory. Read more about RStudio projects here.

Background Reading

This workshop will be most useful to those with some basic understanding of spatial & hierarchical data. Please read the following materials if you would like additional background on these topics.

• Read more about the HDF5 viewer
• A brief introduction to Lidar Data
• About the basic Lidar Derived Data Products - CHM, DEM, DSM
• A brief intro to the Hierarchical Data Format (HDF5) format.
• A brief intro to Hyperspectral Remote Sensing data format.
• A 5-minute video on imagery/optical remote sensing.
• Documentation for the raster package in R.

Workshop Instructors

• Leah Wasser @leahawasser, Supervising Scientist, NEON, Inc
• Natalie Robinson, Staff Scientist, NEON, Inc

Workshop Fearless Instruction Assistants

• Claire Lunch @dr_lunch, Staff Scientist
• Kate Thibault @fluby, Senior Staff Scientist
• Christine Laney @cmlaney, Staff Scientist, NEON, Inc
• Mike Smorul @msmorul, Associate Director of Cyberinfrastructure, SESYNC
• Philippe Marchand, Scientific Support Specialist, SESYNC

Social Media

Please tweet using the hashtag #WorkWithData during this workshop! Also you can tweet at @NEON_Sci !

Schedule

Location: ESA Annual Meeting, Baltimore, Maryland

The schedule below is subject to change.

Additional Set Up Instructions

R & RStudio

Prior to the workshop you should have R and, preferably, RStudio installed on your computer.

Setting Up R & RStudio

Windows R/RStudio Setup

• Download R for Windows here
• Run the .exe file that was just downloaded
• Go to the RStudio Download page
• Under Installers select RStudio X.XX.XXX - Windows Vista/7/8/10
• Double click the file to install it

Once R and RStudio are installed, click to open RStudio. If you don't get any error messages you are set. If there is an error message, you will need to re-install the program.

Mac R/RStudio Setup

• Go to CRAN and click on Download R for (Mac) OS X
• Select the .pkg file for the version of OS X that you have and the file will download.
• Double click on the file that was downloaded and R will install
• Go to the RStudio Download page
• Under Installers select RStudio 0.98.1103 - Mac OS X XX.X (64-bit) to download it.
• Once it's downloaded, double click the file to install it

Once R and RStudio are installed, click to open RStudio. If you don't get any error messages you are set. If there is an error message, you will need to re-install the program.

Linux R/RStudio Setup

• R is available through most Linux package managers. You can download the binary files for your distribution from CRAN. Or you can use your package manager (e.g. for Debian/Ubuntu run sudo apt-get install r-base and for Fedora run sudo yum install R).
• To install RStudio, go to the RStudio Download page
• Under Installers select the version for your distribution.
• Once it's downloaded, double click the file to install it

Once R and RStudio are installed, click to open RStudio. If you don't get any error messages you are set. If there is an error message, you will need to re-install the program.

Set Working Directory to Downloaded Data

1) Download Data

After clicking on the Download Data button, the data will automatically download to the computer.

2) Locate .zip file

Second, find the downloaded .zip file. Many browsers save downloaded files to your computer’s Downloads directory. If you have previously specified a specific directory (folder) for downloaded files, the .zip file will download there.

3) Move to **data** directory

Third, move the downloaded file to a directory called data within the Documents directory on your computer. You can choose to place the data in other locations, however, you will need to remember to set your R Working Directory to that location and not as we demonstrate in the workshop.

4) Unzip/uncompress

Fourth, we need to unzip/uncompress the file so that the data files can be accessed. Use your favorite tool that can unpackage/open .zip files (e.g., winzip, Archive Utility, etc). The files will now be accessible in three directories:

These directories contain all of the subdirectories and files that we will use in this workshop.

5) Set working directory

Fifth, we need to set the working directory in R to this data directory that is parent to the directories containing the data we want. For complete directions, on how to do that check out the Set A Working Directory in R tutorial.

Install HDFView

The free HDFView application allows you to explore the contents of an HDF5 file.

To install HDFView:

1. Click to go to the download page.
2. From the section titled HDF-Java 2.1x Pre-Built Binary Distributions select the HDFView download option that matches the operating system and computer setup (32 bit vs 64 bit) that you have. The download will start automatically.
3. Open the downloaded file.
   • Mac - You may want to add the HDFView application to your Applications directory.
   • Windows - Unzip the file, open the folder, run the .exe file, and follow directions to complete installation.
4. Open HDFView to ensure that the program installed correctly.

 Data Tip: The HDFView application requires Java to be up to date. If you are having issues opening HDFView, try to update Java first!

QGIS (Optional)

QGIS is a cross-platform Open Source Geographic Information system.

Online LiDAR Data/las Viewer (Optional)

Plas.io is a open source LiDAR data viewer developed by Martin Isenberg of Las Tools and several of his colleagues.