Series

Get Started with NEON Data: A Series of Data Tutorials

This Data Tutorial Series is designed to provide you with an introduction to NEON and then provide an overview of how to access and use NEON data. We propose the following order to go through these materials, note, one resource is Data Tutorials linked in the Table of Contents to the right.

Foundational Skills and Tools to Access NEON Data

Start with an Introduction to the National Ecological Observatory Network (NEON) to learn more about the resources that NEON provides.
Use the Access NEON Data using The Data Portal Visual Guide to explore the NEON Data Portal [direct download]
Then start in R with Download and Explore NEON Data to learn the basics of accessing NEON data programmatically.
Learn more important functionality in Use the neonUtilities Package to Access NEON Data.
Optional: Learn more about Using the NEON API in R if you want to write API calls and access data programmatically with the neonUtilties package.

Introductions To Working With Different Data Types

Start working with an exemplar Observational Sampling (OS) data set in Work With NEON's Plant Phenology Data.
Or choose an exemplar Instrumented Sampling (IS) data set in Work with NEON's Single-Aspirated Air Temperature Data.
Combine the two in Plot Continuous & Discrete Data Together.
Choose an intro to working with budled eddy/flux data.

Introduction to the National Ecological Observatory Network (NEON)

Last Updated: Oct 7, 2020

Here we will provide an overview of the National Ecological Observatory Network (NEON). Please carefully read through these materials and links that discuss NEON’s mission and design.

Data Institute participants: As you review this information, please consider the capstone project that you may want to work on at the Institute. At the end of week one, you will document an initial research question or idea and associated data needed to address that question, that you may want to explore while at the NEON Data Institute.

Learning Objectives

At the end of this activity, you will be able to:

Explain the mission of the National Ecological Observatory Network (NEON).
Explain the how sites are located within the NEON project design.
Explain the different types of data that will be collected and provided by NEON.

The NEON Project Mission & Design

To capture ecological heterogeneity across the United States, NEON’s design divides the continent into 20 statistically different eco-climatic domains. Each NEON field site is located within an eco-climatic domain.

The Science and Design of NEON

To gain a better understanding of the broad scope fo NEON watch this 4 minute long video.

Please, read the following page about NEON's mission.

Data Institute Participants -- Thought Question: How might/does the NEON project intersect with your current research or future career goals?

NEON's Spatial Design

The Spatial Design of NEON

Watch this 4:22 minute video exploring the spatial design of NEON field sites.

Please read the following page about NEON's Spatial Design:

NEON Field Site Locations

Explore the NEON Field Site map taking note of the locations of

Aquatic & terrestrial field sites.
Core & relocatable field sites.

Click here to view the NEON Field Site Map

Explore the NEON field site map. Do the following:

Zoom in on a study area of interest to see if there are any NEON field sites that are nearby.
Click the "More" button in the upper right hand corner of the map to filter sites by name, site host, domain or state.
Select one field site of interest.
- Click on the marker in the map.
- Then click on the name of the field site to jump to the field site landing page.

NOTE: During summer/fall 2018 these field site pages are being updated, during this time the amount of information on each site may vary.

Data Institute Participant -- Thought Questions: Use the map above to answer these questions. Consider the research question that you may explore as your Capstone Project at the Institute or about a current project that you are working on and answer the following questions:

Are there NEON field sites that are in study regions of interest to you?
What domains are the sites located in?
What NEON field sites do your current research or Capstone Project ideas coincide with?
Is the site(s) core or relocatable?
Is it/are they terrestrial or aquatic?
Are there data available for the NEON field site(s) that you are most interested in? What kind of data are available?

Data Tip: You can download maps, kmz, or shapefiles of the field sites here.

NEON Data

How NEON Collects Data

Watch this 3:06 minute video exploring the data that NEON collects.

Read the Data Collection Methods page to learn more about the different types of data that NEON collects and provides. Then, follow the links below to learn more about each collection method:

All data collection protocols and processing documents are publicly available. Read more about the standardized protocols and how to access these documents.

Specimens & Samples

NEON also collects samples and specimens from which the other data products are based. These samples are also available for research and education purposes. Learn more: NEON Biorepository.

Airborne Remote Sensing

Watch this 5 minute video to better understand the NEON Airborne Observation Platform (AOP).

Data Institute Participant – Thought Questions: Consider either your current or future research or the question you’d like to address at the Institute.

Which types of NEON data may be more useful to address these questions?
What non-NEON data resources could be combined with NEON data to help address your question?
What challenges, if any, could you foresee when beginning to work with these data?

Data Tip: NEON also provides support to your own research including proposals to fly the AOP over other study sites, a mobile tower/instrumentation setup and others. Learn more here the Assignable Assets programs .

Access NEON Data

NEON data are processed and go through quality assurance quality control checks at NEON headquarters in Boulder, CO. NEON carefully documents every aspect of sampling design, data collection, processing and delivery. This documentation is freely available through the NEON data portal.

Visit the NEON Data Portal - data.neonscience.org
Read more about the quality assurance and quality control processes for NEON data and how the data are processed from raw data to higher level data products.
Explore NEON Data Products. On the page for each data product in the catalog you can find the basic information about the product, find the data collection and processing protocols, and link directly to downloading the data.
Additionally, some types of NEON data are also available through the data portals of other organizations. For example, NEON Terrestrial Insect DNA Barcoding Data is available through the Barcode of Life Datasystem (BOLD). Or NEON phenocam images are available from the Phenocam network site. More details on where else the data are available from can be found in the Availability and Download section on the Product Details page for each data product (visit Explore Data Products to access individual Product Details pages).

Pathways to access NEON Data

There are several ways to access data from NEON:

Via the NEON data portal. Explore and download data. Note that much of the tabular data is available in zipped .csv files for each month and site of interest. To combine these files, use the neonUtilities package (R tutorial, Python tutorial).
Use R or Python to programmatically access the data. NEON and community members have created code packages to directly access the data through an API. Learn more about the available resources by reading the Code Resources page or visiting the NEONScience GitHub repo.
Using the NEON API. Access NEON data directly using a custom API call.
Access NEON data through partner's portals. Where NEON data directly overlap with other community resources, NEON data can be accessed through the portals. Examples include Phenocam, BOLD, Ameriflux, and others. You can learn more in the documentation for individual data products.

Data Institute Participant – Thought Questions: Use the Data Portal tools to investigate the data availability for the field sites you’ve already identified in the previous Thought Questions.

What types of aquatic/terrestrial data are currently available? Remote sensing data?
Of these, what type of data are you most interested in working with for your project while at the Institute.
For what time period does the data cover?
What format is the downloadable file available in?
Where is the metadata to support this data?

Data Institute Participants: Intro to NEON Culmination Activity

Write up a brief summary of a project that you might want to explore while at the Data Institute in Boulder, CO. Include the types of NEON (and other data) that you will need to implement this project. Save this summary as you will be refining and adding to your ideas over the next few weeks.

The goal of this activity if for you to begin to think about the capstone project that you wish to work on while at the Data Institute. This project will ideally be performed in groups, so over the next few weeks you'll have a chance to view the other project proposals and merge projects to collaborate with your colleagues.

Download and Explore NEON Data

Authors: Claire K. Lunch

Last Updated: Apr 8, 2021

This tutorial covers downloading NEON data, using the Data Portal and the neonUtilities R package, as well as basic instruction in beginning to explore and work with the downloaded data, including guidance in navigating data documentation.

NEON data

There are 3 basic categories of NEON data:

Remote sensing (AOP) - Data collected by the airborne observation platform, e.g. LIDAR, surface reflectance
Observational (OS) - Data collected by a human in the field, or in an analytical laboratory, e.g. beetle identification, foliar isotopes
Instrumentation (IS) - Data collected by an automated, streaming sensor, e.g. net radiation, soil carbon dioxide. This category also includes the eddy covariance (EC) data, which are processed and structured in a unique way, distinct from other instrumentation data (see Tutorial for EC data for details).

This lesson covers all three types of data. The download procedures are similar for all types, but data navigation differs significantly by type.

Objectives

After completing this activity, you will be able to:

Download NEON data using the neonUtilities package.
Understand downloaded data sets and load them into R for analyses.

Things You’ll Need To Complete This Tutorial

To complete this tutorial you will need the most current version of R and, preferably, RStudio loaded on your computer.

Install R Packages

neonUtilities: Basic functions for accessing NEON data
raster: Raster package; needed for remote sensing data

Both of these packages can be installed from CRAN:

install.packages("neonUtilities")
install.packages("raster")

Additional Resources

Tutorial for neonUtilities. Some overlap with this tutorial but goes into more detail about the neonUtilities package.
Tutorial for using neonUtilities from a Python environment.
GitHub repository for neonUtilities

Getting started: Download data from the Portal and load packages

Go to the NEON Data Portal and download some data! Almost any IS or OS data product can be used for this section of the tutorial, but we will proceed assuming you've downloaded Photosynthetically Active Radiation (PAR) (DP1.00024.001) data. For optimal results, download three months of data from one site. The downloaded file should be a zip file named NEON_par.zip. For this tutorial, we will be using PAR data from the Wind River Experimental Forest (WREF) in Washington state from September-November 2019.

Now switch over to R and load all the packages installed above.

# load packages
library(neonUtilities)
library(raster)

# Set global option to NOT convert all character variables to factors
options(stringsAsFactors=F)

Stack the downloaded data files: stackByTable()

The stackByTable() function will unzip and join the files in the downloaded zip file.

# Modify the file path to match the path to your zip file
stackByTable("~/Downloads/NEON_par.zip")

In the same directory as the zipped file, you should now have an unzipped folder of the same name. When you open this you will see a new folder called stackedFiles, which should contain five files: PARPAR_30min.csv, PARPAR_1min.csv, sensor_positions.csv, variables.csv, and readme.txt.

We'll look at these files in more detail below.

Download files and load directly to R: loadByProduct()

In the section above, we downloaded a .zip file from the data portal to our downloads folder, then used the stackByTable() function to transform those data into a usable format. However, there is a faster way to load data directly into the R Global Environment using loadByProduct().

The most popular function in neonUtilities is loadByProduct(). This function downloads data from the NEON API, merges the site-by-month files, and loads the resulting data tables into the R environment, assigning each data type to the appropriate R class. This is a popular choice because it ensures you're always working with the latest data, and it ends with ready-to-use tables in R. However, if you use it in a workflow you run repeatedly, keep in mind it will re-download the data every time.

loadByProduct() works on most observational (OS) and sensor (IS) data, but not on surface-atmosphere exchange (SAE) data, remote sensing (AOP) data, and some of the data tables in the microbial data products. For functions that download AOP data, see the byFileAOP() and byTileAOP() sections in this tutorial. For functions that work with SAE data, see the NEON eddy flux data tutorial.

The inputs to loadByProduct() control which data to download and how to manage the processing:

dpID: the data product ID, e.g. DP1.00002.001
site: defaults to "all", meaning all sites with available data; can be a vector of 4-letter NEON site codes, e.g. c("HARV","CPER","ABBY").
startdate and enddate: defaults to NA, meaning all dates with available data; or a date in the form YYYY-MM, e.g. 2017-06. Since NEON data are provided in month packages, finer scale querying is not available. Both start and end date are inclusive.
package: either basic or expanded data package. Expanded data packages generally include additional information about data quality, such as chemical standards and quality flags. Not every data product has an expanded package; if the expanded package is requested but there isn't one, the basic package will be downloaded.
avg: defaults to "all", to download all data; or the number of minutes in the averaging interval. See example below; only applicable to IS data.
savepath: the file path you want to download to; defaults to the working directory.
check.size: T or F: should the function pause before downloading data and warn you about the size of your download? Defaults to T; if you are using this function within a script or batch process you will want to set it to F.
nCores: Number of cores to use for parallel processing. Defaults to 1, i.e. no parallelization.
forceParallel: If the data volume to be processed does not meet minimum requirements to run in parallel, this overrides.

The dpID is the data product identifier of the data you want to download. The DPID can be found on the Explore Data Products page. It will be in the form DP#.#####.###

Here, we'll download aquatic plant chemistry data from three lake sites: Prairie Lake (PRLA), Suggs Lake (SUGG), and Toolik Lake (TOOK).

apchem <- loadByProduct(dpID="DP1.20063.001", 
                  site=c("PRLA","SUGG","TOOK"), 
                  package="expanded", check.size=T)

The object returned by loadByProduct() is a named list of data frames. To work with each of them, select them from the list using the $ operator.

names(apchem)
View(apchem$apl_plantExternalLabDataPerSample)

If you prefer to extract each table from the list and work with it as an independent object, you can use the list2env() function:

list2env(apchem, .GlobalEnv)

## <environment: R_GlobalEnv>

If you want to be able to close R and come back to these data without re-downloading, you'll want to save the tables locally. We recommend also saving the variables file, both so you'll have it to refer to, and so you can use it with readTableNEON() (see below).

write.csv(apl_clipHarvest, 
          "~/Downloads/apl_clipHarvest.csv", 
          row.names=F)
write.csv(apl_biomass, 
          "~/Downloads/apl_biomass.csv", 
          row.names=F)
write.csv(apl_plantExternalLabDataPerSample, 
          "~/Downloads/apl_plantExternalLabDataPerSample.csv", 
          row.names=F)
write.csv(variables_20063, 
          "~/Downloads/variables_20063.csv", 
          row.names=F)

But, if you want to save files locally and load them into R (or another platform) each time you run a script, instead of downloading from the API every time, you may prefer to use zipsByProduct() and stackByTable() instead of loadByProduct(), as we did in the first section above. Details can be found in our neonUtilities tutorial. You can also try out the community-developed neonstore package, which is designed for maintaining a local store of the NEON data you use.

Download remote sensing data: byFileAOP() and byTileAOP()

Remote sensing data files are very large, so downloading them can take a long time. byFileAOP() and byTileAOP() enable easier programmatic downloads, but be aware it can take a very long time to download large amounts of data.

Input options for the AOP functions are:

dpID: the data product ID, e.g. DP1.00002.001
site: the 4-letter code of a single site, e.g. HARV
year: the 4-digit year to download
savepath: the file path you want to download to; defaults to the working directory
check.size: T or F: should the function pause before downloading data and warn you about the size of your download? Defaults to T; if you are using this function within a script or batch process you will want to set it to F.
easting: byTileAOP() only. Vector of easting UTM coordinates whose corresponding tiles you want to download
northing: byTileAOP() only. Vector of northing UTM coordinates whose corresponding tiles you want to download
buffer: byTileAOP() only. Size in meters of buffer to include around coordinates when deciding which tiles to download

Here, we'll download one tile of Ecosystem structure (Canopy Height Model) (DP3.30015.001) from WREF in 2017.

byTileAOP("DP3.30015.001", site="WREF", year="2017", check.size = T,
          easting=580000, northing=5075000, savepath="~/Downloads")

In the directory indicated in savepath, you should now have a folder named DP3.30015.001 with several nested subfolders, leading to a tif file of a canopy height model tile. We'll look at this in more detail below.

Navigate data downloads: IS

Let's take a look at the PAR data we downloaded earlier. We'll read in the 30-minute file using the function readTableNEON(), which uses the variables.csv file to assign data types to each column of data:

par30 <- readTableNEON(
  dataFile="~/Downloads/NEON_par/stackedFiles/PARPAR_30min.csv", 
  varFile="~/Downloads/NEON_par/stackedFiles/variables_00024.csv")
View(par30)

The first four columns are added by stackByTable() when it merges files across sites, months, and tower heights. The final column, publicationDate, is the date-time stamp indicating when the data were published. This can be used as an indicator for whether data have been updated since the last time you downloaded them.

The remaining columns are described by the variables file:

parvar <- read.csv("~/Downloads/NEON_par/stackedFiles/variables_00024.csv")
View(parvar)

The variables file shows you the definition and units for each column of data.

Now that we know what we're looking at, let's plot PAR from the top tower level:

plot(PARMean~startDateTime, 
     data=par30[which(par30$verticalPosition=="080"),],
     type="l")

Looks good! The sun comes up and goes down every day, and some days are cloudy. If you want to dig in a little deeper, try plotting PAR from lower tower levels on the same axes to see light attenuation through the canopy.

Navigate data downloads: OS

Let's take a look at the aquatic plant data. OS data products are simple in that the data generally tabular, and data volumes are lower than the other NEON data types, but they are complex in that almost all consist of multiple tables containing information collected at different times in different ways. For example, samples collected in the field may be shipped to a laboratory for analysis. Data associated with the field collection will appear in one data table, and the analytical results will appear in another. Complexity in working with OS data usually involves bringing data together from multiple measurements or scales of analysis.

As with the IS data, the variables file can tell you more about the data. OS data also come with a validation file, which contains information about the validation and controlled data entry that were applied to the data:

View(variables_20063)

View(validation_20063)

OS data products each come with a Data Product User Guide, which can be downloaded with the data, or accessed from the document library on the Data Portal, or the Product Details page for the data product. The User Guide is designed to give a basic introduction to the data product, including a brief summary of the protocol and descriptions of data format and structure.

To get started with the aquatic plant chemistry data, let's take a look at carbon isotope ratios in plants across the three sites we downloaded. The chemical analytes are reported in the apl_plantExternalLabDataPerSample table, and the table is in long format, with one record per sample per analyte, so we'll subset to only the carbon isotope analyte:

boxplot(analyteConcentration~siteID, 
        data=apl_plantExternalLabDataPerSample, 
        subset=analyte=="d13C",
        xlab="Site", ylab="d13C")

We see plants at Suggs and Toolik are quite low in 13C, with more spread at Toolik than Suggs, and plants at Prairie Lake are relatively enriched. Clearly the next question is what species these data represent. But taxonomic data aren't present in the apl_plantExternalLabDataPerSample table, they're in the apl_biomass table. We'll need to join the two tables to get chemistry by taxon.

The Data Relationships section of the User Guide can help you determine which fields to use as the key to join the tables. Here, sampleID is the joining variable. We'll also include the basic spatial variables, to avoid creating unnecessary duplicates of those columns.

apct <- merge(apl_biomass, 
              apl_plantExternalLabDataPerSample, 
              by=c("sampleID","namedLocation",
                   "domainID","siteID"))

Using the merged data, now we can plot carbon isotope ratio for each taxon.

boxplot(analyteConcentration~scientificName, 
        data=apct, subset=analyte=="d13C", 
        xlab=NA, ylab="d13C", 
        las=2, cex.axis=0.7)

And now we can see most of the sampled plants have carbon isotope ratios around -30, with just two species accounting for most of the more enriched samples.

Navigate data downloads: AOP

To work with AOP data, the best bet is the raster package. It has functionality for most analyses you might want to do.

We'll use it to read in the tile we downloaded:

chm <- raster("~/Downloads/DP3.30015.001/2017/FullSite/D16/2017_WREF_1/L3/DiscreteLidar/CanopyHeightModelGtif/NEON_D16_WREF_DP3_580000_5075000_CHM.tif")

The raster package includes plotting functions:

plot(chm, col=topo.colors(6))

Now we can see canopy height across the downloaded tile; the tallest trees are over 60 meters, not surprising in the Pacific Northwest. There is a clearing or clear cut in the lower right corner.

Get Lesson Code

NEON-download-explore.R

Use the neonUtilities Package to Access NEON Data

Authors: Claire K. Lunch, Megan A. Jones

Last Updated: Apr 8, 2021

This tutorial goes over how to use the neonUtilities R package (formerly the neonDataStackR package).

The package contains several functions:

stackByTable(): Takes zip files downloaded from the Data Portal or downloaded by zipsByProduct(), unzips them, and joins the monthly files by data table to create a single file per table.
zipsByProduct(): A wrapper for the NEON API; downloads data based on data product and site criteria. Stores downloaded data in a format that can then be joined by stackByTable().
loadByProduct(): Combines the functionality of zipsByProduct() and stackByTable(): Downloads the specified data, stacks the files, and loads the files to the R environment.
getPackage(): A wrapper for the NEON API; downloads one site-by-month zip file at a time.
byFileAOP(): A wrapper for the NEON API; downloads remote sensing data based on data product, site, and year criteria. Preserves the file structure of the original data.
byTileAOP(): Downloads remote sensing data for the specified data product, subset to tiles that intersect a list of coordinates.
readTableNEON(): Reads NEON data tables into R, using the variables file to assign R classes to each column.
transformFileToGeoCSV(): Converts any NEON data file in csv format into a new file with GeoCSV headers.

If you are only interested in joining data files downloaded from the NEON Data Portal, you will only need to use stackByTable(). Follow the instructions in the first section of the Download and Explore tutorial.

neonUtilities package

This package is intended to provide a toolbox of basic functionality for working with NEON data. It currently contains the functions listed above, but it is under development and more will be added in the future. To report bugs or request new features, post an issue in the GitHub repo issues page.

First, we must install and load the neonUtilities package.

# install neonUtilities - can skip if already installed
install.packages("neonUtilities")
# load neonUtilities
library(neonUtilities)

Download files and load directly to R: loadByProduct()

The most popular function in neonUtilities is loadByProduct(). This function downloads data from the NEON API, merges the site-by-month files, and loads the resulting data tables into the R environment, assigning each data type to the appropriate R class. It combines the actions of the zipsByProduct(), stackByTable(), and readTableNEON() functions, described below.

This is a popular choice because it ensures you're always working with the latest data, and it ends with ready-to-use tables in R. However, if you use it in a workflow you run repeatedly, keep in mind it will re-download the data every time.

The inputs to loadByProduct() control which data to download and how to manage the processing:

dpID: the data product ID, e.g. DP1.00002.001
site: defaults to "all", meaning all sites with available data; can be a vector of 4-letter NEON site codes, e.g. c("HARV","CPER","ABBY").
startdate and enddate: defaults to NA, meaning all dates with available data; or a date in the form YYYY-MM, e.g. 2017-06. Since NEON data are provided in month packages, finer scale querying is not available. Both start and end date are inclusive.
package: either basic or expanded data package. Expanded data packages generally include additional information about data quality, such as chemical standards and quality flags. Not every data product has an expanded package; if the expanded package is requested but there isn't one, the basic package will be downloaded.
avg: defaults to "all", to download all data; or the number of minutes in the averaging interval. See example below; only applicable to IS data.
savepath: the file path you want to download to; defaults to the working directory.
check.size: T or F: should the function pause before downloading data and warn you about the size of your download? Defaults to T; if you are using this function within a script or batch process you will want to set it to F.
nCores: Number of cores to use for parallel processing. Defaults to 1, i.e. no parallelization.

The dpID is the data product identifier of the data you want to download. The DPID can be found on the Explore Data Products page. It will be in the form DP#.#####.###

Let's get triple-aspirated air temperature data (DP1.00003.001) from Moab and Onaqui (MOAB and ONAQ), from May--August 2018, and name the data object trip.temp:

trip.temp <- loadByProduct(dpID="DP1.00003.001", 
                           site=c("MOAB","ONAQ"),
                           startdate="2018-05", 
                           enddate="2018-08")


Continuing will download files totaling approximately 7.994569 MB. Do you want to proceed y/n: y
Downloading 8 files
  |========================================================================================================| 100%

Unpacking zip files using 1 cores.
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s  
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s  
Stacking operation across a single core.
Stacking table TAAT_1min
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s  
Stacking table TAAT_30min
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s  
Merged the most recent publication of sensor position files for each site and saved to /stackedFiles
Copied the most recent publication of variable definition file to /stackedFiles
Finished: Stacked 2 data tables and 2 metadata tables!
Stacking took 0.8517601 secs

The object returned by loadByProduct() is a named list of data frames. To work with each of them, select them from the list using the $ operator.

names(trip.temp)
View(trip.temp$TAAT_30min)

If you prefer to extract each table from the list and work with it as an independent object, you can use the list2env() function:

list2env(trip.temp, .GlobalEnv)

For more details about the contents of the data tables and metadata tables, check out the Download and Explore tutorial.

write.csv(TAAT_30min, 
          "~/Downloads/TAAT_30min.csv", 
          row.names=F)
write.csv(variables_00003, 
          "~/Downloads/variables_00003.csv", 
          row.names=F)

Join data files: stackByTable()

The function stackByTable() joins the month-by-site files from a data download. The output will yield data grouped into new files by table name. For example, the single aspirated air temperature data product contains 1 minute and 30 minute interval data. The output from this function is one .csv with 1 minute data and one .csv with 30 minute data.

Depending on your file size this function may run for a while. For example, in testing for this tutorial, 124 MB of temperature data took about 4 minutes to stack. A progress bar will display while the stacking is in progress.

Download the Data

To stack data from the Portal, first download the data of interest from the NEON Data Portal. To stack data downloaded from the API, see the zipsByProduct() section below.

Your data will download from the Portal in a single zipped file.

The stacking function will only work on zipped Comma Separated Value (.csv) files and not the NEON data stored in other formats (HDF5, etc).

Run `stackByTable()`

The example data below are single-aspirated air temperature.

To run the stackByTable() function, input the file path to the downloaded and zipped file.

# stack files - Mac OSX file path shown
stackByTable(filepath="~neon/data/NEON_temp-air-single.zip")


Unpacking zip files
  |=========================================================================================| 100%
Stacking table SAAT_1min
  |=========================================================================================| 100%
Stacking table SAAT_30min
  |=========================================================================================| 100%
Finished: All of the data are stacked into  2  tables!
Copied the first available variable definition file to /stackedFiles and renamed as variables.csv
Stacked SAAT_1min which has 424800 out of the expected 424800 rows (100%).
Stacked SAAT_30min which has 14160 out of the expected 14160 rows (100%).
Stacking took 6.233922 secs
All unzipped monthly data folders have been removed.

From the single-aspirated air temperature data we are given two final tables. One with 1 minute intervals: SAAT_1min and one for 30 minute intervals: SAAT_30min.

In the same directory as the zipped file, you should now have an unzipped directory of the same name. When you open this you will see a new directory called stackedFiles. This directory contains one or more .csv files (depends on the data product you are working with) with all the data from the months & sites you downloaded. There will also be a single copy of the associated variables, validation, and sensor_positions files, if applicable (validation files are only available for observational data products, and sensor position files are only available for instrument data products).

These .csv files are now ready for use with the program of your choice.

To read the data tables into R, we recommend using readTableNEON(), which will assign each column to the relevant R class, based on the metadata in the variables file. This ensures time stamps and missing data are interpreted correctly in R.

SAAT30 <- readTableNEON(
  dataFile='~/stackedFiles/SAAT_30min.csv',
  varFile='~/stackedFiles/variables_00002.csv'
)

Other options

Other input options in stackByTable() are:

savepath : allows you to specify the file path where you want the stacked files to go, overriding the default.
saveUnzippedFiles : allows you to keep the unzipped, unstacked files from an intermediate stage of the process; by default they are discarded.

Example usage:

stackByTable(filepath="~neon/data/NEON_temp-air-single.zip", 
             savepath="~data/allTemperature", saveUnzippedFiles=T)

Download files to be stacked: zipsByProduct()

The function zipsByProduct() is a wrapper for the NEON API, it downloads zip files for the data product specified and stores them in a format that can then be passed on to stackByTable().

Input options for zipsByProduct() are the same as those for loadByProduct() described above, except for nCores.

Here, we'll download single-aspirated air temperature (DP1.00002.001) data from Wind River Experimental Forest (WREF) for April and May of 2019.

zipsByProduct(dpID="DP1.00002.001", site="WREF", 
              startdate="2019-04", enddate="2019-05",
              package="basic", check.size=T)


Continuing will download files totaling approximately 12.750557 MB. Do you want to proceed y/n: y
Downloading 2 files
  |========================================================================================================| 100%
2 files downloaded to /Users/neon/filesToStack00002

Downloaded files can now be passed to stackByTable() to be stacked.

stackByTable(filepath="/Users/neon/filesToStack00002")

For many sensor data products, download sizes can get very large, and stackByTable() takes a long time. The 1-minute or 2-minute files are much larger than the longer averaging intervals, so if you don't need high- frequency data, the avg input option lets you choose which averaging interval to download.

This option is only applicable to sensor (IS) data, since OS data are not averaged.

Download only the 30-minute data for single-aspirated air temperature at WREF:

zipsByProduct(dpID="DP1.00002.001", site="WREF", 
              startdate="2019-04", enddate="2019-05",
              package="basic", avg=30, check.size=T)


Continuing will download files totaling approximately 2.101936 MB. Do you want to proceed y/n: y
Downloading 17 files
  |========================================================================================================| 100%
17 files downloaded to /Users/neon/filesToStack00002

The 30-minute files can be stacked as usual, and can be read into R using readTableNEON():

stackByTable(filepath="/Users/neon/filesToStack00002")
SAAT30 <- readTableNEON(
  dataFile='/Users/neon/filesToStack00002/stackedFiles/SAAT_30min.csv',
  varFile='/Users/neon/filesToStack00002/stackedFiles/variables_00002.csv'
)

Download a single zip file: getPackage()

If you only need a single site-month (e.g., to test code you're writing), the getPackage() function can be used to download a single zip file. Here we'll download the November 2017 temperature data from HARV.

getPackage("DP1.00002.001", site_code="HARV", 
           year_month="2017-11", package="basic")

The file should now be saved to your working directory.

Download remote sensing files: byFileAOP()

Remote sensing data files can be very large, and NEON remote sensing (AOP) data are stored in a directory structure that makes them easier to navigate. byFileAOP() downloads AOP files from the API while preserving their directory structure. This provides a convenient way to access AOP data programmatically.

Be aware that downloads from byFileAOP() can take a VERY long time, depending on the data you request and your connection speed. You may need to run the function and then leave your machine on and downloading for an extended period of time.

Here the example download is the Ecosystem Structure data product at Hop Brook (HOPB) in 2017; we use this as the example because it's a relatively small year-site-product combination.

byFileAOP("DP3.30015.001", site="HOPB", 
          year="2017", check.size=T)


Continuing will download 36 files totaling approximately 140.3 MB . Do you want to proceed y/n: y
trying URL 'https://neon-aop-product.s3.data.neonscience.org:443/2017/FullSite/D01/2017_HOPB_2/L3/DiscreteLidar/CanopyHeightModelGtif/NEON_D01_HOPB_DP3_716000_4704000_CHM.tif?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180410T233031Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=pub-internal-read%2F20180410%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=92833ebd10218f4e2440cb5ea78d1c8beac4ee4c10be5c6aeefb72d18cf6bd78'
Content type 'application/octet-stream' length 4009489 bytes (3.8 MB)
==================================================
downloaded 3.8 MB

(Further URLs omitted for space. Function returns a message 
  for each URL it attempts to download from)

Successfully downloaded  36  files.
NEON_D01_HOPB_DP3_716000_4704000_CHM.tif downloaded to /Users/neon/DP3.30015.001/2017/FullSite/D01/2017_HOPB_2/L3/DiscreteLidar/CanopyHeightModelGtif
NEON_D01_HOPB_DP3_716000_4705000_CHM.tif downloaded to /Users/neon/DP3.30015.001/2017/FullSite/D01/2017_HOPB_2/L3/DiscreteLidar/CanopyHeightModelGtif

(Further messages omitted for space.)

The files should now be downloaded to a new folder in your working directory.

Download remote sensing files for specific coordinates: byTileAOP()

Often when using remote sensing data, we only want data covering a certain area - usually the area where we have coordinated ground sampling. byTileAOP() queries for data tiles containing a specified list of coordinates. It only works for the tiled, AKA mosaicked, versions of the remote sensing data, i.e. the ones with data product IDs beginning with "DP3".

Here, we'll download tiles of vegetation indices data (DP3.30026.001) corresponding to select observational sampling plots. For more information about accessing NEON spatial data, see the API tutorial and the in-development geoNEON package.

For now, assume we've used the API to look up the plot centroids of plots SOAP_009 and SOAP_011 at the Soaproot Saddle site. You can also look these up in the Spatial Data folder of the document library. The coordinates of the two plots in UTMs are 298755,4101405 and 299296,4101461. These are 40x40m plots, so in looking for tiles that contain the plots, we want to include a 20m buffer. The "buffer" is actually a square, it's a delta applied equally to both the easting and northing coordinates.

byTileAOP(dpID="DP3.30026.001", site="SOAP", 
          year="2018", easting=c(298755,299296),
          northing=c(4101405,4101461),
          buffer=20)


trying URL 'https://neon-aop-product.s3.data.neonscience.org:443/2018/FullSite/D17/2018_SOAP_3/L3/Spectrometer/VegIndices/NEON_D17_SOAP_DP3_299000_4101000_VegIndices.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20190313T225238Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20190313%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=e9ae6858242b48df0677457e31ea3d86b2f20ac2cf43d5fc02847bbaf2e1da47'
Content type 'application/octet-stream' length 47798759 bytes (45.6 MB)
==================================================
downloaded 45.6 MB

(Further URLs omitted for space. Function returns a message 
  for each URL it attempts to download from)

Successfully downloaded  2  files.
NEON_D17_SOAP_DP3_298000_4101000_VegIndices.zip downloaded to /Users/neon/DP3.30026.001/2018/FullSite/D17/2018_SOAP_3/L3/Spectrometer/VegIndices
NEON_D17_SOAP_DP3_299000_4101000_VegIndices.zip downloaded to /Users/neon/DP3.30026.001/2018/FullSite/D17/2018_SOAP_3/L3/Spectrometer/VegIndices

The 2 tiles covering the SOAP_009 and SOAP_011 plots have
been downloaded.

Convert files to GeoCSV: transformFileToGeoCSV()

transformFileToGeoCSV() takes a NEON csv file, plus its corresponding variables file, and writes out a new version of the file with GeoCSV headers. This allows for compatibility with data provided by UNAVCO and other facilities.

Inputs to transformFileToGeoCSV() are the file path to the data file, the file path to the variables file, and the file path where you want to write out the new version. It works on both single site-month files and on stacked files.

In this example, we'll convert the November 2017 temperature data
from HARV that we downloaded with getPackage() earlier. First, you'll need to unzip the file so you can get to the data files. Then we'll select the file for the tower top, which we can identify by the 050 in the VER field (see the file naming conventions page for more information).

transformFileToGeoCSV("~/NEON.D01.HARV.DP1.00002.001.000.050.030.SAAT_30min.2017-11.basic.20171207T181046Z.csv", 
                      "~/NEON.D01.HARV.DP1.00002.001.variables.20171207T181046Z.csv",
                      "~/SAAT_30min_geo.csv")

Get Lesson Code

neonUtilities.R

Using the NEON API in R

Authors: [Claire K. Lunch]

Last Updated: Mar 9, 2021

This is a tutorial in pulling data from the NEON API or Application Programming Interface. The tutorial uses R and the R package httr, but the core information about the API is applicable to other languages and approaches.

NEON data

There are 3 basic categories of NEON data:

Observational - Data collected by a human in the field, or in an analytical laboratory, e.g. beetle identification, foliar isotopes
Instrumentation - Data collected by an automated, streaming sensor, e.g. net radiation, soil carbon dioxide
Remote sensing - Data collected by the airborne observation platform, e.g. LIDAR, surface reflectance

This lesson covers all three types of data, but we recommend proceeding through the lesson in order and not skipping ahead, since the query principles are explained in the first section, on observational data.

Objectives

After completing this activity, you will be able to:

Pull observational, instrumentation, and geolocation data from the NEON API.
Transform API-accessed data from JSON to tabular format for analyses.

Things You’ll Need To Complete This Tutorial

To complete this tutorial you will need the most current version of R and, preferably, RStudio loaded on your computer.

Install R Packages

httr: install.packages("httr")
jsonlite: install.packages("jsonlite")
dplyr: install.packages("dplyr")
devtools: install.packages("devtools")
downloader: install.packages("downloader")
neonUtilities: install.packages("neonUtilities")
geoNEON: devtools::install_github("NEONScience/NEON-geolocation/geoNEON")

Note, you must have devtools installed & loaded, prior to loading geoNEON.

Additional Resources

What is an API?

If you are unfamiliar with the concept of an API, think of an API as a ‘middle person' that provides a communication path for a software application to obtain information from a digital data source. APIs are becoming a very common means of sharing digital information. Many of the apps that you use on your computer or mobile device to produce maps, charts, reports, and other useful forms of information pull data from multiple sources using APIs. In the ecological and environmental sciences, many researchers use APIs to programmatically pull data into their analyses. (Quoted from the NEON Observatory Blog story: API and data availability viewer now live on the NEON data portal.)

Anatomy of an API call

An example API call: http://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2015-07

This includes the base URL, endpoint, and target.

Base URL:

http://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2015-07

Specifics are appended to this in order to get the data or metadata you're looking for, but all calls to this API will include the base URL. For the NEON API, this is http://data.neonscience.org/api/v0 -- not clickable, because the base URL by itself will take you nowhere!

Endpoints:

http://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2015-07

What type of data or metadata are you looking for?

~/products Information about one or all of NEON's data products
~/sites Information about data availability at the site specified in the call
~/locations Spatial data for the NEON locations specified in the call
~/data Data! By product, site, and date (in monthly chunks).

Targets:

http://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2015-07

The specific data product, site, or location you want to get data for.

Observational data (OS)

Which product do you want to get data for? Consult the Explore Data Products page.

We'll pick Breeding landbird point counts, DP1.10003.001

First query the products endpoint of the API to find out which sites and dates have data available. In the products endpoint, the target is the numbered identifier for the data product:

# Load the necessary libraries
library(httr)
library(jsonlite)
library(dplyr, quietly=T)
library(downloader)

# Request data using the GET function & the API call
req <- GET("http://data.neonscience.org/api/v0/products/DP1.10003.001")
req

## Response [https://data.neonscience.org/api/v0/products/DP1.10003.001]
##   Date: 2021-03-09 23:50
##   Status: 200
##   Content-Type: application/json;charset=UTF-8
##   Size: 37.4 kB

The object returned from GET() has many layers of information. Entering the name of the object gives you some basic information about what you downloaded.

The content() function returns the contents in the form of a highly nested list. This is typical of JSON-formatted data returned by APIs. We can use the names() function to view the different types of information within this list.

# View requested data
req.content <- content(req, as="parsed")
names(req.content$data)

##  [1] "productCodeLong"              "productCode"                  "productCodePresentation"     
##  [4] "productName"                  "productDescription"           "productStatus"               
##  [7] "productCategory"              "productHasExpanded"           "productScienceTeamAbbr"      
## [10] "productScienceTeam"           "productPublicationFormatType" "productAbstract"             
## [13] "productDesignDescription"     "productStudyDescription"      "productBasicDescription"     
## [16] "productExpandedDescription"   "productSensor"                "productRemarks"              
## [19] "themes"                       "changeLogs"                   "specs"                       
## [22] "keywords"                     "releases"                     "siteCodes"

You can see all of the infoamtion by running the line print(req.content), but this will result in a very long printout in your console. Instead, you can view list items individually. Here, we highlight a couple of interesting examples:

# View Abstract
req.content$data$productAbstract

## [1] "This data product contains the quality-controlled, native sampling resolution data from NEON's breeding landbird sampling. Breeding landbirds are defined as “smaller birds (usually exclusive of raptors and upland game birds) not usually associated with aquatic habitats” (Ralph et al. 1993). The breeding landbird point counts product provides records of species identification of all individuals observed during the 6-minute count period, as well as metadata which can be used to model detectability, e.g., weather, distances from observers to birds, and detection methods. The NEON point count method is adapted from the Integrated Monitoring in Bird Conservation Regions (IMBCR): Field protocol for spatially-balanced sampling of landbird populations (Hanni et al. 2017; http://bit.ly/2u2ChUB). For additional details, see protocol [NEON.DOC.014041](http://data.neonscience.org/api/v0/documents/NEON.DOC.014041vF): TOS Protocol and Procedure: Breeding Landbird Abundance and Diversity and science design [NEON.DOC.000916](http://data.neonscience.org/api/v0/documents/NEON.DOC.000916vB): TOS Science Design for Breeding Landbird Abundance and Diversity.\n\nLatency: The expected time from data and/or sample collection in the field to data publication is as follows, for each of the data tables (in days) in the downloaded data package. See the Data Product User Guide for more information.\n \nbrd_countdata: 120\n\nbrd_perpoint: 120\n\nbrd_personnel: 120\n\nbrd_references: 120"

# View Available months and associated URLs for Onaqui, Utah - ONAQ
req.content$data$siteCodes[[27]]

## $siteCode
## [1] "ONAQ"
## 
## $availableMonths
## $availableMonths[[1]]
## [1] "2017-05"
## 
## $availableMonths[[2]]
## [1] "2018-05"
## 
## $availableMonths[[3]]
## [1] "2018-06"
## 
## $availableMonths[[4]]
## [1] "2019-05"
## 
## $availableMonths[[5]]
## [1] "2020-05"
## 
## 
## $availableDataUrls
## $availableDataUrls[[1]]
## [1] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2017-05"
## 
## $availableDataUrls[[2]]
## [1] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2018-05"
## 
## $availableDataUrls[[3]]
## [1] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2018-06"
## 
## $availableDataUrls[[4]]
## [1] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2019-05"
## 
## $availableDataUrls[[5]]
## [1] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2020-05"
## 
## 
## $availableReleases
## $availableReleases[[1]]
## $availableReleases[[1]]$release
## [1] "PROVISIONAL"
## 
## $availableReleases[[1]]$availableMonths
## $availableReleases[[1]]$availableMonths[[1]]
## [1] "2020-05"
## 
## 
## 
## $availableReleases[[2]]
## $availableReleases[[2]]$release
## [1] "RELEASE-2021"
## 
## $availableReleases[[2]]$availableMonths
## $availableReleases[[2]]$availableMonths[[1]]
## [1] "2017-05"
## 
## $availableReleases[[2]]$availableMonths[[2]]
## [1] "2018-05"
## 
## $availableReleases[[2]]$availableMonths[[3]]
## [1] "2018-06"
## 
## $availableReleases[[2]]$availableMonths[[4]]
## [1] "2019-05"

To get a more accessible view of which sites have data for which months, you'll need to extract data from the nested list. There are a variety of ways to do this, in this tutorial we'll explore a couple of them. Here we'll use fromJSON(), in the jsonlite package, which doesn't fully flatten the nested list, but gets us the part we need. To use it, we need a text version of the content. The text version is not as human readable but is readable by the fromJSON() function.

# make this JSON readable -> "text"
req.text <- content(req, as="text")

# Flatten data frame to see available data. 
avail <- jsonlite::fromJSON(req.text, simplifyDataFrame=T, flatten=T)
avail

## $data
## $data$productCodeLong
## [1] "NEON.DOM.SITE.DP1.10003.001"
## 
## $data$productCode
## [1] "DP1.10003.001"
## 
## $data$productCodePresentation
## [1] "NEON.DP1.10003"
## 
## $data$productName
## [1] "Breeding landbird point counts"
## 
## $data$productDescription
## [1] "Count, distance from observer, and taxonomic identification of breeding landbirds observed during point counts"
## 
## $data$productStatus
## [1] "ACTIVE"
## 
## $data$productCategory
## [1] "Level 1 Data Product"
## 
## $data$productHasExpanded
## [1] TRUE
## 
## $data$productScienceTeamAbbr
## [1] "TOS"
## 
## $data$productScienceTeam
## [1] "Terrestrial Observation System (TOS)"
## 
## $data$productPublicationFormatType
## [1] "TOS Data Product Type"
## 
## $data$productAbstract
## [1] "This data product contains the quality-controlled, native sampling resolution data from NEON's breeding landbird sampling. Breeding landbirds are defined as “smaller birds (usually exclusive of raptors and upland game birds) not usually associated with aquatic habitats” (Ralph et al. 1993). The breeding landbird point counts product provides records of species identification of all individuals observed during the 6-minute count period, as well as metadata which can be used to model detectability, e.g., weather, distances from observers to birds, and detection methods. The NEON point count method is adapted from the Integrated Monitoring in Bird Conservation Regions (IMBCR): Field protocol for spatially-balanced sampling of landbird populations (Hanni et al. 2017; http://bit.ly/2u2ChUB). For additional details, see protocol [NEON.DOC.014041](http://data.neonscience.org/api/v0/documents/NEON.DOC.014041vF): TOS Protocol and Procedure: Breeding Landbird Abundance and Diversity and science design [NEON.DOC.000916](http://data.neonscience.org/api/v0/documents/NEON.DOC.000916vB): TOS Science Design for Breeding Landbird Abundance and Diversity.\n\nLatency: The expected time from data and/or sample collection in the field to data publication is as follows, for each of the data tables (in days) in the downloaded data package. See the Data Product User Guide for more information.\n \nbrd_countdata: 120\n\nbrd_perpoint: 120\n\nbrd_personnel: 120\n\nbrd_references: 120"
## 
## $data$productDesignDescription
## [1] "Depending on the size of the site, sampling for this product occurs at either randomly distributed individual points or grids of nine points each. At larger sites, point count sampling occurs at five to ten 9-point grids, with grid centers collocated with distributed base plot centers (where plant, beetle, and/or soil sampling may also occur), if possible. At smaller sites (i.e., sites that cannot accommodate a minimum of 5 grids) point counts occur at the southwest corner (point 21) of 5-25 distributed base plots. Point counts are conducted once per breeding season at large sites and twice per breeding season at smaller sites. Point counts are six minutes long, with each minute tracked by the observer, following a two-minute settling-in period. All birds are recorded to species and sex, whenever possible, and the distance to each individual or flock is measured with a laser rangefinder, except in the case of flyovers."
## 
## $data$productStudyDescription
## [1] "This sampling occurs at all NEON terrestrial sites."
## 
## $data$productBasicDescription
## [1] "The basic package contains the per point metadata table that includes data pertaining to the observer and the weather conditions and the count data table that includes all of the observational data."
## 
## $data$productExpandedDescription
## [1] "The expanded package includes two additional tables and two additional fields within the count data table. The personnel table provides institutional information about each observer, as well as their performance on identification quizzes, where available. The references tables provides the list of resources used by an observer to identify birds. The additional fields in the countdata table are family and nativeStatusCode, which are derived from the NEON master list of birds."
## 
## $data$productSensor
## NULL
## 
## $data$productRemarks
## [1] "Queries for this data product will return data collected during the date range specified for `brd_perpoint` and `brd_countdata`, but will return data from all dates for `brd_personnel` (quiz scores may occur over time periods which are distinct from when sampling occurs) and `brd_references` (which apply to a broad range of sampling dates). A record from `brd_perPoint` should have 6+ child records in `brd_countdata`, at least one per pointCountMinute. Duplicates or missing data may exist where protocol and/or data entry aberrations have occurred; users should check data carefully for anomalies before joining tables. Taxonomic IDs of species of concern have been 'fuzzed'; see data package readme files for more information."
## 
## $data$themes
## [1] "Organisms, Populations, and Communities"
## 
## $data$changeLogs
##      id parentIssueID            issueDate         resolvedDate       dateRangeStart         dateRangeEnd
## 1 16607            NA 2020-10-28T00:00:00Z 2020-01-01T00:00:00Z 2013-01-01T00:00:00Z 2020-01-01T00:00:00Z
## 2 17938            NA 2021-01-06T00:00:00Z                 <NA> 2020-03-23T00:00:00Z 2021-06-01T00:00:00Z
##   locationAffected
## 1              All
## 2              All
##                                                                                                                                                                                                            issue
## 1                                                                                                                                 There was not a way to indicate that a scheduled sampling event did not occur.
## 2 Safety measures to protect personnel during the COVID-19 pandemic resulted in reduced or eliminated sampling activities for extended periods at NEON sites. Data availability may be reduced during this time.
##                                                                                                                                                                                                                                                                                                                                   resolution
## 1 The fields samplingImpracticalRemarks and samplingImpractical were added prior to the 2020 field season. The contractor supplies the samplingImpracticalRemarks field, and this field autopopulates the samplingImpractical field. The samplingImpractical field has a value other than OK if something prevented sampling from occurring.
## 2                                                                                                                                                                                                                                                                                                                                       <NA>
## 
## $data$specs
##   specId             specNumber        specType specSize
## 1   3656      NEON.DOC.000916vC application/pdf  2827241
## 2   5183 NEON_bird_userGuide_vB application/pdf   295419
## 3   3729      NEON.DOC.014041vJ application/pdf  5026183
##                                                         specDescription
## 1      TOS Science Design for Breeding Landbird Abundance and Diversity
## 2     NEON USER GUIDE TO BREEDING LANDBIRD POINT COUNTS (DP1.10003.001)
## 3 TOS Protocol and Procedure: Breeding Landbird Abundance and Diversity
## 
## $data$keywords
##  [1] "vertebrates"           "birds"                 "diversity"             "taxonomy"             
##  [5] "community composition" "distance sampling"     "avian"                 "species composition"  
##  [9] "population"            "Aves"                  "Chordata"              "point counts"         
## [13] "landbirds"             "invasive"              "introduced"            "native"               
## [17] "animals"               "Animalia"             
## 
## $data$releases
##        release       generationDate                                                       url
## 1 RELEASE-2021 2021-01-23T02:30:02Z https://data.neonscience.org/api/v0/releases/RELEASE-2021
##   productDoi.generationDate                     productDoi.url
## 1      2021-01-25T18:14:30Z https://doi.org/10.48443/s730-dy13
## 
## $data$siteCodes
##    siteCode
## 1      ABBY
## 2      BARR
## 3      BART
## 4      BLAN
## 5      BONA
## 6      CLBJ
## 7      CPER
## 8      DCFS
## 9      DEJU
## 10     DELA
## 11     DSNY
## 12     GRSM
## 13     GUAN
## 14     HARV
## 15     HEAL
## 16     JERC
## 17     JORN
## 18     KONA
## 19     KONZ
## 20     LAJA
## 21     LENO
## 22     MLBS
## 23     MOAB
## 24     NIWO
## 25     NOGP
## 26     OAES
## 27     ONAQ
## 28     ORNL
## 29     OSBS
## 30     PUUM
## 31     RMNP
## 32     SCBI
## 33     SERC
## 34     SJER
## 35     SOAP
## 36     SRER
## 37     STEI
## 38     STER
## 39     TALL
## 40     TEAK
## 41     TOOL
## 42     TREE
## 43     UKFS
## 44     UNDE
## 45     WOOD
## 46     WREF
## 47     YELL
##                                                                                      availableMonths
## 1                                                        2017-05, 2017-06, 2018-06, 2018-07, 2019-05
## 2                                                                          2017-07, 2018-07, 2019-06
## 3                                      2015-06, 2016-06, 2017-06, 2018-06, 2019-06, 2020-06, 2020-07
## 4                                      2017-05, 2017-06, 2018-05, 2018-06, 2019-05, 2019-06, 2020-06
## 5                                               2017-06, 2018-06, 2018-07, 2019-06, 2020-06, 2020-07
## 6                                               2017-05, 2018-04, 2019-04, 2019-05, 2020-04, 2020-05
## 7                             2013-06, 2015-05, 2016-05, 2017-05, 2017-06, 2018-05, 2019-06, 2020-05
## 8                                               2017-06, 2017-07, 2018-07, 2019-06, 2019-07, 2020-07
## 9                                                                 2017-06, 2018-06, 2019-06, 2020-06
## 10                                                       2015-06, 2017-06, 2018-05, 2019-06, 2020-05
## 11                                                       2015-06, 2016-05, 2017-05, 2018-05, 2019-05
## 12                                              2016-06, 2017-05, 2017-06, 2018-05, 2019-05, 2020-06
## 13                                              2015-05, 2017-05, 2018-05, 2019-05, 2019-06, 2020-07
## 14                                     2015-05, 2015-06, 2016-06, 2017-06, 2018-06, 2019-06, 2020-06
## 15                                              2017-06, 2018-06, 2018-07, 2019-06, 2019-07, 2020-06
## 16                                                       2016-06, 2017-05, 2018-06, 2019-06, 2020-05
## 17                                              2017-04, 2017-05, 2018-04, 2018-05, 2019-04, 2020-05
## 18                                                       2018-05, 2018-06, 2019-06, 2020-05, 2020-06
## 19                                                       2017-06, 2018-05, 2018-06, 2019-06, 2020-05
## 20                                                       2017-05, 2018-05, 2019-05, 2019-06, 2020-07
## 21                                                                2017-06, 2018-05, 2019-06, 2020-05
## 22                                                                         2018-06, 2019-05, 2020-05
## 23                                              2015-06, 2017-05, 2018-05, 2019-05, 2020-05, 2020-06
## 24                                                       2015-07, 2017-07, 2018-07, 2019-07, 2020-07
## 25                                                                2017-07, 2018-07, 2019-07, 2020-07
## 26                                              2017-05, 2017-06, 2018-04, 2018-05, 2019-05, 2020-05
## 27                                                       2017-05, 2018-05, 2018-06, 2019-05, 2020-05
## 28                                              2016-05, 2016-06, 2017-05, 2018-06, 2019-05, 2020-05
## 29                                                       2016-05, 2017-05, 2018-05, 2019-05, 2020-06
## 30                                                                                           2018-04
## 31                            2017-06, 2017-07, 2018-06, 2018-07, 2019-06, 2019-07, 2020-06, 2020-07
## 32 2015-06, 2016-05, 2016-06, 2017-05, 2017-06, 2018-05, 2018-06, 2019-05, 2019-06, 2020-05, 2020-06
## 33                                              2017-05, 2017-06, 2018-05, 2019-05, 2020-05, 2020-06
## 34                                                                         2017-04, 2018-04, 2019-04
## 35                                                                         2017-05, 2018-05, 2019-05
## 36                                                       2017-05, 2018-04, 2018-05, 2019-04, 2020-04
## 37                            2016-05, 2016-06, 2017-06, 2018-05, 2018-06, 2019-05, 2019-06, 2020-06
## 38                            2013-06, 2015-05, 2016-05, 2017-05, 2018-05, 2019-05, 2019-06, 2020-06
## 39                                     2015-06, 2016-07, 2017-06, 2018-06, 2019-05, 2020-05, 2020-06
## 40                                                                2017-06, 2018-06, 2019-06, 2019-07
## 41                                                                         2017-06, 2018-07, 2019-06
## 42                                                       2016-06, 2017-06, 2018-06, 2019-06, 2020-06
## 43                                                       2017-06, 2018-06, 2019-06, 2020-05, 2020-06
## 44                                              2016-06, 2016-07, 2017-06, 2018-06, 2019-06, 2020-06
## 45                                              2015-07, 2017-07, 2018-07, 2019-06, 2019-07, 2020-07
## 46                                                                         2018-06, 2019-05, 2019-06
## 47                                                                         2018-06, 2019-06, 2020-06
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        availableDataUrls
## 1                                                                                                                                                                                                                                                                                                                                                                                                                                https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2019-05
## 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/BARR/2017-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/BARR/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/BARR/2019-06
## 3                                                                                                                                                                                                                                                                                      https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2020-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2020-07
## 4                                                                                                                                                                                                                                                                                      https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BLAN/2020-06
## 5                                                                                                                                                                                                                                                                                                                                                           https://data.neonscience.org/api/v0/data/DP1.10003.001/BONA/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BONA/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BONA/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/BONA/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BONA/2020-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/BONA/2020-07
## 6                                                                                                                                                                                                                                                                                                                                                           https://data.neonscience.org/api/v0/data/DP1.10003.001/CLBJ/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/CLBJ/2018-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/CLBJ/2019-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/CLBJ/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/CLBJ/2020-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/CLBJ/2020-05
## 7                                                                                                                                                                                                                 https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2013-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2015-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/CPER/2020-05
## 8                                                                                                                                                                                                                                                                                                                                                           https://data.neonscience.org/api/v0/data/DP1.10003.001/DCFS/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DCFS/2017-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/DCFS/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/DCFS/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DCFS/2019-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/DCFS/2020-07
## 9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     https://data.neonscience.org/api/v0/data/DP1.10003.001/DEJU/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DEJU/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DEJU/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DEJU/2020-06
## 10                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/DELA/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DELA/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DELA/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/DELA/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DELA/2020-05
## 11                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/DSNY/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/DSNY/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/DSNY/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/DSNY/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/DSNY/2019-05
## 12                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/GRSM/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/GRSM/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GRSM/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/GRSM/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GRSM/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GRSM/2020-06
## 13                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/GUAN/2015-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GUAN/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GUAN/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GUAN/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/GUAN/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/GUAN/2020-07
## 14                                                                                                                                                                                                                                                                                     https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2015-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HARV/2020-06
## 15                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/HEAL/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HEAL/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HEAL/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/HEAL/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/HEAL/2019-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/HEAL/2020-06
## 16                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/JERC/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/JERC/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/JERC/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/JERC/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/JERC/2020-05
## 17                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/JORN/2017-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/JORN/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/JORN/2018-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/JORN/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/JORN/2019-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/JORN/2020-05
## 18                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/KONA/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONA/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONA/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONA/2020-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONA/2020-06
## 19                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/KONZ/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONZ/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONZ/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONZ/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/KONZ/2020-05
## 20                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/LAJA/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/LAJA/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/LAJA/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/LAJA/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/LAJA/2020-07
## 21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    https://data.neonscience.org/api/v0/data/DP1.10003.001/LENO/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/LENO/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/LENO/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/LENO/2020-05
## 22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         https://data.neonscience.org/api/v0/data/DP1.10003.001/MLBS/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/MLBS/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/MLBS/2020-05
## 23                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/MOAB/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/MOAB/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/MOAB/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/MOAB/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/MOAB/2020-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/MOAB/2020-06
## 24                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/NIWO/2015-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NIWO/2017-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NIWO/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NIWO/2019-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NIWO/2020-07
## 25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    https://data.neonscience.org/api/v0/data/DP1.10003.001/NOGP/2017-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NOGP/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NOGP/2019-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/NOGP/2020-07
## 26                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/OAES/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OAES/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/OAES/2018-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/OAES/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OAES/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OAES/2020-05
## 27                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ONAQ/2020-05
## 28                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/ORNL/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ORNL/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/ORNL/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ORNL/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/ORNL/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/ORNL/2020-05
## 29                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/OSBS/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OSBS/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OSBS/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OSBS/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/OSBS/2020-06
## 30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   https://data.neonscience.org/api/v0/data/DP1.10003.001/PUUM/2018-04
## 31                                                                                                                                                                                                                https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2017-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2019-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2020-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/RMNP/2020-07
## 32 https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2020-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SCBI/2020-06
## 33                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/SERC/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SERC/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/SERC/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SERC/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SERC/2020-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SERC/2020-06
## 34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         https://data.neonscience.org/api/v0/data/DP1.10003.001/SJER/2017-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/SJER/2018-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/SJER/2019-04
## 35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         https://data.neonscience.org/api/v0/data/DP1.10003.001/SOAP/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SOAP/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SOAP/2019-05
## 36                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/SRER/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SRER/2018-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/SRER/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/SRER/2019-04, https://data.neonscience.org/api/v0/data/DP1.10003.001/SRER/2020-04
## 37                                                                                                                                                                                                                https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/STEI/2020-06
## 38                                                                                                                                                                                                                https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2013-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2015-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2016-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2017-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2018-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/STER/2020-06
## 39                                                                                                                                                                                                                                                                                     https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2015-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2016-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2020-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/TALL/2020-06
## 40                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    https://data.neonscience.org/api/v0/data/DP1.10003.001/TEAK/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TEAK/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TEAK/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TEAK/2019-07
## 41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         https://data.neonscience.org/api/v0/data/DP1.10003.001/TOOL/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TOOL/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/TOOL/2019-06
## 42                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/TREE/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TREE/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TREE/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TREE/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/TREE/2020-06
## 43                                                                                                                                                                                                                                                                                                                                                                                                                               https://data.neonscience.org/api/v0/data/DP1.10003.001/UKFS/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UKFS/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UKFS/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UKFS/2020-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/UKFS/2020-06
## 44                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/UNDE/2016-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UNDE/2016-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/UNDE/2017-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UNDE/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UNDE/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/UNDE/2020-06
## 45                                                                                                                                                                                                                                                                                                                                                          https://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2015-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2017-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2018-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2019-07, https://data.neonscience.org/api/v0/data/DP1.10003.001/WOOD/2020-07
## 46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         https://data.neonscience.org/api/v0/data/DP1.10003.001/WREF/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/WREF/2019-05, https://data.neonscience.org/api/v0/data/DP1.10003.001/WREF/2019-06
## 47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         https://data.neonscience.org/api/v0/data/DP1.10003.001/YELL/2018-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/YELL/2019-06, https://data.neonscience.org/api/v0/data/DP1.10003.001/YELL/2020-06
##                                                                                                               availableReleases
## 1                                                                     RELEASE-2021, 2017-05, 2017-06, 2018-06, 2018-07, 2019-05
## 2                                                                                       RELEASE-2021, 2017-07, 2018-07, 2019-06
## 3                                      PROVISIONAL, RELEASE-2021, 2020-06, 2020-07, 2015-06, 2016-06, 2017-06, 2018-06, 2019-06
## 4                                      PROVISIONAL, RELEASE-2021, 2020-06, 2017-05, 2017-06, 2018-05, 2018-06, 2019-05, 2019-06
## 5                                               PROVISIONAL, RELEASE-2021, 2020-06, 2020-07, 2017-06, 2018-06, 2018-07, 2019-06
## 6                                               PROVISIONAL, RELEASE-2021, 2020-04, 2020-05, 2017-05, 2018-04, 2019-04, 2019-05
## 7                             PROVISIONAL, RELEASE-2021, 2020-05, 2013-06, 2015-05, 2016-05, 2017-05, 2017-06, 2018-05, 2019-06
## 8                                               PROVISIONAL, RELEASE-2021, 2020-07, 2017-06, 2017-07, 2018-07, 2019-06, 2019-07
## 9                                                                 PROVISIONAL, RELEASE-2021, 2020-06, 2017-06, 2018-06, 2019-06
## 10                                                       PROVISIONAL, RELEASE-2021, 2020-05, 2015-06, 2017-06, 2018-05, 2019-06
## 11                                                                    RELEASE-2021, 2015-06, 2016-05, 2017-05, 2018-05, 2019-05
## 12                                              PROVISIONAL, RELEASE-2021, 2020-06, 2016-06, 2017-05, 2017-06, 2018-05, 2019-05
## 13                                              PROVISIONAL, RELEASE-2021, 2020-07, 2015-05, 2017-05, 2018-05, 2019-05, 2019-06
## 14                                     PROVISIONAL, RELEASE-2021, 2020-06, 2015-05, 2015-06, 2016-06, 2017-06, 2018-06, 2019-06
## 15                                              PROVISIONAL, RELEASE-2021, 2020-06, 2017-06, 2018-06, 2018-07, 2019-06, 2019-07
## 16                                                       PROVISIONAL, RELEASE-2021, 2020-05, 2016-06, 2017-05, 2018-06, 2019-06
## 17                                              PROVISIONAL, RELEASE-2021, 2020-05, 2017-04, 2017-05, 2018-04, 2018-05, 2019-04
## 18                                                       PROVISIONAL, RELEASE-2021, 2020-05, 2020-06, 2018-05, 2018-06, 2019-06
## 19                                                       PROVISIONAL, RELEASE-2021, 2020-05, 2017-06, 2018-05, 2018-06, 2019-06
## 20                                                       PROVISIONAL, RELEASE-2021, 2020-07, 2017-05, 2018-05, 2019-05, 2019-06
## 21                                                                PROVISIONAL, RELEASE-2021, 2020-05, 2017-06, 2018-05, 2019-06
## 22                                                                         PROVISIONAL, RELEASE-2021, 2020-05, 2018-06, 2019-05
## 23                                              PROVISIONAL, RELEASE-2021, 2020-05, 2020-06, 2015-06, 2017-05, 2018-05, 2019-05
## 24                                                       PROVISIONAL, RELEASE-2021, 2020-07, 2015-07, 2017-07, 2018-07, 2019-07
## 25                                                                PROVISIONAL, RELEASE-2021, 2020-07, 2017-07, 2018-07, 2019-07
## 26                                              PROVISIONAL, RELEASE-2021, 2020-05, 2017-05, 2017-06, 2018-04, 2018-05, 2019-05
## 27                                                       PROVISIONAL, RELEASE-2021, 2020-05, 2017-05, 2018-05, 2018-06, 2019-05
## 28                                              PROVISIONAL, RELEASE-2021, 2020-05, 2016-05, 2016-06, 2017-05, 2018-06, 2019-05
## 29                                                       PROVISIONAL, RELEASE-2021, 2020-06, 2016-05, 2017-05, 2018-05, 2019-05
## 30                                                                                                        RELEASE-2021, 2018-04
## 31                            PROVISIONAL, RELEASE-2021, 2020-06, 2020-07, 2017-06, 2017-07, 2018-06, 2018-07, 2019-06, 2019-07
## 32 PROVISIONAL, RELEASE-2021, 2020-05, 2020-06, 2015-06, 2016-05, 2016-06, 2017-05, 2017-06, 2018-05, 2018-06, 2019-05, 2019-06
## 33                                              PROVISIONAL, RELEASE-2021, 2020-05, 2020-06, 2017-05, 2017-06, 2018-05, 2019-05
## 34                                                                                      RELEASE-2021, 2017-04, 2018-04, 2019-04
## 35                                                                                      RELEASE-2021, 2017-05, 2018-05, 2019-05
## 36                                                       PROVISIONAL, RELEASE-2021, 2020-04, 2017-05, 2018-04, 2018-05, 2019-04
## 37                            PROVISIONAL, RELEASE-2021, 2020-06, 2016-05, 2016-06, 2017-06, 2018-05, 2018-06, 2019-05, 2019-06
## 38                            PROVISIONAL, RELEASE-2021, 2020-06, 2013-06, 2015-05, 2016-05, 2017-05, 2018-05, 2019-05, 2019-06
## 39                                     PROVISIONAL, RELEASE-2021, 2020-05, 2020-06, 2015-06, 2016-07, 2017-06, 2018-06, 2019-05
## 40                                                                             RELEASE-2021, 2017-06, 2018-06, 2019-06, 2019-07
## 41                                                                                      RELEASE-2021, 2017-06, 2018-07, 2019-06
## 42                                                       PROVISIONAL, RELEASE-2021, 2020-06, 2016-06, 2017-06, 2018-06, 2019-06
## 43                                                       PROVISIONAL, RELEASE-2021, 2020-05, 2020-06, 2017-06, 2018-06, 2019-06
## 44                                              PROVISIONAL, RELEASE-2021, 2020-06, 2016-06, 2016-07, 2017-06, 2018-06, 2019-06
## 45                                              PROVISIONAL, RELEASE-2021, 2020-07, 2015-07, 2017-07, 2018-07, 2019-06, 2019-07
## 46                                                                                      RELEASE-2021, 2018-06, 2019-05, 2019-06
## 47                                                                         PROVISIONAL, RELEASE-2021, 2020-06, 2018-06, 2019-06

The object contains a lot of information about the data product, including:

keywords under $data$keywords,
references for documentation under $data$specs,
data availability by site and month under $data$siteCodes, and
specific URLs for the API calls for each site and month under $data$siteCodes$availableDataUrls.

We need $data$siteCodes to tell us what we can download. $data$siteCodes$availableDataUrls allows us to avoid writing the API calls ourselves in the next steps.

# get data availability list for the product
bird.urls <- unlist(avail$data$siteCodes$availableDataUrls)
length(bird.urls) #total number of URLs

## [1] 252

bird.urls[1:10] #show first 10 URLs available

##  [1] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2017-05"
##  [2] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2017-06"
##  [3] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2018-06"
##  [4] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2018-07"
##  [5] "https://data.neonscience.org/api/v0/data/DP1.10003.001/ABBY/2019-05"
##  [6] "https://data.neonscience.org/api/v0/data/DP1.10003.001/BARR/2017-07"
##  [7] "https://data.neonscience.org/api/v0/data/DP1.10003.001/BARR/2018-07"
##  [8] "https://data.neonscience.org/api/v0/data/DP1.10003.001/BARR/2019-06"
##  [9] "https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2015-06"
## [10] "https://data.neonscience.org/api/v0/data/DP1.10003.001/BART/2016-06"

These are the URLs showing us what files are available for each month where there are data.

Let's look at the bird data from Woodworth (WOOD) site from July 2015. We can do this by using the above code but now specifying which site/date we want using the grep() function.

Note that if there were only one month of data from a site, you could leave off the date in the function. If you want date from more than one site/month you need to iterate this code, GET fails if you give it more than one URL.

# get data availability for WOOD July 2015
brd <- GET(bird.urls[grep("WOOD/2015-07", bird.urls)])
brd.files <- jsonlite::fromJSON(content(brd, as="text"))

# view just the available data files 
brd.files$data$files

##                                                                               name   size
## 1                          NEON.D09.WOOD.DP1.10003.001.readme.20210123T023002Z.txt   9207
## 2                      NEON.D09.WOOD.DP0.10003.001.validation.20201223T141702Z.csv  11357
## 3                NEON.D09.WOOD.DP0.10003.001.categoricalCodes.20201223T141702Z.csv   9521
## 4                NEON.D09.WOOD.DP1.10003.001.2015-07.expanded.20201223T141702Z.zip  87636
## 5          NEON.Bird_Conservancy_of_the_Rockies.brd_personnel.20201223T141702Z.csv  63796
## 6  NEON.D09.WOOD.DP1.10003.001.brd_countdata.2015-07.expanded.20201223T141702Z.csv 371251
## 7           NEON.D09.WOOD.DP1.10003.001.EML.20150701-20150705.20210123T023002Z.xml 178787
## 8   NEON.D09.WOOD.DP1.10003.001.brd_perpoint.2015-07.expanded.20201223T141702Z.csv  23883
## 9         NEON.D09.WOOD.DP1.10003.001.brd_references.expanded.20201223T141702Z.csv   1178
## 10                      NEON.D09.WOOD.DP1.10003.001.variables.20201223T141702Z.csv   8670
## 11                  NEON.D09.WOOD.DP1.10003.001.2015-07.basic.20201223T141702Z.zip  71662
## 12    NEON.D09.WOOD.DP1.10003.001.brd_countdata.2015-07.basic.20201223T141702Z.csv 350528
## 13     NEON.D09.WOOD.DP1.10003.001.brd_perpoint.2015-07.basic.20201223T141702Z.csv  23883
## 14                     NEON.D09.WOOD.DP0.10003.001.validation.20201223T141702Z.csv  11357
## 15                         NEON.D09.WOOD.DP1.10003.001.readme.20210123T023002Z.txt   8913
## 16          NEON.D09.WOOD.DP1.10003.001.EML.20150701-20150705.20210123T023002Z.xml 161547
## 17               NEON.D09.WOOD.DP0.10003.001.categoricalCodes.20201223T141702Z.csv   9521
## 18                      NEON.D09.WOOD.DP1.10003.001.variables.20201223T141702Z.csv   8670
##                                 md5 crc32
## 1  6ed97825bf5b1fc8537dc81cfd531872    NA
## 2  e5b23aad062e41b93631d8c47278a1db    NA
## 3  115177f66ca9b88b28a473f7d476d23c    NA
## 4  31c7ef400f956b364a2af17de244bc17    NA
## 5  893ed864cf0d421da609e856bd3b028f    NA
## 6  3606c59c60c9eb3da0edd7ccd732b230    NA
## 7  e5352a63118c5ce3ad78110a497f5393    NA
## 8  bed5c479043d45cf81f67bdc9936f0bf    NA
## 9  0bcf6823e84b58e1f1de655b51087cd0    NA
## 10 437dd6434e1cd8538dc9676dcf6ca7f9    NA
## 11 022598a3792de3e1366465f0142ac9e5    NA
## 12 555d89c54bd03b8af08be121341d5e56    NA
## 13 bed5c479043d45cf81f67bdc9936f0bf    NA
## 14 e5b23aad062e41b93631d8c47278a1db    NA
## 15 04e8bb1969e9b2b630d10679afe073a7    NA
## 16 16674c19c1f66b3d662eafd3a3128ec3    NA
## 17 115177f66ca9b88b28a473f7d476d23c    NA
## 18 437dd6434e1cd8538dc9676dcf6ca7f9    NA
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   url
## 1                 https://neon-prod-pub-1.s3.data.neonscience.org/release/tag/RELEASE-2021/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.readme.20210123T023002Z.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=547681725eef6816ffb823004ead962dfd1aacf92f255ca325158b8f2465a519
## 2                                      https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP0.10003.001.validation.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=fce32f120631af83bb9c81728d58709637895ee4dcb308c9d9587e38438bc922
## 3                                https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP0.10003.001.categoricalCodes.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=b6a440902a89347a78c8164ad84feaa41d116d7dfcc33e932ad6b2d66923dfd9
## 4                                https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.2015-07.expanded.20201223T141702Z.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=c2ac5b87f4671e5eea523938506f409bc98ee3c9660d99a577fd0b88fc4f0e86
## 5                          https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.Bird_Conservancy_of_the_Rockies.brd_personnel.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=a66745da0c7bc67ff5b1d47f79c2565c973b8af93b12616117c097958104f455
## 6                  https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.brd_countdata.2015-07.expanded.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=26f085b9282668c09b3683bebf61e7fe13b9a904ad8d89b1cb7ce17ecf9666e0
## 7  https://neon-prod-pub-1.s3.data.neonscience.org/release/tag/RELEASE-2021/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.EML.20150701-20150705.20210123T023002Z.xml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=de9341178a05bb3cf8d7e88efccf1029af7a7085786fee25363450c8ca1b395b
## 8                   https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.brd_perpoint.2015-07.expanded.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=bce008aa377243041e683aa4eea371c7918cc836c32e490923c06a577e3355a0
## 9                         https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.brd_references.expanded.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=4fa46fe4f9ddd23d609e0f770d0ce927970acc3a1a79469392befc864bab52a5
## 10                                      https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/expanded/NEON.D09.WOOD.DP1.10003.001.variables.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=43b2ad84e0b93dd7b4b4c4308c786909bdc39ec5e3c9074955925f6f84395145
## 11                                     https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP1.10003.001.2015-07.basic.20201223T141702Z.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=28c62ce9891ec21d2b9790e4ec4e425acfc9e1a0133b0998ee68fb4a066a22ee
## 12                       https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP1.10003.001.brd_countdata.2015-07.basic.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=6179554c8778edcd382085bf6cae8992ddec8e1e0f5cb7f8d68c4319ff799a0b
## 13                        https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP1.10003.001.brd_perpoint.2015-07.basic.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=bc866da78bf5d1c3fe6b6ca251d2bfa1cddfd7c94ac9c88d8d0bed4f256751da
## 14                                        https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP0.10003.001.validation.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=14911e9366063510d0ef08e5a920f9db6dc84355e6deab10899e4c921dc40e9b
## 15                   https://neon-prod-pub-1.s3.data.neonscience.org/release/tag/RELEASE-2021/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP1.10003.001.readme.20210123T023002Z.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=6a20f66863db4214d7d8334c35612acc8d89b739de6bd5ddc22324d06686b4a2
## 16    https://neon-prod-pub-1.s3.data.neonscience.org/release/tag/RELEASE-2021/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP1.10003.001.EML.20150701-20150705.20210123T023002Z.xml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=fdf5b851c875b87a84c9138b932525b9381e195c1ff04926be5a72e5f144636a
## 17                                  https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP0.10003.001.categoricalCodes.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=1959b9dd1183e735e149e4671d08fc4f9730d1a37d0a4187c0109077f6c769e3
## 18                                         https://neon-prod-pub-1.s3.data.neonscience.org/NEON.DOM.SITE.DP1.10003.001/WOOD/20150701T000000--20150801T000000/basic/NEON.D09.WOOD.DP1.10003.001.variables.20201223T141702Z.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210309T235036Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=pub-internal-read%2F20210309%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=8423d5aa9449dd9bab8374bcec801ef59b4647c45f1ac3f7c761d15839a87924

In this output, name and url are key fields. It provides us with the names of the files available for this site and month, and URLs where we can get the files. We'll use the file names to pick which ones we want.

The available files include both data and metadata, and both the basic and expanded data packages. Typically the expanded package includes additional quality or uncertainty data, either in additional files or additional fields than in the basic files. Basic and expanded data packages are available for most NEON data products (some only have basic). Metadata are described by file name below.

The format for most of the file names is:

NEON.[domain number].[site code].[data product ID].[file-specific name]. [date of file creation]

Some files omit the domain and site, since they're not specific to a location, like the data product readme. The date of file creation uses the ISO6801 format, in this case 20170720T182547Z, and can be used to determine whether data have been updated since the last time you downloaded.

Available files in our query for July 2015 at Woodworth are all of the following (leaving off the initial NEON.D09.WOOD.10003.001):

~.2015-07.expanded.20170720T182547Z.zip: zip of all files in the expanded package
~.brd_countdata.2015-07.expanded.20170720T182547Z.csv: count data table, expanded package version: counts of birds at each point
~.brd_perpoint.2015-07.expanded.20170720T182547Z.csv: point data table, expanded package version: metadata at each observation point
NEON.Bird Conservancy of the Rockies.brd_personnel.csv: personnel data table, accuracy scores for bird observers
~.2015-07.basic.20170720T182547Z.zip: zip of all files in the basic package
~.brd_countdata.2015-07.basic.20170720T182547Z.csv: count data table, basic package version: counts of birds at each point
~.brd_perpoint.2015-07.basic.20170720T182547Z.csv: point data table, basic package version: metadata at each observation point
NEON.DP1.10003.001_readme.txt: readme for the data product (not specific to dates or location). Appears twice in the list, since it's in both the basic and expanded package
~.20150101-20160613.xml: Ecological Metadata Language (EML) file. Appears twice in the list, since it's in both the basic and expanded package
~.validation.20170720T182547Z.csv: validation file for the data product, lists input data and data entry rules. Appears twice in the list, since it's in both the basic and expanded package
~.variables.20170720T182547Z.csv: variables file for the data product, lists data fields in downloaded tables. Appears twice in the list, since it's in both the basic and expanded package

We'll get the data tables for the point data and count data in the basic package. The list of files doesn't return in the same order every time, so we won't use position in the list to select. Plus, we want code we can re-use when getting data from other sites and other months. So we select files based on the data table name and the package name.

# Get both files
brd.count <- read.delim(brd.files$data$files$url
                        [intersect(grep("countdata", 
                                        brd.files$data$files$name),
                                    grep("basic", 
                                         brd.files$data$files$name))], 
                        sep=",")

brd.point <- read.delim(brd.files$data$files$url
                        [intersect(grep("perpoint", 
                                        brd.files$data$files$name),
                                    grep("basic", 
                                         brd.files$data$files$name))], 
                        sep=",")

Now we have the data and can access it in R. Just to show that the files we pulled have actual data in them, let's make a quick graphic:

# Cluster by species 
clusterBySp <- brd.count %>%
  dplyr::group_by(scientificName) %>%
  dplyr::summarise(total=sum(clusterSize, na.rm=T))

# Reorder so list is ordered most to least abundance
clusterBySp <- clusterBySp[order(clusterBySp$total, decreasing=T),]

# Plot
barplot(clusterBySp$total, names.arg=clusterBySp$scientificName, 
        ylab="Total", cex.names=0.5, las=2)

Wow! There are lots of Agelaius phoeniceus (Red-winged Blackbirds) at WOOD in July.

Instrumentation data (IS)

The process is essentially the same for sensor data. We'll do the same series of queries for Soil Temperature, DP1.00041.001. Let's use data from Moab in March 2017 this time.

# Request soil temperature data availability info
req.soil <- GET("http://data.neonscience.org/api/v0/products/DP1.00041.001")

# make this JSON readable
# Note how we've change this from two commands into one here
avail.soil <- jsonlite::fromJSON(content(req.soil, as="text"), simplifyDataFrame=T, flatten=T)

# get data availability list for the product
temp.urls <- unlist(avail.soil$data$siteCodes$availableDataUrls)

# get data availability from location/date of interest
tmp <- GET(temp.urls[grep("MOAB/2017-08", temp.urls)])
tmp.files <- jsonlite::fromJSON(content(tmp, as="text"))
length(tmp.files$data$files$name) # There are a lot of available files

## [1] 190

tmp.files$data$files$name[1:10]   # Let's print the first 10

##  [1] "NEON.D13.MOAB.DP1.00041.001.005.501.030.ST_30_minute.2017-08.basic.20200620T054008Z.csv"
##  [2] "NEON.D13.MOAB.DP1.00041.001.002.504.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv" 
##  [3] "NEON.D13.MOAB.DP1.00041.001.004.505.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv" 
##  [4] "NEON.D13.MOAB.DP1.00041.001.001.501.030.ST_30_minute.2017-08.basic.20200620T054008Z.csv"
##  [5] "NEON.D13.MOAB.DP1.00041.001.003.508.030.ST_30_minute.2017-08.basic.20200620T054008Z.csv"
##  [6] "NEON.D13.MOAB.DP1.00041.001.001.504.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv" 
##  [7] "NEON.D13.MOAB.DP1.00041.001.004.504.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv" 
##  [8] "NEON.D13.MOAB.DP1.00041.001.003.508.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv" 
##  [9] "NEON.D13.MOAB.DP1.00041.001.003.506.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv" 
## [10] "NEON.D13.MOAB.DP1.00041.001.001.507.001.ST_1_minute.2017-08.basic.20200620T054008Z.csv"

These file names start and end the same way as the observational files, but the middle is a little more cryptic. The structure from beginning to end is:

NEON.[domain number].[site code].[data product ID]. [soil plot number].[depth].[averaging interval].[data table name]. [year]-[month].[data package].[date of file creation]

So NEON.D13.MOAB.DP1.00041.001.00000.003.506.030.ST_30_minute. 2017-08.expanded.20200620T054008Z.csv is the:

NEON (NEON.)
Domain 13 (.D13.)
Moab field site (.MOAB.)
soil temperature data (.DP1.00041.001.)
collected in Soil Plot 3, (.003.)
at the 6th depth below the surface (.506.)
and reported as a 30-minute mean of (.030. and .ST_30_minute.)
only for the period of Aug 2017 (.2017-08.)
and provided in the basic data package (.basic.)
published on June 20, 2020 at 05:40:08 GMT (.20200620T054008Z.).

More information about interpreting file names can be found on the Data Formats page.

Let's get data (and the URL) for only the plot and depth described above by selecting 003.506.030 and the word basic in the file name.

Go get it:

soil.temp <- read.delim(tmp.files$data$files$url
                        [intersect(grep("003.506.030", 
                                        tmp.files$data$files$name),
                                   grep("basic", 
                                        tmp.files$data$files$name))], 
                        sep=",")

Now we have the data and can use it to conduct our analyses. To take a quick look at it, let's plot the mean soil temperature by date.

# plot temp ~ date
plot(soil.temp$soilTempMean~as.POSIXct(soil.temp$startDateTime, 
                                       format="%Y-%m-%d T %H:%M:%S Z"), 
     pch=".", xlab="Date", ylab="T")

As we'd expect we see daily fluctuation in soil temperature, and some change in temperature over the month as well.

Remote sensing data (AOP)

Again, the process of determining which sites and time periods have data, and finding the URLs for those data, is the same as for the other data types. We'll go looking for High resolution orthorectified camera imagery, DP1.30010, and we'll look at the flight over San Joaquin Experimental Range (SJER) in March 2017.

# Request camera data availability info
req.aop <- GET("http://data.neonscience.org/api/v0/products/DP1.30010.001")

# make this JSON readable
# Note how we've changed this from two commands into one here
avail.aop <- jsonlite::fromJSON(content(req.aop, as="text"), 
                      simplifyDataFrame=T, flatten=T)

# get data availability list for the product
cam.urls <- unlist(avail.aop$data$siteCodes$availableDataUrls)

# get data availability from location/date of interest
cam <- GET(cam.urls[intersect(grep("SJER", cam.urls),
                              grep("2017", cam.urls))])
cam.files <- jsonlite::fromJSON(content(cam, as="text"))

# this list of files is very long, so we'll just look at the first ten
head(cam.files$data$files$name, 10)

##  [1] "17032816_EH021656(20170328184327)-0507_ort.tif" "17032816_EH021656(20170328191148)-0721_ort.tif"
##  [3] "17032816_EH021656(20170328200008)-1155_ort.tif" "17032816_EH021656(20170328190210)-0653_ort.tif"
##  [5] "17032816_EH021656(20170328195154)-1065_ort.tif" "17032816_EH021656(20170328195740)-1122_ort.tif"
##  [7] "17032816_EH021656(20170328175153)-0067_ort.tif" "17032816_EH021656(20170328194702)-1023_ort.tif"
##  [9] "17032816_EH021656(20170328195246)-1077_ort.tif" "17032816_EH021656(20170328175804)-0128_ort.tif"

File names for AOP data are more variable than for IS or OS data; different AOP data products use different naming conventions. File formats differ by product as well.

This particular product, camera imagery, is stored in TIFF files. For a full list of AOP data products, their naming conventions, and their file formats, see .

Instead of reading a TIFF into R, we'll download it to the working directory. This is one option for getting AOP files from the API; if you plan to work with the files in R, you'll need to know how to read the relevant file types into R. We hope to add tutorials for this in the near future.

To download the TIFF file, we use the downloader package, and we'll select a file based on the time stamp in the file name: 20170328192931

download(cam.files$data$files$url[grep("20170328192931", 
                                       cam.files$data$files$name)],
         paste(getwd(), "/SJER_image.tif", sep=""), mode="wb")

The image, below, of the San Joaquin Experimental Range should now be in your working directory.

An example of camera data (DP1.30010.001) from the San Joaquin Experimental Range. Source: National Ecological Observatory Network (NEON)

Geolocation data

You may have noticed some of the spatial data referenced above are a bit vague, e.g. "soil plot 2, 4th depth below the surface."

How to get spatial data and what to do with it depends on which type of data you're working with.

Instrumentation data (both aquatic and terrestrial)

The sensor_positions files, which are included in the list of available files, contain spatial coordinates for each sensor in the data. See the final section of the Geolocation tutorial for guidance in using these files.

Observational data - Aquatic

Latitude, longitude, elevation, and associated uncertainties are included in data downloads. Most products also include an "additional coordinate uncertainty" that should be added to the provided uncertainty. Additional spatial data, such as northing and easting, can be downloaded from the API.

Observational data - Terrestrial

Latitude, longitude, elevation, and associated uncertainties are included in data downloads. These are the coordinates and uncertainty of the sampling plot; for many protocols it is possible to calculate a more precise location. Instructions for doing this are in the respective data product user guides, and code is in the geoNEON package on GitHub.

Querying a single named location

Let's look at the named locations in the bird data we downloaded above. To do this, look for the field called namedLocation, which is present in all observational data products, both aquatic and terrestrial.

# view named location
head(brd.point$namedLocation)

## [1] "WOOD_013.birdGrid.brd" "WOOD_013.birdGrid.brd" "WOOD_013.birdGrid.brd" "WOOD_013.birdGrid.brd"
## [5] "WOOD_013.birdGrid.brd" "WOOD_013.birdGrid.brd"

Here we see the first six entries in the namedLocation column which tells us the names of the Terrestrial Observation plots where the bird surveys were conducted.

We can query the locations endpoint of the API for the first named location, WOOD_013.birdGrid.brd.

# location data 
req.loc <- GET("http://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd")

# make this JSON readable
brd.WOOD_013 <- jsonlite::fromJSON(content(req.loc, as="text"))
brd.WOOD_013

## $data
## $data$locationName
## [1] "WOOD_013.birdGrid.brd"
## 
## $data$locationDescription
## [1] "Plot \"WOOD_013\" at site \"WOOD\""
## 
## $data$locationType
## [1] "OS Plot - brd"
## 
## $data$domainCode
## [1] "D09"
## 
## $data$siteCode
## [1] "WOOD"
## 
## $data$locationDecimalLatitude
## [1] 47.13912
## 
## $data$locationDecimalLongitude
## [1] -99.23243
## 
## $data$locationElevation
## [1] 579.31
## 
## $data$locationUtmEasting
## [1] 482375.7
## 
## $data$locationUtmNorthing
## [1] 5220650
## 
## $data$locationUtmHemisphere
## [1] "N"
## 
## $data$locationUtmZone
## [1] 14
## 
## $data$alphaOrientation
## [1] 0
## 
## $data$betaOrientation
## [1] 0
## 
## $data$gammaOrientation
## [1] 0
## 
## $data$xOffset
## [1] 0
## 
## $data$yOffset
## [1] 0
## 
## $data$zOffset
## [1] 0
## 
## $data$offsetLocation
## NULL
## 
## $data$locationProperties
##                             locationPropertyName locationPropertyValue
## 1                    Value for Coordinate source            GeoXH 6000
## 2               Value for Coordinate uncertainty                  0.28
## 3                              Value for Country          unitedStates
## 4                               Value for County              Stutsman
## 5                Value for Elevation uncertainty                  0.48
## 6                   Value for Filtered positions                   121
## 7                       Value for Geodetic datum                 WGS84
## 8     Value for Horizontal dilution of precision                     1
## 9                    Value for Maximum elevation                579.31
## 10                   Value for Minimum elevation                569.79
## 11 Value for National Land Cover Database (2001)   grasslandHerbaceous
## 12                     Value for Plot dimensions           500m x 500m
## 13                             Value for Plot ID              WOOD_013
## 14                           Value for Plot size                250000
## 15                        Value for Plot subtype              birdGrid
## 16                           Value for Plot type           distributed
## 17    Value for Positional dilution of precision                   2.4
## 18            Value for Reference Point Position                    B2
## 19                        Value for Slope aspect                238.91
## 20                      Value for Slope gradient                  2.83
## 21                     Value for Soil type order             Mollisols
## 22                      Value for State province                    ND
## 23               Value for Subtype Specification            ninePoints
## 24                            Value for UTM Zone                   14N
## 
## $data$locationParent
## [1] "WOOD"
## 
## $data$locationParentUrl
## [1] "https://data.neonscience.org/api/v0/locations/WOOD"
## 
## $data$locationChildren
## [1] "WOOD_013.birdGrid.brd.B2" "WOOD_013.birdGrid.brd.A2" "WOOD_013.birdGrid.brd.C3"
## [4] "WOOD_013.birdGrid.brd.A3" "WOOD_013.birdGrid.brd.B3" "WOOD_013.birdGrid.brd.C1"
## [7] "WOOD_013.birdGrid.brd.A1" "WOOD_013.birdGrid.brd.B1" "WOOD_013.birdGrid.brd.C2"
## 
## $data$locationChildrenUrls
## [1] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.B2"
## [2] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.A2"
## [3] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.C3"
## [4] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.A3"
## [5] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.B3"
## [6] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.C1"
## [7] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.A1"
## [8] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.B1"
## [9] "https://data.neonscience.org/api/v0/locations/WOOD_013.birdGrid.brd.C2"

Note spatial information under $data$[nameOfCoordinate] and under $data$locationProperties. Also note $data$locationChildren: these are the finer scale locations that can be used to calculate precise spatial data for bird observations.

For convenience, we'll use the geoNEON package to make the calculations. First we'll use getLocByName() to get the additional spatial information available through the API, and look at the spatial resolution available in the initial download:

# load the geoNEON package
library(geoNEON)

# extract the spatial data
brd.point.loc <- getLocByName(brd.point)

##

|
| | 0% |
|============== | 14% |
|============================ | 29% |
|========================================== | 43% |
|======================================================= | 57% |
|===================================================================== | 71% |
|=================================================================================== | 86% |
|=================================================================================================| 100%

# plot bird point locations 
# note that decimal degrees is also an option in the data
symbols(brd.point.loc$easting, brd.point.loc$northing, 
        circles=brd.point.loc$coordinateUncertainty, 
        xlab="Easting", ylab="Northing", tck=0.01, inches=F)

And use getLocTOS() to calculate the point locations of observations.

brd.point.pt <- getLocTOS(brd.point, "brd_perpoint")

##

|
| | 0% |
|== | 2% |
|=== | 3% |
|===== | 5% |
|====== | 6% |
|======== | 8% |
|========= | 10% |
|=========== | 11% |
|============ | 13% |
|============== | 14% |
|=============== | 16% |
|================= | 17% |
|================== | 19% |
|==================== | 21% |
|====================== | 22% |
|======================= | 24% |
|========================= | 25% |
|========================== | 27% |
|============================ | 29% |
|============================= | 30% |
|=============================== | 32% |
|================================ | 33% |
|================================== | 35% |
|=================================== | 37% |
|===================================== | 38% |
|====================================== | 40% |
|======================================== | 41% |
|========================================== | 43% |
|=========================================== | 44% |
|============================================= | 46% |
|============================================== | 48% |
|================================================ | 49% |
|================================================= | 51% |
|=================================================== | 52% |
|==================================================== | 54% |
|====================================================== | 56% |
|======================================================= | 57% |
|========================================================= | 59% |
|=========================================================== | 60% |
|============================================================ | 62% |
|============================================================== | 63% |
|=============================================================== | 65% |
|================================================================= | 67% |
|================================================================== | 68% |
|==================================================================== | 70% |
|===================================================================== | 71% |
|======================================================================= | 73% |
|======================================================================== | 75% |
|========================================================================== | 76% |
|=========================================================================== | 78% |
|============================================================================= | 79% |
|=============================================================================== | 81% |
|================================================================================ | 83% |
|================================================================================== | 84% |
|=================================================================================== | 86% |
|===================================================================================== | 87% |
|====================================================================================== | 89% |
|======================================================================================== | 90% |
|========================================================================================= | 92% |
|=========================================================================================== | 94% |
|============================================================================================ | 95% |
|============================================================================================== | 97% |
|=============================================================================================== | 98% |
|=================================================================================================| 100%

# plot bird point locations 
# note that decimal degrees is also an option in the data
symbols(brd.point.pt$adjEasting, brd.point.pt$adjNorthing, 
        circles=brd.point.pt$adjCoordinateUncertainty, 
        xlab="Easting", ylab="Northing", tck=0.01, inches=F)

Now you can see the individual points where the respective point counts were located.

Taxonomy

NEON maintains accepted taxonomies for many of the taxonomic identification data we collect. NEON taxonomies are available for query via the API; they are also provided via an interactive user interface, the Taxon Viewer.

NEON taxonomy data provides the reference information for how NEON validates taxa; an identification must appear in the taxonomy lists in order to be accepted into the NEON database. Additions to the lists are reviewed regularly. The taxonomy lists also provide the author of the scientific name, and the reference text used.

The taxonomy endpoint of the API works a little bit differently from the other endpoints. In the "Anatomy of an API Call" section above, each endpoint has a single type of target - a data product number, a named location name, etc. For taxonomic data, there are multiple query options, and some of them can be used in combination. For example, a query for taxa in the Pinaceae family:

http://data.neonscience.org/api/v0/taxonomy/?family=Pinaceae

The available types of queries are listed in the taxonomy section of the API web page. Briefly, they are:

taxonTypeCode: Which of the taxonomies maintained by NEON are you looking for? BIRD, FISH, PLANT, etc. Cannot be used in combination with the taxonomic rank queries.
each of the major taxonomic ranks from genus through kingdom
scientificname: Genus + specific epithet (+ authority). Search is by exact match only, see final example below.
verbose: Do you want the short (false) or long (true) response
offset: Skip this number of items in the list. Defaults to 50.
limit: Result set will be truncated at this length. Defaults to 50.

Staff on the NEON project have plans to modify the settings for offset and limit, such that offset will default to 0 and limit will default to ∞, but in the meantime users will want to set these manually. They are set to non-default values in the examples below.

For the first example, let's query for the loon family, Gaviidae, in the bird taxonomy. Note that query parameters are case-sensitive.

loon.req <- GET("http://data.neonscience.org/api/v0/taxonomy/?family=Gaviidae&offset=0&limit=500")

Parse the results into a list using fromJSON():

loon.list <- jsonlite::fromJSON(content(loon.req, as="text"))

And look at the $data element of the results, which contains:

The full taxonomy of each taxon
The short taxon code used by NEON (taxonID/acceptedTaxonID)
The author of the scientific name (scientificNameAuthorship)
The vernacular name, if applicable
The reference text used (nameAccordingToID)

The terms used for each field are matched to Darwin Core (dwc) and the Global Biodiversity Information Facility (gbif) terms, where possible, and the matches are indicated in the column headers.

loon.list$data

##   taxonTypeCode taxonID acceptedTaxonID dwc:scientificName dwc:scientificNameAuthorship dwc:taxonRank
## 1          BIRD    ARLO            ARLO      Gavia arctica                   (Linnaeus)       species
## 2          BIRD    COLO            COLO        Gavia immer                   (Brunnich)       species
## 3          BIRD    PALO            PALO     Gavia pacifica                   (Lawrence)       species
## 4          BIRD    RTLO            RTLO     Gavia stellata                (Pontoppidan)       species
## 5          BIRD    YBLO            YBLO      Gavia adamsii                 (G. R. Gray)       species
##   dwc:vernacularName    dwc:nameAccordingToID dwc:kingdom dwc:phylum dwc:class   dwc:order dwc:family
## 1        Arctic Loon doi: 10.1642/AUK-15-73.1    Animalia   Chordata      Aves Gaviiformes   Gaviidae
## 2        Common Loon doi: 10.1642/AUK-15-73.1    Animalia   Chordata      Aves Gaviiformes   Gaviidae
## 3       Pacific Loon doi: 10.1642/AUK-15-73.1    Animalia   Chordata      Aves Gaviiformes   Gaviidae
## 4  Red-throated Loon doi: 10.1642/AUK-15-73.1    Animalia   Chordata      Aves Gaviiformes   Gaviidae
## 5 Yellow-billed Loon doi: 10.1642/AUK-15-73.1    Animalia   Chordata      Aves Gaviiformes   Gaviidae
##   dwc:genus gbif:subspecies gbif:variety
## 1     Gavia              NA           NA
## 2     Gavia              NA           NA
## 3     Gavia              NA           NA
## 4     Gavia              NA           NA
## 5     Gavia              NA           NA

To get the entire list for a particular taxonomic type, use the taxonTypeCode query. Be cautious with this query, the PLANT taxonomic list has several hundred thousand entries.

For an example, let's look up the small mammal taxonomic list, which is one of the shorter ones, and use the verbose=true option to see a more extensive list of taxon data, including many taxon ranks that aren't populated for these taxa. For space here, we display only the first 10 taxa:

mam.req <- GET("http://data.neonscience.org/api/v0/taxonomy/?taxonTypeCode=SMALL_MAMMAL&offset=0&limit=500&verbose=true")
mam.list <- jsonlite::fromJSON(content(mam.req, as="text"))
mam.list$data[1:10,]

##    taxonTypeCode taxonID acceptedTaxonID               dwc:scientificName dwc:scientificNameAuthorship
## 1   SMALL_MAMMAL    AMHA            AMHA        Ammospermophilus harrisii          Audubon and Bachman
## 2   SMALL_MAMMAL    AMIN            AMIN       Ammospermophilus interpres                      Merriam
## 3   SMALL_MAMMAL    AMLE            AMLE        Ammospermophilus leucurus                      Merriam
## 4   SMALL_MAMMAL    AMLT            AMLT Ammospermophilus leucurus tersus                      Goldman
## 5   SMALL_MAMMAL    AMNE            AMNE         Ammospermophilus nelsoni                      Merriam
## 6   SMALL_MAMMAL    AMSP            AMSP             Ammospermophilus sp.                         <NA>
## 7   SMALL_MAMMAL    APRN            APRN            Aplodontia rufa nigra                       Taylor
## 8   SMALL_MAMMAL    APRU            APRU                  Aplodontia rufa                   Rafinesque
## 9   SMALL_MAMMAL    ARAL            ARAL                Arborimus albipes                      Merriam
## 10  SMALL_MAMMAL    ARLO            ARLO            Arborimus longicaudus                         True
##    dwc:taxonRank            dwc:vernacularName taxonProtocolCategory dwc:nameAccordingToID
## 1        species     Harriss Antelope Squirrel         opportunistic  isbn: 978 0801882210
## 2        species       Texas Antelope Squirrel         opportunistic  isbn: 978 0801882210
## 3        species Whitetailed Antelope Squirrel         opportunistic  isbn: 978 0801882210
## 4     subspecies                          <NA>         opportunistic  isbn: 978 0801882210
## 5        species     Nelsons Antelope Squirrel         opportunistic  isbn: 978 0801882210
## 6          genus                          <NA>         opportunistic  isbn: 978 0801882210
## 7     subspecies                          <NA>            non-target  isbn: 978 0801882210
## 8        species                      Sewellel            non-target  isbn: 978 0801882210
## 9        species              Whitefooted Vole                target  isbn: 978 0801882210
## 10       species                 Red Tree Vole                target  isbn: 978 0801882210
##                                                                                                                                                 dwc:nameAccordingToTitle
## 1  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 2  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 3  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 4  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 5  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 6  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 7  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 8  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 9  Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
## 10 Wilson D. E. and D. M. Reeder. 2005. Mammal Species of the World; A Taxonomic and Geographic Reference. Third edition. Johns Hopkins University Press; Baltimore, MD.
##    dwc:kingdom gbif:subkingdom gbif:infrakingdom gbif:superdivision gbif:division gbif:subdivision
## 1     Animalia              NA                NA                 NA            NA               NA
## 2     Animalia              NA                NA                 NA            NA               NA
## 3     Animalia              NA                NA                 NA            NA               NA
## 4     Animalia              NA                NA                 NA            NA               NA
## 5     Animalia              NA                NA                 NA            NA               NA
## 6     Animalia              NA                NA                 NA            NA               NA
## 7     Animalia              NA                NA                 NA            NA               NA
## 8     Animalia              NA                NA                 NA            NA               NA
## 9     Animalia              NA                NA                 NA            NA               NA
## 10    Animalia              NA                NA                 NA            NA               NA
##    gbif:infradivision gbif:parvdivision gbif:superphylum dwc:phylum gbif:subphylum gbif:infraphylum
## 1                  NA                NA               NA   Chordata             NA               NA
## 2                  NA                NA               NA   Chordata             NA               NA
## 3                  NA                NA               NA   Chordata             NA               NA
## 4                  NA                NA               NA   Chordata             NA               NA
## 5                  NA                NA               NA   Chordata             NA               NA
## 6                  NA                NA               NA   Chordata             NA               NA
## 7                  NA                NA               NA   Chordata             NA               NA
## 8                  NA                NA               NA   Chordata             NA               NA
## 9                  NA                NA               NA   Chordata             NA               NA
## 10                 NA                NA               NA   Chordata             NA               NA
##    gbif:superclass dwc:class gbif:subclass gbif:infraclass gbif:superorder dwc:order gbif:suborder
## 1               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 2               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 3               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 4               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 5               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 6               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 7               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 8               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 9               NA  Mammalia            NA              NA              NA  Rodentia            NA
## 10              NA  Mammalia            NA              NA              NA  Rodentia            NA
##    gbif:infraorder gbif:section gbif:subsection gbif:superfamily    dwc:family gbif:subfamily gbif:tribe
## 1               NA           NA              NA               NA     Sciuridae        Xerinae  Marmotini
## 2               NA           NA              NA               NA     Sciuridae        Xerinae  Marmotini
## 3               NA           NA              NA               NA     Sciuridae        Xerinae  Marmotini
## 4               NA           NA              NA               NA     Sciuridae        Xerinae  Marmotini
## 5               NA           NA              NA               NA     Sciuridae        Xerinae  Marmotini
## 6               NA           NA              NA               NA     Sciuridae        Xerinae  Marmotini
## 7               NA           NA              NA               NA Aplodontiidae           <NA>       <NA>
## 8               NA           NA              NA               NA Aplodontiidae           <NA>       <NA>
## 9               NA           NA              NA               NA    Cricetidae    Arvicolinae       <NA>
## 10              NA           NA              NA               NA    Cricetidae    Arvicolinae       <NA>
##    gbif:subtribe        dwc:genus dwc:subgenus gbif:subspecies gbif:variety gbif:subvariety gbif:form
## 1             NA Ammospermophilus         <NA>              NA           NA              NA        NA
## 2             NA Ammospermophilus         <NA>              NA           NA              NA        NA
## 3             NA Ammospermophilus         <NA>              NA           NA              NA        NA
## 4             NA Ammospermophilus         <NA>              NA           NA              NA        NA
## 5             NA Ammospermophilus         <NA>              NA           NA              NA        NA
## 6             NA Ammospermophilus         <NA>              NA           NA              NA        NA
## 7             NA       Aplodontia         <NA>              NA           NA              NA        NA
## 8             NA       Aplodontia         <NA>              NA           NA              NA        NA
## 9             NA        Arborimus         <NA>              NA           NA              NA        NA
## 10            NA        Arborimus         <NA>              NA           NA              NA        NA
##    gbif:subform speciesGroup dwc:specificEpithet dwc:infraspecificEpithet
## 1            NA         <NA>            harrisii                     <NA>
## 2            NA         <NA>           interpres                     <NA>
## 3            NA         <NA>            leucurus                     <NA>
## 4            NA         <NA>            leucurus                   tersus
## 5            NA         <NA>             nelsoni                     <NA>
## 6            NA         <NA>                 sp.                     <NA>
## 7            NA         <NA>                rufa                    nigra
## 8            NA         <NA>                rufa                     <NA>
## 9            NA         <NA>             albipes                     <NA>
## 10           NA         <NA>         longicaudus                     <NA>

To get information about a single taxon, use the scientificname query. This query will not do a fuzzy match, so you need to query the exact name of the taxon in the NEON taxonomy. Because of this, the query will be most useful when you already have NEON data in hand and are looking for more information about a specific taxon. Querying on scientificname is unlikely to be an efficient way to figure out if NEON recognizes a particular taxon.

In addition, scientific names contain spaces, which are not allowed in a URL. The spaces need to be replaced with the URL encoding replacement, %20.

For an example, let's look up the little sand verbena, Abronia minor Standl. Searching for Abronia minor will fail, because the NEON taxonomy for this species includes the authority. The search will also fail with spaces. Search for Abronia%20minor%20Standl., and in this case we can omit offset and limit because we know there can only be a single result:

am.req <- GET("http://data.neonscience.org/api/v0/taxonomy/?scientificname=Abronia%20minor%20Standl.")
am.list <- jsonlite::fromJSON(content(am.req, as="text"))
am.list$data

##   taxonTypeCode taxonID acceptedTaxonID    dwc:scientificName dwc:scientificNameAuthorship dwc:taxonRank
## 1         PLANT   ABMI2           ABMI2 Abronia minor Standl.                      Standl.       species
##    dwc:vernacularName                       dwc:nameAccordingToID dwc:kingdom    dwc:phylum     dwc:class
## 1 little sand verbena http://plants.usda.gov (accessed 8/25/2014)     Plantae Magnoliophyta Magnoliopsida
##        dwc:order    dwc:family dwc:genus gbif:subspecies gbif:variety
## 1 Caryophyllales Nyctaginaceae   Abronia              NA           NA

Stacking NEON data

At the top of this tutorial, we installed the neonUtilities package. This is a custom R package that stacks the monthly files provided by the NEON data portal into a single continuous file for each type of data table in the download.

For a guide to using neonUtilities on data downloaded from the portal, check out the neonUtilities tutorial.

Get Lesson Code

NEON-API-How-To.R

Work With NEON's Plant Phenology Data

Authors: Megan A. Jones, Natalie Robinson, Lee Stanish

Last Updated: Apr 8, 2021

Many organisms, including plants, show patterns of change across seasons - the different stages of this observable change are called phenophases. In this tutorial we explore how to work with NEON plant phenophase data.

Objectives

After completing this activity, you will be able to:

work with NEON Plant Phenology Observation data.
use dplyr functions to filter data.
plot time series data in a bar plot using ggplot the function.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio loaded on your computer to complete this tutorial.

Install R Packages

neonUtilities: install.packages("neonUtilities")
ggplot2: install.packages("ggplot2")
dplyr: install.packages("dplyr")

More on Packages in R – Adapted from Software Carpentry.

Download Data

This tutorial is designed to have you download data directly from the NEON portal API using the neonUtilities package. However, you can also directly download this data, prepackaged, from FigShare. This data set includes all the files needed for the Work with NEON OS & IS Data - Plant Phenology & Temperature tutorial series. The data are in the format you would receive if downloading them using the zipsByProduct() function in the neonUtilities package.

Direct Download: NEON Phenology & Temp Time Series Teaching Data Subset (v2 - 2017-2019 data) (12 MB)

Additional Resources

NEON data portal
NEON Plant Phenology Observations data product user guide
RStudio's data wrangling (dplyr/tidyr) cheatsheet
NEONScience GitHub Organization
nneo API wrapper on CRAN

Plants change throughout the year - these are phenophases. Why do they change?

Explore Phenology Data

The following sections provide a brief overview of the NEON plant phenology observation data. When designing a research project using this data, you need to consult the documents associated with this data product and not rely solely on this summary.

The following description of the NEON Plant Phenology Observation data is modified from the data product user guide.

NEON Plant Phenology Observation Data

NEON collects plant phenology data and provides it as NEON data product DP1.10055.001.

The plant phenology observations data product provides in-situ observations of the phenological status and intensity of tagged plants (or patches) during discrete observations events.

Sampling occurs at all terrestrial field sites at site and season specific intervals. During Phase I (dominant species) sampling (pre-2021), three species with 30 individuals each are sampled. In 2021, Phase II (community) sampling will begin, with <=20 species with 5 or more individuals sampled will occur.

Status-based Monitoring

NEON employs status-based monitoring, in which the phenological condition of an individual is reported any time that individual is observed. At every observations bout, records are generated for every phenophase that is occurring and for every phenophase not occurring. With this approach, events (such as leaf emergence in Mediterranean climates, or flowering in many desert species) that may occur multiple times during a single year, can be captured. Continuous reporting of phenophase status enables quantification of the duration of phenophases rather than just their date of onset while allows enabling the explicit quantification of uncertainty in phenophase transition dates that are introduced by monitoring in discrete temporal bouts.

Specific products derived from this sampling include the observed phenophase status (whether or not a phenophase is occurring) and the intensity of phenophases for individuals in which phenophase status = ‘yes’. Phenophases reported are derived from the USA National Phenology Network (USA-NPN) categories. The number of phenophases observed varies by growth form and ranges from 1 phenophase (cactus) to 7 phenophases (semi-evergreen broadleaf). In this tutorial we will focus only on the state of the phenophase, not the phenophase intensity data.

Phenology Transects

Plant phenology observations occurs at all terrestrial NEON sites along an 800 meter square loop transect (primary) and within a 200 m x 200 m plot located within view of a canopy level, tower-mounted, phenology camera.

Diagram of a phenology transect layout, with meter layout marked. Point-level geolocations are recorded at eight reference
points along the perimeter; plot-level geolocation at the plot centroid(star). — Diagram of a phenology transect layout, with meter layout marked. Point-level geolocations are recorded at eight reference points along the perimeter, plot-level geolocation at the plot centroid (star). Source: National Ecological Observatory Network (NEON)

Timing of Observations

At each site, there are:

~50 observation bouts per year.
no more that 100 sampling points per phenology transect.
no more than 9 sampling points per phenocam plot.
1 annual measurement per year to collect annual size and disease status measurements from each sampling point.

Available Data Tables

In the downloaded data packet, data are available in two main files

phe_statusintensity: Plant phenophase status and intensity data
phe_perindividual: Geolocation and taxonomic identification for phenology plants
phe_perindividualperyear: recorded once a year, essentially the "metadata" about the plant: DBH, height, etc.

There are other files in each download including a readme with information on the data product and the download; a variables file that defines the term descriptions, data types, and units; a validation file with data entry validation and parsing rules; and an XML with machine readable metadata.

Stack NEON Data

NEON data are delivered in a site and year-month format. When you download data, you will get a single zipped file containing a directory for each month and site that you've requested data for. Dealing with these separate tables from even one or two sites over a 12 month period can be a bit overwhelming. Luckily NEON provides an R package neonUtilities that takes the unzipped downloaded file and joining the data files. The teaching data downloaded with this tutorial is already stacked. If you are working with other NEON data, please go through the tutorial to stack the data in R or in Python and then return to this tutorial.

Work with NEON Data

When we do this for phenology data we get three files, one for each data table, with all the data from your site and date range of interest.

First, we need to set up our R environment.

# install needed package (only uncomment & run if not already installed)
#install.packages("neonUtilities")
#install.packages("dplyr")
#install.packages("ggplot2")

# load needed packages
library(neonUtilities)
library(dplyr)
library(ggplot2)


options(stringsAsFactors=F) #keep strings as character type not factors

# set working directory to ensure R can find the file we wish to import and where
# we want to save our files. Be sure to move the download into your working directory!
wd <- "~/Documents/data/" # Change this to match your local environment
setwd(wd)

Let's start by loading our data of interest. For this series, we'll work with date from the NEON Domain 02 sites:

Blandy Farm (BLAN)
Smithsonian Conservation Biology Institute (SCBI)
Smithsonian Environmental Research Center (SERC)

And we'll use data from January 2017 to December 2019. This downloads over 9MB of data. If this is too large, use a smaller date range. If you opt to do this, your figures and some output may look different later in the tutorial.

With this information, we can download our data using the neonUtilities package. If you are not using a NEON token to download your data, remove the token = Sys.getenv(NEON_TOKEN) line of code (learn more about NEON API tokens in the Using an API Token when Accessing NEON Data with neonUtilities tutorial).

If you are using the data downloaded at the start of the tutorial, use the commented out code in the second half of this code chunk.

## Two options for accessing data - programmatic or from the example dataset
# Read data from data portal 

phe <- loadByProduct(dpID = "DP1.10055.001", site=c("BLAN","SCBI","SERC"), 
                                         startdate = "2017-01", enddate="2019-12", 
                                         token = Sys.getenv("NEON_TOKEN"),
                                         check.size = F) 

## Finding available files
##

|
| | 0% |
|= | 1% |
|== | 2% |
|== | 3% |
|=== | 4% |
|==== | 5% |
|===== | 6% |
|====== | 7% |
|======= | 8% |
|======= | 9% |
|======== | 11% |
|========= | 12% |
|========== | 13% |
|=========== | 14% |
|============ | 15% |
|============ | 16% |
|============= | 17% |
|============== | 18% |
|=============== | 19% |
|================ | 20% |
|================= | 21% |
|================= | 22% |
|================== | 23% |
|=================== | 24% |
|==================== | 25% |
|===================== | 26% |
|====================== | 27% |
|====================== | 28% |
|======================= | 29% |
|======================== | 31% |
|========================= | 32% |
|========================== | 33% |
|=========================== | 34% |
|=========================== | 35% |
|============================ | 36% |
|============================= | 37% |
|============================== | 38% |
|=============================== | 39% |
|================================ | 40% |
|================================ | 41% |
|================================= | 42% |
|================================== | 43% |
|=================================== | 44% |
|==================================== | 45% |
|===================================== | 46% |
|===================================== | 47% |
|====================================== | 48% |
|======================================= | 49% |
|======================================== | 51% |
|========================================= | 52% |
|========================================== | 53% |
|========================================== | 54% |
|=========================================== | 55% |
|============================================ | 56% |
|============================================= | 57% |
|============================================== | 58% |
|=============================================== | 59% |
|=============================================== | 60% |
|================================================ | 61% |
|================================================= | 62% |
|================================================== | 63% |
|=================================================== | 64% |
|==================================================== | 65% |
|==================================================== | 66% |
|===================================================== | 67% |
|====================================================== | 68% |
|======================================================= | 69% |
|======================================================== | 71% |
|========================================================= | 72% |
|========================================================= | 73% |
|========================================================== | 74% |
|=========================================================== | 75% |
|============================================================ | 76% |
|============================================================= | 77% |
|============================================================== | 78% |
|============================================================== | 79% |
|=============================================================== | 80% |
|================================================================ | 81% |
|================================================================= | 82% |
|================================================================== | 83% |
|=================================================================== | 84% |
|=================================================================== | 85% |
|==================================================================== | 86% |
|===================================================================== | 87% |
|====================================================================== | 88% |
|======================================================================= | 89% |
|======================================================================== | 91% |
|======================================================================== | 92% |
|========================================================================= | 93% |
|========================================================================== | 94% |
|=========================================================================== | 95% |
|============================================================================ | 96% |
|============================================================================= | 97% |
|============================================================================= | 98% |
|============================================================================== | 99% |
|===============================================================================| 100% ## ## Downloading files totaling approximately 11.661285 MB ## Downloading 95 files ## |
| | 0% |
|= | 1% |
|== | 2% |
|=== | 3% |
|=== | 4% |
|==== | 5% |
|===== | 6% |
|====== | 7% |
|======= | 9% |
|======== | 10% |
|======== | 11% |
|========= | 12% |
|========== | 13% |
|=========== | 14% |
|============ | 15% |
|============= | 16% |
|============= | 17% |
|============== | 18% |
|=============== | 19% |
|================ | 20% |
|================= | 21% |
|================== | 22% |
|================== | 23% |
|=================== | 24% |
|==================== | 26% |
|===================== | 27% |
|====================== | 28% |
|======================= | 29% |
|======================== | 30% |
|======================== | 31% |
|========================= | 32% |
|========================== | 33% |
|=========================== | 34% |
|============================ | 35% |
|============================= | 36% |
|============================= | 37% |
|============================== | 38% |
|=============================== | 39% |
|================================ | 40% |
|================================= | 41% |
|================================== | 43% |
|================================== | 44% |
|=================================== | 45% |
|==================================== | 46% |
|===================================== | 47% |
|====================================== | 48% |
|======================================= | 49% |
|======================================== | 50% |
|======================================== | 51% |
|========================================= | 52% |
|========================================== | 53% |
|=========================================== | 54% |
|============================================ | 55% |
|============================================= | 56% |
|============================================= | 57% |
|============================================== | 59% |
|=============================================== | 60% |
|================================================ | 61% |
|================================================= | 62% |
|================================================== | 63% |
|================================================== | 64% |
|=================================================== | 65% |
|==================================================== | 66% |
|===================================================== | 67% |
|====================================================== | 68% |
|======================================================= | 69% |
|======================================================= | 70% |
|======================================================== | 71% |
|========================================================= | 72% |
|========================================================== | 73% |
|=========================================================== | 74% |
|============================================================ | 76% |
|============================================================= | 77% |
|============================================================= | 78% |
|============================================================== | 79% |
|=============================================================== | 80% |
|================================================================ | 81% |
|================================================================= | 82% |
|================================================================== | 83% |
|================================================================== | 84% |
|=================================================================== | 85% |
|==================================================================== | 86% |
|===================================================================== | 87% |
|====================================================================== | 88% |
|======================================================================= | 89% |
|======================================================================= | 90% |
|======================================================================== | 91% |
|========================================================================= | 93% |
|========================================================================== | 94% |
|=========================================================================== | 95% |
|============================================================================ | 96% |
|============================================================================ | 97% |
|============================================================================= | 98% |
|============================================================================== | 99% |
|===============================================================================| 100% ## ## Unpacking zip files using 1 cores. ## Stacking operation across a single core. ## Stacking table phe_perindividual ## Stacking table phe_statusintensity ## Stacking table phe_perindividualperyear ## Copied the most recent publication of validation file to /stackedFiles ## Copied the most recent publication of categoricalCodes file to /stackedFiles ## Copied the most recent publication of variable definition file to /stackedFiles ## Finished: Stacked 3 data tables and 3 metadata tables! ## Stacking took 1.394884 secs

# if you aren't sure you can handle the data file size use check.size = T. 

# save dataframes from the downloaded list
ind <- phe$phe_perindividual  #individual information
status <- phe$phe_statusintensity  #status & intensity info


##If choosing to use example dataset downloaded from this tutorial: 

# Stack multiple files within the downloaded phenology data
#stackByTable("NEON-pheno-temp-timeseries_v2/filesToStack10055", folder = T)

# read in data - readTableNEON uses the variables file to assign the correct
# data type for each variable
#ind <- readTableNEON('NEON-pheno-temp-timeseries_v2/filesToStack10055/stackedFiles/phe_perindividual.csv', 'NEON-pheno-temp-timeseries_v2/filesToStack10055/stackedFiles/variables_10055.csv')

#status <- readTableNEON('NEON-pheno-temp-timeseries_v2/filesToStack10055/stackedFiles/phe_statusintensity.csv', 'NEON-pheno-temp-timeseries_v2/filesToStack10055/stackedFiles/variables_10055.csv')

Let's explore the data. Let's get to know what the ind dataframe looks like.

# What are the fieldnames in this dataset?
names(ind)

##  [1] "uid"                         "namedLocation"              
##  [3] "domainID"                    "siteID"                     
##  [5] "plotID"                      "decimalLatitude"            
##  [7] "decimalLongitude"            "geodeticDatum"              
##  [9] "coordinateUncertainty"       "elevation"                  
## [11] "elevationUncertainty"        "subtypeSpecification"       
## [13] "transectMeter"               "directionFromTransect"      
## [15] "ninetyDegreeDistance"        "sampleLatitude"             
## [17] "sampleLongitude"             "sampleGeodeticDatum"        
## [19] "sampleCoordinateUncertainty" "sampleElevation"            
## [21] "sampleElevationUncertainty"  "date"                       
## [23] "editedDate"                  "individualID"               
## [25] "taxonID"                     "scientificName"             
## [27] "identificationQualifier"     "taxonRank"                  
## [29] "nativeStatusCode"            "growthForm"                 
## [31] "vstTag"                      "samplingProtocolVersion"    
## [33] "measuredBy"                  "identifiedBy"               
## [35] "recordedBy"                  "remarks"                    
## [37] "dataQF"                      "publicationDate"            
## [39] "release"

# Unsure of what some of the variables are you? Look at the variables table!
View(phe$variables_10055)
# if using the pre-downloaded data, you need to read in the variables file 
# or open and look at it on your desktop
#var <- read.csv('NEON-pheno-temp-timeseries_v2/filesToStack10055/stackedFiles/variables_10055.csv')
#View(var)

# how many rows are in the data?
nrow(ind)

## [1] 433

# look at the first six rows of data.
#head(ind) #this is a good function to use but looks messy so not rendering it 

# look at the structure of the dataframe.
str(ind)

## 'data.frame':    433 obs. of  39 variables:
##  $ uid                        : chr  "76bf37d9-c834-43fc-a430-83d87e4b9289" "cf0239bb-2953-44a8-8fd2-051539be5727" "833e5f41-d5cb-4550-ba60-e6f000a2b1b6" "6c2e348d-d19e-4543-9d22-0527819ee964" ...
##  $ namedLocation              : chr  "BLAN_061.phenology.phe" "BLAN_061.phenology.phe" "BLAN_061.phenology.phe" "BLAN_061.phenology.phe" ...
##  $ domainID                   : chr  "D02" "D02" "D02" "D02" ...
##  $ siteID                     : chr  "BLAN" "BLAN" "BLAN" "BLAN" ...
##  $ plotID                     : chr  "BLAN_061" "BLAN_061" "BLAN_061" "BLAN_061" ...
##  $ decimalLatitude            : num  39.1 39.1 39.1 39.1 39.1 ...
##  $ decimalLongitude           : num  -78.1 -78.1 -78.1 -78.1 -78.1 ...
##  $ geodeticDatum              : chr  NA NA NA NA ...
##  $ coordinateUncertainty      : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ elevation                  : num  183 183 183 183 183 183 183 183 183 183 ...
##  $ elevationUncertainty       : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ subtypeSpecification       : chr  "primary" "primary" "primary" "primary" ...
##  $ transectMeter              : num  491 464 537 15 753 506 527 305 627 501 ...
##  $ directionFromTransect      : chr  "Left" "Right" "Left" "Left" ...
##  $ ninetyDegreeDistance       : num  0.5 4 2 3 2 1 2 3 2 3 ...
##  $ sampleLatitude             : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ sampleLongitude            : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ sampleGeodeticDatum        : chr  "WGS84" "WGS84" "WGS84" "WGS84" ...
##  $ sampleCoordinateUncertainty: num  NA NA NA NA NA NA NA NA NA NA ...
##  $ sampleElevation            : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ sampleElevationUncertainty : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ date                       : POSIXct, format: "2016-04-20" "2017-02-24" ...
##  $ editedDate                 : POSIXct, format: "2016-05-09" "2018-05-02" ...
##  $ individualID               : chr  "NEON.PLA.D02.BLAN.06290" "NEON.PLA.D02.BLAN.06501" "NEON.PLA.D02.BLAN.06204" "NEON.PLA.D02.BLAN.06223" ...
##  $ taxonID                    : chr  "RHDA" "SOAL6" "RHDA" "LOMA6" ...
##  $ scientificName             : chr  "Rhamnus davurica Pall." "Solidago altissima L." "Rhamnus davurica Pall." "Lonicera maackii (Rupr.) Herder" ...
##  $ identificationQualifier    : chr  NA NA NA NA ...
##  $ taxonRank                  : chr  "species" "species" "species" "species" ...
##  $ nativeStatusCode           : chr  "I" "N" "I" "I" ...
##  $ growthForm                 : chr  "Deciduous broadleaf" "Forb" "Deciduous broadleaf" "Deciduous broadleaf" ...
##  $ vstTag                     : chr  NA NA NA NA ...
##  $ samplingProtocolVersion    : chr  NA "NEON.DOC.014040vJ" "NEON.DOC.014040vJ" "NEON.DOC.014040vJ" ...
##  $ measuredBy                 : chr  "jcoloso@neoninc.org" "jward@battelleecology.org" "alandes@field-ops.org" "alandes@field-ops.org" ...
##  $ identifiedBy               : chr  "shackley@neoninc.org" "llemmon@field-ops.org" "llemmon@field-ops.org" "llemmon@field-ops.org" ...
##  $ recordedBy                 : chr  "shackley@neoninc.org" NA NA NA ...
##  $ remarks                    : chr  "Nearly dead shaded out" "no entry" "no entry" "no entry" ...
##  $ dataQF                     : chr  NA NA NA NA ...
##  $ publicationDate            : chr  "20201218T103411Z" "20201218T103411Z" "20201218T103411Z" "20201218T103411Z" ...
##  $ release                    : chr  "RELEASE-2021" "RELEASE-2021" "RELEASE-2021" "RELEASE-2021" ...

Notice that the neonUtilities package read the data type from the variables file and then automatically converts the data to the correct date type in R.

(Note that if you first opened your data file in Excel, you might see 06/14/2014 as the format instead of 2014-06-14. Excel can do some ~~weird~~ interesting things to dates.)

Phenology status

Now let's look at the status data.

# What variables are included in this dataset?
names(status)

##  [1] "uid"                           "namedLocation"                
##  [3] "domainID"                      "siteID"                       
##  [5] "plotID"                        "date"                         
##  [7] "editedDate"                    "dayOfYear"                    
##  [9] "individualID"                  "phenophaseName"               
## [11] "phenophaseStatus"              "phenophaseIntensityDefinition"
## [13] "phenophaseIntensity"           "samplingProtocolVersion"      
## [15] "measuredBy"                    "recordedBy"                   
## [17] "remarks"                       "dataQF"                       
## [19] "publicationDate"               "release"

nrow(status)

## [1] 219357

#head(status)   #this is a good function to use but looks messy so not rendering it 
str(status)

## 'data.frame':    219357 obs. of  20 variables:
##  $ uid                          : chr  "b69ada55-41d1-41c7-9031-149c54de51f9" "9be6f7ad-4422-40ac-ba7f-e32e0184782d" "58e7aeaf-163c-4ea2-ad75-db79a580f2f8" "efe7ca02-d09e-4964-b35d-aebdac8f3efb" ...
##  $ namedLocation                : chr  "BLAN_061.phenology.phe" "BLAN_061.phenology.phe" "BLAN_061.phenology.phe" "BLAN_061.phenology.phe" ...
##  $ domainID                     : chr  "D02" "D02" "D02" "D02" ...
##  $ siteID                       : chr  "BLAN" "BLAN" "BLAN" "BLAN" ...
##  $ plotID                       : chr  "BLAN_061" "BLAN_061" "BLAN_061" "BLAN_061" ...
##  $ date                         : POSIXct, format: "2017-02-24" "2017-02-24" ...
##  $ editedDate                   : POSIXct, format: "2017-03-31" "2017-03-31" ...
##  $ dayOfYear                    : num  55 55 55 55 55 55 55 55 55 55 ...
##  $ individualID                 : chr  "NEON.PLA.D02.BLAN.06229" "NEON.PLA.D02.BLAN.06226" "NEON.PLA.D02.BLAN.06222" "NEON.PLA.D02.BLAN.06223" ...
##  $ phenophaseName               : chr  "Leaves" "Leaves" "Leaves" "Leaves" ...
##  $ phenophaseStatus             : chr  "no" "no" "no" "no" ...
##  $ phenophaseIntensityDefinition: chr  NA NA NA NA ...
##  $ phenophaseIntensity          : chr  NA NA NA NA ...
##  $ samplingProtocolVersion      : chr  NA NA NA NA ...
##  $ measuredBy                   : chr  "llemmon@neoninc.org" "llemmon@neoninc.org" "llemmon@neoninc.org" "llemmon@neoninc.org" ...
##  $ recordedBy                   : chr  "llemmon@neoninc.org" "llemmon@neoninc.org" "llemmon@neoninc.org" "llemmon@neoninc.org" ...
##  $ remarks                      : chr  NA NA NA NA ...
##  $ dataQF                       : chr  "legacyData" "legacyData" "legacyData" "legacyData" ...
##  $ publicationDate              : chr  "20201217T203824Z" "20201217T203824Z" "20201217T203824Z" "20201217T203824Z" ...
##  $ release                      : chr  "RELEASE-2021" "RELEASE-2021" "RELEASE-2021" "RELEASE-2021" ...

# date range
min(status$date)

## [1] "2017-02-24 GMT"

max(status$date)

## [1] "2019-12-12 GMT"

Clean up the Data

remove duplicates (full rows)
convert to date format
retain only the most recent editedDate in the perIndividual and status table.

Remove Duplicates

The individual table (ind) file is included in each site by month-year file. As a result when all the tables are stacked there are many duplicates.

Let's remove any duplicates that exist.

# drop UID as that will be unique for duplicate records
ind_noUID <- select(ind, -(uid))

## Error in (function (classes, fdef, mtable) : unable to find an inherited method for function 'select' for signature '"data.frame"'

status_noUID <- select(status, -(uid))

## Error in (function (classes, fdef, mtable) : unable to find an inherited method for function 'select' for signature '"data.frame"'

# remove duplicates
## expect many

ind_noD <- distinct(ind_noUID)

## Error in distinct(ind_noUID): object 'ind_noUID' not found

nrow(ind_noD)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'nrow': object 'ind_noD' not found

status_noD<-distinct(status_noUID)

## Error in distinct(status_noUID): object 'status_noUID' not found

nrow(status_noD)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'nrow': object 'status_noD' not found

Variable Overlap between Tables

From the initial inspection of the data we can see there is overlap in variable names between the fields.

Let's see what they are.

# where is there an intersection of names
intersect(names(status_noD), names(ind_noD))

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'intersect': object 'status_noD' not found

There are several fields that overlap between the datasets. Some of these are expected to be the same and will be what we join on.

However, some of these will have different values in each table. We want to keep those distinct value and not join on them. Therefore, we can rename these fields before joining:

date
editedDate
measuredBy
recordedBy
samplingProtocolVersion
remarks
dataQF
publicationDate

Now we want to rename the variables that would have duplicate names. We can rename all the variables in the status object to have "Stat" at the end of the variable name.

# in Status table rename like columns 
status_noD <- rename(status_noD, dateStat=date, 
                                         editedDateStat=editedDate, measuredByStat=measuredBy, 
                                         recordedByStat=recordedBy, 
                                         samplingProtocolVersionStat=samplingProtocolVersion, 
                                         remarksStat=remarks, dataQFStat=dataQF, 
                                         publicationDateStat=publicationDate)

## Error in rename(status_noD, dateStat = date, editedDateStat = editedDate, : object 'status_noD' not found

Filter to last editedDate

The individual (ind) table contains all instances that any of the location or taxonomy data of an individual was updated. Therefore there are many rows for some individuals. We only want the latest editedDate on ind.

# retain only the max of the date for each individualID
ind_last <- ind_noD %>%
    group_by(individualID) %>%
    filter(editedDate==max(editedDate))

## Error in eval(lhs, parent, parent): object 'ind_noD' not found

# oh wait, duplicate dates, retain only the most recent editedDate
ind_lastnoD <- ind_last %>%
    group_by(editedDate, individualID) %>%
    filter(row_number()==1)

## Error in eval(lhs, parent, parent): object 'ind_last' not found

Join Dataframes

Now we can join the two data frames on all the variables with the same name. We use a left_join() from the dpylr package because we want to match all the rows from the "left" (first) dataframe to any rows that also occur in the "right" (second) dataframe.

Check out RStudio's data wrangling (dplyr/tidyr) cheatsheet for other types of joins.

# Create a new dataframe "phe_ind" with all the data from status and some from ind_lastnoD
phe_ind <- left_join(status_noD, ind_lastnoD)

## Error in left_join(status_noD, ind_lastnoD): object 'status_noD' not found

Now that we have clean datasets we can begin looking into our particular data to address our research question: do plants show patterns of changes in phenophase across season?

Patterns in Phenophase

From our larger dataset (several sites, species, phenophases), let's create a dataframe with only the data from a single site, species, and phenophase and call it phe_1sp.

Select Site(s) of Interest

To do this, we'll first select our site of interest. Note how we set this up with an object that is our site of interest. This will allow us to more easily change which site or sites if we want to adapt our code later.

# set site of interest
siteOfInterest <- "SCBI"

# use filter to select only the site of Interest 
## using %in% allows one to add a vector if you want more than one site. 
## could also do it with == instead of %in% but won't work with vectors

phe_1st <- filter(phe_ind, siteID %in% siteOfInterest)

## Error in filter(phe_ind, siteID %in% siteOfInterest): object 'phe_ind' not found

Select Species of Interest

Now we may only want to view a single species or a set of species. Let's first look at the species that are present in our data. We could do this just by looking at the taxonID field which give the four letter UDSA plant code for each species. But if we don't know all the plant codes, we can get a bit fancier and view both

# see which species are present - taxon ID only
unique(phe_1st$taxonID)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1st' not found

# or see which species are present with taxon ID + species name
unique(paste(phe_1st$taxonID, phe_1st$scientificName, sep=' - ')) 

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1st' not found

For now, let's choose only the flowering tree Liriodendron tulipifera (LITU). By writing it this way, we could also add a list of species to the speciesOfInterest object to select for multiple species.

speciesOfInterest <- "LITU"

#subset to just "LITU"
# here just use == but could also use %in%
phe_1sp <- filter(phe_1st, taxonID==speciesOfInterest)

## Error in filter(phe_1st, taxonID == speciesOfInterest): object 'phe_1st' not found

# check that it worked
unique(phe_1sp$taxonID)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1sp' not found

Select Phenophase of Interest

And, perhaps a single phenophase.

# see which phenophases are present
unique(phe_1sp$phenophaseName)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1sp' not found

phenophaseOfInterest <- "Leaves"

#subset to just the phenosphase of interest 
phe_1sp <- filter(phe_1sp, phenophaseName %in% phenophaseOfInterest)

## Error in filter(phe_1sp, phenophaseName %in% phenophaseOfInterest): object 'phe_1sp' not found

# check that it worked
unique(phe_1sp$phenophaseName)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1sp' not found

Select only Primary Plots

NEON plant phenology observations are collected along two types of plots.

Primary plots: an 800 meter square phenology loop transect
Phenocam plots: a 200 m x 200 m plot located within view of a canopy level, tower-mounted, phenology camera

In the data, these plots are differentiated by the subtypeSpecification. Depending on your question you may want to use only one or both of these plot types. For this activity, we're going to only look at the primary plots.

Data Tip: How do I learn this on my own? Read the Data Product User Guide and use the variables files with the data download to find the corresponding variables names.

# what plots are present?
unique(phe_1sp$subtypeSpecification)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1sp' not found

# filter
phe_1spPrimary <- filter(phe_1sp, subtypeSpecification == 'primary')

## Error in filter(phe_1sp, subtypeSpecification == "primary"): object 'phe_1sp' not found

# check that it worked
unique(phe_1spPrimary$subtypeSpecification)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'phe_1spPrimary' not found

Total in Phenophase of Interest

The phenophaseState is recorded as "yes" or "no" that the individual is in that phenophase. The phenophaseIntensity are categories for how much of the individual is in that state. For now, we will stick with phenophaseState.

We can now calculate the total number of individuals with that state. We use n_distinct(indvidualID) count the individuals (and not the records) in case there are duplicate records for an individual.

But later on we'll also want to calculate the percent of the observed individuals in the "leaves" status, therefore, we're also adding in a step here to retain the sample size so that we can calculate % later.

Here we use pipes %>% from the dpylr package to "pass" objects onto the next function.

# Calculate sample size for later use
sampSize <- phe_1spPrimary %>%
  group_by(dateStat) %>%
  summarise(numInd= n_distinct(individualID))

## Error in eval(lhs, parent, parent): object 'phe_1spPrimary' not found

# Total in status by day for distinct individuals
inStat <- phe_1spPrimary%>%
  group_by(dateStat, phenophaseStatus)%>%
  summarise(countYes=n_distinct(individualID))

## Error in eval(lhs, parent, parent): object 'phe_1spPrimary' not found

inStat <- full_join(sampSize, inStat, by="dateStat")

## Error in full_join(sampSize, inStat, by = "dateStat"): object 'sampSize' not found

# Retain only Yes
inStat_T <- filter(inStat, phenophaseStatus %in% "yes")

## Error in filter(inStat, phenophaseStatus %in% "yes"): object 'inStat' not found

# check that it worked
unique(inStat_T$phenophaseStatus)

## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unique': object 'inStat_T' not found

Now that we have the data we can plot it.

Plot with ggplot

The ggplot() function within the ggplot2 package gives us considerable control over plot appearance. Three basic elements are needed for ggplot() to work:

The data_frame: containing the variables that we wish to plot,
aes (aesthetics): which denotes which variables will map to the x-, y- (and other) axes,
geom_XXXX (geometry): which defines the data's graphical representation (e.g. points (geom_point), bars (geom_bar), lines (geom_line), etc).

The syntax begins with the base statement that includes the data_frame (inStat_T) and associated x (date) and y (n) variables to be plotted:

ggplot(inStat_T, aes(date, n))

Data Tip: For a more detailed introduction to using ggplot(), visit Time Series 05: Plot Time Series with ggplot2 in R tutorial.

Bar Plots with ggplot

To successfully plot, the last piece that is needed is the geometry type. To create a bar plot, we set the geom element from to geom_bar().

The default setting for a ggplot bar plot - geom_bar() - is a histogram designated by stat="bin". However, in this case, we want to plot count values. We can use geom_bar(stat="identity") to force ggplot to plot actual values.

# plot number of individuals in leaf
phenoPlot <- ggplot(inStat_T, aes(dateStat, countYes)) +
    geom_bar(stat="identity", na.rm = TRUE) 

## Error in ggplot(inStat_T, aes(dateStat, countYes)): object 'inStat_T' not found

phenoPlot

## Error in eval(expr, envir, enclos): object 'phenoPlot' not found

# Now let's make the plot look a bit more presentable
phenoPlot <- ggplot(inStat_T, aes(dateStat, countYes)) +
    geom_bar(stat="identity", na.rm = TRUE) +
    ggtitle("Total Individuals in Leaf") +
    xlab("Date") + ylab("Number of Individuals") +
    theme(plot.title = element_text(lineheight=.8, face="bold", size = 20)) +
    theme(text = element_text(size=18))

## Error in ggplot(inStat_T, aes(dateStat, countYes)): object 'inStat_T' not found

phenoPlot

## Error in eval(expr, envir, enclos): object 'phenoPlot' not found

We could also covert this to percentage and plot that.

# convert to percent
inStat_T$percent<- ((inStat_T$countYes)/inStat_T$numInd)*100

## Error in eval(expr, envir, enclos): object 'inStat_T' not found

# plot percent of leaves
phenoPlot_P <- ggplot(inStat_T, aes(dateStat, percent)) +
    geom_bar(stat="identity", na.rm = TRUE) +
    ggtitle("Proportion in Leaf") +
    xlab("Date") + ylab("% of Individuals") +
    theme(plot.title = element_text(lineheight=.8, face="bold", size = 20)) +
    theme(text = element_text(size=18))

## Error in ggplot(inStat_T, aes(dateStat, percent)): object 'inStat_T' not found

phenoPlot_P

## Error in eval(expr, envir, enclos): object 'phenoPlot_P' not found

The plots demonstrate the nice expected pattern of increasing leaf-out, peak, and drop-off.

Drivers of Phenology

Now that we see that there are differences in and shifts in phenophases, what are the drivers of phenophases?

The NEON phenology measurements track sensitive and easily observed indicators of biotic responses to climate variability by monitoring the timing and duration of phenological stages in plant communities. Plant phenology is affected by forces such as temperature, timing and duration of pest infestations and disease outbreaks, water fluxes, nutrient budgets, carbon dynamics, and food availability and has feedbacks to trophic interactions, carbon sequestration, community composition and ecosystem function. (quoted from Plant Phenology Observations user guide.)

Filter by Date

In the next part of this series, we will be exploring temperature as a driver of phenology. Temperature date is quite large (NEON provides this in 1 minute or 30 minute intervals) so let's trim our phenology date down to only one year so that we aren't working with as large a data.

Let's filter to just 2018 data.

# use filter to select only the date of interest 
phe_1sp_2018 <- filter(inStat_T, dateStat >= "2018-01-01" & dateStat <= "2018-12-31")

## Error in filter(inStat_T, dateStat >= "2018-01-01" & dateStat <= "2018-12-31"): object 'inStat_T' not found

# did it work?
range(phe_1sp_2018$dateStat)

## Error in eval(expr, envir, enclos): object 'phe_1sp_2018' not found

How does that look?

# Now let's make the plot look a bit more presentable
phenoPlot18 <- ggplot(phe_1sp_2018, aes(dateStat, countYes)) +
    geom_bar(stat="identity", na.rm = TRUE) +
    ggtitle("Total Individuals in Leaf") +
    xlab("Date") + ylab("Number of Individuals") +
    theme(plot.title = element_text(lineheight=.8, face="bold", size = 20)) +
    theme(text = element_text(size=18))

## Error in ggplot(phe_1sp_2018, aes(dateStat, countYes)): object 'phe_1sp_2018' not found

phenoPlot18

## Error in eval(expr, envir, enclos): object 'phenoPlot18' not found

Now that we've filtered down to just the 2018 data from SCBI for LITU in leaf, we may want to save that subsetted data for another use. To do that you can write the data frame to a .csv file.

You do not need to follow this step if you are continuing on to the next tutorials in this series as you already have the data frame in your environment. Of course if you close R and then come back to it, you will need to re-load this data and instructions for that are provided in the relevant tutorials.

# Write .csv - this step is optional 
# This will write to your current working directory, change as desired.
write.csv( phe_1sp_2018 , file="NEONpheno_LITU_Leaves_SCBI_2018.csv", row.names=F)

#If you are using the downloaded example date, this code will write it to the 
# pheno data file. Note - this file is already a part of the download.

#write.csv( phe_1sp_2018 , file="NEON-pheno-temp-timeseries_v2/NEONpheno_LITU_Leaves_SCBI_2018.csv", row.names=F)

Get Lesson Code

01-explore-phenology-data.R

Work with NEON's Single-Aspirated Air Temperature Data

Authors: Lee Stanish, Megan A. Jones, Natalie Robinson

Last Updated: Apr 8, 2021

In this tutorial, we explore the NEON single-aspirated air temperature data. We then discuss how to interpret the variables, how to work with date-time and date formats, and finally how to plot the data.

This tutorial is part of a series on how to work with both discrete and continuous time series data with NEON plant phenology and temperature data products.

Objectives

After completing this activity, you will be able to:

work with "stacked" NEON Single-Aspirated Air Temperature data.
correctly format date-time data.
use dplyr functions to filter data.
plot time series data in scatter plots using ggplot function.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio loaded on your computer to complete this tutorial.

Install R Packages

neonUtilities: install.packages("neonUtilities")
ggplot2: install.packages("ggplot2")
dplyr: install.packages("dplyr")
tidyr: install.packages("tidyr")

More on Packages in R – Adapted from Software Carpentry.

Download Data

Direct Download: NEON Phenology & Temp Time Series Teaching Data Subset (v2 - 2017-2019 data) (12 MB)

Additional Resources

NEON data portal
RStudio's data wrangling (dplyr/tidyr) cheatsheet
NEONScience GitHub Organization
nneo API wrapper on CRAN
RStudio's data wrangling (dplyr/tidyr) cheatsheet
Hadley Wickham's documentation on the ggplot2 package.
Winston Chang's Cookbook for R site based on his R Graphics Cookbook text.

Explore NEON Air Temperature Data

Air temperature is continuously monitored by NEON by two methods. At terrestrial sites temperature for the top of the tower will be derived from a triple redundant aspirated air temperature sensor. This is provided as NEON data product DP1.00003.001. Single Aspirated Air Temperature Sensors (SAATS) are deployed to develop temperature profiles at the tower at NEON terrestrial sites and on the meteorological stations at NEON aquatic sites. This is provided as NEON data product DP1.00002.001. These data are also available as part of the NEON Mobile Deployment Platforms.

When designing a research project using this data, you should consult the documents associated with this or any data product and not rely solely on this summary.

Single-aspirated Air Temperature

Air temperature profiles are ascertained by deploying SAATS at various heights on NEON tower infrastructure. Air temperature at aquatic sites is measured using a single SAAT at a standard height of 3m above ground level. Air temperature for this data product is provided as one- and thirty-minute averages of 1 Hz observations. Temperature observations are made using platinum resistance thermometers, which are housed in a fan aspirated shield to reduce radiative bias. The temperature is measured in Ohms and subsequently converted to degrees Celsius during data processing. Details on the conversion can be found in the associated Algorithm Theoretic Basis Document (ATBD) for any instrumented data product.

Available Data Tables

The SAAT data product has two available data tables that are delivered for each site and month-year selected. In addition, there are several metadata files that provide you with additional useful information.

a readme with information on the data product and the download;
a variables file that defines the term descriptions, data types, and units;
a validation file with data entry validation and parsing rules; and
an XML file with machine readable metadata.

For the data tables, there are both a 1-minute average and a 30-minute average available. If you download data directly off the portal, you will get one of these files for each level on the tower for each site & month-year selected.

File Naming Conventions

It is important to understand the file names to know which file is which. The readme associated with the data provides the following information:

The file naming convention for sensor data files is NEON.DOM.SITE.DPL.PRNUM.REV.TERMS.HOR.VER.TMI.DESC

where:

DOM refers to the domain of data acquisition (D01 or D20)
SITE refers to the standardized four-character alphabetic code of the site of data acquisition.
DPL refers to the data product processing level
PRNUM refers to the data product number (see Explore Data Products.)
REV refers to the revision number of the data product. (001 = initial REV, Engineering-Grade or Provisional; 101 = initial REV, Science-Grade)
TERMS is used in data product numbering to identify a sub-product or discrete vector of metadata. Since each download file typically contains several sub-products, this field is set to 00000 in the file name to maintain consistency with the data product numbering scheme.
HOR refers to measurement locations within one horizontal plane.
VER refers to measurement locations within one vertical plane. For example, if eight temperature measurements are collected, one at each tower level, the number in the VER field would range from 010-080.
TMI is the Temporal Index; refers to the temporal representation, averaging period, or coverage of the data product (e.g., minute, hour, month, year, sub-hourly, day, lunar month, single instance, seasonal, annual, multi-annual)
DESC is an abbreviated description of the data product

Therefore, we can interpret the following .csv file name

NEON.D02.SERC.DP1.00002.001.00000.000.010.030.SAAT_30min.csv

as NEON data from Smithsonian Environmental Research Center (SERC) located in Domain 02 (D02). The specific data product is level 1 data product (DP1) and is single aspirated temperature data (00002). There have not been revisions, there are no associated terms, and there is no differentiation in horizontal plane. This data comes from the first (010) vertical level of the tower. The temporal interval is 30-minute averaged data (030; the other option in our data is 1 minute averaging. Finally there is the abbreviated description that is more human readable and tells us again that it is single-aspirated air temperature at 30 minute averages.

Access NEON Data

There are several ways to access NEON data, directly from the NEON data portal, access through a data partner (select data products only), writing code to directly pull data from the NEON API, or, as we'll use here, using the neonUtilities package which is a wrapper for the API with useful function to make working with the data easier.

Data Downloaded Direct from Portal

If you prefer to work with data that are downloaded from the data portal, please review the Getting started and Stack the downloaded data sections of the Download and Explore NEON Data tutorial. This will get you to the point where you can upload your data from sites or dates of interest and resume this tutorial.

Import Data

First, we need to set up our environment with the packages needed for this tutorial.

# Install needed package (only uncomment & run if not already installed)
#install.packages("neonUtilities")
#install.packages("ggplot2")
#install.packages("dplyr")
#install.packages("tidyr")


# Load required libraries
library(neonUtilities)  # for accessing NEON data
library(ggplot2)  # for plotting
library(dplyr)  # for data munging
library(tidyr)  # for data munging


# set working directory to ensure R can find the file we wish to import and where
# we want to save our files. Be sure to move the download into your working directory!
wd <- "~/Documents/data/" # Change this to match your local environment
setwd(wd)

Data of Interest

This tutorial is part of series working with discrete, plant phenology data and (near) continuous temperature data. Our overall "research" question is to see if there is any correlation between the plant phenology and the temperature. Therefore, we will want to work with data that aligns with the plant phenology data that we worked with in the first tutorial. If you are only interested in working with the temperature data, you do not need to complete the previous tutorial.

Our data of interest will be the temperature data from 2018 from NEON's Smithsonian Conservation Biology Institute (SCBI) field site located in Virginia near the northern terminus of the Blue Ridge Mountains.

NEON single aspirated air temperature data is available in two averaging intervals, 1 minute and 30 minute intervals. Which data you want to work with is going to depend on your research questions. Here, we're going to only download and work with the 30 minute interval data as we're primarily interest in longer term (daily, weekly, annual) patterns.

This will download 7.7 MiB of data. check.size is set to false (F) to improve flow of the script but is always a good idea to view the size with true (T) before downloading a new dataset.

If you are using the data downloaded at the start of the tutorial, use the commented out code in the second half of this code chunk.

# download data of interest - Single Aspirated Air Temperature
saat<-loadByProduct(dpID="DP1.00002.001", site="SCBI", 
                                        startdate="2018-01", enddate="2018-12", 
                                        package="basic", 
                                        avg = "30",
                                        token = Sys.getenv("NEON_TOKEN"),
                                        check.size = F)

## Input parameter avg is deprecated; use timeIndex to download by time interval.
## Finding available files
##

|
| | 0% |
|======= | 8% |
|============= | 17% |
|==================== | 25% |
|========================== | 33% |
|================================= | 42% |
|======================================== | 50% |
|============================================== | 58% |
|===================================================== | 67% |
|=========================================================== | 75% |
|================================================================== | 83% |
|======================================================================== | 92% |
|===============================================================================| 100% ## ## Downloading files totaling approximately 8.056716 MB ## Downloading 63 files ## |
| | 0% |
|= | 2% |
|=== | 3% |
|==== | 5% |
|===== | 6% |
|====== | 8% |
|======== | 10% |
|========= | 11% |
|========== | 13% |
|=========== | 15% |
|============= | 16% |
|============== | 18% |
|=============== | 19% |
|================= | 21% |
|================== | 23% |
|=================== | 24% |
|==================== | 26% |
|====================== | 27% |
|======================= | 29% |
|======================== | 31% |
|========================= | 32% |
|=========================== | 34% |
|============================ | 35% |
|============================= | 37% |
|=============================== | 39% |
|================================ | 40% |
|================================= | 42% |
|================================== | 44% |
|==================================== | 45% |
|===================================== | 47% |
|====================================== | 48% |
|======================================== | 50% |
|========================================= | 52% |
|========================================== | 53% |
|=========================================== | 55% |
|============================================= | 56% |
|============================================== | 58% |
|=============================================== | 60% |
|================================================ | 61% |
|================================================== | 63% |
|=================================================== | 65% |
|==================================================== | 66% |
|====================================================== | 68% |
|======================================================= | 69% |
|======================================================== | 71% |
|========================================================= | 73% |
|=========================================================== | 74% |
|============================================================ | 76% |
|============================================================= | 77% |
|============================================================== | 79% |
|================================================================ | 81% |
|================================================================= | 82% |
|================================================================== | 84% |
|==================================================================== | 85% |
|===================================================================== | 87% |
|====================================================================== | 89% |
|======================================================================= | 90% |
|========================================================================= | 92% |
|========================================================================== | 94% |
|=========================================================================== | 95% |
|============================================================================ | 97% |
|============================================================================== | 98% |
|===============================================================================| 100% ## ## Stacking operation across a single core. ## Stacking table SAAT_30min ## Merged the most recent publication of sensor position files for each site and saved to /stackedFiles ## Copied the most recent publication of variable definition file to /stackedFiles ## Finished: Stacked 1 data tables and 2 metadata tables! ## Stacking took 0.4314961 secs

##If choosing to use example dataset downloaded from this tutorial: 

# Stack multiple files within the downloaded phenology data
#stackByTable("NEON-pheno-temp-timeseries_v2/filesToStack00002", folder = T)

# read in data - readTableNEON uses the variables file to assign the correct
# data type for each variable
#SAAT_30min <- readTableNEON('NEON-pheno-temp-timeseries_v2/filesToStack00002/stackedFiles/SAAT_30min.csv', 'NEON-pheno-temp-timeseries_v2/filesToStack00002/stackedFiles/variables_00002.csv')

Explore Temperature Data

Now that you have the data, let's take a look at the structure and understand what's in the data. The data (saat) come in as a large list of four items.

# View the list
View(saat)

# if using the pre-downloaded data, you need to read in the variables file 
# or open and look at it on your desktop
#var <- read.csv('NEON-pheno-temp-timeseries_v2/filesToStack00002/stackedFiles/variables_00002.csv')
#View(var)

So what exactly are these four files and why would you want to use them?

data file(s): There will always be one or more dataframes that include the primary data of the data product you downloaded. Since we downloaded only the 30 minute averaged data we only have one data table SAAT_30min.
readme_xxxxx: The readme file, with the corresponding 5 digits from the data product number, provides you with important information relevant to the data product and the specific instance of downloading the data.
sensor_positions_xxxxx: this file contains information about the coordinates of each sensor, relative to a reference location.
variables_xxxxx: this file contains all the variables found in the associated data table(s). This includes full definitions, units, and other important information.

Since we want to work with the individual files, let's create individual objects from the large list. There are several ways to do this, including the following two.

# if using the pre-downloaded data - you can skip this part.
# assign individual dataFrames in the list as an object
#SAAT_30min <- saat$SAAT_30min

# unlist all objects
list2env(saat, .GlobalEnv)

## <environment: R_GlobalEnv>

Now we the four files as separate R objects. But what is in our data file?

# what is in the data?
str(SAAT_30min)

## 'data.frame':    87600 obs. of  16 variables:
##  $ domainID           : chr  "D02" "D02" "D02" "D02" ...
##  $ siteID             : chr  "SCBI" "SCBI" "SCBI" "SCBI" ...
##  $ horizontalPosition : chr  "000" "000" "000" "000" ...
##  $ verticalPosition   : chr  "010" "010" "010" "010" ...
##  $ startDateTime      : POSIXct, format: "2018-01-01 00:00:00" "2018-01-01 00:30:00" ...
##  $ endDateTime        : POSIXct, format: "2018-01-01 00:30:00" "2018-01-01 01:00:00" ...
##  $ tempSingleMean     : num  -11.8 -11.8 -12 -12.2 -12.4 ...
##  $ tempSingleMinimum  : num  -12.1 -12.2 -12.3 -12.6 -12.8 ...
##  $ tempSingleMaximum  : num  -11.4 -11.3 -11.3 -11.7 -12.1 ...
##  $ tempSingleVariance : num  0.0208 0.0315 0.0412 0.0393 0.0361 0.0289 0.0126 0.0211 0.0115 0.0022 ...
##  $ tempSingleNumPts   : num  1800 1800 1800 1800 1800 1800 1800 1800 1800 1800 ...
##  $ tempSingleExpUncert: num  0.13 0.13 0.13 0.13 0.129 ...
##  $ tempSingleStdErMean: num  0.0034 0.0042 0.0048 0.0047 0.0045 0.004 0.0026 0.0034 0.0025 0.0011 ...
##  $ finalQF            : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ publicationDate    : chr  "20200621T115323Z" "20200621T115323Z" "20200621T115323Z" "20200621T115323Z" ...
##  $ release            : chr  "undetermined" "undetermined" "undetermined" "undetermined" ...

Quality Flags

The sensor data undergo a variety of quality assurance and quality control checks. Data can pass or fail these many checks. The expanded data package includes all of these quality flags, which can allow you to decide if not passing one of the checks will significantly hamper your research and if you should therefore remove the data from your analysis. The data that we are using is the basic data package and only includes the finalQF flag.

A pass of the check is 0, while a fail is 1. Let's see if we have data with a quality flag.

# Are there quality flags in your data? Count 'em up

sum(SAAT_30min$finalQF==1)

## [1] 20501

How do we want to deal with this quality flagged data. This may depend on why it is flagged and what questions you are asking. The expanded data package will be useful for determining this.

For our demonstration purposes here we will keep the flagged data for now.

What about null (NA) data?

# Are there NA's in your data? Count 'em up
sum(is.na(SAAT_30min$tempSingleMean) )

## [1] 19616

mean(SAAT_30min$tempSingleMean)

## [1] NA

Why was there no output?

We had previously seen that there are NA values in the temperature data. Given there are NA values, R, by default, won't calculate a mean (and many other summary statistics) as the NA values could skew the data.

na.rm=TRUE

tells R to ignore them for calculation,etc

# create new dataframe without NAs
SAAT_30min_noNA <- SAAT_30min %>%
    drop_na(tempSingleMean)  # tidyr function

# alternate base R
# SAAT_30min_noNA <- SAAT_30min[!is.na(SAAT_30min$tempSingleMean),]

# did it work?
sum(is.na(SAAT_30min_noNA$tempSingleMean))

## [1] 0

Scatterplots with ggplot

We can use ggplot to create scatter plots. Which data should we plot, as we have several options?

tempSingleMean: the mean temperature for the interval
tempSingleMinimum: the minimum temperature during the interval
tempSingleMaximum: the maximum temperature for the interval

Depending on exactly what question you are asking you may prefer to use one over the other. For many applications, the mean temperature of the one or 30 minute interval will provide the best representation of the data.

Let's plot it. (This is a plot of a large amount of data. It can take 1-2 mins to process. It is not essential for completing the next steps if this takes too much of your computer memory.)

# plot temp data
tempPlot <- ggplot(SAAT_30min, aes(startDateTime, tempSingleMean)) +
    geom_point() +
    ggtitle("Single Aspirated Air Temperature") +
    xlab("Date") + ylab("Temp (C)") +
    theme(plot.title = element_text(lineheight=.8, face="bold", size = 20)) +
    theme(text = element_text(size=18))

tempPlot

## Warning: Removed 19616 rows containing missing values (geom_point).

Scatter plot of mean temperatures for the year 2018 at the Smithsonian Conservation Biology Institute (SCBI). Plotted data shows erroneous sensor readings occured during late April/May 2018.

Given all the data -- 68,000+ observations -- it took a little while for that to plot.

What patterns can you see in the data?

Something odd seems to have happened in late April/May 2018. Since it is unlikely Virginia experienced -50C during this time, these are probably erroneous sensor readings and why we should probably remove data that are flagged with those quality flags.

Right now we are also looking at all the data points in the dataset. However, we may want to view or aggregate the data differently:

aggregated data: min, mean, or max over a some duration
the number of days since a freezing temperatures
or some other segregation of the data.

Given that in the previous tutorial, Work With NEON's Plant Phenology Data, we were working with phenology data collected on a daily scale let's aggregate to that level.

To make this plot better, lets do two things

1) Remove flagged data 2) Aggregate to a daily mean.

Subset to remove quality flagged data

We previously saw a fair number of data points that were flagged. Now we'll subset the data to remove those data points.

# subset and add C to name for "clean"
SAAT_30minC <- filter(SAAT_30min_noNA, SAAT_30min_noNA$finalQF==0)

# Do any quality flags remain? Count 'em up
sum(SAAT_30minC$finalQF==1)

## [1] 0

Now we can plot it with the clean data.

# plot temp data
tempPlot <- ggplot(SAAT_30minC, aes(startDateTime, tempSingleMean)) +
    geom_point() +
    ggtitle("Single Aspirated Air Temperature") +
    xlab("Date") + ylab("Temp (C)") +
    theme(plot.title = element_text(lineheight=.8, face="bold", size = 20)) +
    theme(text = element_text(size=18))

tempPlot

Scatter plot of mean temperatures for the year 2018 at the Smithsonian Conservation Biology Institute (SCBI). Plotted data now has been cleaned of the erroneous sensor readings by filtering out flagged data.

That looks better! But we still have the 30 min data.

Aggregate Data by Day

We can use the dplyr package functions to aggregate the data. However, we have to choose what product we want from the aggregation. Again, you might want daily minimum temps, mean temperature or maximum temps depending on your question.

In the context of phenology, minimum temperatures might be very important if you are interested in a species that is very frost susceptible. Any days with a minimum temperature below 0C could dramatically change the phenophase. For other species or climates, maximum thresholds may be very import. Or you might be most interested in the daily mean.

For this tutorial, let's stick with maximum daily temperature (of the interval means).

# convert to date, easier to work with
SAAT_30minC$Date <- as.Date(SAAT_30minC$startDateTime)

# did it work
str(SAAT_30minC$Date)

##  Date[1:67099], format: "2018-01-01" "2018-01-01" "2018-01-01" "2018-01-01" "2018-01-01" "2018-01-01" ...

# max of mean temp each day
temp_day <- SAAT_30minC %>%
    group_by(Date) %>%
    distinct(Date, .keep_all=T) %>%
    mutate(dayMax=max(tempSingleMean))

Now we can plot the cleaned up daily temperature.

# plot Air Temperature Data across 2018 using daily data
tempPlot_dayMax <- ggplot(temp_day, aes(Date, dayMax)) +
    geom_point() +
    ggtitle("Daily Max Air Temperature") +
    xlab("") + ylab("Temp (C)") +
    theme(plot.title = element_text(lineheight=.8, face="bold", size = 20)) +
    theme(text = element_text(size=18))

tempPlot_dayMax

Scatter plot of daily maximum temperatures(of 30 minute interval means) for the year 2018 at the Smithsonian Conservation Biology Institute (SCBI).

Thought questions:

What do we gain by this visualization?
What do we loose over the 30 minute intervals?

ggplot - Subset by Time

Sometimes we want to scale the x- or y-axis to a particular time subset without subsetting the entire data_frame. To do this, we can define start and end times. We can then define these limits in the scale_x_date object as follows:

scale_x_date(limits=start.end) +

Let's plot just the first three months of the year.

# Define Start and end times for the subset as R objects that are the time class
startTime <- as.Date("2018-01-01")
endTime <- as.Date("2018-03-31")

# create a start and end time R object
start.end <- c(startTime,endTime)
str(start.end)

##  Date[1:2], format: "2018-01-01" "2018-03-31"

# View data for first 3 months only
# And we'll add some color for a change. 
tempPlot_dayMax3m <- ggplot(temp_day, aes(Date, dayMax)) +
           geom_point(color="blue", size=1) +  # defines what points look like
           ggtitle("Air Temperature\n Jan - March") +
           xlab("Date") + ylab("Air Temperature (C)")+ 
           (scale_x_date(limits=start.end, 
                date_breaks="1 week",
                date_labels="%b %d"))

tempPlot_dayMax3m

## Warning: Removed 268 rows containing missing values (geom_point).

Scatter plot showing daily maximum temperatures(of 30 minute interval means) from the beginning of January 2018 through the end of March 2018 at the Smithsonian Conservation Biology Institute (SCBI).

Now we have the temperature data matching our Phenology data from the previous tutorial, we want to save it to our computer to use in future analyses (or the next tutorial). This is optional if you are continuing as you already have this data in R.

# Write .csv - this step is optional 
# This will write to your current working directory, change as desired.
write.csv( temp_day , file="NEONsaat_daily_SCBI_2018.csv", row.names=F)

#If you are using the downloaded example date, this code will write it to the 
# pheno data file. Note - this file is already a part of the download.

#write.csv(temp_day , file="NEON-pheno-temp-timeseries_v2/NEONsaat_daily_SCBI_2018.csv", row.names=F)

Get Lesson Code

02-drivers-pheno-change-temp.R

Plot Continuous & Discrete Data Together

Authors: Lee Stanish, Megan A. Jones, Natalie Robinson

Last Updated: Apr 8, 2021

This tutorial discusses ways to plot plant phenology (discrete time series) and single-aspirated temperature (continuous time series) together. It uses data frames created in the first two parts of this series, Work with NEON OS & IS Data - Plant Phenology & Temperature. If you have not completed these tutorials, please download the dataset below.

Objectives

After completing this tutorial, you will be able to:

plot multiple figures together with grid.arrange()
plot only a subset of dates

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio loaded on your computer to complete this tutorial.

Install R Packages

neonUtilities: install.packages("neonUtilities")
ggplot2: install.packages("ggplot2")
dplyr: install.packages("dplyr")
gridExtra: install.packages("gridExtra")

More on Packages in R – Adapted from Software Carpentry.

Download Data

Direct Download: NEON Phenology & Temp Time Series Teaching Data Subset (v2 - 2017-2019 data) (12 MB)

To start, we need to set up our R environment. If you're continuing from the previous tutorial in this series, you'll only need to load the new packages.

# Install needed package (only uncomment & run if not already installed)
#install.packages("dplyr")
#install.packages("ggplot2")
#install.packages("scales")

# Load required libraries
library(ggplot2)
library(dplyr)
library(gridExtra)
library(scales)

options(stringsAsFactors=F) #keep strings as character type not factors

# set working directory to ensure R can find the file we wish to import and where
# we want to save our files. Be sure to move the download into your working directory!
wd <- "~/Documents/data/" # Change this to match your local environment
setwd(wd)

If you don't already have the R objects, temp_day and phe_1sp_2018, loaded you'll need to load and format those data. If you do, you can skip this code.

# Read in data -> if in series this is unnecessary
temp_day <- read.csv(paste0(wd,'NEON-pheno-temp-timeseries/NEONsaat_daily_SCBI_2018.csv'))

phe_1sp_2018 <- read.csv(paste0(wd,'NEON-pheno-temp-timeseries/NEONpheno_LITU_Leaves_SCBI_2018.csv'))

# Convert dates
temp_day$Date <- as.Date(temp_day$Date)
# use dateStat - the date the phenophase status was recorded
phe_1sp_2018$dateStat <- as.Date(phe_1sp_2018$dateStat)

Separate Plots, Same Panel

In this dataset, we have phenology and temperature data from the Smithsonian Conservation Biology Institute (SCBI) NEON field site. There are a variety of ways we may want to look at this data, including aggregated at the site level, by a single plot, or viewing all plots at the same time but in separate plots. In the Work With NEON's Plant Phenology Data and the Work with NEON's Single-Aspirated Air Temperature Data tutorials, we created separate plots of the number of individuals who had leaves at different times of the year and the temperature in 2018.

However, plot the data next to each other to aid comparisons. The grid.arrange() function from the gridExtra package can help us do this.

# first, create one plot 
phenoPlot <- ggplot(phe_1sp_2018, aes(dateStat, countYes)) +
    geom_bar(stat="identity", na.rm = TRUE) +
    ggtitle("Total Individuals in Leaf") +
    xlab("") + ylab("Number of Individuals")

# create second plot of interest
tempPlot_dayMax <- ggplot(temp_day, aes(Date, dayMax)) +
    geom_point() +
    ggtitle("Daily Max Air Temperature") +
    xlab("Date") + ylab("Temp (C)")

# Then arrange the plots - this can be done with >2 plots as well.
grid.arrange(phenoPlot, tempPlot_dayMax)

Now, we can see both plots in the same window. But, hmmm... the x-axis on both plots is kinda wonky. We want the same spacing in the scale across the year (e.g., July in one should line up with July in the other) plus we want the dates to display in the same format(e.g. 2016-07 vs. Jul vs Jul 2018).

Format Dates in Axis Labels

The date format parameter can be adjusted with scale_x_date. Let's format the x-axis ticks so they read "month" (%b) in both graphs. We will use the syntax:

scale_x_date(labels=date_format("%b"")

Rather than re-coding the entire plot, we can add the scale_x_date element to the plot object phenoPlot we just created.

Data Tip:

1) You can type ?strptime into the R console to find a list of date format conversion specifications (e.g. %b = month). Type scale_x_date for a list of parameters that allow you to format dates on the x-axis.

2) If you are working with a date & time class (e.g. POSIXct), you can use scale_x_datetime instead of scale_x_date.

# format x-axis: dates
phenoPlot <- phenoPlot + 
  (scale_x_date(breaks = date_breaks("1 month"), labels = date_format("%b")))

tempPlot_dayMax <- tempPlot_dayMax +
  (scale_x_date(breaks = date_breaks("1 month"), labels = date_format("%b")))

# New plot. 
grid.arrange(phenoPlot, tempPlot_dayMax)

But this only solves one of the problems, we still have a different range on the x-axis which makes it harder to see trends.

Align data sets with different start dates

Now let's work to align the values on the x-axis. We can do this in two ways, 1) setting the x-axis to have the same date range or 2) by filtering the dataset itself to only include the overlapping data. Depending on what you are trying to demonstrate and if you're doing additional analyses and want only the overlapping data, you may prefer one over the other. Let's try both.

Set range of x-axis

Alternatively, we can set the x-axis range for both plots by adding the limits parameter to the scale_x_date() function.

# first, lets recreate the full plot and add in the 
phenoPlot_setX <- ggplot(phe_1sp_2018, aes(dateStat, countYes)) +
    geom_bar(stat="identity", na.rm = TRUE) +
    ggtitle("Total Individuals in Leaf") +
    xlab("") + ylab("Number of Individuals") +
    scale_x_date(breaks = date_breaks("1 month"), 
                  labels = date_format("%b"),
                  limits = as.Date(c('2018-01-01','2018-12-31')))

# create second plot of interest
tempPlot_dayMax_setX <- ggplot(temp_day, aes(Date, dayMax)) +
    geom_point() +
    ggtitle("Daily Max Air Temperature") +
    xlab("Date") + ylab("Temp (C)") +
    scale_x_date(date_breaks = "1 month", 
                 labels=date_format("%b"),
                  limits = as.Date(c('2018-01-01','2018-12-31')))

# Plot
grid.arrange(phenoPlot_setX, tempPlot_dayMax_setX)

Now we can really see the pattern over the full year. This emphasizes the point that during much of the late fall, winter, and early spring none of the trees have leaves on them (or that data were not collected - this plot would not distinguish between the two).

Subset one data set to match other

Alternatively, we can simply filter the dataset with the larger date range so the we only plot the data from the overlapping dates.

# filter to only having overlapping data
temp_day_filt <- filter(temp_day, Date >= min(phe_1sp_2018$date) & 
                         Date <= max(phe_1sp_2018$date))

# Check 
range(phe_1sp_2018$date)

## [1] "2018-04-13" "2018-11-20"

range(temp_day_filt$Date)

## [1] "2018-04-13" "2018-11-20"

#plot again
tempPlot_dayMaxFiltered <- ggplot(temp_day_filt, aes(Date, dayMax)) +
    geom_point() +
    scale_x_date(breaks = date_breaks("months"), labels = date_format("%b")) +
    ggtitle("Daily Max Air Temperature") +
    xlab("Date") + ylab("Temp (C)")


grid.arrange(phenoPlot, tempPlot_dayMaxFiltered)

With this plot, we really look at the area of overlap in the plotted data (but this does cut out the time where the data are collected but not plotted).

Same plot with two Y-axes

What about layering these plots and having two y-axes (right and left) that have the different scale bars?

Some argue that you should not do this as it can distort what is actually going on with the data. The author of the ggplot2 package is one of these individuals. Therefore, you cannot use ggplot() to create a single plot with multiple y-axis scales. You can read his own discussion of the topic on this StackOverflow post.

However, individuals have found work arounds for these plots. The code below is provided as a demonstration of this capability. Note, by showing this code here, we don't necessarily endorse having plots with two y-axes.

This code is adapted from code by Jake Heare.

# Source: http://heareresearch.blogspot.com/2014/10/10-30-2014-dual-y-axis-graph-ggplot2_30.html

# Additional packages needed
library(gtable)
library(grid)


# Plot 1: Pheno data as bars, temp as scatter
grid.newpage()
phenoPlot_2 <- ggplot(phe_1sp_2018, aes(dateStat, countYes)) +
  geom_bar(stat="identity", na.rm = TRUE) +
  scale_x_date(breaks = date_breaks("1 month"), labels = date_format("%b")) +
  ggtitle("Total Individuals in Leaf vs. Temp (C)") +
  xlab(" ") + ylab("Number of Individuals") +
  theme_bw()+
  theme(legend.justification=c(0,1),
        legend.position=c(0,1),
        plot.title=element_text(size=25,vjust=1),
        axis.text.x=element_text(size=20),
        axis.text.y=element_text(size=20),
        axis.title.x=element_text(size=20),
        axis.title.y=element_text(size=20))


tempPlot_dayMax_corr_2 <- ggplot() +
  geom_point(data = temp_day_filt, aes(Date, dayMax),color="red") +
  scale_x_date(breaks = date_breaks("months"), labels = date_format("%b")) +
  xlab("") + ylab("Temp (C)") +
  theme_bw() %+replace% 
  theme(panel.background = element_rect(fill = NA),
        panel.grid.major.x=element_blank(),
        panel.grid.minor.x=element_blank(),
        panel.grid.major.y=element_blank(),
        panel.grid.minor.y=element_blank(),
        axis.text.y=element_text(size=20,color="red"),
        axis.title.y=element_text(size=20))

g1<-ggplot_gtable(ggplot_build(phenoPlot_2))
g2<-ggplot_gtable(ggplot_build(tempPlot_dayMax_corr_2))

pp<-c(subset(g1$layout,name=="panel",se=t:r))
g<-gtable_add_grob(g1, g2$grobs[[which(g2$layout$name=="panel")]],pp$t,pp$l,pp$b,pp$l)

ia<-which(g2$layout$name=="axis-l")
ga <- g2$grobs[[ia]]
ax <- ga$children[[2]]
ax$widths <- rev(ax$widths)
ax$grobs <- rev(ax$grobs)
ax$grobs[[1]]$x <- ax$grobs[[1]]$x - unit(1, "npc") + unit(0.15, "cm")
g <- gtable_add_cols(g, g2$widths[g2$layout[ia, ]$l], length(g$widths) - 1)
g <- gtable_add_grob(g, ax, pp$t, length(g$widths) - 1, pp$b)

grid.draw(g)

# Plot 2: Both pheno data and temp data as line graphs
grid.newpage()
phenoPlot_3 <- ggplot(phe_1sp_2018, aes(dateStat, countYes)) +
  geom_line(na.rm = TRUE) +
  scale_x_date(breaks = date_breaks("months"), labels = date_format("%b")) +
  ggtitle("Total Individuals in Leaf vs. Temp (C)") +
  xlab("Date") + ylab("Number of Individuals") +
  theme_bw()+
  theme(legend.justification=c(0,1),
        legend.position=c(0,1),
        plot.title=element_text(size=25,vjust=1),
        axis.text.x=element_text(size=20),
        axis.text.y=element_text(size=20),
        axis.title.x=element_text(size=20),
        axis.title.y=element_text(size=20))

tempPlot_dayMax_corr_3 <- ggplot() +
  geom_line(data = temp_day_filt, aes(Date, dayMax),color="red") +
  scale_x_date(breaks = date_breaks("months"), labels = date_format("%b")) +
  xlab("") + ylab("Temp (C)") +
  theme_bw() %+replace% 
  theme(panel.background = element_rect(fill = NA),
        panel.grid.major.x=element_blank(),
        panel.grid.minor.x=element_blank(),
        panel.grid.major.y=element_blank(),
        panel.grid.minor.y=element_blank(),
        axis.text.y=element_text(size=20,color="red"),
        axis.title.y=element_text(size=20))

g1<-ggplot_gtable(ggplot_build(phenoPlot_3))
g2<-ggplot_gtable(ggplot_build(tempPlot_dayMax_corr_3))

pp<-c(subset(g1$layout,name=="panel",se=t:r))
g<-gtable_add_grob(g1, g2$grobs[[which(g2$layout$name=="panel")]],pp$t,pp$l,pp$b,pp$l)

ia<-which(g2$layout$name=="axis-l")
ga <- g2$grobs[[ia]]
ax <- ga$children[[2]]
ax$widths <- rev(ax$widths)
ax$grobs <- rev(ax$grobs)
ax$grobs[[1]]$x <- ax$grobs[[1]]$x - unit(1, "npc") + unit(0.15, "cm")
g <- gtable_add_cols(g, g2$widths[g2$layout[ia, ]$l], length(g$widths) - 1)
g <- gtable_add_grob(g, ax, pp$t, length(g$widths) - 1, pp$b)

grid.draw(g)

Get Lesson Code

03-plot-discrete-continuous-data-pheno-temp.R

Introduction to working with NEON eddy flux data

Authors: [Claire K. Lunch]

Last Updated: Mar 12, 2021

This data tutorial provides an introduction to working with NEON eddy flux data, using the neonUtilities R package. If you are new to NEON data, we recommend starting with a more general tutorial, such as the neonUtilities tutorial or the Download and Explore tutorial. Some of the functions and techniques described in those tutorials will be used here, as well as functions and data formats that are unique to the eddy flux system.

This tutorial assumes general familiarity with eddy flux data and associated concepts.

1. Setup

Start by installing and loading packages and setting options. To work with the NEON flux data, we need the rhdf5 package, which is hosted on Bioconductor, and requires a different installation process than CRAN packages:

install.packages('BiocManager')
BiocManager::install('rhdf5')
install.packages('neonUtilities')




options(stringsAsFactors=F)

library(neonUtilities)

Use the zipsByProduct() function from the neonUtilities package to download flux data from two sites and two months. The transformations and functions below will work on any time range and site(s), but two sites and two months allows us to see all the available functionality while minimizing download size.

Inputs to the zipsByProduct() function:

dpID: DP4.00200.001, the bundled eddy covariance product
package: basic (the expanded package is not covered in this tutorial)
site: NIWO = Niwot Ridge and HARV = Harvard Forest
startdate: 2018-06 (both dates are inclusive)
enddate: 2018-07 (both dates are inclusive)
savepath: modify this to something logical on your machine
check.size: T if you want to see file size before downloading, otherwise F

The download may take a while, especially if you're on a slow network. For faster downloads, consider using an API token.

zipsByProduct(dpID="DP4.00200.001", package="basic", 
              site=c("NIWO", "HARV"), 
              startdate="2018-06", enddate="2018-07",
              savepath="~/Downloads", 
              check.size=F)

2. Data Levels

There are five levels of data contained in the eddy flux bundle. For full details, refer to the NEON algorithm document.

Briefly, the data levels are:

Level 0' (dp0p): Calibrated raw observations
Level 1 (dp01): Time-aggregated observations, e.g. 30-minute mean gas concentrations
Level 2 (dp02): Time-interpolated data, e.g. rate of change of a gas concentration
Level 3 (dp03): Spatially interpolated data, i.e. vertical profiles
Level 4 (dp04): Fluxes

The dp0p data are available in the expanded data package and are beyond the scope of this tutorial.

The dp02 and dp03 data are used in storage calculations, and the dp04 data include both the storage and turbulent components. Since many users will want to focus on the net flux data, we'll start there.

3. Extract Level 4 data (Fluxes!)

To extract the Level 4 data from the HDF5 files and merge them into a single table, we'll use the stackEddy() function from the neonUtilities package.

stackEddy() requires two inputs:

filepath: Path to a file or folder, which can be any one of:
1. A zip file of eddy flux data downloaded from the NEON data portal
2. A folder of eddy flux data downloaded by the zipsByProduct() function
3. The folder of files resulting from unzipping either of 1 or 2
4. One or more HDF5 files of NEON eddy flux data
level: dp01-4

Input the filepath you downloaded to using zipsByProduct() earlier, including the filestoStack00200 folder created by the function, and dp04:

flux <- stackEddy(filepath="~/Downloads/filesToStack00200",
                 level="dp04")

We now have an object called flux. It's a named list containing four tables: one table for each site's data, and variables and objDesc tables.

names(flux)

## [1] "HARV"      "NIWO"      "variables" "objDesc"

Let's look at the contents of one of the site data files:

head(flux$NIWO)

##               timeBgn             timeEnd data.fluxCo2.nsae.flux data.fluxCo2.stor.flux data.fluxCo2.turb.flux
## 1 2018-06-01 00:00:00 2018-06-01 00:29:59              0.1713858            -0.06348163              0.2348674
## 2 2018-06-01 00:30:00 2018-06-01 00:59:59              0.9251711             0.08748146              0.8376896
## 3 2018-06-01 01:00:00 2018-06-01 01:29:59              0.5005812             0.02231698              0.4782642
## 4 2018-06-01 01:30:00 2018-06-01 01:59:59              0.8032820             0.25569306              0.5475889
## 5 2018-06-01 02:00:00 2018-06-01 02:29:59              0.4897685             0.23090472              0.2588638
## 6 2018-06-01 02:30:00 2018-06-01 02:59:59              0.9223979             0.06228581              0.8601121
##   data.fluxH2o.nsae.flux data.fluxH2o.stor.flux data.fluxH2o.turb.flux data.fluxMome.turb.veloFric
## 1              15.876622              3.3334970              12.543125                   0.2047081
## 2               8.089274             -1.2063258               9.295600                   0.1923735
## 3               5.290594             -4.4190781               9.709672                   0.1200918
## 4               9.190214              0.2030371               8.987177                   0.1177545
## 5               3.111909              0.1349363               2.976973                   0.1589189
## 6               4.613676             -0.3929445               5.006621                   0.1114406
##   data.fluxTemp.nsae.flux data.fluxTemp.stor.flux data.fluxTemp.turb.flux data.foot.stat.angZaxsErth
## 1               4.7565505              -1.4575094               6.2140599                    94.2262
## 2              -0.2717454               0.3403877              -0.6121331                   355.4252
## 3              -4.2055147               0.1870677              -4.3925824                   359.8013
## 4             -13.3834484              -2.4904300             -10.8930185                   137.7743
## 5              -5.1854815              -0.7514531              -4.4340284                   188.4799
## 6              -7.7365481              -1.9046775              -5.8318707                   183.1920
##   data.foot.stat.distReso data.foot.stat.veloYaxsHorSd data.foot.stat.veloZaxsHorSd data.foot.stat.veloFric
## 1                    8.34                    0.7955893                    0.2713232               0.2025427
## 2                    8.34                    0.8590177                    0.2300000               0.2000000
## 3                    8.34                    1.2601763                    0.2300000               0.2000000
## 4                    8.34                    0.7332641                    0.2300000               0.2000000
## 5                    8.34                    0.7096286                    0.2300000               0.2000000
## 6                    8.34                    0.3789859                    0.2300000               0.2000000
##   data.foot.stat.distZaxsMeasDisp data.foot.stat.distZaxsRgh data.foot.stat.distZaxsAbl
## 1                            8.34                 0.04105708                       1000
## 2                            8.34                 0.27991938                       1000
## 3                            8.34                 0.21293225                       1000
## 4                            8.34                 0.83400000                       1000
## 5                            8.34                 0.83400000                       1000
## 6                            8.34                 0.83400000                       1000
##   data.foot.stat.distXaxs90 data.foot.stat.distXaxsMax data.foot.stat.distYaxs90 qfqm.fluxCo2.nsae.qfFinl
## 1                    325.26                     133.44                     25.02                        1
## 2                    266.88                     108.42                     50.04                        1
## 3                    275.22                     116.76                     66.72                        1
## 4                    208.50                      83.40                     75.06                        1
## 5                    208.50                      83.40                     66.72                        1
## 6                    208.50                      83.40                     41.70                        1
##   qfqm.fluxCo2.stor.qfFinl qfqm.fluxCo2.turb.qfFinl qfqm.fluxH2o.nsae.qfFinl qfqm.fluxH2o.stor.qfFinl
## 1                        1                        1                        1                        1
## 2                        1                        1                        1                        0
## 3                        1                        1                        1                        0
## 4                        1                        1                        1                        0
## 5                        1                        1                        1                        0
## 6                        1                        1                        1                        1
##   qfqm.fluxH2o.turb.qfFinl qfqm.fluxMome.turb.qfFinl qfqm.fluxTemp.nsae.qfFinl qfqm.fluxTemp.stor.qfFinl
## 1                        1                         0                         0                         0
## 2                        1                         0                         1                         0
## 3                        1                         1                         0                         0
## 4                        1                         1                         0                         0
## 5                        1                         0                         0                         0
## 6                        1                         0                         0                         0
##   qfqm.fluxTemp.turb.qfFinl qfqm.foot.turb.qfFinl
## 1                         0                     0
## 2                         1                     0
## 3                         0                     0
## 4                         0                     0
## 5                         0                     0
## 6                         0                     0

The variables and objDesc tables can help you interpret the column headers in the data table. The objDesc table contains definitions for many of the terms used in the eddy flux data product, but it isn't complete. To get the terms of interest, we'll break up the column headers into individual terms and look for them in the objDesc table:

term <- unlist(strsplit(names(flux$NIWO), split=".", fixed=T))
flux$objDesc[which(flux$objDesc$Object %in% term),]

##          Object
## 138 angZaxsErth
## 171        data
## 343      qfFinl
## 420        qfqm
## 604     timeBgn
## 605     timeEnd
##                                                                                                         Description
## 138                                                                                                 Wind direction 
## 171                                                                                          Represents data fields
## 343       The final quality flag indicating if the data are valid for the given aggregation period (1=fail, 0=pass)
## 420 Quality flag and quality metrics, represents quality flags and quality metrics that accompany the provided data
## 604                                                                    The beginning time of the aggregation period
## 605                                                                          The end time of the aggregation period

For the terms that aren't captured here, fluxCo2, fluxH2o, and fluxTemp are self-explanatory. The flux components are

turb: Turbulent flux
stor: Storage
nsae: Net surface-atmosphere exchange

The variables table contains the units for each field:

flux$variables

##    category   system variable             stat           units
## 1      data  fluxCo2     nsae          timeBgn              NA
## 2      data  fluxCo2     nsae          timeEnd              NA
## 3      data  fluxCo2     nsae             flux umolCo2 m-2 s-1
## 4      data  fluxCo2     stor          timeBgn              NA
## 5      data  fluxCo2     stor          timeEnd              NA
## 6      data  fluxCo2     stor             flux umolCo2 m-2 s-1
## 7      data  fluxCo2     turb          timeBgn              NA
## 8      data  fluxCo2     turb          timeEnd              NA
## 9      data  fluxCo2     turb             flux umolCo2 m-2 s-1
## 10     data  fluxH2o     nsae          timeBgn              NA
## 11     data  fluxH2o     nsae          timeEnd              NA
## 12     data  fluxH2o     nsae             flux           W m-2
## 13     data  fluxH2o     stor          timeBgn              NA
## 14     data  fluxH2o     stor          timeEnd              NA
## 15     data  fluxH2o     stor             flux           W m-2
## 16     data  fluxH2o     turb          timeBgn              NA
## 17     data  fluxH2o     turb          timeEnd              NA
## 18     data  fluxH2o     turb             flux           W m-2
## 19     data fluxMome     turb          timeBgn              NA
## 20     data fluxMome     turb          timeEnd              NA
## 21     data fluxMome     turb         veloFric           m s-1
## 22     data fluxTemp     nsae          timeBgn              NA
## 23     data fluxTemp     nsae          timeEnd              NA
## 24     data fluxTemp     nsae             flux           W m-2
## 25     data fluxTemp     stor          timeBgn              NA
## 26     data fluxTemp     stor          timeEnd              NA
## 27     data fluxTemp     stor             flux           W m-2
## 28     data fluxTemp     turb          timeBgn              NA
## 29     data fluxTemp     turb          timeEnd              NA
## 30     data fluxTemp     turb             flux           W m-2
## 31     data     foot     stat          timeBgn              NA
## 32     data     foot     stat          timeEnd              NA
## 33     data     foot     stat      angZaxsErth             deg
## 34     data     foot     stat         distReso               m
## 35     data     foot     stat    veloYaxsHorSd           m s-1
## 36     data     foot     stat    veloZaxsHorSd           m s-1
## 37     data     foot     stat         veloFric           m s-1
## 38     data     foot     stat distZaxsMeasDisp               m
## 39     data     foot     stat      distZaxsRgh               m
## 40     data     foot     stat      distZaxsAbl               m
## 41     data     foot     stat       distXaxs90               m
## 42     data     foot     stat      distXaxsMax               m
## 43     data     foot     stat       distYaxs90               m
## 44     qfqm  fluxCo2     nsae          timeBgn              NA
## 45     qfqm  fluxCo2     nsae          timeEnd              NA
## 46     qfqm  fluxCo2     nsae           qfFinl              NA
## 47     qfqm  fluxCo2     stor           qfFinl              NA
## 48     qfqm  fluxCo2     stor          timeBgn              NA
## 49     qfqm  fluxCo2     stor          timeEnd              NA
## 50     qfqm  fluxCo2     turb          timeBgn              NA
## 51     qfqm  fluxCo2     turb          timeEnd              NA
## 52     qfqm  fluxCo2     turb           qfFinl              NA
## 53     qfqm  fluxH2o     nsae          timeBgn              NA
## 54     qfqm  fluxH2o     nsae          timeEnd              NA
## 55     qfqm  fluxH2o     nsae           qfFinl              NA
## 56     qfqm  fluxH2o     stor           qfFinl              NA
## 57     qfqm  fluxH2o     stor          timeBgn              NA
## 58     qfqm  fluxH2o     stor          timeEnd              NA
## 59     qfqm  fluxH2o     turb          timeBgn              NA
## 60     qfqm  fluxH2o     turb          timeEnd              NA
## 61     qfqm  fluxH2o     turb           qfFinl              NA
## 62     qfqm fluxMome     turb          timeBgn              NA
## 63     qfqm fluxMome     turb          timeEnd              NA
## 64     qfqm fluxMome     turb           qfFinl              NA
## 65     qfqm fluxTemp     nsae          timeBgn              NA
## 66     qfqm fluxTemp     nsae          timeEnd              NA
## 67     qfqm fluxTemp     nsae           qfFinl              NA
## 68     qfqm fluxTemp     stor           qfFinl              NA
## 69     qfqm fluxTemp     stor          timeBgn              NA
## 70     qfqm fluxTemp     stor          timeEnd              NA
## 71     qfqm fluxTemp     turb          timeBgn              NA
## 72     qfqm fluxTemp     turb          timeEnd              NA
## 73     qfqm fluxTemp     turb           qfFinl              NA
## 74     qfqm     foot     turb          timeBgn              NA
## 75     qfqm     foot     turb          timeEnd              NA
## 76     qfqm     foot     turb           qfFinl              NA

Let's plot some data! First, a brief aside about time stamps, since these are time series data.

Time stamps

NEON sensor data come with time stamps for both the start and end of the averaging period. Depending on the analysis you're doing, you may want to use one or the other; for general plotting, re-formatting, and transformations, I prefer to use the start time, because there are some small inconsistencies between data products in a few of the end time stamps.

Note that all NEON data use UTC time, aka Greenwich Mean Time. This is true across NEON's instrumented, observational, and airborne measurements. When working with NEON data, it's best to keep everything in UTC as much as possible, otherwise it's very easy to end up with data in mismatched times, which can cause insidious and hard-to-detect problems. In the code below, time stamps and time zones have been handled by stackEddy() and loadByProduct(), so we don't need to do anything additional. But if you're writing your own code and need to convert times, remember that if the time zone isn't specified, R will default to the local time zone it detects on your operating system.

plot(flux$NIWO$data.fluxCo2.nsae.flux~flux$NIWO$timeBgn, 
     pch=".", xlab="Date", ylab="CO2 flux")

There is a clear diurnal pattern, and an increase in daily carbon uptake as the growing season progresses.

Let's trim down to just two days of data to see a few other details.

plot(flux$NIWO$data.fluxCo2.nsae.flux~flux$NIWO$timeBgn, 
     pch=20, xlab="Date", ylab="CO2 flux",
     xlim=c(as.POSIXct("2018-07-07", tz="GMT"),
            as.POSIXct("2018-07-09", tz="GMT")),
    ylim=c(-20,20), xaxt="n")
axis.POSIXct(1, x=flux$NIWO$timeBgn, 
             format="%Y-%m-%d %H:%M:%S")

Note the timing of C uptake; the UTC time zone is clear here, where uptake occurs at times that appear to be during the night.

4. Merge flux data with other sensor data

Many of the data sets we would use to interpret and model flux data are measured as part of the NEON project, but are not present in the eddy flux data product bundle. In this section, we'll download PAR data and merge them with the flux data; the steps taken here can be applied to any of the NEON instrumented (IS) data products.

Download PAR data

To get NEON PAR data, use the loadByProduct() function from the neonUtilities package. loadByProduct() takes the same inputs as zipsByProduct(), but it loads the downloaded data directly into the current R environment.

Let's download PAR data matching the Niwot Ridge flux data. The inputs needed are:

dpID: DP1.00024.001
site: NIWO
startdate: 2018-06
enddate: 2018-07
package: basic
timeIndex: 30

The new input here is timeIndex=30, which downloads only the 30-minute data. Since the flux data are at a 30-minute resolution, we can save on download time by disregarding the 1-minute data files (which are of course 30 times larger). The timeIndex input can be left off if you want to download all available averaging intervals.

pr <- loadByProduct("DP1.00024.001", site="NIWO", 
                    timeIndex=30, package="basic", 
                    startdate="2018-06", enddate="2018-07",
                    check.size=F)

pr is another named list, and again, metadata and units can be found in the variables table. The PARPAR_30min table contains a verticalPosition field. This field indicates the position on the tower, with 10 being the first tower level, and 20, 30, etc going up the tower.

Join PAR to flux data

We'll connect PAR data from the tower top to the flux data.

pr.top <- pr$PARPAR_30min[which(pr$PARPAR_30min$verticalPosition==
                                max(pr$PARPAR_30min$verticalPosition)),]

As noted above, loadByProduct() automatically converts time stamps to a recognized date-time format when it reads the data. However, the field names for the time stamps differ between the flux data and the other meteorological data: the start of the averaging interval is timeBgn in the flux data and startDateTime in the PAR data.

Let's create a new variable in the PAR data:

pr.top$timeBgn <- pr.top$startDateTime

And now use the matching time stamp fields to merge the flux and PAR data.

fx.pr <- merge(pr.top, flux$NIWO, by="timeBgn")

And now we can plot net carbon exchange as a function of light availability:

plot(fx.pr$data.fluxCo2.nsae.flux~fx.pr$PARMean,
     pch=".", ylim=c(-20,20),
     xlab="PAR", ylab="CO2 flux")

If you're interested in data in the eddy covariance bundle besides the net flux data, the rest of this tutorial will guide you through how to get those data out of the bundle.

5. Vertical profile data (Level 3)

The Level 3 (dp03) data are the spatially interpolated profiles of the rates of change of CO₂, H₂O, and temperature. Extract the Level 3 data from the HDF5 file using stackEddy() with the same syntax as for the Level 4 data.

prof <- stackEddy(filepath="~/Downloads/filesToStack00200/",
                 level="dp03")

As with the Level 4 data, the result is a named list with data tables for each site.

head(prof$NIWO)

##      timeBgn             timeEnd data.co2Stor.rateRtioMoleDryCo2.X0.1.m data.co2Stor.rateRtioMoleDryCo2.X0.2.m
## 1 2018-06-01 2018-06-01 00:29:59                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X0.3.m data.co2Stor.rateRtioMoleDryCo2.X0.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X0.5.m data.co2Stor.rateRtioMoleDryCo2.X0.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X0.7.m data.co2Stor.rateRtioMoleDryCo2.X0.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X0.9.m data.co2Stor.rateRtioMoleDryCo2.X1.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X1.1.m data.co2Stor.rateRtioMoleDryCo2.X1.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X1.3.m data.co2Stor.rateRtioMoleDryCo2.X1.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X1.5.m data.co2Stor.rateRtioMoleDryCo2.X1.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X1.7.m data.co2Stor.rateRtioMoleDryCo2.X1.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X1.9.m data.co2Stor.rateRtioMoleDryCo2.X2.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X2.1.m data.co2Stor.rateRtioMoleDryCo2.X2.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X2.3.m data.co2Stor.rateRtioMoleDryCo2.X2.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X2.5.m data.co2Stor.rateRtioMoleDryCo2.X2.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X2.7.m data.co2Stor.rateRtioMoleDryCo2.X2.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X2.9.m data.co2Stor.rateRtioMoleDryCo2.X3.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X3.1.m data.co2Stor.rateRtioMoleDryCo2.X3.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X3.3.m data.co2Stor.rateRtioMoleDryCo2.X3.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X3.5.m data.co2Stor.rateRtioMoleDryCo2.X3.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X3.7.m data.co2Stor.rateRtioMoleDryCo2.X3.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X3.9.m data.co2Stor.rateRtioMoleDryCo2.X4.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X4.1.m data.co2Stor.rateRtioMoleDryCo2.X4.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X4.3.m data.co2Stor.rateRtioMoleDryCo2.X4.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X4.5.m data.co2Stor.rateRtioMoleDryCo2.X4.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X4.7.m data.co2Stor.rateRtioMoleDryCo2.X4.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X4.9.m data.co2Stor.rateRtioMoleDryCo2.X5.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X5.1.m data.co2Stor.rateRtioMoleDryCo2.X5.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X5.3.m data.co2Stor.rateRtioMoleDryCo2.X5.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X5.5.m data.co2Stor.rateRtioMoleDryCo2.X5.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X5.7.m data.co2Stor.rateRtioMoleDryCo2.X5.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X5.9.m data.co2Stor.rateRtioMoleDryCo2.X6.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X6.1.m data.co2Stor.rateRtioMoleDryCo2.X6.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X6.3.m data.co2Stor.rateRtioMoleDryCo2.X6.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X6.5.m data.co2Stor.rateRtioMoleDryCo2.X6.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X6.7.m data.co2Stor.rateRtioMoleDryCo2.X6.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X6.9.m data.co2Stor.rateRtioMoleDryCo2.X7.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X7.1.m data.co2Stor.rateRtioMoleDryCo2.X7.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X7.3.m data.co2Stor.rateRtioMoleDryCo2.X7.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X7.5.m data.co2Stor.rateRtioMoleDryCo2.X7.6.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X7.7.m data.co2Stor.rateRtioMoleDryCo2.X7.8.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X7.9.m data.co2Stor.rateRtioMoleDryCo2.X8.m
## 1                          -0.0002681938                        -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X8.1.m data.co2Stor.rateRtioMoleDryCo2.X8.2.m
## 1                          -0.0002681938                          -0.0002681938
##   data.co2Stor.rateRtioMoleDryCo2.X8.3.m data.co2Stor.rateRtioMoleDryCo2.X8.4.m
## 1                          -0.0002681938                          -0.0002681938
##   data.h2oStor.rateRtioMoleDryH2o.X0.1.m data.h2oStor.rateRtioMoleDryH2o.X0.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X0.3.m data.h2oStor.rateRtioMoleDryH2o.X0.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X0.5.m data.h2oStor.rateRtioMoleDryH2o.X0.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X0.7.m data.h2oStor.rateRtioMoleDryH2o.X0.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X0.9.m data.h2oStor.rateRtioMoleDryH2o.X1.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X1.1.m data.h2oStor.rateRtioMoleDryH2o.X1.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X1.3.m data.h2oStor.rateRtioMoleDryH2o.X1.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X1.5.m data.h2oStor.rateRtioMoleDryH2o.X1.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X1.7.m data.h2oStor.rateRtioMoleDryH2o.X1.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X1.9.m data.h2oStor.rateRtioMoleDryH2o.X2.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X2.1.m data.h2oStor.rateRtioMoleDryH2o.X2.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X2.3.m data.h2oStor.rateRtioMoleDryH2o.X2.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X2.5.m data.h2oStor.rateRtioMoleDryH2o.X2.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X2.7.m data.h2oStor.rateRtioMoleDryH2o.X2.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X2.9.m data.h2oStor.rateRtioMoleDryH2o.X3.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X3.1.m data.h2oStor.rateRtioMoleDryH2o.X3.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X3.3.m data.h2oStor.rateRtioMoleDryH2o.X3.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X3.5.m data.h2oStor.rateRtioMoleDryH2o.X3.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X3.7.m data.h2oStor.rateRtioMoleDryH2o.X3.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X3.9.m data.h2oStor.rateRtioMoleDryH2o.X4.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X4.1.m data.h2oStor.rateRtioMoleDryH2o.X4.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X4.3.m data.h2oStor.rateRtioMoleDryH2o.X4.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X4.5.m data.h2oStor.rateRtioMoleDryH2o.X4.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X4.7.m data.h2oStor.rateRtioMoleDryH2o.X4.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X4.9.m data.h2oStor.rateRtioMoleDryH2o.X5.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X5.1.m data.h2oStor.rateRtioMoleDryH2o.X5.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X5.3.m data.h2oStor.rateRtioMoleDryH2o.X5.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X5.5.m data.h2oStor.rateRtioMoleDryH2o.X5.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X5.7.m data.h2oStor.rateRtioMoleDryH2o.X5.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X5.9.m data.h2oStor.rateRtioMoleDryH2o.X6.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X6.1.m data.h2oStor.rateRtioMoleDryH2o.X6.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X6.3.m data.h2oStor.rateRtioMoleDryH2o.X6.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X6.5.m data.h2oStor.rateRtioMoleDryH2o.X6.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X6.7.m data.h2oStor.rateRtioMoleDryH2o.X6.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X6.9.m data.h2oStor.rateRtioMoleDryH2o.X7.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X7.1.m data.h2oStor.rateRtioMoleDryH2o.X7.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X7.3.m data.h2oStor.rateRtioMoleDryH2o.X7.4.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X7.5.m data.h2oStor.rateRtioMoleDryH2o.X7.6.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X7.7.m data.h2oStor.rateRtioMoleDryH2o.X7.8.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X7.9.m data.h2oStor.rateRtioMoleDryH2o.X8.m
## 1                            0.000315911                          0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X8.1.m data.h2oStor.rateRtioMoleDryH2o.X8.2.m
## 1                            0.000315911                            0.000315911
##   data.h2oStor.rateRtioMoleDryH2o.X8.3.m data.h2oStor.rateRtioMoleDryH2o.X8.4.m data.tempStor.rateTemp.X0.1.m
## 1                            0.000315911                            0.000315911                 -0.0001014444
##   data.tempStor.rateTemp.X0.2.m data.tempStor.rateTemp.X0.3.m data.tempStor.rateTemp.X0.4.m
## 1                 -0.0001014444                 -0.0001014444                 -0.0001014444
##   data.tempStor.rateTemp.X0.5.m data.tempStor.rateTemp.X0.6.m data.tempStor.rateTemp.X0.7.m
## 1                 -0.0001014444                 -0.0001050874                  -0.000111159
##   data.tempStor.rateTemp.X0.8.m data.tempStor.rateTemp.X0.9.m data.tempStor.rateTemp.X1.m
## 1                 -0.0001172305                 -0.0001233021               -0.0001293737
##   data.tempStor.rateTemp.X1.1.m data.tempStor.rateTemp.X1.2.m data.tempStor.rateTemp.X1.3.m
## 1                 -0.0001354453                 -0.0001415168                 -0.0001475884
##   data.tempStor.rateTemp.X1.4.m data.tempStor.rateTemp.X1.5.m data.tempStor.rateTemp.X1.6.m
## 1                   -0.00015366                 -0.0001597315                 -0.0001658031
##   data.tempStor.rateTemp.X1.7.m data.tempStor.rateTemp.X1.8.m data.tempStor.rateTemp.X1.9.m
## 1                 -0.0001718747                 -0.0001779463                 -0.0001840178
##   data.tempStor.rateTemp.X2.m data.tempStor.rateTemp.X2.1.m data.tempStor.rateTemp.X2.2.m
## 1                -0.000185739                 -0.0001869767                 -0.0001882144
##   data.tempStor.rateTemp.X2.3.m data.tempStor.rateTemp.X2.4.m data.tempStor.rateTemp.X2.5.m
## 1                 -0.0001894521                 -0.0001906899                 -0.0001919276
##   data.tempStor.rateTemp.X2.6.m data.tempStor.rateTemp.X2.7.m data.tempStor.rateTemp.X2.8.m
## 1                 -0.0001931653                 -0.0001944031                 -0.0001956408
##   data.tempStor.rateTemp.X2.9.m data.tempStor.rateTemp.X3.m data.tempStor.rateTemp.X3.1.m
## 1                 -0.0001968785               -0.0001981162                  -0.000199354
##   data.tempStor.rateTemp.X3.2.m data.tempStor.rateTemp.X3.3.m data.tempStor.rateTemp.X3.4.m
## 1                 -0.0002005917                 -0.0002018294                 -0.0002030672
##   data.tempStor.rateTemp.X3.5.m data.tempStor.rateTemp.X3.6.m data.tempStor.rateTemp.X3.7.m
## 1                 -0.0002043049                 -0.0002055426                 -0.0002067803
##   data.tempStor.rateTemp.X3.8.m data.tempStor.rateTemp.X3.9.m data.tempStor.rateTemp.X4.m
## 1                 -0.0002080181                 -0.0002092558               -0.0002104935
##   data.tempStor.rateTemp.X4.1.m data.tempStor.rateTemp.X4.2.m data.tempStor.rateTemp.X4.3.m
## 1                 -0.0002117313                  -0.000212969                 -0.0002142067
##   data.tempStor.rateTemp.X4.4.m data.tempStor.rateTemp.X4.5.m data.tempStor.rateTemp.X4.6.m
## 1                 -0.0002154444                 -0.0002172161                 -0.0002189878
##   data.tempStor.rateTemp.X4.7.m data.tempStor.rateTemp.X4.8.m data.tempStor.rateTemp.X4.9.m
## 1                 -0.0002207595                 -0.0002225312                 -0.0002243029
##   data.tempStor.rateTemp.X5.m data.tempStor.rateTemp.X5.1.m data.tempStor.rateTemp.X5.2.m
## 1               -0.0002260746                 -0.0002278463                  -0.000229618
##   data.tempStor.rateTemp.X5.3.m data.tempStor.rateTemp.X5.4.m data.tempStor.rateTemp.X5.5.m
## 1                 -0.0002313896                 -0.0002331613                  -0.000234933
##   data.tempStor.rateTemp.X5.6.m data.tempStor.rateTemp.X5.7.m data.tempStor.rateTemp.X5.8.m
## 1                 -0.0002367047                 -0.0002384764                 -0.0002402481
##   data.tempStor.rateTemp.X5.9.m data.tempStor.rateTemp.X6.m data.tempStor.rateTemp.X6.1.m
## 1                 -0.0002420198               -0.0002437915                 -0.0002455631
##   data.tempStor.rateTemp.X6.2.m data.tempStor.rateTemp.X6.3.m data.tempStor.rateTemp.X6.4.m
## 1                 -0.0002473348                 -0.0002491065                 -0.0002508782
##   data.tempStor.rateTemp.X6.5.m data.tempStor.rateTemp.X6.6.m data.tempStor.rateTemp.X6.7.m
## 1                 -0.0002526499                 -0.0002544216                 -0.0002561933
##   data.tempStor.rateTemp.X6.8.m data.tempStor.rateTemp.X6.9.m data.tempStor.rateTemp.X7.m
## 1                  -0.000257965                 -0.0002597367               -0.0002615083
##   data.tempStor.rateTemp.X7.1.m data.tempStor.rateTemp.X7.2.m data.tempStor.rateTemp.X7.3.m
## 1                   -0.00026328                 -0.0002650517                 -0.0002668234
##   data.tempStor.rateTemp.X7.4.m data.tempStor.rateTemp.X7.5.m data.tempStor.rateTemp.X7.6.m
## 1                 -0.0002685951                 -0.0002703668                 -0.0002721385
##   data.tempStor.rateTemp.X7.7.m data.tempStor.rateTemp.X7.8.m data.tempStor.rateTemp.X7.9.m
## 1                 -0.0002739102                 -0.0002756819                 -0.0002774535
##   data.tempStor.rateTemp.X8.m data.tempStor.rateTemp.X8.1.m data.tempStor.rateTemp.X8.2.m
## 1               -0.0002792252                 -0.0002809969                 -0.0002827686
##   data.tempStor.rateTemp.X8.3.m data.tempStor.rateTemp.X8.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X0.1.m
## 1                 -0.0002845403                  -0.000286312                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X0.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X0.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X0.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X0.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X0.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X0.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X0.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X0.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X1.m qfqm.co2Stor.rateRtioMoleDryCo2.X1.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X1.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X1.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X1.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X1.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X1.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X1.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X1.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X1.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X2.m qfqm.co2Stor.rateRtioMoleDryCo2.X2.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X2.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X2.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X2.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X2.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X2.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X2.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X2.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X2.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X3.m qfqm.co2Stor.rateRtioMoleDryCo2.X3.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X3.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X3.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X3.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X3.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X3.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X3.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X3.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X3.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X4.m qfqm.co2Stor.rateRtioMoleDryCo2.X4.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X4.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X4.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X4.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X4.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X4.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X4.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X4.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X4.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X5.m qfqm.co2Stor.rateRtioMoleDryCo2.X5.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X5.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X5.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X5.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X5.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X5.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X5.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X5.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X5.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X6.m qfqm.co2Stor.rateRtioMoleDryCo2.X6.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X6.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X6.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X6.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X6.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X6.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X6.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X6.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X6.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X7.m qfqm.co2Stor.rateRtioMoleDryCo2.X7.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X7.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X7.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X7.4.m qfqm.co2Stor.rateRtioMoleDryCo2.X7.5.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X7.6.m qfqm.co2Stor.rateRtioMoleDryCo2.X7.7.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X7.8.m qfqm.co2Stor.rateRtioMoleDryCo2.X7.9.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X8.m qfqm.co2Stor.rateRtioMoleDryCo2.X8.1.m
## 1                                    1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X8.2.m qfqm.co2Stor.rateRtioMoleDryCo2.X8.3.m
## 1                                      1                                      1
##   qfqm.co2Stor.rateRtioMoleDryCo2.X8.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X0.1.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X0.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X0.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X0.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X0.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X0.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X0.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X0.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X0.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X1.m qfqm.h2oStor.rateRtioMoleDryH2o.X1.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X1.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X1.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X1.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X1.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X1.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X1.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X1.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X1.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X2.m qfqm.h2oStor.rateRtioMoleDryH2o.X2.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X2.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X2.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X2.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X2.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X2.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X2.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X2.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X2.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X3.m qfqm.h2oStor.rateRtioMoleDryH2o.X3.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X3.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X3.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X3.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X3.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X3.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X3.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X3.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X3.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X4.m qfqm.h2oStor.rateRtioMoleDryH2o.X4.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X4.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X4.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X4.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X4.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X4.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X4.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X4.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X4.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X5.m qfqm.h2oStor.rateRtioMoleDryH2o.X5.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X5.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X5.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X5.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X5.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X5.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X5.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X5.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X5.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X6.m qfqm.h2oStor.rateRtioMoleDryH2o.X6.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X6.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X6.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X6.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X6.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X6.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X6.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X6.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X6.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X7.m qfqm.h2oStor.rateRtioMoleDryH2o.X7.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X7.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X7.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X7.4.m qfqm.h2oStor.rateRtioMoleDryH2o.X7.5.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X7.6.m qfqm.h2oStor.rateRtioMoleDryH2o.X7.7.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X7.8.m qfqm.h2oStor.rateRtioMoleDryH2o.X7.9.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X8.m qfqm.h2oStor.rateRtioMoleDryH2o.X8.1.m
## 1                                    1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X8.2.m qfqm.h2oStor.rateRtioMoleDryH2o.X8.3.m
## 1                                      1                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.X8.4.m qfqm.tempStor.rateTemp.X0.1.m qfqm.tempStor.rateTemp.X0.2.m
## 1                                      1                             0                             0
##   qfqm.tempStor.rateTemp.X0.3.m qfqm.tempStor.rateTemp.X0.4.m qfqm.tempStor.rateTemp.X0.5.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X0.6.m qfqm.tempStor.rateTemp.X0.7.m qfqm.tempStor.rateTemp.X0.8.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X0.9.m qfqm.tempStor.rateTemp.X1.m qfqm.tempStor.rateTemp.X1.1.m
## 1                             0                           0                             0
##   qfqm.tempStor.rateTemp.X1.2.m qfqm.tempStor.rateTemp.X1.3.m qfqm.tempStor.rateTemp.X1.4.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X1.5.m qfqm.tempStor.rateTemp.X1.6.m qfqm.tempStor.rateTemp.X1.7.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X1.8.m qfqm.tempStor.rateTemp.X1.9.m qfqm.tempStor.rateTemp.X2.m
## 1                             0                             0                           0
##   qfqm.tempStor.rateTemp.X2.1.m qfqm.tempStor.rateTemp.X2.2.m qfqm.tempStor.rateTemp.X2.3.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X2.4.m qfqm.tempStor.rateTemp.X2.5.m qfqm.tempStor.rateTemp.X2.6.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X2.7.m qfqm.tempStor.rateTemp.X2.8.m qfqm.tempStor.rateTemp.X2.9.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X3.m qfqm.tempStor.rateTemp.X3.1.m qfqm.tempStor.rateTemp.X3.2.m
## 1                           0                             0                             0
##   qfqm.tempStor.rateTemp.X3.3.m qfqm.tempStor.rateTemp.X3.4.m qfqm.tempStor.rateTemp.X3.5.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X3.6.m qfqm.tempStor.rateTemp.X3.7.m qfqm.tempStor.rateTemp.X3.8.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X3.9.m qfqm.tempStor.rateTemp.X4.m qfqm.tempStor.rateTemp.X4.1.m
## 1                             0                           0                             0
##   qfqm.tempStor.rateTemp.X4.2.m qfqm.tempStor.rateTemp.X4.3.m qfqm.tempStor.rateTemp.X4.4.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X4.5.m qfqm.tempStor.rateTemp.X4.6.m qfqm.tempStor.rateTemp.X4.7.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X4.8.m qfqm.tempStor.rateTemp.X4.9.m qfqm.tempStor.rateTemp.X5.m
## 1                             0                             0                           0
##   qfqm.tempStor.rateTemp.X5.1.m qfqm.tempStor.rateTemp.X5.2.m qfqm.tempStor.rateTemp.X5.3.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X5.4.m qfqm.tempStor.rateTemp.X5.5.m qfqm.tempStor.rateTemp.X5.6.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X5.7.m qfqm.tempStor.rateTemp.X5.8.m qfqm.tempStor.rateTemp.X5.9.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X6.m qfqm.tempStor.rateTemp.X6.1.m qfqm.tempStor.rateTemp.X6.2.m
## 1                           0                             0                             0
##   qfqm.tempStor.rateTemp.X6.3.m qfqm.tempStor.rateTemp.X6.4.m qfqm.tempStor.rateTemp.X6.5.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X6.6.m qfqm.tempStor.rateTemp.X6.7.m qfqm.tempStor.rateTemp.X6.8.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X6.9.m qfqm.tempStor.rateTemp.X7.m qfqm.tempStor.rateTemp.X7.1.m
## 1                             0                           0                             0
##   qfqm.tempStor.rateTemp.X7.2.m qfqm.tempStor.rateTemp.X7.3.m qfqm.tempStor.rateTemp.X7.4.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X7.5.m qfqm.tempStor.rateTemp.X7.6.m qfqm.tempStor.rateTemp.X7.7.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X7.8.m qfqm.tempStor.rateTemp.X7.9.m qfqm.tempStor.rateTemp.X8.m
## 1                             0                             0                           0
##   qfqm.tempStor.rateTemp.X8.1.m qfqm.tempStor.rateTemp.X8.2.m qfqm.tempStor.rateTemp.X8.3.m
## 1                             0                             0                             0
##   qfqm.tempStor.rateTemp.X8.4.m
## 1                             0
##  [ reached 'max' / getOption("max.print") -- omitted 5 rows ]

6. Un-interpolated vertical profile data (Level 2)

The Level 2 data are interpolated in time but not in space. They contain the rates of change at each of the measurement heights.

Again, they can be extracted from the HDF5 files using stackEddy() with the same syntax:

prof.l2 <- stackEddy(filepath="~/Downloads/filesToStack00200/",
                 level="dp02")



head(prof.l2$HARV)

##   verticalPosition             timeBgn             timeEnd data.co2Stor.rateRtioMoleDryCo2.mean
## 1              010 2018-06-01 00:00:00 2018-06-01 00:29:59                                  NaN
## 2              010 2018-06-01 00:30:00 2018-06-01 00:59:59                          0.002666576
## 3              010 2018-06-01 01:00:00 2018-06-01 01:29:59                         -0.011224223
## 4              010 2018-06-01 01:30:00 2018-06-01 01:59:59                          0.006133056
## 5              010 2018-06-01 02:00:00 2018-06-01 02:29:59                         -0.019554655
## 6              010 2018-06-01 02:30:00 2018-06-01 02:59:59                         -0.007855632
##   data.h2oStor.rateRtioMoleDryH2o.mean data.tempStor.rateTemp.mean qfqm.co2Stor.rateRtioMoleDryCo2.qfFinl
## 1                                  NaN                2.583333e-05                                      1
## 2                                  NaN               -2.008056e-04                                      1
## 3                                  NaN               -1.901111e-04                                      1
## 4                                  NaN               -7.419444e-05                                      1
## 5                                  NaN               -1.537083e-04                                      1
## 6                                  NaN               -1.874861e-04                                      1
##   qfqm.h2oStor.rateRtioMoleDryH2o.qfFinl qfqm.tempStor.rateTemp.qfFinl
## 1                                      1                             0
## 2                                      1                             0
## 3                                      1                             0
## 4                                      1                             0
## 5                                      1                             0
## 6                                      1                             0

Note that here, as in the PAR data, there is a verticalPosition field. It has the same meaning as in the PAR data, indicating the tower level of the measurement.

7. Calibrated raw data (Level 1)

Level 1 (dp01) data are calibrated, and aggregated in time, but otherwise untransformed. Use Level 1 data for raw gas concentrations and atmospheric stable isotopes.

Using stackEddy() to extract Level 1 data requires additional inputs. The Level 1 files are too large to simply pull out all the variables by default, and they include multiple averaging intervals, which can't be merged. So two additional inputs are needed:

avg: The averaging interval to extract
var: One or more variables to extract

What variables are available, at what averaging intervals? Another function in the neonUtilities package, getVarsEddy(), returns a list of HDF5 file contents. It requires only one input, a filepath to a single NEON HDF5 file:

vars <- getVarsEddy("~/Downloads/filesToStack00200/NEON.D01.HARV.DP4.00200.001.nsae.2018-07.basic.20201020T201317Z.h5")
head(vars)

##    site level category system hor ver tmi       name       otype   dclass   dim  oth
## 5  HARV  dp01     data   amrs 000 060 01m angNedXaxs H5I_DATASET COMPOUND 43200 <NA>
## 6  HARV  dp01     data   amrs 000 060 01m angNedYaxs H5I_DATASET COMPOUND 43200 <NA>
## 7  HARV  dp01     data   amrs 000 060 01m angNedZaxs H5I_DATASET COMPOUND 43200 <NA>
## 9  HARV  dp01     data   amrs 000 060 30m angNedXaxs H5I_DATASET COMPOUND  1440 <NA>
## 10 HARV  dp01     data   amrs 000 060 30m angNedYaxs H5I_DATASET COMPOUND  1440 <NA>
## 11 HARV  dp01     data   amrs 000 060 30m angNedZaxs H5I_DATASET COMPOUND  1440 <NA>

Inputs to var can be any values from the name field in the table returned by getVarsEddy(). Let's take a look at CO₂ and H₂O, ¹³C in CO₂ and ¹⁸O in H₂O, at 30-minute aggregation. Let's look at Harvard Forest for these data, since deeper canopies generally have more interesting profiles:

iso <- stackEddy(filepath="~/Downloads/filesToStack00200/",
               level="dp01", var=c("rtioMoleDryCo2","rtioMoleDryH2o",
                                   "dlta13CCo2","dlta18OH2o"), avg=30)



head(iso$HARV)

##   verticalPosition             timeBgn             timeEnd data.co2Stor.rtioMoleDryCo2.mean
## 1              010 2018-06-01 00:00:00 2018-06-01 00:29:59                         509.3375
## 2              010 2018-06-01 00:30:00 2018-06-01 00:59:59                         502.2736
## 3              010 2018-06-01 01:00:00 2018-06-01 01:29:59                         521.6139
## 4              010 2018-06-01 01:30:00 2018-06-01 01:59:59                         469.6317
## 5              010 2018-06-01 02:00:00 2018-06-01 02:29:59                         484.7725
## 6              010 2018-06-01 02:30:00 2018-06-01 02:59:59                         476.8554
##   data.co2Stor.rtioMoleDryCo2.min data.co2Stor.rtioMoleDryCo2.max data.co2Stor.rtioMoleDryCo2.vari
## 1                        451.4786                        579.3518                         845.0795
## 2                        463.5470                        533.6622                         161.3652
## 3                        442.8649                        563.0518                         547.9924
## 4                        432.6588                        508.7463                         396.8379
## 5                        436.2842                        537.4641                         662.9449
## 6                        443.7055                        515.6598                         246.6969
##   data.co2Stor.rtioMoleDryCo2.numSamp data.co2Turb.rtioMoleDryCo2.mean data.co2Turb.rtioMoleDryCo2.min
## 1                                 235                               NA                              NA
## 2                                 175                               NA                              NA
## 3                                 235                               NA                              NA
## 4                                 175                               NA                              NA
## 5                                 235                               NA                              NA
## 6                                 175                               NA                              NA
##   data.co2Turb.rtioMoleDryCo2.max data.co2Turb.rtioMoleDryCo2.vari data.co2Turb.rtioMoleDryCo2.numSamp
## 1                              NA                               NA                                  NA
## 2                              NA                               NA                                  NA
## 3                              NA                               NA                                  NA
## 4                              NA                               NA                                  NA
## 5                              NA                               NA                                  NA
## 6                              NA                               NA                                  NA
##   data.h2oStor.rtioMoleDryH2o.mean data.h2oStor.rtioMoleDryH2o.min data.h2oStor.rtioMoleDryH2o.max
## 1                              NaN                             NaN                             NaN
## 2                              NaN                             NaN                             NaN
## 3                              NaN                             NaN                             NaN
## 4                              NaN                             NaN                             NaN
## 5                              NaN                             NaN                             NaN
## 6                              NaN                             NaN                             NaN
##   data.h2oStor.rtioMoleDryH2o.vari data.h2oStor.rtioMoleDryH2o.numSamp data.h2oTurb.rtioMoleDryH2o.mean
## 1                               NA                                   0                               NA
## 2                               NA                                   0                               NA
## 3                               NA                                   0                               NA
## 4                               NA                                   0                               NA
## 5                               NA                                   0                               NA
## 6                               NA                                   0                               NA
##   data.h2oTurb.rtioMoleDryH2o.min data.h2oTurb.rtioMoleDryH2o.max data.h2oTurb.rtioMoleDryH2o.vari
## 1                              NA                              NA                               NA
## 2                              NA                              NA                               NA
## 3                              NA                              NA                               NA
## 4                              NA                              NA                               NA
## 5                              NA                              NA                               NA
## 6                              NA                              NA                               NA
##   data.h2oTurb.rtioMoleDryH2o.numSamp data.isoCo2.dlta13CCo2.mean data.isoCo2.dlta13CCo2.min
## 1                                  NA                         NaN                        NaN
## 2                                  NA                   -11.40646                    -14.992
## 3                                  NA                         NaN                        NaN
## 4                                  NA                   -10.69318                    -14.065
## 5                                  NA                         NaN                        NaN
## 6                                  NA                   -11.02814                    -13.280
##   data.isoCo2.dlta13CCo2.max data.isoCo2.dlta13CCo2.vari data.isoCo2.dlta13CCo2.numSamp
## 1                        NaN                          NA                              0
## 2                     -8.022                   1.9624355                            305
## 3                        NaN                          NA                              0
## 4                     -7.385                   1.5766385                            304
## 5                        NaN                          NA                              0
## 6                     -7.966                   0.9929341                            308
##   data.isoCo2.rtioMoleDryCo2.mean data.isoCo2.rtioMoleDryCo2.min data.isoCo2.rtioMoleDryCo2.max
## 1                             NaN                            NaN                            NaN
## 2                        458.3546                        415.875                        531.066
## 3                             NaN                            NaN                            NaN
## 4                        439.9582                        415.777                        475.736
## 5                             NaN                            NaN                            NaN
## 6                        446.5563                        420.845                        468.312
##   data.isoCo2.rtioMoleDryCo2.vari data.isoCo2.rtioMoleDryCo2.numSamp data.isoCo2.rtioMoleDryH2o.mean
## 1                              NA                                  0                             NaN
## 2                        953.2212                                306                        22.11830
## 3                              NA                                  0                             NaN
## 4                        404.0365                                306                        22.38925
## 5                              NA                                  0                             NaN
## 6                        138.7560                                309                        22.15731
##   data.isoCo2.rtioMoleDryH2o.min data.isoCo2.rtioMoleDryH2o.max data.isoCo2.rtioMoleDryH2o.vari
## 1                            NaN                            NaN                              NA
## 2                       21.85753                       22.34854                      0.01746926
## 3                            NaN                            NaN                              NA
## 4                       22.09775                       22.59945                      0.02626762
## 5                            NaN                            NaN                              NA
## 6                       22.06641                       22.26493                      0.00277579
##   data.isoCo2.rtioMoleDryH2o.numSamp data.isoH2o.dlta18OH2o.mean data.isoH2o.dlta18OH2o.min
## 1                                  0                         NaN                        NaN
## 2                                 85                   -12.24437                    -12.901
## 3                                  0                         NaN                        NaN
## 4                                 84                   -12.04580                    -12.787
## 5                                  0                         NaN                        NaN
## 6                                 80                   -11.81500                    -12.375
##   data.isoH2o.dlta18OH2o.max data.isoH2o.dlta18OH2o.vari data.isoH2o.dlta18OH2o.numSamp
## 1                        NaN                          NA                              0
## 2                    -11.569                  0.03557313                            540
## 3                        NaN                          NA                              0
## 4                    -11.542                  0.03970481                            539
## 5                        NaN                          NA                              0
## 6                    -11.282                  0.03498614                            540
##   data.isoH2o.rtioMoleDryH2o.mean data.isoH2o.rtioMoleDryH2o.min data.isoH2o.rtioMoleDryH2o.max
## 1                             NaN                            NaN                            NaN
## 2                        20.89354                       20.36980                       21.13160
## 3                             NaN                            NaN                            NaN
## 4                        21.12872                       20.74663                       21.33272
## 5                             NaN                            NaN                            NaN
## 6                        20.93480                       20.63463                       21.00702
##   data.isoH2o.rtioMoleDryH2o.vari data.isoH2o.rtioMoleDryH2o.numSamp qfqm.co2Stor.rtioMoleDryCo2.qfFinl
## 1                              NA                                  0                                  1
## 2                     0.025376207                                540                                  1
## 3                              NA                                  0                                  1
## 4                     0.017612293                                540                                  1
## 5                              NA                                  0                                  1
## 6                     0.003805751                                540                                  1
##   qfqm.co2Turb.rtioMoleDryCo2.qfFinl qfqm.h2oStor.rtioMoleDryH2o.qfFinl qfqm.h2oTurb.rtioMoleDryH2o.qfFinl
## 1                                 NA                                  1                                 NA
## 2                                 NA                                  1                                 NA
## 3                                 NA                                  1                                 NA
## 4                                 NA                                  1                                 NA
## 5                                 NA                                  1                                 NA
## 6                                 NA                                  1                                 NA
##   qfqm.isoCo2.dlta13CCo2.qfFinl qfqm.isoCo2.rtioMoleDryCo2.qfFinl qfqm.isoCo2.rtioMoleDryH2o.qfFinl
## 1                             1                                 1                                 1
## 2                             0                                 0                                 0
## 3                             1                                 1                                 1
## 4                             0                                 0                                 0
## 5                             1                                 1                                 1
## 6                             0                                 0                                 0
##   qfqm.isoH2o.dlta18OH2o.qfFinl qfqm.isoH2o.rtioMoleDryH2o.qfFinl ucrt.co2Stor.rtioMoleDryCo2.mean
## 1                             1                                 1                       10.0248527
## 2                             0                                 0                        1.1077243
## 3                             1                                 1                        7.5181428
## 4                             0                                 0                        8.4017805
## 5                             1                                 1                        0.9465824
## 6                             0                                 0                        1.3629090
##   ucrt.co2Stor.rtioMoleDryCo2.vari ucrt.co2Stor.rtioMoleDryCo2.se ucrt.co2Turb.rtioMoleDryCo2.mean
## 1                        170.28091                      1.8963340                               NA
## 2                         34.29589                      0.9602536                               NA
## 3                        151.35746                      1.5270503                               NA
## 4                         93.41077                      1.5058703                               NA
## 5                         14.02753                      1.6795958                               NA
## 6                          8.50861                      1.1873064                               NA
##   ucrt.co2Turb.rtioMoleDryCo2.vari ucrt.co2Turb.rtioMoleDryCo2.se ucrt.h2oStor.rtioMoleDryH2o.mean
## 1                               NA                             NA                               NA
## 2                               NA                             NA                               NA
## 3                               NA                             NA                               NA
## 4                               NA                             NA                               NA
## 5                               NA                             NA                               NA
## 6                               NA                             NA                               NA
##   ucrt.h2oStor.rtioMoleDryH2o.vari ucrt.h2oStor.rtioMoleDryH2o.se ucrt.h2oTurb.rtioMoleDryH2o.mean
## 1                               NA                             NA                               NA
## 2                               NA                             NA                               NA
## 3                               NA                             NA                               NA
## 4                               NA                             NA                               NA
## 5                               NA                             NA                               NA
## 6                               NA                             NA                               NA
##   ucrt.h2oTurb.rtioMoleDryH2o.vari ucrt.h2oTurb.rtioMoleDryH2o.se ucrt.isoCo2.dlta13CCo2.mean
## 1                               NA                             NA                         NaN
## 2                               NA                             NA                   0.5812574
## 3                               NA                             NA                         NaN
## 4                               NA                             NA                   0.3653442
## 5                               NA                             NA                         NaN
## 6                               NA                             NA                   0.2428672
##   ucrt.isoCo2.dlta13CCo2.vari ucrt.isoCo2.dlta13CCo2.se ucrt.isoCo2.rtioMoleDryCo2.mean
## 1                         NaN                        NA                             NaN
## 2                   0.6827844                0.08021356                       16.931819
## 3                         NaN                        NA                             NaN
## 4                   0.3761155                0.07201605                       10.078698
## 5                         NaN                        NA                             NaN
## 6                   0.1544487                0.05677862                        7.140787
##   ucrt.isoCo2.rtioMoleDryCo2.vari ucrt.isoCo2.rtioMoleDryCo2.se ucrt.isoCo2.rtioMoleDryH2o.mean
## 1                             NaN                            NA                             NaN
## 2                       614.01630                      1.764965                      0.08848440
## 3                             NaN                            NA                             NaN
## 4                       196.99445                      1.149078                      0.08917388
## 5                             NaN                            NA                             NaN
## 6                        55.90843                      0.670111                              NA
##   ucrt.isoCo2.rtioMoleDryH2o.vari ucrt.isoCo2.rtioMoleDryH2o.se ucrt.isoH2o.dlta18OH2o.mean
## 1                             NaN                            NA                         NaN
## 2                      0.01226428                   0.014335993                  0.02544454
## 3                             NaN                            NA                         NaN
## 4                      0.01542679                   0.017683602                  0.01373503
## 5                             NaN                            NA                         NaN
## 6                              NA                   0.005890447                  0.01932110
##   ucrt.isoH2o.dlta18OH2o.vari ucrt.isoH2o.dlta18OH2o.se ucrt.isoH2o.rtioMoleDryH2o.mean
## 1                         NaN                        NA                             NaN
## 2                 0.003017400               0.008116413                      0.06937514
## 3                         NaN                        NA                             NaN
## 4                 0.002704220               0.008582764                      0.08489408
## 5                         NaN                        NA                             NaN
## 6                 0.002095066               0.008049170                      0.02813808
##   ucrt.isoH2o.rtioMoleDryH2o.vari ucrt.isoH2o.rtioMoleDryH2o.se
## 1                             NaN                            NA
## 2                     0.009640249                   0.006855142
## 3                             NaN                            NA
## 4                     0.008572288                   0.005710986
## 5                             NaN                            NA
## 6                     0.002551672                   0.002654748

Let's plot vertical profiles of CO₂ and ¹³C in CO₂ on a single day.

Here we'll use the time stamps in a different way, using grep() to select all of the records for a single day. And discard the verticalPosition values that are string values - those are the calibration gases.

iso.d <- iso$HARV[grep("2018-06-25", iso$HARV$timeBgn, fixed=T),]
iso.d <- iso.d[-which(is.na(as.numeric(iso.d$verticalPosition))),]

ggplot is well suited to these types of data, let's use it to plot the profiles. If you don't have the package yet, use install.packages() to install it first.

library(ggplot2)

Now we can plot CO₂ relative to height on the tower, with separate lines for each time interval.

g <- ggplot(iso.d, aes(y=verticalPosition)) + 
  geom_path(aes(x=data.co2Stor.rtioMoleDryCo2.mean, 
                group=timeBgn, col=timeBgn)) + 
  theme(legend.position="none") + 
  xlab("CO2") + ylab("Tower level")
g

And the same plot for ¹³C in CO₂:

g <- ggplot(iso.d, aes(y=verticalPosition)) + 
  geom_path(aes(x=data.isoCo2.dlta13CCo2.mean, 
                group=timeBgn, col=timeBgn)) + 
  theme(legend.position="none") + 
  xlab("d13C") + ylab("Tower level")
g

The legends are omitted for space, see if you can use the concentration and isotope ratio buildup and drawdown below the canopy to work out the times of day the different colors represent.

Get Lesson Code

eddy_intro.R

Learning Objectives

The NEON Project Mission & Design

The Science and Design of NEON

NEON's Spatial Design

The Spatial Design of NEON

NEON Field Site Locations

NEON Data

How NEON Collects Data

Specimens & Samples

Airborne Remote Sensing

Access NEON Data

Pathways to access NEON Data

NEON data

Objectives

Things You’ll Need To Complete This Tutorial

Install R Packages

Additional Resources

Getting started: Download data from the Portal and load packages

Stack the downloaded data files: stackByTable()

Download files and load directly to R: loadByProduct()

Download remote sensing data: byFileAOP() and byTileAOP()

Navigate data downloads: IS

Navigate data downloads: OS

Navigate data downloads: AOP

Get Lesson Code

neonUtilities package

Download files and load directly to R: loadByProduct()

Join data files: stackByTable()

Download the Data

Run stackByTable()

Other options

Download files to be stacked: zipsByProduct()

Download a single zip file: getPackage()

Download remote sensing files: byFileAOP()

Download remote sensing files for specific coordinates: byTileAOP()

Convert files to GeoCSV: transformFileToGeoCSV()

Get Lesson Code

NEON data

Objectives

Things You’ll Need To Complete This Tutorial

Install R Packages

Additional Resources

What is an API?

Anatomy of an API call

Base URL:

Endpoints:

Targets:

Observational data (OS)

Instrumentation data (IS)

Remote sensing data (AOP)

Geolocation data

Instrumentation data (both aquatic and terrestrial)

Observational data - Aquatic

Observational data - Terrestrial

Querying a single named location

Taxonomy

Stacking NEON data

Get Lesson Code

Objectives

Things You’ll Need To Complete This Tutorial

Install R Packages

Download Data

Additional Resources

Explore Phenology Data

NEON Plant Phenology Observation Data

Status-based Monitoring

Phenology Transects

Timing of Observations

Available Data Tables

Stack NEON Data

Work with NEON Data

Phenology status

Clean up the Data

Remove Duplicates

Variable Overlap between Tables

Filter to last editedDate

Join Dataframes

Patterns in Phenophase

Select Site(s) of Interest

Select Species of Interest

Run `stackByTable()`