Intro to AOP Datasets in Google Earth Engine (GEE) using Python

AOP data in GEE

Google Earth Engine is a platform idea for carrying out continental and planetary scale geospatial analyses. It has a multi-pedabyte catalog of satellite imagery and geospatial datasets, and is a powerful tool for comparing and scaling up NEON airborne datasets.

The NEON data products can currently be accessed through the projects/neon-prod-earthengine/assets/ folder with an appended prefix of the data product ID (DPID) similar to what you see on the NEON data portal. The table below shows the corresponding prefixes to use for the available data products.

To access the NEON AOP data you can read in the Image Collection ee.ImageCollection and filter by the date and other properties of interest.

First, import the relevant Earth Engine Python packages, earthengine-api (ee) and geemap. This lesson was written using geemap version 0.22.1. If you have an older version installed, used the command below to update.

GEE Python API

import ee, geemap

In order to use Earth Engine from within this Jupyter Notebook, we need to first Authenticate (which requires generating a token) and then Initialize, as below. For more detailed instructions on the Authentication process, please refer to the: geemap text book Earth Engine authentication section.

geemap.__version__

'0.22.1'

ee.Authenticate()

To authorize access needed by Earth Engine, open the following

The authorization workflow will generate a code, which you should paste in the box below.

When the token is entered, you should receive the notice: "Successfully saved authorization token."

ee.Initialize()

Now that you've logged in to your Earth Engine account, you can start using the Python API with the ee package.

First let's read in the Surface Directional Reflectance (SDR) Image Collection and take a look at the available data.

# Read in the NEON AOP Surface Directional Reflectance (SDR) Collection:
sdrCol = ee.ImageCollection('projects/neon-prod-earthengine/assets/DP3-30006-001')

# List all available sites in the NEON SDR image collection:
print('All Available NEON SDR Images:')

# Get the number of SDR images
sdrCount = sdrCol.size()
print('Count: ', str(sdrCount.getInfo())+'\n')

print(sdrCol.aggregate_array('system:index').getInfo())

All Available NEON SDR Images:
Count:  25

['2013_CPER_1_SDR', '2016_CLBJ_1_SDR', '2016_GRSM_2_SDR', '2016_HARV_3_SDR', '2017_CLBJ_2_SDR', '2017_CPER_3_SDR', '2017_GRSM_3_SDR', '2017_SERC_3_SDR', '2018_CLBJ_3_SDR', '2019_CLBJ_4_SDR', '2019_HARV_6_SDR', '2019_HEAL_3_SDR', '2019_JORN_3_SDR', '2019_SOAP_4_SDR', '2020_CPER_5_SDR', '2020_CPER_7_SDR', '2020_NIWO_4_SDR', '2021_ABBY_4_SDR', '2021_CLBJ_5_SDR', '2021_CPER_8_SDR', '2021_GRSM_5_SDR', '2021_HEAL_4_SDR', '2021_JORN_4_SDR', '2021_SJER_5_SDR', '2021_SOAP_5_SDR']

We can also look for data for a specified site - for example look at all the years of AOP SDR data available for a given site.

# See the years of data available for the specified site:
site = 'CPER'

# Get the flight year and site information
flightYears = sdrCol.aggregate_array('FLIGHT_YEAR').getInfo()
sites = sdrCol.aggregate_array('NEON_SITE').getInfo()

print('\nYears of SDR data available in GEE for',site+':')
print([year_site[0] for year_site in zip(flightYears,sites) if site in year_site])

Years of SDR data available in GEE for CPER:
[2013, 2017, 2020, 2020, 2021]

Let's take a look at another dataset, the high-resolution RGB camera imagery:

# Read in the NEON AOP Camera (RGB) Collection:
rgbCol = ee.ImageCollection('projects/neon-prod-earthengine/assets/DP3-30010-001')

# List all available sites in the NEON SDR image collection:
print('All Available NEON Camera Images:')
# Get the number of RGB images
rgbCount = rgbCol.size()
print('Count: ', str(sdrCount.getInfo())+'\n')
print(rgbCol.aggregate_array('system:index').getInfo())

All Available NEON Camera Images:
Count:  25

['2016_HARV_3_RGB', '2017_CLBJ_2_RGB', '2017_GRSM_3_RGB', '2017_SERC_3_RGB', '2018_CLBJ_3_RGB', '2018_TEAK_3_RGB', '2019_BART_5_RGB', '2019_CLBJ_4_RGB', '2019_HARV_6_RGB', '2019_HEAL_3_RGB', '2019_JORN_3_RGB', '2019_SOAP_4_RGB', '2020_CPER_7_RGB', '2020_NIWO_4_RGB', '2020_RMNP_3_RGB', '2020_UKFS_5_RGB', '2020_UNDE_4_RGB', '2020_YELL_3_RGB', '2021_ABBY_4_RGB', '2021_BLAN_4_RGB', '2021_BONA_4_RGB', '2021_CLBJ_5_RGB', '2021_DEJU_4_RGB', '2021_DELA_6_RGB', '2021_HEAL_4_RGB', '2021_JERC_6_RGB', '2021_JORN_4_RGB', '2021_LENO_6_RGB', '2021_MLBS_4_RGB', '2021_OSBS_6_RGB', '2021_SERC_5_RGB', '2021_SJER_5_RGB', '2021_SOAP_5_RGB', '2021_TALL_6_RGB', '2021_WREF_4_RGB']

Similarly, you can read in the DEM and CHM collections as follows:

# Read in the NEON AOP DEM Collection (this includes the DTM and DSM as 2 bands)
demCol = ee.ImageCollection('projects/neon-prod-earthengine/assets/DP3-30024-001')

# Read in the NEON AOP CHM Collection
chmCol = ee.ImageCollection('projects/neon-prod-earthengine/assets/DP3-30015-001')

Explore Image Properties

Now that we've read in a couple of image collections, let's take a look at some of the image properties using geemap.image_props:

props = geemap.image_props(sdrCol.first())

# Optionally display all properties by uncommenting the line below
# props

# Display the property names for the first 15 properties:
props.keys().getInfo()[:15]

['AOP_VISIT_NUMBER',
 'DESCRIPTION',
 'FLIGHT_YEAR',
 'IMAGE_DATE',
 'NEON_DATA_PROD_ID',
 'NEON_DATA_PROD_URL',
 'NEON_DOMAIN',
 'NEON_SITE',
 'NOMINAL_SCALE',
 'PRODUCT_TYPE',
 'SCALING_FACTOR',
 'SENSOR_NAME',
 'SENSOR_NUMBER',
 'WL_FWHM_B001',
 'WL_FWHM_B002']

You can also look at all the Image properties by typing props. This generates a long output, so we will just show a portion of the output from that:

AOP_VISIT_NUMBER:1
DESCRIPTION:Orthorectified surface directional reflectance (0-1 unitless, scaled by 10000) ...
FLIGHT_YEAR:2013
IMAGE_DATE:2013-06-25
NEON_DATA_PROD_ID:DP3.30006.001
NEON_DATA_PROD_URL:https://data.neonscience.org/data-products/DP3.30006.001
NEON_DOMAIN:D10
NEON_SITE:CPER
NOMINAL_SCALE:1
PRODUCT_TYPE:SDR
SCALING_FACTOR:10000
SENSOR_NAME:AVIRIS-NG
SENSOR_NUMBER:NIS1
WL_FWHM_B001:382.3465,5.8456
    
system:asset_size:68059.439009 MB
system:band_names: List (442 elements)
system:id:projects/neon-prod-earthengine/assets/DP3-30006-001/2013_CPER_1_SDR
system:index:2013_CPER_1_SDR
system:time_end:2013-06-25 10:42:05
system:time_start:2013-06-25 08:30:45
system:version:1689911980211725

The image properties contain some additional relevant information about the dataset, and are variables you can filter on to select a subset of the data. A lot of these properties are self-explanatory, but some of them may be less apparent. A short description of a few properties is outlined below. Note that when the datasets become part of the Google Public Datasets, you will be able to see descriptions of the properties in GEE.

• SENSOR_NAME: The name of the hyperspectral sensor. All NEON sensors are JPL AVIRIS-NG sensors.
• SENSOR_NUMBER: The payload number, NIS1 = NEON Imaging Spectrometer, Payload 1. NEON Operates 3 separate payloads, each with a unique hyperspectral sensor (as well as unique LiDAR and Camera sensors).
• WL_FWHM_B###: Center Wavelength (WL) and Full Width Half Max (FWHM) of the band, both in nm

In addition there are some system properties, including information about the size of the asset, the band names (most of these are just band numbers but the QA bands have more descriptive names), as well as the start and end time of the collection.

Filter an Image Collection

One of the most useful aspects of having AOP data ingested in Earth Engine is the ability to filter by properties, such as the site name, dates, sensors, etc. In this next section, we will show how to filter datasets to extract only data of interest. We'll use the NEON's Smithsonian Environmental Research Center (SERC), in Maryland.

# See the years of data available for the specified site:
site = 'SERC'

# Get the flight year and site information
flightYears = rgbCol.aggregate_array('FLIGHT_YEAR').getInfo()
sites = rgbCol.aggregate_array('NEON_SITE').getInfo()

print('\nYears of RGB data available in GEE for',site+':')
print([year_site[0] for year_site in zip(flightYears,sites) if site in year_site])

Years of RGB data available in GEE for SERC:
[2017, 2021]

# Get the flight year and site information
flightYears = sdrCol.aggregate_array('FLIGHT_YEAR').getInfo()
sites = sdrCol.aggregate_array('NEON_SITE').getInfo()

print('\nYears of SDR data available in GEE for',site+':')
print([year_site[0] for year_site in zip(flightYears,sites) if site in year_site])

Years of SDR data available in GEE for SERC:
[2017]

# Specify the start and end dates
year = 2017
start_date = ee.Date.fromYMD(year, 1, 1) 
end_date = start_date.advance(1, "year")

# Filter the RGB image collection on the site and dates
SERC_RGB_2017 = rgbCol.filterDate(start_date, end_date).filterMetadata('NEON_SITE', 'equals', site).mosaic()

Add Data Layers to the Map

In order to visualize and interact with the dataset, we can use geemap.Map() as follows. This will create a blank Map of the world, and then when you run geemap code to add layers, the Map will update - so note this is a little different from standard Jupyter Notebook output, where the figures typically show up below each code chunk.

Map = geemap.Map()
Map

We'll start by reading in an RGB image over the Smithsonian Environmental Research Center (SERC) and adding the layer to the Map.

# Specify center location of SERC: 38.890131, -76.560014
# NEON field site information can be found on the NEON website here > https://www.neonscience.org/field-sites/serc
geo = ee.Geometry.Point([-76.56, 38.89])

# Set the visualization parameters so contrast is maximized
rgb_vis_params = {'min': 0, 'max': 250, 'gamma': 0.8}

# Add the SERC RGB data as a layer to the Map
Map.addLayer(SERC_RGB_2017, rgb_vis_params, 'SERC 2017 RGB Camera');

# Center the map on SERC and set zoom level
Map.centerObject(geo, 12);

Surface Directional Reflectance (SDR)

Next let's take a look at one of the SDR datasets. We will pull in only the data bands, for this example.

SDR Data Bands

# Read in the first image of the SDR Image Collection
sdrImage = sdrCol.filterDate(start_date, end_date).filterMetadata('NEON_SITE', 'equals', site).mosaic()

# Read in only the data bands, all of which start with "B", eg. "B001"
sdrData = sdrImage.select('B.*')

# Set the visualization parameters so contrast is maximized, and display RGB bands (true-color image)
sdrVisParams = {'min':0, 'max':1200, 'gamma':0.9, 'bands':['B053','B035','B019']};

# Add the SERC RGB data as a layer to the Map
Map.addLayer(sdrData, sdrVisParams, 'SERC 2017 SDR');

SDR QA Bands

Before we used a regular expression to pull out only the bands starting with "B". We can also take a look at the remaining bands using a similar expression, but this time excluding bands starting with "B". These comprise all of the QA-related bands that provide additional information and context about the data bands. This next chunk of code prints out the IDs of all the QA bands.

# Read in only the QA bands, none of which start with "B"
sdrQA_bands = sdrImage.select('[^B].*')

# Create a dictionary of the band information
sdrQA_band_info = sdrQA_bands.getInfo()
type(sdrQA_band_info)

# Loop through the band info dictionary to print out the ID of each band
print('QA band IDs:\n')
for i in range(len(sdrQA_band_info['bands'])):
    print(sdrQA_band_info['bands'][i]['id'])

QA band IDs:

Aerosol_Optical_Depth
Aspect
Cast_Shadow
Dark_Dense_Vegetation_Classification
Haze_Cloud_Water_Map
Illumination_Factor
Path_Length
Sky_View_Factor
Slope
Smooth_Surface_Elevation
Visibility_Index_Map
Water_Vapor_Column
to-sensor_Azimuth_Angle
to-sensor_Zenith_Angle
Weather_Quality_Indicator
Acquisition_Date

Most of these QA bands are related to the Atmospheric Correction (ATCOR), one of the processing steps in converting Radiance to Reflectance. For more details on this process, NEON provides an Algorithm Theoretical Basis Document (ATBD), which is available from the data portal. https://data.neonscience.org/api/v0/documents/NEON.DOC.001288vB?inline=true

Bands that may be most useful for standard analyses are the Weather_Quality_Indicator and Acquisition_Date. The weather quality indicator includes information about the cloud conditions during the flight, as reported by the flight operators, where 1 corresponds to <10% cloud cover, 2 corresponds to 10-50% cloud cover, and 3 corresponds to >50% cloud cover. We recommend using only clear-sky data for a typical analysis, as it comprises the highest quality reflectance data.

You may be interested in the acquisition date if, for example, you are linking field data collected on a specific date, or are interested in finding satellite data collected close in time to the AOP imagery.

The next chunks of code show how to add the Weather QA bands to the Map Layer.

Weather Quality Indicator Band

sdrWeatherQA = sdrCol.select(['Weather_Quality_Indicator']);

# Define a palette for the weather - to match NEON AOP's weather color conventions (green-yellow-red)
gyrPalette = ['00ff00', # green (<10% cloud cover)
              'ffff00', # yellow (10-50% cloud cover)
              'ff0000']; # red (>50% cloud cover)
             
# Visualization parameters to display the weather bands (cloud conditions) with the green-yellow-red palette
weatherVisParams = {'min': 1, 'max': 3, 'palette': gyrPalette, 'opacity': 0.3};

# Add the SERC RGB data as a layer to the Map
Map.addLayer(sdrWeatherQA, weatherVisParams, 'SERC 2017 Weather QA');

In 2017 most of the SERC site was collected in "green" (<10% cloud cover) sky conditions, with the exception of a small portion on the western side of the flight box. When working with this dataset, we recommend only using this clear-sky portion, where clouds have not obscured the reflectance values.

Recap

In this lesson, you learned how to access the four NEON AOP Image Collections that are available in GEE: 1. Surface Directional Reflectance (SDR), 2. High-resolution Camera (RGB), and LiDAR-derived rasters 3. Digital Elevation Models (DEM - DTM/Terrain and DSM/Surface) and 4. Ecosystem Structure (Canopy Height Model - CHM). You generated code to determine which AOP datasets are available in GEE for a given Image Collection. You explored the SDR image properties and learned how to filter on specified metadata to pull out a subset of Images or a single Image from an Image Collection. You learned how to use geemap to add data layers to the interactive map panel. And lastly, you learned how to select and visualize the Weather_Quality_Indicator band, which is a useful first step in working with AOP reflectance data.

This is a great starting point for your own research!