Skip to main content
NSF NEON, Operated by Battelle

Main navigation

  • About Us
    • Overview
      • Spatial and Temporal Design
      • History
    • Vision and Management
    • Advisory Groups
      • Science, Technology & Education Advisory Committee
      • Technical Working Groups (TWGs)
    • FAQ
    • Contact Us
      • Contact NEON Biorepository
      • Field Offices
    • User Accounts
    • Staff
    • Code of Conduct

    About Us

  • Data & Samples
    • Data Portal
      • Explore Data Products
      • Data Availability Charts
      • Spatial Data & Maps
      • Document Library
      • API & GraphQL
      • Prototype Data
      • External Lab Data Ingest (restricted)
    • Data Themes
      • Biogeochemistry
      • Ecohydrology
      • Land Cover and Processes
      • Organisms, Populations, and Communities
    • Samples & Specimens
      • Discover and Use NEON Samples
        • Sample Types
        • Sample Repositories
        • Sample Explorer
        • Megapit and Distributed Initial Characterization Soil Archives
      • Sample Processing
      • Sample Quality
      • Taxonomic Lists
    • Collection Methods
      • Protocols & Standardized Methods
      • Airborne Remote Sensing
        • Flight Box Design
        • Flight Schedules and Coverage
        • Daily Flight Reports
          • AOP Flight Report Sign Up
        • Camera
        • Imaging Spectrometer
        • Lidar
      • Automated Instruments
        • Site Level Sampling Design
        • Sensor Collection Frequency
        • Instrumented Collection Types
          • Meteorology
          • Phenocams
          • Soil Sensors
          • Ground Water
          • Surface Water
      • Observational Sampling
        • Site Level Sampling Design
        • Sampling Schedules
        • Observation Types
          • Aquatic Organisms
            • Aquatic Microbes
            • Fish
            • Macroinvertebrates & Zooplankton
            • Periphyton, Phytoplankton, and Aquatic Plants
          • Terrestrial Organisms
            • Birds
            • Ground Beetles
            • Mosquitoes
            • Small Mammals
            • Soil Microbes
            • Terrestrial Plants
            • Ticks
          • Hydrology & Geomorphology
            • Discharge
            • Geomorphology
          • Biogeochemistry
          • DNA Sequences
          • Pathogens
          • Sediments
          • Soils
            • Soil Descriptions
        • Optimizing the Observational Sampling Designs
    • Data Notifications
    • Data Guidelines and Policies
      • Acknowledging and Citing NEON
      • Publishing Research Outputs
      • Usage Policies
    • Data Management
      • Data Availability
      • Data Formats and Conventions
      • Data Processing
      • Data Quality
      • Data Product Bundles
      • Data Product Revisions and Releases
        • Release 2021
        • Release 2022
        • Release 2023
        • Release 2024
        • Release-2025
      • NEON and Google
      • Externally Hosted Data

    Data & Samples

  • Field Sites
    • About Field Sites and Domains
    • Explore Field Sites
    • Site Management Data Product

    Field Sites

  • Impact
    • Observatory Blog
    • Case Studies
    • Papers & Publications
    • Newsroom
      • NEON in the News
      • Newsletter Archive
      • Newsletter Sign Up

    Impact

  • Resources
    • Getting Started with NEON Data & Resources
    • Documents and Communication Resources
      • Papers & Publications
      • Document Library
      • Outreach Materials
    • Code Hub
      • Code Resources Guidelines
      • Code Resources Submission
      • NEON's GitHub Organization Homepage
    • Learning Hub
      • Science Videos
      • Tutorials
      • Workshops & Courses
      • Teaching Modules
    • Research Support Services
      • Field Site Coordination
      • Letters of Support
      • Mobile Deployment Platforms
      • Permits and Permissions
      • AOP Flight Campaigns
      • Research Support FAQs
      • Research Support Projects
    • Funding Opportunities

    Resources

  • Get Involved
    • Advisory Groups
      • Science, Technology & Education Advisory Committee
      • Technical Working Groups
    • Upcoming Events
    • NEON Ambassador Program
      • Exploring NEON-Derived Data Products Workshop Series
    • Research and Collaborations
      • Environmental Data Science Innovation and Inclusion Lab
      • Collaboration with DOE BER User Facilities and Programs
      • EFI-NEON Ecological Forecasting Challenge
      • NEON Great Lakes User Group
      • NEON Science Summit
      • NCAR-NEON-Community Collaborations
        • NCAR-NEON Community Steering Committee
    • Community Engagement
      • How Community Feedback Impacts NEON Operations
    • Science Seminars and Data Skills Webinars
      • Past Years
    • Work Opportunities
      • Careers
      • Seasonal Fieldwork
      • Internships
        • Intern Alumni
    • Partners

    Get Involved

  • My Account
  • Search

Search

Git 05: Git Add Changes - Commit

This tutorial reviews how to add and commit changes to a Git repo.

## Learning Objectives At the end of this activity, you will be able to:
  • Add new files or changes to existing files to your repo.
  • Document changes using the commit command with a message describing what has changed.
  • Describe the difference between git add and git commit.
  • Sync changes to your local repository with the repostored on GitHub.com.
  • Use and interpret the output from the following commands:
    • git status
    • git add
    • git commit
    • git push

Additional Resources

  • Diagram of Git Commands -- this diagram includes more commands than we will learn in this series but includes all that we use for our standard workflow.
  • GitHub Help Learning Git resources
  • Information on branches in Git -- we do not focus on the use of branches in Git or GitHub, however, if you want more information on this structure, this Git documentation may be of use.

In the previous lesson, we created a markdown (.md) file in our forked version of the DI-NEON-participants central repo. In order for Git to recognize this new file and track it, we need to:

  1. Add the file to the repository using git add.
  2. Commit the file to the repository as a set of changes to the repo (in this case, a new document with some text content) using git commit.
  3. Push or sync the changes we've made locally with our forked repo hosted on github.com using git push.
Graphic showing distributed version control workflow. After the repository has been cloned to your local computeryou can work on any file in the repository. You can use git pull to pull changes in your fork on github.com to your computer to ensure both repositories are in sync. Edits to the file on your computer will not be recognized by Git until you add and commit them as tracked changes in your repository.
After a Git repo has been cloned locally, you can now work on any file in the repo. You use git pull to pull changes in your fork on github.com down to your computer to ensure both repos are in sync. Edits to a file on your computer are not recognized by Git until you "add" and "commit" them as tracked changes in your repo. Source: National Ecological Observatory Network (NEON)

Check Repository Status -- git status

Let's first run through some basic commands to get going with Git at the command line. First, it's always a good idea to check the status of your repository. This allows us to see any changes that have occurred.

Do the following:

  1. Open bash if it's not already open.
  2. Navigate to the DI-NEON-participants repository in bash.
  3. Type: git status.

The commands that you type into bash should look like the code below:

# Change directory
# The directory containing the git repo that you wish to work in.
$ cd ~/Documents/GitHub/neon-data-repository-2016

# check the status of the repo
$ git status

Output:

On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git checkout -- <file>..." to discard changes in working directory)

Untracked files:
 (use "git add <file>..." to include in what will be committed)

_posts/ExampleFile.md

Let's make sense of the output of the git status command.

  • On branch master: This tells us that we are on the master branch of the repo. Don't worry too much about branches just yet. We will work on the master branch throughout the Data Institute.
  • Changes not staged for commit: This lists any file(s) that is/are currently being tracked by Git but have new changes that need to be added for Git to track.
  • Untracked file: These are all new files that have never been added to or tracked by Git.

Use git status anytime to view any untracked changes that have occurred, what is being tracked and what is not currently being tracked.

Add a File - git add

Next, let's add the Markdown file containing our bio and short project summary using the command git add FileName.md. Replace FileName.md with the name of your markdown file.

# add a file, so that changes are tracked
$ git add ExampleBioFile.md

# check status again
$ git status

On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
 (use "git reset HEAD <file>..." to unstage)

	new file:  _posts/ExampleBioFile.md

Understand the output:

  • Changes to be committed: This lists the new files or files with changes that have been added to the Git tracking system but need to be committed as actual changes in the git repository history.
**Data Tip:** If you want to delete a file from your repo, you can do so using `git rm file-name-here.fileExtension`. If you delete a file in the finder (Mac) or Windows Explorer, you will still have to use `git add` at the command line to tell git that a file has been removed from the repo, and to track that "change".

Commit Changes - git commit

When we add a file in the command line, we are telling Git to recognize that a change has occurred. The file moves to a "staging" area where Git recognizes a change has happened but the change has not yet been formally documented. When we want to permanently document those changes, we commit the change. A single commit will work for all files that are currently added to and in the Git staging area (anything in green when we check the status).

Commit Messages

When we commit a change to the Git version control system, we need to add a commit message. This message describes the changes made in the commit. This commit message is helpful to us when we review commit history to see what has changed over time and when those changes occurred. Be sure that your message covers the change.

**Data Tip:** It is good practice to keep commit messages to fewer than 50 characters.
# commit changes with message
$ git commit -m “new example file for demonstration”

[master e3cd622] new example file for demonstration
 1 file changed, 56 insertions(+), 4 deletions(-)
 create mode 100644 _posts/ExampleFile.md

Understand the output: Each commit will look slightly different but the important parts include:

  • master xxxxxxx this is the unique identifier for this set of changes or this commit. You will always be able to track this specific commit (this specific set of changes) using this identifier.
  • _ file change, _ insertions(+), _ deletion (-) this tells us how many files have changed and the number of type of changes made to the files including: insertions, and deletions.
**Data Tip:** It is a good idea to use `git status` frequently as you are working with Git in the shell. This allows you to keep track of change that you've made and what Git is actually tracking.

Why Add, then Commit?

Graphic showing the workflow of using the git add and git commit command. You can think of Git as taking snapshots of changes over the
life of a project. Git add specifies what will go in a snapshot, and git commit then actually takes the snapshot and makes a permanent record of it.
You can think of Git as taking snapshots of changes over the life of a project. git add specifies what will go in a snapshot (putting things in the staging area), and git commit then actually takes the snapshot and makes a permanent record of it (as a commit). Image and caption source: Software Carpentry

To understand what is going on with git add and git commit it is important to understand that Git has a staging area that we add items to with git add. Changes are not actually documented and permanently tracked until we commit them. This allows us to commit specific groups of files at the same time if we wish. For instance, we may decide to add and commit all R scripts together. And Markdown files in another, separate commit.

Transfer Changes (Commits) from a Local Repo to a GitHub Repo - git push

When we are done editing our files and have committed the changes locally, we are ready to transfer or sync these changes to our forked repo on github.com. To do this we need to push our changes from the local Git version control to the remote GitHub repo.

To sync local changes with github.com, we can do the following:

  1. Check the status of our repo using git status. Are all of the changes added and committed to the repo?
  2. Use git push origin master. origin tells Git to push the files to the originating repo which in this case - is our fork on github.com which we originally cloned to our local computer. master is the repo branch that you are currently working on.
**Data Tip:** Note about branches in Git: We won't cover branches in these tutorials, however, a Git repo can consist of many branches. You can think about a branch, like an additional copy of a repo where you can work on changes and updates.

Let's push the changes that we made to the local version of our Git repo to our fork, in our github.com account.

# check the repo status
$ git status

On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

# transfer committed changes to the forked repo
git push origin master

Counting objects: 1, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.51 KiB | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To https://github.com/mjones01/DI-NEON-participants.git
   5022aca..e3cd622  master -> master

NOTE: You may be asked for your username and password! This is your github.com username and password.

Understand the output:

  • Pay attention to the repository URL - the "origin" is the repository that the commit was pushed to, here https://github.com/mjones01/DI-NEON-participants.git. Note that because this repo is a fork, your URL will have your GitHub username in it instead of "mjones01".
**Data Tip:** You can use Git and connect to GitHub directly in the RStudio interface. If interested, read this R-bloggers How-To.

View Commits in GitHub

Let’s view our recent commit in our forked repo on GitHub.

  1. Go to github.com and navigate to your forked Data Institute repo - DI-NEON-participants.
  2. Click on the commits link at the top of the page.
  3. Look at the commits - do you see your recent commit message that you typed into bash on your computer?
  4. Next, click on the <>CODE link which is ABOVE the commits link in github.
  5. Is the Markdown file that you added and committed locally at the command line on your computer, there in the same directory (participants/pre-institute2-git) that you saved it on your laptop?
Screenshot of a forked NEON Data Instituterepository on github.com displaying an example .md file within the repository.
An example .md file located within the participants/2017-RemoteSensing/pre-institute2-git of a Data Institute repo fork. Source: National Ecological Observatory Network (NEON)

Is Your File in the NEON Central Repo Yet?

Next, do the following:

  1. Navigate to the NEON central NEONScience/DI-NEON-participants repo. (The easiest method to do this is to click the link at the top of the page under your repo name).
  2. Look for your file in the same directory. Is your new file there? If not, why?

Remember the structure of our workflow.

We’ve added changes from our local repo on our computer and pushed them to our fork on github.com. But this fork is in our individual user account, not NEONS. This fork is separate from the central repo. Changes to a fork in our github.com account do not automatically transfer to the central repo. We need to sync them! We will learn how to sync these two repos in the next tutorial Git 06: Syncing GitHub Repos with Pull Requests .

Summary Workflow - Committing Changes

On your computer, within your local copy of the Git repo:

  • Create a new markdown file and edit it in your favorite text editor.

On your computer, in shell (at the command line):

  • git status
  • git add FileName
  • git status - make sure everything is added and ready for commit
  • `git commit -m “messageHere”
  • git push origin master

On the github.com website:

  • Check to make sure commit is added.
  • Check to see if the file that you added is visible online in your Git repo.

Have questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

Git 03: Git Clone - Work Locally On Your Computer

This tutorial covers how to clone a github.com repo to your computer so that you can work locally on files within the repo.

## Learning Objectives At the end of this activity, you will be able to:
  • Be able to use the git clone command to create a local version of a GitHub repository on your computer.

Additional Resources

  • Diagram of Git Commands -- this diagram includes more commands than we will cover in this series but includes all that we use for our standard workflow.
  • GitHub Help Learning Git resources.

Clone - Copy Repo To Your Computer

In the previous tutorial, we used the github.com interface to fork the central NEON repo. By forking the NEON repo, we created a copy of it in our github.com account.

Graphic showing a fork of the central repository, which creates an exact copy of the repository in our own github account.
When you fork a repository on the github.com website, you are creating a duplicate copy of it in your github.com account. This is useful as a backup of the material. It also allows you to edit the material without modifying the original repository. Source: National Ecological Observatory Network (NEON)

Now we will learn how to create a local version of our forked repo on our laptop, so that we can efficiently add to and edit repo content.

Graphic showing the workflow of creating a clone from the forked copy of the central repository, which creates an exact copy of the forked repository to your own computer. This process allows you to make edits to the documents on your own computer, and also serves as another backup of the materials.
When you clone a repository to your local computer, you are creating a copy of that same repo locally on your computer. This allows you to edit files on your computer. And, of course, it is also yet another backup of the material! Source: National Ecological Observatory Network (NEON)

Copy Repo URL

Start from the github.com interface:

  1. Navigate to the repo that you want to clone (copy) to your computer -- this should be YOUR-USER-NAME/DI-NEON-participants.
  2. Click on the Clone or Download dropdown button and copy the URL of the repo.
Screenshot of the NEON Data Institute forked repository on your personal github.com account. The image highlights the clone or download button, which allows you to copy the URL that you will need to clone the repository or download the files in the repository as a .zip file.
The clone or download drop down allows you to copy the URL that you will need to clone a repository. Download allows you to download a .zip file containing all of the files in the repo. Source: National Ecological Observatory Network (NEON).

Then on your local computer:

  1. Your computer should already be setup with Git and a bash shell interface. If not, please refer to the Institute setup materials before continuing.
  2. Open bash on your computer and navigate to the local GitHub directory that you created using the Set-up Materials.

To do this, at the command prompt, type:

$ cd ~/Documents/GitHub

Note: If you have stored your GitHub directory in a location that is different

  • i.e. it is not /Documents/GitHub, be sure to adjust the above code to represent the actual path to the GitHub directory on your computer.

Now use git clone to clone, or create a copy of, the entire repo in the GitHub directory on your computer.

# clone the forked repo to our computer
$ git clone https://github.com/neon/DI-NEON-participants.git
**Data Tip:** Are you a Windows user and are having a hard time copying the URL into shell? You can copy and paste in the shell environment **after** you have the feature turned on. Right click on your bash shell window (at the top) and select "properties". Make sure "quick edit" is checked. You should now be able to copy and paste within the bash environment.

The output shows you what is being cloned to your computer.

Cloning into 'DI-NEON-participants.git'...
remote: Counting objects: 3808, done.
remote: Total 3808 (delta 0), reused 0 (delta 0), pack-reused 3808
Receiving objects: 100% (3808/3808), 2.92 MiB | 2.17 MiB/s, done.
Resolving deltas: 100% (2185/2185), done.
Checking connectivity... done.
$

Note: The output numbers that you see on your computer, representing the total file size, etc, may differ from the example provided above.

View the New Repo

Next, let's make sure the repository is created on your computer in the location where you think it is.

At the command line, type ls to list the contents of the current directory.

# view directory contents
$ ls

Next, navigate to your copy of the data institute repo using cd or change directory:

# navigate to the NEON participants repository
$ cd DI-NEON-participants

# view repository contents
$ ls

404.md			_includes		code
ISSUE_TEMPLATE.md	_layouts		images
README.md		_posts			index.md
_config.yml		_site			institute-materials
_data			assets			org

Alternatively, we can view the local repo DI-NEON-participants in a finder (Mac) or Windows Explorer (Windows) window. Simply open your Documents in a window and navigate to the new local repo.

Using either method, we can see that the file structure of our cloned repo exactly mirrors the file structure of our forked GitHub repo.

**Thought Question:** Is the cloned version of this repo that you just created on your laptop, a direct copy of the NEON central repo -OR- of your forked version of the NEON central repo?

Summary Workflow -- Create a Local Repo

In the github.com interface:

  • Copy URL of the repo you want to work on locally

In shell:

  • git clone URLhere

Note: that you can copy the URL of your repository directly from GitHub.


Got questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

Git 02: GitHub.com - Repos & Forks

In this tutorial, we will fork, or create a copy in your github.com account, an existing GitHub repository. We will also explore the github.com interface.

## Learning Objectives At the end of this activity, you will be able to:
  • Create a GitHub account.
  • Know how to navigate to and between GitHub repositories.
  • Create your own fork, or copy, a GitHub repository.
  • Explain the relationship between your forked repository and the master repository it was created from.

Additional Resources

  • Diagram of Git Commands -- this diagram includes more commands than we will learn in this series but includes all that we use for our standard workflow.
  • GitHub Help Learning Git resources

Create An Account

If you do not already have a GitHub account, go to GitHub and sign up for your free account. Pick a username that you like! This username is what your colleagues will see as you work with them in GitHub and Git.

Take a minute to setup your account. If you want to make your account more recognizable, be sure to add a profile picture to your account!

If you already have a GitHub account, simply sign in.

**Data Tip:** Are you a student? Sign up for the Student Developer Pack and get the Git Personal account free (with unlimited private repos and other discounts/options; normally $7/month).

Navigate GitHub

Repositories, AKA Repos

Let's first discuss the repository or "repo". (The cool kids say repo, so we will jump on the git cool kid bandwagon) and use "repo" from here on in. According to the GitHub glossary:

A repository is the most basic element of GitHub. They're easiest to imagine as a project's folder. A repository contains all of the project files (including documentation), and stores each file's revision history. Repositories can have multiple collaborators and can be either public or private.

In the Data Institute, we will share our work in the DI-NEON-participants repo.

Find an Existing Repo

The first thing that you'll need to do is find the DI-NEON-participants repo. You can find repos in two ways:

  1. Type “DI-NEON-participants” in the github.com search bar to find the repository.
  2. Use the repository URL if you have it - like so: https://github.com/NEONScience/DI-NEON-participants.

Navigation of a Repo Page

Once you have found the Data Institute participants repo, take 5 minutes to explore it.

Git Repo Names

First, get to know the repository naming convention. Repository names all take the format:

OrganizationName/RepositoryName

So the full name of our repository is:

NEONScience/DI-NEON-participants

Header Tabs

At the top of the page you'll notice a series of tabs. Please focus on the following 3 for now:

  • Code: Click here to view structure & contents of the repo.
  • Issues: Submit discussion topics, or problems that you are having with the content in the repo, here.
  • Pull Requests: Submit changes to the repo for review / acceptance. We will explore pull requests more in the Git 06 tutorial.
Screenshot of the NEON Data Institute central repository on github.com highlighting the search bar, and six tabs below the repository name including: Code, Issues, Pull Request, Pulse, Graphics, and Settings.
Screenshot of the NEON Data Institute central repository (note, there has been a slight change in the repo name). The github.com search bar is at the top of the page. Notice there are 6 "tabs" below the repo name including: Code, Issues, Pull Request, Pulse, Graphics and Settings. NOTE: Because you are not an administrator for this repo, you will not see the "Settings" tab in your browser. Source: National Ecological Observatory Network (NEON)

Other Text Links

A bit further down the page, you'll notice a few other links:

  • commits: a commit is a saved and documented change to the content or structure of the repo. The commit history contains all changes that have been made to that repo. We will discuss commits more in Git 05: Git Add Changes -- Commits .

Fork a Repository

Next, let's discuss the concept of a fork on the github.com site. A fork is a copy of the repo that you create in your account. You can fork any repo at any time by clicking the fork button in the upper right hand corner on github.com.

Graphic showing the fork button as it appears on the upper right hand corner of the github website.
Click on the "Fork" button to fork any repo. Source: GitHub Guides.
Graphic showing a fork of the central repository, which creates an exact copy of the repository in our own github account.
When we fork a repo in github.com, we are telling Git to create an exact copy of the repo that we're forking in our own github.com account. Once the repo is in our own account, we can edit it as we now own that fork. Note that a fork is just a copy of the repo on github.com. Source: National Ecological Observatory Network (NEON)
## Activity: Fork the NEON Data Institute Participants Repo

Create your own fork of the DI-NEON-participants now.

**Data Tip:** You can change the name of a forked repo and it will still be connected to the central repo from which it was forked. For now, leave it the same.

Check Out Your Data Institute Fork

Now, check out your new fork. Its name should be:

YOUR-USER-NAME/DI-NEON-participants.

It can get confusing sometimes moving between a central repo:

  • https://github.com/NEONScience/DI-NEON-participants

and your forked repo:

  • https://github.com/YOUR-USER-NAME/DI-NEON-participants

A good way to figure out which repo you are viewing is to look at the name of the repo. Does it contain your username? Or your colleagues'? Or NEON's?

Your Fork vs the Central Repo

Your fork is an exact copy, or completely in sync with, the NEON central repo. You could confirm this by comparing your fork to the NEON central repository using the pull request option. We will learn about pull requests in Git06: Sync GitHub Repos with Pull Requests. For now, take our word for it.

The fork will remain in sync with the NEON central repo until:

  1. You begin to make changes to your forked copy of the repo.
  2. The central repository is changed or updated by a collaborator.

If you make changes to your forked repo, the changes will not be added to the NEON central repo until you sync your fork with the NEON central repo.

Summary Workflow -- Fork a GitHub Repository

On the github.com website:

  • Navigate to desired repo that you want to fork.
  • Click Fork button.

Have questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

Git 01: Intro to Git Version Control

In this page, you will be introduced to the importance of version control in scientific workflows.

## Learning Objectives At the end of this activity, you will be able to:
  • Explain what version control is and how it can be used.
  • Explain why version control is important.
  • Discuss the basics of how the Git version control system works.
  • Discuss how GitHub can be used as a collaboration tool.

The text and graphics in the first three sections were borrowed, with some modifications, from Software Carpentry's Version Control with Git lessons.

What is Version Control?

A version control system maintains a record of changes to code and other content. It also allows us to revert changes to a previous point in time.

Cartoon showing a graduate student and his advisor going through several iterations, and respective changes to the name, of a document.
Many of us have used the "append a date" to a file name version of version control at some point in our lives. Source: "Piled Higher and Deeper" by Jorge Cham www.phdcomics.com

Types of Version control

There are many forms of version control. Some not as good:

  • Save a document with a new date (we’ve all done it, but it isn’t efficient)
  • Google Docs "history" function (not bad for some documents, but limited in scope).

Some better:

  • Mercurial
  • Subversion
  • Git - which we’ll be learning much more about in this series.
**Thought Question:** Do you currently implement any form of version control in your work?

More Resources:

Visit the version control Wikipedia list of version control platforms.

  • Read the Git documentation explaining the progression of version control systems.

Why Version Control is Important

Version control facilitates two important aspects of many scientific workflows:

  1. The ability to save and review or revert to previous versions.
  2. The ability to collaborate on a single project.

This means that you don’t have to worry about a collaborator (or your future self) overwriting something important. It also allows two people working on the same document to efficiently combine ideas and changes.

**Thought Questions:** Think of a specific time when you weren’t using version control that it would have been useful.
  • Why would version control have been helpful to your project & work flow?
  • What were the consequences of not having a version control system in place?

How Version Control Systems Works

Simple Version Control Model

A version control system keeps track of what has changed in one or more files over time. The way this tracking occurs, is slightly different between various version control tools including git, mercurial and svn. However the principle is the same.

Version control systems begin with a base version of a document. They then save the committed changes that you make. You can think of version control as a tape: if you rewind the tape and start at the base document, then you can play back each change and end up with your latest version.

Graphic showing how a version control system saves changes to a document, sequentially, as you add and commit them to the system.
A version control system saves changes to a document, sequentially, as you add and commit them to the system. Source: Software Carpentry

Once you think of changes as separate from the document itself, you can then think about “playing back” different sets of changes onto the base document. You can then retrieve, or revert to, different versions of the document.

The benefit of version control when you are in a collaborative environment is that two users can make independent changes to the same document.

Graphic showing how different versions of the same document can be saved within a version control system.
Different versions of the same document can be saved within a version control system. Source: Software Carpentry

If there aren’t conflicts between the users changes (a conflict is an area where both users modified the same part of the same document in different ways) you can review two sets of changes on the same base document.

Graphic showing how two sets of changes to the same bsae document can be reviewed together within a version control system if there are no conflicts.
Two sets of changes to the same base document can be reviewed together, within a version control system if there are no conflicts (areas where both users modified the same part of the same document in different ways). Changes submitted by both users can then be merged together. Source: Software Carpentry

A version control system is a tool that keeps track of these changes for us. Each version of a file can be viewed and reverted to at any time. That way if you add something that you end up not liking or delete something that you need, you can simply go back to a previous version.

Git & GitHub - A Distributed Version Control Model

GitHub uses a distributed version control model. This means that there can be many copies (or forks in GitHub world) of the repository.

Graphic showing one of the advantages of using a distributed version control system. In a version control system, there are many copies of the repository, thus if any server or computer dies, any of the client repositories can be copied and used to restore the data.
One advantage of a distributed version control system is that there are many copies of the repository. Thus, if any server or computer dies, any of the client repositories can be copied and used to restore the data! Every clone (or fork) is a full backup of all the data. Source: Pro Git by Scott Chacon & Ben Straub

Have a look at the graphic below. Notice that in the example, there is a "central" version of our repository. Joe, Sue and Eve are all working together to update the central repository. Because they are using a distributed system, each user (Joe, Sue and Eve) has their own copy of the repository and can contribute to the central copy of the repository at any time.

Graphic showing an example of the use of a distributed version control system. In this example, Joe, Sue, and Eve are all working together to update a central repository. Each user has their own copy of the repository, and can contribute to the central repository at a any time.
Distributed version control models allow many users to contribute to the same central document. Source: Better Explained

Create A Working Copy of a Git Repo - Fork

There are many different Git and GitHub workflows. In the NEON Data Institute, we will use a distributed workflow with a Central Repository. This allows us all (all of the Institute participants) to work independently. We can then contribute our changes to update the Central (NEON) Repository. Our collaborative workflow goes like this:

  1. NEON "owns" the Central Repository.
  2. You will create a copy of this repository (known as a fork) in your own GitHub account.
  3. You will then clone (copy) the repository to your local computer. You will do your work locally on your laptop.
  4. When you are ready to submit your changes to the NEON repository, you will:
    • Sync your local copy of the repository with NEON's central repository so you have the most up to date version, and then,
    • Push the changes you made to your local copy (or fork) of the repository to NEON's main repository.

Each participant in the institute will be contributing to the NEON central repository using the same workflow! Pretty cool stuff.

Graphic showing the workflow of working with the NEON central repository. Workflow includes: Forking or creating a copy of the central repository into your personal github account. Cloning your fork to your local computer, where you can make edits. Pushing or transferring those edits back to your local fork, and submitting a pull request to update the central repository.
The NEON central repository is the final working version of our project. You can fork or create a copy of this repository into your github.com account. You can then copy or clone your fork, to your local computer where you can make edits. When you are done working, you can push or transfer those edits back to your local fork. When you are read to update the NEON central repository, you submit a pull request. We will walk through the steps of this workflow over the next few lessons. Source: National Ecological Observatory Network (NEON)

Let's get some terms straight before we go any further.

  • Central repository - the central repository is what all participants will add to. It is the "final working version" of the project.
  • Your forked repository - is a "personal” working copy of the central repository stored in your GitHub account. This is called a fork. When you are happy with your work, you update your repo from the central repo, then you can update your changes to the central NEON repository.
  • Your local repository - this is a local version of your fork on your own computer. You will most often do all of your work locally on your computer.
**Data Tip:** Other Workflows -- There are many other git workflows. Read more about other workflows. This resource mentions Bitbucket, another web-based hosting service like GitHub.

Additional Resources:

Further documentation for and how-to-use direction for Git, is provided by the Git Pro version 2 book by Scott Chacon and Ben Straub , available in print or online. If you enjoy learning from videos, the site hosts several.

Vector 02: Plot Multiple Shapefiles and Create Custom Legends in Base Plot in R

This tutorial builds upon the previous tutorial, to work with shapefile attributes in R and explores how to plot multiple shapefiles using base R graphics. It then covers how to create a custom legend with colors and symbols that match your plot.

Learning Objectives

After completing this tutorial, you will be able to:

  • Plot multiple shapefiles using base R graphics.
  • Apply custom symbology to spatial objects in a plot in R.
  • Customize a baseplot legend in R.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and preferably RStudio loaded on your computer to complete this tutorial.

Install R Packages

  • raster: install.packages("raster")
  • rgdal: install.packages("rgdal")
  • sp: install.packages("sp")

More on Packages in R – Adapted from Software Carpentry.

Download Data

NEON Teaching Data Subset: Site Layout Shapefiles

These vector data provide information on the site characterization and infrastructure at the National Ecological Observatory Network's Harvard Forest field site. The Harvard Forest shapefiles are from the Harvard Forest GIS & Map archives. US Country and State Boundary layers are from the US Census Bureau.

Download Dataset

Set Working Directory: This lesson assumes that you have set your working directory to the location of the downloaded and unzipped data subsets.

An overview of setting the working directory in R can be found here.

R Script & Challenge Code: NEON data lessons often contain challenges that reinforce learned skills. If available, the code for challenge solutions is found in the downloadable R script of the entire lesson, available in the footer of each lesson page.

Load the Data

To work with vector data in R, we can use the rgdal library. The raster package also allows us to explore metadata using similar commands for both raster and vector files.

We will import three shapefiles. The first is our AOI or area of interest boundary polygon that we worked with in Open and Plot Shapefiles in R. The second is a shapefile containing the location of roads and trails within the field site. The third is a file containing the Harvard Forest Fisher tower location. These latter two we worked with in the Explore Shapefile Attributes & Plot Shapefile Objects by Attribute Value in R tutorial.

# load packages
# rgdal: for vector work; sp package should always load with rgdal. 
library(rgdal)  
# raster: for metadata/attributes- vectors or rasters
library(raster)   

# set working directory to data folder
# setwd("pathToDirHere")

# Import a polygon shapefile 
aoiBoundary_HARV <- readOGR("NEON-DS-Site-Layout-Files/HARV",
                            "HarClip_UTMZ18", stringsAsFactors = T)

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HarClip_UTMZ18"
## with 1 features
## It has 1 fields
## Integer64 fields read as strings:  id

# Import a line shapefile
lines_HARV <- readOGR( "NEON-DS-Site-Layout-Files/HARV", "HARV_roads", stringsAsFactors = T)

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HARV_roads"
## with 13 features
## It has 15 fields

# Import a point shapefile 
point_HARV <- readOGR("NEON-DS-Site-Layout-Files/HARV",
                      "HARVtower_UTM18N", stringsAsFactors = T)

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HARVtower_UTM18N"
## with 1 features
## It has 14 fields

Plot Data

In the Explore Shapefile Attributes & Plot Shapefile Objects by Attribute Value in R tutorial we created a plot where we customized the width of each line in a spatial object according to a factor level or category. To do this, we create a vector of colors containing a color value for EACH feature in our spatial object grouped by factor level or category.

# view the factor levels
levels(lines_HARV$TYPE)

## [1] "boardwalk"  "footpath"   "stone wall" "woods road"

# create vector of line width values
lineWidth <- c(2,4,3,8)[lines_HARV$TYPE]
# view vector
lineWidth

##  [1] 8 4 4 3 3 3 3 3 3 2 8 8 8

# create a color palette of 4 colors - one for each factor level
roadPalette <- c("blue","green","grey","purple")
roadPalette

## [1] "blue"   "green"  "grey"   "purple"

# create a vector of colors - one for each feature in our vector object
# according to its attribute value
roadColors <- c("blue","green","grey","purple")[lines_HARV$TYPE]
roadColors

##  [1] "purple" "green"  "green"  "grey"   "grey"   "grey"   "grey"  
##  [8] "grey"   "grey"   "blue"   "purple" "purple" "purple"

# create vector of line width values
lineWidth <- c(2,4,3,8)[lines_HARV$TYPE]
# view vector
lineWidth

##  [1] 8 4 4 3 3 3 3 3 3 2 8 8 8

# in this case, boardwalk (the first level) is the widest.
plot(lines_HARV, 
     col=roadColors,
     main="NEON Harvard Forest Field Site\n Roads & Trails \nLine Width Varies by Type Attribute Value",
     lwd=lineWidth)

Roads and trails at NEON Harvard Forest Field Site with color and line width varied by attribute factor value.

**Data Tip:** Given we have a factor with 4 levels, we can create a vector of numbers, each of which specifies the thickness of each feature in our `SpatialLinesDataFrame` by factor level (category): `c(6,4,1,2)[lines_HARV$TYPE]`

Add Plot Legend

In the the previous tutorial, we also learned how to add a basic legend to our plot.

  • bottomright: We specify the location of our legend by using a default keyword. We could also use top, topright, etc.
  • levels(objectName$attributeName): Label the legend elements using the categories of levels in an attribute (e.g., levels(lines_HARV$TYPE) means use the levels boardwalk, footpath, etc).
  • fill=: apply unique colors to the boxes in our legend. palette() is the default set of colors that R applies to all plots.

Let's add a legend to our plot.

plot(lines_HARV, 
     col=roadColors,
     main="NEON Harvard Forest Field Site\n Roads & Trails\n Default Legend")

# we can use the color object that we created above to color the legend objects
roadPalette

## [1] "blue"   "green"  "grey"   "purple"

# add a legend to our map
legend("bottomright", 
       legend=levels(lines_HARV$TYPE), 
       fill=roadPalette, 
       bty="n", # turn off the legend border
       cex=.8) # decrease the font / legend size

Roads and trails at NEON Harvard Forest Field Site with color varied by attribute factor value and with a default legend.

However, what if we want to create a more complex plot with many shapefiles and unique symbols that need to be represented clearly in a legend?

Plot Multiple Vector Layers

Now, let's create a plot that combines our tower location (point_HARV), site boundary (aoiBoundary_HARV) and roads (lines_HARV) spatial objects. We will need to build a custom legend as well.

To begin, create a plot with the site boundary as the first layer. Then layer the tower location and road data on top using add=TRUE.

# Plot multiple shapefiles
plot(aoiBoundary_HARV, 
     col = "grey93", 
     border="grey",
     main="NEON Harvard Forest Field Site")

plot(lines_HARV, 
     col=roadColors,
     add = TRUE)

plot(point_HARV, 
     add  = TRUE, 
     pch = 19, 
     col = "purple")

Roads and tower location at NEON Harvard Forest Field Site with color varied by attribute type.

# assign plot to an object for easy modification!
plot_HARV<- recordPlot()

Customize Your Legend

Next, let's build a custom legend using the symbology (the colors and symbols) that we used to create the plot above. To do this, we will need to build three things:

  1. A list of all "labels" (the text used to describe each element in the legend to use in the legend.
  2. A list of colors used to color each feature in our plot.
  3. A list of symbols to use in the plot. NOTE: we have a combination of points, lines and polygons in our plot. So we will need to customize our symbols!

Let's create objects for the labels, colors and symbols so we can easily reuse them. We will start with the labels.

# create a list of all labels
labels <- c("Tower", "AOI", levels(lines_HARV$TYPE))
labels

## [1] "Tower"      "AOI"        "boardwalk"  "footpath"   "stone wall"
## [6] "woods road"

# render plot
plot_HARV

# add a legend to our map
legend("bottomright", 
       legend=labels, 
       bty="n", # turn off the legend border
       cex=.8) # decrease the font / legend size

Roads and tower location at NEON Harvard Forest Field Site with color varied by attribute type and with a modified legend.

Now we have a legend with the labels identified. Let's add colors to each legend element next. We can use the vectors of colors that we created earlier to do this.

# we have a list of colors that we used above - we can use it in the legend
roadPalette

## [1] "blue"   "green"  "grey"   "purple"

# create a list of colors to use 
plotColors <- c("purple", "grey", roadPalette)
plotColors

## [1] "purple" "grey"   "blue"   "green"  "grey"   "purple"

# render plot
plot_HARV

# add a legend to our map
legend("bottomright", 
       legend=labels, 
       fill=plotColors,
       bty="n", # turn off the legend border
       cex=.8) # decrease the font / legend size

Roads and tower location at NEON Harvard Forest Field Site with color and a modified legend varied by attribute type.

Great, now we have a legend! However, this legend uses boxes to symbolize each element in the plot. It might be better if the lines were symbolized as a line and the points were symbolized as a point symbol. We can customize this using pch= in our legend: 16 is a point symbol, 15 is a box.

**Data Tip:** To view a short list of `pch` symbols, type `?pch` into the R console.
# create a list of pch values
# these are the symbols that will be used for each legend value
# ?pch will provide more information on values
plotSym <- c(16,15,15,15,15,15)
plotSym

## [1] 16 15 15 15 15 15

# Plot multiple shapefiles
plot_HARV

# to create a custom legend, we need to fake it
legend("bottomright", 
       legend=labels,
       pch=plotSym, 
       bty="n", 
       col=plotColors,
       cex=.8)

Roads and tower location at NEON Harvard Forest Field Site with color and a modified legend varied by attribute type; each symbol on the legend corresponds to the shapefile type (i.e., tower = point).

Now we've added a point symbol to represent our point element in the plot. However it might be more useful to use line symbols in our legend rather than squares to represent the line data. We can create line symbols, using lty = (). We have a total of 6 elements in our legend:

  1. A Tower Location
  2. An Area of Interest (AOI)
  3. and 4 Road types (levels)

The lty list designates, in order, which of those elements should be designated as a line (1) and which should be designated as a symbol (NA). Our object will thus look like lty = c(NA,NA,1,1,1,1). This tells R to only use a line element for the 3-6 elements in our legend.

Once we do this, we still need to modify our pch element. Each line element (3-6) should be represented by a NA value - this tells R to not use a symbol, but to instead use a line.

# create line object
lineLegend = c(NA,NA,1,1,1,1)
lineLegend

## [1] NA NA  1  1  1  1

plotSym <- c(16,15,NA,NA,NA,NA)
plotSym

## [1] 16 15 NA NA NA NA

# plot multiple shapefiles
plot_HARV

# build a custom legend
legend("bottomright", 
       legend=labels, 
       lty = lineLegend,
       pch=plotSym, 
       bty="n", 
       col=plotColors,
       cex=.8)

Roads and tower location at NEON Harvard Forest Field Site with color and a modified legend varied by attribute type; each symbol on the legend corresponds to the shapefile type [i.e., tower = point, roads = lines].

### Challenge: Plot Polygon by Attribute
  1. Using the NEON-DS-Site-Layout-Files/HARV/PlotLocations_HARV.shp shapefile, create a map of study plot locations, with each point colored by the soil type (soilTypeOr). How many different soil types are there at this particular field site? Overlay this layer on top of the lines_HARV layer (the roads). Create a custom legend that applies line symbols to lines and point symbols to the points.

  2. Modify the plot above. Tell R to plot each point, using a different symbol of pch value. HINT: to do this, create a vector object of symbols by factor level using the syntax described above for line width: c(15,17)[lines_HARV$soilTypeOr]. Overlay this on top of the AOI Boundary. Create a custom legend.

Roads and study plots at NEON Harvard Forest Field Site with color and a modified legend varied by attribute type; each symbol on the legend corresponds to the shapefile type [i.e., soil plots = points, roads = lines].Roads and study plots at NEON Harvard Forest Field Site with color and a modified legend varied by attribute type; each symbol on the legend corresponds to the shapefile type [i.e., soil plots = points, roads = lines], and study plots symbols vary by soil type.

Publish Code - From R Markdown to HTML with knitr

In this tutorial, we will cover the R knitr package that is used to convert R Markdown into a rendered document (HTML, PDF, etc).

Learning Objectives

At the end of this activity, you will:

  • Be able to produce (‘knit’) an HTML file from a R Markdown file.
  • Know how to modify chunk options to change the output in your HTML file.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio loaded on your computer to complete this tutorial.

Install R Packages

  • knitr: install.packages("knitr")
  • rmarkdown: install.packages("rmarkdown")

Share & Publish Results Directly from Your Code!

The knitr package allow us to:

  • Publish & share preliminary results with collaborators.
  • Create professional reports that document our workflow and results directly from our code, reducing the risk of accidental copy and paste or transcription errors.
  • Document our workflow to facilitate reproducibility.
  • Efficiently change code outputs (figures, files) given changes in the data, methods, etc.

Publish from Rmd files with knitr

To complete this tutorial you need:

  1. The R knitr package to complete this tutorial. If you need help installing packages, visit the R packages tutorial.
  2. An R Markdown document that contains a YAML header, code chunks and markdown segments. If you don't have an .Rmd file, visit the R Markdown tutorial to create one.
**When To Knit**: Knitting is a useful exercise throughout your scientific workflow. It allows you to see what your outputs look like and also to test that your code runs without errors. The time required to knit depends on the length and complexity of the script and the size of your data.

How to Knit

RStudio window with R Markdown template of new document and 'Knit HTML' button 
circled.
Location of the knit button in RStudio in Version 0.99.486. Source: National Ecological Observatory Network (NEON)

To knit in RStudio, click the knit pull down button. You want to use the
knit HTML for this lesson.

When you click the Knit HTML button, a window will open in your console titled R Markdown. This pane shows the knitting progress. The output (HTML in this case) file will automatically be saved in the current working directory. If there is an error in the code, an error message will appear with a line number in the R Console to help you diagnose the problem.

**Data Tip:** You can run `knitr` from the command prompt using: `render(“input.Rmd”, “all”)`.

Activity: Knit Script

Knit the .Rmd file that you built in the last tutorial. What does it look like?

View the Output

RStudio windows of R Markdown file, with activity content added, 
and HTML document with text, code, output and Digital Surface Model plot figure.
R Markdown (left) and the resultant HTML (right) after knitting. Source: National Ecological Observatory Network (NEON)

When knitting is complete, the new HTML file produced will automatically open.

Notice that information from the YAML header (title, author, date) is printed at the top of the HTML document. Then the HTML shows the text, code, and results of the code that you included in the RMD document.

Data Institute Participants: Complete Week 2 Assignment

  • Read this week’s assignment closely.
  • Be sure to carefully check your knitr output to make sure it is rendering the way you think it should!
  • When you are complete, submit your .Rmd and .html files to the NEON Institute participants GitHub repository (NEONScience/DI-NEON-participants).
  • The files will have automatically saved to your R working directory, you will need to transfer the files to the /participants/pre-institute3-rmd/ directory and submitted via a pull request.

Document Code with R Markdown

You will need to have the rmarkdown and knitr packages installed on your computer prior to completing this tutorial. Refer to the setup materials to get these installed.

Learning Objectives

At the end of this activity, you will:

  • Know how to create an R Markdown file in RStudio.
  • Be able to write a script with text and R code chunks.
  • Create an R Markdown document ready to be ‘knit’ into an HTML document to share your code and results.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio loaded on your computer to complete this tutorial.

Install R Packages

  • knitr: install.packages("knitr")
  • rmarkdown: install.packages("rmarkdown")
  • raster: install.packages("raster")
  • rgdal: install.packages("rgdal")

Download Data

Download NEON Teaching Data Subset: TEAK-Data Institute 2016

The LiDAR and imagery data used to create this raster teaching data subset were collected over the National Ecological Observatory Network's (NEON) Lower Teakettle field site and processed at NEON headquarters. The entire dataset can be accessed by request from the NEON Data Portal.

Download Dataset

You will want to create a data directory for all the Data Institute teaching datasets. We suggest the pathway be ~/Documents/data/NEONDI-2016 or the equivalent for your operating system. Once you've downloaded and unzipped the dataset, move it to this directory.

Mac OS Finder window with directory structure showing 'NEONDI-2016' folder contents
The data directory with the teaching data subset. This is the suggested organization for all Data Institute teaching data subsets. Source: National Ecological Observatory Network (NEON)

Additional Resources

  • R Markdown Cheatsheet: a very handy reference for using R Markdown
  • R Markdown Reference Guide: a more expensive reference for R Markdown
  • Introduction to R Markdown by Garrett Grolemund: a tutorial for learning R Markdown

Create an Rmd File

RMarkdown in RStudio Video

Our goal in this series is to document our workflow. We can do this by

  1. Creating an R Markdown (RMD) file in R studio and
  2. Rendering that RMD file to HTML using knitr.

Watch this 6:38 minute video below to learn more about how you can convert an R Markdown file to HTML (or other formats) using knitr in RStudio. The text size in the video is small so you may want to watch the video in full screen mode.

Now that you have a sense of how R Markdown can be used in RStudio, you are ready to create your own RMD document. Do the following:

  1. Create a new R Markdown file and choose HTML as the desired output format.
  2. Enter a Title (Explore NEON LiDAR Data) and Author Name (your name). Then click OK.
  3. Save the file using the following format: LastName-institute-week3.rmd NOTE: The document title is not the same as the file name.
  4. Hit the knit button in RStudio (as is done in the video above). What happens?
RStudio window with R Markdown template of new document and 'Knit HTML' button 
circled.
Location of the knit button in RStudio in Version 0.99.486. Source: National Ecological Observatory Network (NEON)

If everything went well, you should have an HTML format (web page) output after you hit the knit button. Note that this HTML output is built from a combination of code and documentation that was written using markdown syntax.

Next, we'll break down the structure of an R Markdown file.

Understand Structure of an R Markdown file

RStudio window with R Markdown template of new document, including header, 
markdown, and code chunk.
Screenshot of a new R Markdown document in RStudio. Notice the different parts of the document. Source: National Ecological Observatory Network (NEON)
**Data Tip:** Screenshots on this page are from RStudio with appearance preferences set to `Twilight` with `Monaco` font. You can change the appearance of your RStudio by **Tools** > **Options** (or **Global Options** depending on the operating system). For more, see the Customizing RStudio page.

Let's next review the structure of an R Markdown (.Rmd) file. There are three main content types:

  • Header: the text at the top of the document, written in YAML format.
  • Markdown sections: text that describes your workflow written using markdown syntax.
  • Code chunks: Chunks of R code that can be run and also can be rendered using knitr to an output document.

Next let's explore each section type.

Header -- YAML block

An R Markdown file always starts with header written using YAML syntax. There are four default elements in the RStudio generated YAML header:

  • title: the title of your document. Note, this is not the same as the file name.
  • author: who wrote the document.
  • date: by default this is the date that the file is created.
  • output: what format will the output be in. We will use HTML.

A YAML header may be structured differently depending upon how your are using it. Learn more on the R Markdown documentation page.

## Activity: R Markdown YAML Customize the header of your `.Rmd` file as follows:
  • Title: Provide a title that fits the code that will be in your RMD.
  • Author: Add your name here.
  • Output: Leave the default output setting: html_document. We will be rendering an HTML file.

R Markdown Text/Markdown Blocks

An RMD document contains a mixture of code chunks and markdown blocks where you can describe aspects of your processing workflow. The markdown blocks use the same markdown syntax that we learned last week in week 2 materials. In these blocks you might describe the data that you are using, how it's being processed and and what the outputs are. You may even add some information that interprets the outputs.

When you render your document to HTML, this markdown will appear as text on the output HTML document.

Learn More about RStudio Markdown Basics

Explore Your R Markdown File

Look closely at the pre-populated markdown and R code chunks in your RMD file.

Does any of the markdown syntax look familiar?

  • Are any words in bold?
  • Are any words in italics?
  • Are any words highlighted as code?

If you are unsure, the answers are at the bottom of this page.

## Activity: R Markdown Text
  1. Remove the template markdown and code chunks added to the RMD file by RStudio. (Be sure to keep the YAML header!)
  2. At the very top of your RMD document - after the YAML header, add the bio and short research description that you wrote last week in markdown syntax to the RMD file.
  3. Between your profile and the research descriptions, add a header that says About My Project (or something similar).
  4. Add a new header stating R Markdown Activity and text below that explaining that this page demonstrates using some of the NEON Teakettle LiDAR data products in R. The wording of this text should clearly describe the code and outputs that you will be adding the page.
**Data Tip**: You can add code output or an R object name to markdown segments of an RMD. For more, view this R Markdown documentation.

Code chunks

Code chunks are where your R code goes. All code chunks start and end with ``` – three backticks or graves. On your keyboard, the backticks can be found on the same key as the tilde. Graves are not the same as an apostrophe!

The initial line of a code chunk must appear as:

 ```{r chunk-name-with-no-spaces} 
# code goes here
 ```

The r part of the chunk header identifies this chunk as an R code chunk and is mandatory. Next to the {r, there is a chunk name. This name is not required for basic knitting however, it is good practice to give each chunk a unique name as it is required for more advanced knitting approaches.

Activity: Add Code Chunks to Your R Markdown File

Continue working on your document. Below the last section that you've just added, create a code chunk that loads the packages required to work with raster data in R.

```{r setup-library }
   
   library(rgdal)
   library(raster)
 
 ```

In R scripts, setting the working directory is normally done once near the beginning of your script. In R Markdown files, knit code chunks behave a little differently, and a warning appears upon kitting a chunk that sets a working directory.

```{r code-setwd}

   # set working directory to ensure R can find the file we wish to import.
   # This will depend on your local environment.
   setwd("~/Documents/data/NEONDI-2016/") 

```
You changed the working directory to ~/Documents/data/NEONDI-2016/ (probably via setwd()). It will be restored to [directory path of current .rmd file]. See the Note section in ?knitr::knit ?knitr::knit

That's a bad sign if you want to set the working directory in one code chunk, and read or write data in another code chunk. To allow for a working data directory that is different from your Rmd file's current directory, you can store the directory path in a string variable.

```{r code-setwd-stringvariable}

	# set working directory as a string variable for use in other code chunks.
	# This will depend on your local environment.
	wd <- "~/Documents/data/NEONDI-2016/"
	setwd(wd)

 ```

The setwd(wd) line could be at the start of a lengthier code chunk that reads from and writes to data files. Alternatively, since the variable will be kept in this document's R environment, it can be used with paste() or paste0() when you need to refer to a filepath. Proceed to the next step for an example of this.

(For further instruction on setting the working directory, see the NEON Data Skills tutorial Set A Working Directory in R.)

Let's add another chunk that loads the TEAK_lidarDSM raster file.

```{r load-dsm-raster }
   # check for the working directory
   getwd()
   # In this new chunk, the working directory has reverted to default upon kitting. 
   # Combining the working directory string variable and 
   # additional path to the file, import a DSM file.
   teak_dsm <- raster(paste0(wd, "NEONdata/D17-California/TEAK/2013/lidar/TEAK_lidarDSM.tif"))
   
 ```

Now run the code in this chunk.

You can run code chunks:

  • Line-by-line: with cursor on current line, Ctrl + Enter (Windows/Linux) or Command + Enter (Mac OS X).
  • By chunk: You can run the entire chunk (or multiple chunks) by clicking on the "Run" button in the upper right corner of the RStudio script panel and choosing the appropriate option (Run Current Chunk, Run Next Chunk). Keyboard shortcuts are available for these options.

Code chunk options

You can also add arguments or options to each code chunk. These arguments allow you to customize how or if you want code to be processed or appear on the output HTML document. Code chunk arguments are added on the first line of a code chunk after the name, within the curly brackets.

The example below, is a code chunk that will not be "run", or evaluated, by R. The code within the chunk will appear on the output document, however there will be no outputs from the code.

```{r intro-option, eval=FALSE}
# the code here will not be processed by R 
# but it will appear on your output document
1+2
 ```

We use eval=FALSE often when the chunk is exporting an file that we don't need to re-export but we want to document the code used to export the file.

Three common code chunk options are:

  • eval = FALSE: Do not evaluate (or run) this code chunk when knitting the RMD document. The code in this chunk will still render in our knitted HTML output, however it will not be evaluated or run by R.
  • echo = FALSE: Hide the code in the output. The code is evaluated when the RMD file is knit, however only the output is rendered on the output document.
  • results = hide: The code chunk will be evaluated but the results of the code will not be rendered on the output document. This is useful if you are viewing the structure of a large object (e.g. outputs of a large data.frame).

Multiple code chunk options can be used for the same chunk. For more on code chunk options, read R Markdown: The Definitive Guide or the knitr documentation.

## Activity: Add More Code to Your R Markdown

Update your RMD file as follows:

  • Add a new code chunk that plots the TEAK_lidarDSM raster object that you imported above. Experiment with plot colors and be sure to add a plot title.
  • Run the code chunk that you just added to your RMD document in R (e.g. run in console, not knitting). Does it create a plot with a title?
  • In another new code chunk, import and plot another raster file from the NEON data subset that you downloaded. The TEAK_lidarCHM is a good raster to plot.
  • Finally, create histograms for both rasters that you've imported into R.
  • Be sure to document your steps as you go using both code comments and markdown syntax in between the code chunks.

For help opening and plotting raster data in R, see the NEON Data Skills tutorial Plot Raster Data in R.

We will knit this document to HTML in the next tutorial.

Now continue on to the next tutorial to learn how to knit this document into a HTML file.

## Answers to the Default Text Markdown Syntax Questions
  • Are any words in bold? - Yes, “Knit” on line 10
  • Are any words in italics? - No
  • Are any words highlighted as code? - Yes, “echo = FALSE” on line 22

Document & Publish Your Workflow: R Markdown & knitr

This tutorial we will work with the knitr and rmarkdown packages within RStudio to learn how to effectively and efficiently document and publish our workflows online.

Learning Objectives

At the end of this activity, you will be able to:

  • Explain why documenting and publishing one's code is important.
  • Describe two tools that enable ease of publishing code & output: R Markdown and the knitr package.

Documentation Is Important

As we read in the Reproducible Science overview, the four facets of reproducible science are:

  • Documentation
  • Organization,
  • Automation and
  • Dissemination.

This week we will learn about the R Markdown file format (and R package) which can be used with the knitr package to document and publish (disseminate) your code and code output.

View Slideshow: Share, Publish & Archive - from the Reproducible Science Curriculum

The Tools We Will Use

R Markdown

“R Markdown is an authoring format that enables easy creation of dynamic documents, presentations, and reports from R. It combines the core syntax of markdown (an easy to write plain text format) with embedded R code chunks that are run so their output can be included in the final document. R Markdown documents are fully reproducible (they can be automatically regenerated whenever underlying R code or data changes)." -- RStudio documentation.

We use markdown syntax in R Markdown (.rmd) files to document workflows and to share data processing, analysis and visualization outputs. We can also use it to create documents that combine R code, output and text.

**Data Tip:** Most of the NEON Data Skills educational resources on this site are built using R Markdown files.

Why R Markdown?

There are many advantages to using R Markdown in your work:

  • Human readable syntax.
  • Simple syntax - it can be learned quickly.
  • All components of your work are clearly documented. You don't have to remember what steps, assumptions, tests were used.
  • You can easily extend or refine analyses by modifying existing or adding new code blocks.
  • Analysis results can be disseminated in various formats including HTML, PDF, slide shows and more.
  • Code and data can be shared with a colleague to replicate the workflow.
**Data Tip:** RPubs is a quick way to share and publish code.

Knitr

The knitr package for R allows us to create readable documents from R Markdown files.

RStudio window with default template of a new R Markdown file. HTML document generated from knit R Markdown file, including 
formatted YAML header, text, code, computed data output and scatter plot.
R Markdown script (left) and the HTML produced from the knit R Markdown script (right). Source: National Ecological Observatory Network (NEON)
>The knitr package was designed to be a transparent engine for dynamic report generation with R -- Yihui Xi -- knitr package creator

In the next tutorial we will learn more about working with the R Markdown format in RStudio.

Download a NEON Teaching Data Subset & Set A Working Directory In R

The primary goal of this tutorial is to explain how to set a working directory in R. The working directory is where your R session interacts with your hard drive. This is where you can read data that you want to use, and save new information such as derived data products, tables, maps, and figures. It is a good practice to store your information in an organized set of directories, so you will often want to change your working directory depending on what kinds of information that you need to access. This tutorial teaches how to download and unzip the data files that accompany many NEON Data Skills tutorials, and also covers the concept of file paths. You can read from top to bottom, or use the menu bar at left to navigate to your desired topic.

Learning Objectives

After completing this tutorial, you will be able to:

  • Be able to download and uncompress NEON Teaching Data Subsets.
  • Be able to set the R working directory.
  • Know the difference between full, base and relative paths.
  • Be able to write out both full and relative paths for a given file or directory.

Things You’ll Need To Complete This Lesson

To complete this lesson you will need the most current version of R and, preferably, RStudio loaded on your computer.

Download Data

NEON Teaching Data Subset: Meteorological Data for Harvard Forest

The data used in this lesson were collected at the National Ecological Observatory Network's Harvard Forest field site. These data are proxy data for what will be available for 30 years on the NEON data portal for the Harvard Forest and other field sites located across the United States.

Download Dataset

NEON Teaching Data Subset: Site Layout Shapefiles

These vector data provide information on the site characterization and infrastructure at the National Ecological Observatory Network's Harvard Forest field site. The Harvard Forest shapefiles are from the Harvard Forest GIS & Map archives. US Country and State Boundary layers are from the US Census Bureau.

Download Dataset

Set Up For NEON Data Skills Tutorials

Many NEON data tutorials utilize teaching data subsets which are hosted on the NEON Data Skills figshare repository. If a data subset is required for a tutorial it can be downloaded at the top of each tutorial in the Download Data section.

Prior to working with any data in R, we must set the working directory to the location of the data files. Setting the working directory tells R where the data files are located on the computer. If the working directory is not correctly set first, when we try to open a file we will get an error telling us that R cannot find the file.

**Data Tip:** All NEON Data Skills tutorials are written assuming the working directory is the parent directory to the uncompressed .zip file of downloaded data. This allows for multiple data subsets to be accessed in the tutorial without resetting the working directory. Generally, these tutorials have a default working directory of **~/Documents/data**. If you are working on a Mac, we suggest that you save your downloaded datasets in a directory with the same name and location so that you don't have to edit the code for the tutorial that you are working on. Most windows machines cannot use the tilde "~" notation, therefore you must define the working directory explicitly.
  • Wondering why we're saying directory instead of folder? See our discussion of Directory vs. Folder in the middle of this tutorial.

Download & Uncompress the Data

1) Download

First, we will download the data to a location on the computer. To download the data for this tutorial, click the blue button Download NEON Teaching Data Subset: Meteorological Data for Harvard Forest within the box at the top of this page.

Note: In other NEON Data Skills tutorials, download all data subsets in the Download Data section prior to starting the tutorial. Here, the second data subset is for those wishing to practice these skills in a Challenge activity and will be downloaded at that time.

Example Download Data box containing two download buttons, one labeled 'Download NEON Teaching Data Subset: Meteorological Data for Harvard Forest' and another labeled 'Download NEON Teaching Data Subset: Site Layout Shapefiles
Screenshot of the Download Data button at the top of NEON Data Skills tutorials. Source: National Ecological Observatory Network (NEON)

After clicking on the Download Data button, the data will automatically download to the computer.

2) Locate .zip file

Second, we need to find the downloaded .zip file. Many browsers default to downloading to the Downloads directory on your computer. Note: You may have previously specified a specific directory (folder) for files downloaded from the internet, if so, the .zip file will download there.

Screenshot of the computer's Downloads folder containing the new NEONDSMetTimeSeries.zip file. Source: National Ecological Observatory Network (NEON)

3) Move to data directory

Third, we must move the data files to the location we want to work with them. We recommend moving the .zip to a dedicated data directory within the Documents directory on your computer. This data directory can then be a repository for all data subsets you use for the NEON Data Skills tutorials. Note: If you chose to store your data in a different directory (e.g., not in ~/Documents/data), modify the directions below with the appropriate file path to your data directory.

4) Unzip/uncompress

Fourth, we need to unzip/uncompress the file so that the data files can be accessed. Use your favorite tool that can unpackage/open .zip files (e.g., winzip, Archive Utility, etc). The files will now be accessible in a directory named NEON-DS-Met-Time-Series containing all the subdirectories and files that make up the dataset or the subdirectories and files will be unzipped directly into the data directory. If the latter happens, they need to be moved into a data/NEON-DS-Met-Time-Series directory.

### Challenge: Download and Unzip Teaching Data Subset Want to make sure you have these steps down! Prepare the **Site Layout Shapefiles Teaching Data Subset** so that the files are accessible and ready to be opened in R.

The directory should be the same as in this screenshot (below). Note that NEON-DS-Site-Lyout-Files directory will only be in your directory if you completed the challenge above. If you did not, your directory should look the same but without that directory.

Screenshot of the neon directory with the nested Documents, data, NEON-DS-Met-Time-Series, and other directories. Source: National Ecological Observatory Network (NEON)

Directory vs. Folder

"Directory" and "Folder" both refer to the same thing. Folder makes a lot of sense when we think of an isolated folder as a "bin" containing many files. However, the analogy to a physical file folder falters when we start thinking about the relationship between different folders and how we tell a computer to find a specific folder. This is why the term directory is often preferred. Any directory (folder) can hold other directories and/or files. When we set the working directory, we are telling the computer the location of the directory (or folder) to start with when looking for other files or directories, or to save any output to.

Full, Base, and Relative Paths

The data downloaded and unzipped in the previous steps are located within a nested set of directories:

  • primary-level/home directory: neon
    • This directory isn't obvious as we are within this directory once we log into the computer.
    • You will see your own user ID.
  • secondary-level directory: neon/Documents
  • tertiary-level directory: neon/Documents/data
  • quaternary-level directory: neon/Documents/data/NEON-DS-Met-Time-Series
  • quaternary-level directory: neon/Documents/data/NEON-DS-Site-Layout-Shapefiles

Full & Base Path

The full path is essentially the complete "directions" for how to find the desired directory or file. It always starts with the home directory or root (e.g., /Users/neon/). A full path is the base path when used to set the working directory to a specific directory. The base path for the NEON-DS-Met-Time-Series directory would be:

 /Users/neon/Documents/data/NEON-DS-Met-Time-Series 
**Data Tip:** File or directory paths and the home directory will appear slightly different in different operating systems. Linux will appear as `/home/neon/`. Windows will be similar to `C:\Documents and Settings\neon\` or `C:\Users\neon\`. The format varies by Windows version. Make special note of the direction of the slashes. Mac OS X and Unix format will appear as `/Users/neon/`. This tutorial will show Mac OS X output unless specifically noted.
### Challenge: Full File Path Write out the full path for the `NEON-DS-Site-Layout-Shapefiles` directory. Use the format of the operating system you are currently using.

Tip: When typing in the Rstudio console or an R script, if you surround your filepath with quotes you can take advantage of the 'tab-completion' feature. To use this feature, begin typing your filepath (e.g., "~/" or "C:") and then hit the tab button, which should pop up a list of possible directories and files that you could be pointing to. This method is awesome for avoiding typos in complex or long filepaths!

Bonus Points: Write the path as for one of the other operating systems.

Relative Path

A relative path is a path to a directory or file that starts from the location determined by the working directory. If our working directory is set to the data directory,

 /Users/neon/Documents/data/

we can then create a relative path for all directories and files within the data directory.

Screenshot of the data directory containing the both NEON Data Skills Teaching Subsets. Source: National Ecological Observatory Network (NEON)

The relative path for the meanNDVI_HARV_2011.csv file would be:

 NEON-DS-Met-Time-Series/HARV/NDVI/meanNDVI_HARV_2011.csv
### Challenge: Relative File Path Use the format of your current operating system:
  1. Write out the full path to for the Boundary-US-State-Mass.shp file.
  2. Write out the relative path for the Boundary-US-State-Mass.shp file assuming that the working directory is set to /Users/neon/Documents/data/.

Bonus: Write the paths as for one of the other operating systems.

The R Working Directory

In R, the working directory is the directory where R starts when looking for any file to open (as directed by a file path) and where it saves any output.

Without a working directory all R scripts would need the full file path written any time we wanted to open or save a file. It is more efficient if we have a base file path set as our working directory and then all file paths written in our scripts only consist of the file path relative to that base path (a relative path).

  • If you are unfamiliar with the term full path, base path, or relative path, please see the section below on Full, Base, and Relative Paths for a more detailed explanation before continuing with this tutorial.

Find a Full Path to a File in Unknown Location

If you are unsure of the path to a specific directory or file, you can find this information for a particular file/directory of interest by looking in the:

  • Windows: Properties, General tab (right click on the file/directory) or in the file path bar at the top of each window (select versions).
  • Mac OS X: Get Info (right clicking/control+click on the file/directory)

The file path may appear as:

Computer > Users > neon > Documents > data > NEON-DS-Met-Time-Series

Copy and paste this information to automatically reformat into the full path to the directory or file:

  • Windows: C:\Users\neon\Documents\data\NEON-DS-Met-Time-Series
  • Mac OS X: /Users/neon/Documents/data/NEON-DS-Met-Time-Series
**Data Tip:** File or directory paths and the home directory will appear slightly different in different operating systems. Linux will appear as `/home/neon/`. Windows will be similar to `C:\Documents and Settings\neon\` or `C:\Users\neon\`. The format varies by Windows version. Make special note of the direction of the slashes. Mac OS X and Unix format will appear as `/Users/neon/`. This tutorial will show Mac OSX output unless specifically noted.

Determine Current Working Directory

Once we are in the R program, we can view the current working directory using the code getwd().

# view current working directory 
getwd()
[1] "/Users/neon"

The working directory is currently set to the home directory /Users/neon. Remember, your current working directory will have a different user name and may appear different based on operating system.

This code can be used at any time to determine the current working directory.

Set the Working Directory

To set our current working directory to the location where our data are located, we can either set the working directory in the R script or use our current GUI to select the working directory.

**Data Tip:** All NEON Data Skills tutorials are written assuming the working directory is the parent directory to the downloaded data (the **data** directory in this tutorial). This allows for multiple data subsets to be accessed in the tutorial without resetting the working directory.

We want to set our working directory to the data directory.

Set the Working Directory: Base Path in Script

We can set the working directory using the code setwd("PATH") where PATH is the full path to the desired directory. You can enter "PATH" as a string (as shown below), or save the file path as a string to a variable (e.g., wd <- "~/Documents/data") and then set the working directory based on that variable (e.g., setwd(wd)).

This latter method is used in many of the NEON Data Skills tutorials because of the advantages that this method provides. First, this method is extermely flexible across different compute environments and formats, including personal computers, Linux-based servers on 'the cloud' (e.g., AWS, CyVerse), and when using Rmarkdown (.Rmd) documents. Second this method allows the tutorial's user to set their working directory once as a string and then to reuse that string as needed to reference input files, or make output files. For example, some functions must have a full filepath to an input file (such as when reading in HDF5 files). Third, this method simplifies the process that NEON uses internally to create and update these tutorials. All in all, saving the working directory as a string variable makes the code more explicit and determanistic without relying on working knowledge of relative filepaths, making your code more producible and easier for an outsider to interpret.

To practice, use these techniques to set your working directory to the directory where you have the data saved, and check that you set the working directory correctly. There is no R output from setwd(). If we want to check that the working directory is correctly set we can use getwd().

Example Windows File Path

Notice the the backslashes used in Windows paths must be changed to slashes in R.

# set the working directory to `data` folder
wd <- "C:/Users/neon/Documents/data"
setwd(wd)

# check to ensure path is correct
getwd()
[1] "C:/Users/neon/Documents/data"

Example Mac OS X File Path

# set the working directory to `data` folder
wd <- "/Users/neon/Documents/data"
setwd(wd)

# check to ensure path is correct
getwd()
[1] "/Users/neon/Documents/data"
**Data Tip:** If using RStudio, you can view the contents of the working directory in the Files pane.
The Files pane in RStudio shows the contents of the current working directory. Source: National Ecological Observatory Network (NEON)

Set the Working Directory: Using RStudio GUI

You can also set the working directory using the Rstudio and/or R graphical user interface (GUI). This method is easy for beginners to learn, but it also makes your code less reproducible because it relies on a person to follow certain instructions, which is a process that introduces human error. It may also be impossible for an observer to determine where your input data are stored, which can make troubleshooting more difficult as well. Use this method when getting started, or when you will find it helpful to use a graphical user interface to navigate your files. Note that this method will run a single line setwd() command in the console when you select your working directory, so you can copy/paste that line into your script for future use!

  1. Go to Session in menu bar,
  2. select Select Working Directory,
  3. select Choose Directory,
  4. in the new window that appears, select the appropriate directory.
How to set the working directory using the RStudio GUI. Source: National Ecological Observatory Network (NEON)

Set the Working Directory: Using R GUI

Windows Operating Systems:

  1. Go to the File menu bar,
  2. select Change dir... or Change Working Directory,
  3. in the new window that appears, select the appropriate directory.
How to set the working directory using the R GUI in Windows. Source: National Ecological Observatory Network (NEON)

Mac Operating Systems:

  1. Go to the Misc menu,
  2. select Change Working Directory,
  3. in the new window that appears, select the appropriate directory.
How to set the working directory using the R GUI in Mac OS X. Source: National Ecological Observatory Network (NEON)

Raster 04: Work With Multi-Band Rasters - Image Data in R

This tutorial explores how to import and plot a multiband raster in R. It also covers how to plot a three-band color image using the plotRGB() function in R.

Learning Objectives

After completing this tutorial, you will be able to:

  • Know how to identify a single vs. a multiband raster file.
  • Be able to import multiband rasters into R using the terra package.
  • Be able to plot multiband color image rasters in R using plotRGB().
  • Understand what a NoData value is in a raster.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio installed on your computer to complete this tutorial.

Install R Packages

  • terra: install.packages("terra")

  • neonUtilities: install.packages("neonUtilities")

  • More on Packages in R – Adapted from Software Carpentry.

Data to Download

Data required for this tutorial will be downloaded using neonUtilities in the lesson.

The LiDAR and imagery data used in this lesson were collected over the National Ecological Observatory Network's Harvard Forest (HARV) field site.

The entire dataset can be accessed from the NEON Data Portal.


Set Working Directory: This lesson assumes that you have set your working directory to the location of the downloaded and unzipped data subsets.

An overview of setting the working directory in R can be found here.

R Script & Challenge Code: NEON data lessons often contain challenges that reinforce skills. If available, the code for challenge solutions is found in the downloadable R script of the entire lesson, available in the footer of each lesson page.

The Basics of Imagery - About Spectral Remote Sensing Data

About Raster Bands in R

As discussed in the Intro to Raster Data tutorial, a raster can contain 1 or more bands.

Left: 3D image of a raster with only one band. Right: 3D image showing four separate layers of a multi band raster.
A raster can contain one or more bands. We can use the terra `rast` function to import one single band from a single OR multi-band raster. Source: National Ecological Observatory Network (NEON).

To work with multiband rasters in R, we need to change how we import and plot our data in several ways.

  • To import multiband raster data we will use the stack() function.
  • If our multiband data are imagery that we wish to composite, we can use plotRGB() (instead of plot()) to plot a 3 band raster image.

About MultiBand Imagery

One type of multiband raster dataset that is familiar to many of us is a color image. A basic color image consists of three bands: red, green, and blue. Each band represents light reflected from the red, green or blue portions of the electromagnetic spectrum. The pixel brightness for each band, when composited creates the colors that we see in an image.

A graphic depicting the three different color bands (red, green, and blue) of a satellite image and how they create a basic color image when composited.
A color image consists of 3 bands - red, green and blue. When rendered together in a GIS, or even a tool like Photoshop or any other image software, they create a color image. Source: National Ecological Observatory Network (NEON).

Getting Started with Multi-Band Data in R

To work with multiband raster data we will use the terra package.

# terra package to work with raster data

library(terra)



# package for downloading NEON data

library(neonUtilities)



# package for specifying color palettes

library(RColorBrewer)



# set working directory to ensure R can find the file we wish to import

wd <- "~/data/" # this will depend on your local environment environment

# be sure that the downloaded file is in this directory

setwd(wd)

In this tutorial, the multi-band data that we are working with is imagery collected using the NEON Airborne Observation Platform high resolution camera over the NEON Harvard Forest field site. Each RGB image is a 3-band raster. The same steps would apply to working with a multi-spectral image with 4 or more bands - like Landsat imagery, or even hyperspectral imagery (in geotiff format). We can plot each band of a multi-band image individually.

byTileAOP(dpID='DP3.30010.001', # rgb camera data
          site='HARV',
          year='2022',
          easting=732000,
          northing=4713500,
          check.size=FALSE, # set to TRUE or remove if you want to check the size before downloading
          savepath = wd)

## Downloading files totaling approximately 351.004249 MB 
## Downloading 1 files
## 

|
| | 0% |
|====================================================================================================================================================================| 100% ## Successfully downloaded 1 files to ~/data//DP3.30010.001

Red, green, and blue composite (true color) image of NEON's Harvard Forest (HARV) site

Or we can plot each bands separately as follows:

# Determine the number of bands

num_bands <- nlyr(RGB_HARV)



# Define colors to plot each

# Define color palettes for each band using RColorBrewer

colors <- list(
  brewer.pal(9, "Reds"),
  brewer.pal(9, "Greens"),
  brewer.pal(9, "Blues")
)



# Plot each band in a loop, with the specified colors

for (i in 1:num_bands) {
  plot(RGB_HARV[[i]], main=paste("Band", i), col=colors[[i]])
}

Red bandGreen bandBlue band

Image Raster Data Attributes

We can display some of the attributes about the raster, as shown below:

# Print dimensions

cat("Dimensions:\n")

## Dimensions:

cat("Number of rows:", nrow(RGB_HARV), "\n")

## Number of rows: 10000

cat("Number of columns:", ncol(RGB_HARV), "\n")

## Number of columns: 10000

cat("Number of layers:", nlyr(RGB_HARV), "\n")

## Number of layers: 3

# Print resolution

resolutions <- res(RGB_HARV)

cat("Resolution:\n")

## Resolution:

cat("X resolution:", resolutions[1], "\n")

## X resolution: 0.1

cat("Y resolution:", resolutions[2], "\n")

## Y resolution: 0.1

# Get the extent of the raster

rgb_extent <- ext(RGB_HARV)



# Convert the extent to a string with rounded values

extent_str <- sprintf("xmin: %d, xmax: %d, ymin: %d, ymax: %d", 
                      round(xmin(rgb_extent)), 
                      round(xmax(rgb_extent)), 
                      round(ymin(rgb_extent)), 
                      round(ymax(rgb_extent)))



# Print the extent string

cat("Extent of the raster: \n")

## Extent of the raster:

cat(extent_str, "\n")

## xmin: 732000, xmax: 733000, ymin: 4713000, ymax: 4714000

Let's take a look at the coordinate reference system, or CRS. You can use the parameters describe=TRUE to display this information more succinctly.

crs(RGB_HARV, describe=TRUE)

##                    name authority  code
## 1 WGS 84 / UTM zone 18N      EPSG 32618
##                                                                                                                                                                                                                                                          area
## 1 Between 78°W and 72°W, northern hemisphere between equator and 84°N, onshore and offshore. Bahamas. Canada - Nunavut; Ontario; Quebec. Colombia. Cuba. Ecuador. Greenland. Haiti. Jamaica. Panama. Turks and Caicos Islands. United States (USA). Venezuela
##            extent
## 1 -78, -72, 84, 0

Let's next examine the raster's minimum and maximum values. What is the range of values for each band?

# Replace Inf and -Inf with NA

values(RGB_HARV)[is.infinite(values(RGB_HARV))] <- NA



# Get min and max values for all bands

min_max_values <- minmax(RGB_HARV)



# Print the results

cat("Min and Max Values for All Bands:\n")

## Min and Max Values for All Bands:

print(min_max_values)

##     2022_HARV_7_732000_4713000_image_1 2022_HARV_7_732000_4713000_image_2 2022_HARV_7_732000_4713000_image_3
## min                                  0                                  0                                  0
## max                                255                                255                                255

This raster contains values between 0 and 255. These values represent the intensity of brightness associated with the image band. In the case of a RGB image (red, green and blue), band 1 is the red band. When we plot the red band, larger numbers (towards 255) represent pixels with more red in them (a strong red reflection). Smaller numbers (towards 0) represent pixels with less red in them (less red was reflected). To plot an RGB image, we mix red + green + blue values into one single color to create a full color image - this is the standard color image a digital camera creates.

Challenge: Making Sense of Single Bands of a Multi-Band Image

Go back to the code chunk where you plotted each band separately. Compare the plots of band 1 (red) and band 2 (green). Is the forested area darker or lighter in band 2 (the green band) compared to band 1 (the red band)?

Other Types of Multi-band Raster Data

Multi-band raster data might also contain:

  1. Time series: the same variable, over the same area, over time.
  2. Multi or hyperspectral imagery: image rasters that have 4 or more (multi-spectral) or more than 10-15 (hyperspectral) bands. Check out the NEON Data Skills Imaging Spectroscopy HDF5 in R tutorial to learn how to work with hyperspectral data cubes.

The true color image plotted at the beginning of this lesson looks pretty decent. We can explore whether applying a stretch to the image might improve clarity and contrast using stretch="lin" or stretch="hist".

Graphic depicting stretching pixel brightness values to make a dark satellite image brighter
When the range of pixel brightness values is closer to 0, a darker image is rendered by default. We can stretch the values to extend to the full 0-255 range of potential values to increase the visual contrast of the image.
Graphic depicting stretching pixel brightness values to make a bright satellite image darker
When the range of pixel brightness values is closer to 255, a lighter image is rendered by default. We can stretch the values to extend to the full 0-255 range of potential values to increase the visual contrast of the image.
# What does stretch do?



# Plot the linearly stretched raster

plotRGB(RGB_HARV, stretch="lin")

Composite RGB image of HARV with a linear stretch

# Plot the histogram-stretched raster

plotRGB(RGB_HARV, stretch="hist")

Composite RGB image of HARV with a histogram stretch

In this case, the stretch doesn't enhance the contrast our image significantly given the distribution of reflectance (or brightness) values is distributed well between 0 and 255, and applying a stretch appears to introduce some artificial, almost purple-looking brightness to the image.

Challenge: What Methods Can Be Used on an R Object?

We can view various methods available to call on an R object with methods(class=class(objectNameHere)). Use this to figure out:

  1. What methods can be used to call on the RGB_HARV object?
  2. What methods are available for a single band within RGB_HARV?
  3. Why do you think there is a difference?

Pagination

  • First page
  • Previous page
  • …
  • Page 48
  • Page 49
  • Page 50
  • Page 51
  • Current page 52
  • Page 53
  • Page 54
  • Page 55
  • Page 56
  • …
  • Next page
  • Last page
Subscribe to
NSF NEON, Operated by Battelle

Follow Us:

Join Our Newsletter

Get updates on events, opportunities, and how NEON is being used today.

Subscribe Now

Footer

  • About Us
  • Newsroom
  • Contact Us
  • Terms & Conditions
  • Careers
  • Code of Conduct

Copyright © Battelle, 2025

The National Ecological Observatory Network is a major facility fully funded by the U.S. National Science Foundation.

Any opinions, findings and conclusions or recommendations expressed in this material do not necessarily reflect the views of the U.S. National Science Foundation.