This tutorial reviews how to add and commit changes to a Git repo.
## Learning Objectives
At the end of this activity, you will be able to:
Add new files or changes to existing files to your repo.
Document changes using the commit command with a message describing what has changed.
Describe the difference between git add and git commit.
Sync changes to your local repository with the repostored on GitHub.com.
Use and interpret the output from the following commands:
git status
git add
git commit
git push
Additional Resources
Diagram of Git Commands
-- this diagram includes more commands than we will
learn in this series but includes all that we use for our standard workflow.
Information on branches in Git
-- we do not focus on the use of branches in Git or GitHub, however, if you want
more information on this structure, this Git documentation may be of use.
In the previous lesson, we created a markdown (.md) file in our forked version
of the DI-NEON-participants central repo. In order for Git to recognize this
new file and track it, we need to:
Add the file to the repository using git add.
Commit the file to the repository as a set of changes to the repo (in this case, a new
document with some text content) using git commit.
Push or sync the changes we've made locally with our forked repo hosted on github.com
using git push.
After a Git repo has been cloned locally, you can now work on
any file in the repo. You use git pull to pull changes in your
fork on github.com down to your computer to ensure both repos are in sync.
Edits to a file on your computer are not recognized by Git until you
"add" and "commit" them as tracked changes in your repo.
Source: National Ecological Observatory Network (NEON)
Check Repository Status -- git status
Let's first run through some basic commands to get going with Git at the command
line. First, it's always a good idea to check the status of your repository.
This allows us to see any changes that have occurred.
Do the following:
Open bash if it's not already open.
Navigate to the DI-NEON-participants repository in bash.
Type: git status.
The commands that you type into bash should look like the code below:
# Change directory
# The directory containing the git repo that you wish to work in.
$ cd ~/Documents/GitHub/neon-data-repository-2016
# check the status of the repo
$ git status
Output:
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
Untracked files:
(use "git add <file>..." to include in what will be committed)
_posts/ExampleFile.md
Let's make sense of the output of the git status command.
On branch master: This tells us that we are on the master branch of the
repo. Don't worry too much about branches just yet. We will work on the master branch
throughout the Data Institute.
Changes not staged for commit: This lists any file(s) that is/are currently
being tracked by Git but have new changes that need to be added for Git to track.
Untracked file: These are all new files that have never been added to or
tracked by Git.
Use git status anytime to view any untracked changes that have occurred, what
is being tracked and what is not currently being tracked.
Add a File - git add
Next, let's add the Markdown file containing our bio and short project summary
using the command git add FileName.md. Replace FileName.md with the name
of your markdown file.
# add a file, so that changes are tracked
$ git add ExampleBioFile.md
# check status again
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: _posts/ExampleBioFile.md
Understand the output:
Changes to be committed: This lists the new files or files with changes that
have been added to the Git tracking system but need to be committed as actual changes
in the git repository history.
**Data Tip:** If you want to delete a file from your
repo, you can do so using `git rm file-name-here.fileExtension`. If you delete
a file in the finder (Mac) or Windows Explorer, you will still have to use
`git add` at the command line to tell git that a file has been removed from the
repo, and to track that "change".
Commit Changes - git commit
When we add a file in the command line, we are telling Git to recognize that
a change has occurred. The file moves to a "staging" area where Git
recognizes a change has happened but the change has not yet been formally
documented. When we want to permanently document those changes, we
commit the change. A single commit will work for all files that are currently
added to and in the Git staging area (anything in green when we check the status).
Commit Messages
When we commit a change to the Git version control system, we need to add a commit
message. This message describes the changes made in the commit. This commit
message is helpful to us when we review commit history to see what has changed
over time and when those changes occurred. Be sure that your message
covers the change.
**Data Tip:** It is good practice to keep commit messages to fewer than 50 characters.
# commit changes with message
$ git commit -m “new example file for demonstration”
[master e3cd622] new example file for demonstration
1 file changed, 56 insertions(+), 4 deletions(-)
create mode 100644 _posts/ExampleFile.md
Understand the output:
Each commit will look slightly different but the important parts include:
master xxxxxxx this is the unique identifier for this set of changes or
this commit. You will always be able to track this specific commit (this specific
set of changes) using this identifier.
_ file change, _ insertions(+), _ deletion (-) this tells us how many files
have changed and the number of type of changes made to the files including:
insertions, and deletions.
**Data Tip:**
It is a good idea to use `git status` frequently as you are working with Git
in the shell. This allows you to keep track of change that you've made and what
Git is actually tracking.
Why Add, then Commit?
You can think of Git as taking snapshots of changes over the
life of a project. git add specifies what will go in a snapshot (putting things
in the staging area), and git commit then actually takes the snapshot and
makes a permanent record of it (as a commit). Image and caption source:
Software Carpentry
To understand what is going on with git add and git commit it is important
to understand that Git has a staging area that we add items to with git add.
Changes are not actually documented and permanently tracked until we commit them. This allows
us to commit specific groups of files at the same time if we wish. For instance,
we may decide to add and commit all R scripts together. And Markdown files in another,
separate commit.
Transfer Changes (Commits) from a Local Repo to a GitHub Repo - git push
When we are done editing our files and have committed the changes locally, we
are ready to transfer or sync these changes to our forked repo on github.com. To
do this we need to push our changes from the local Git version control to the
remote GitHub repo.
To sync local changes with github.com, we can do the following:
Check the status of our repo using git status. Are all of the changes added
and committed to the repo?
Use git push origin master. origin tells Git to push the files to the
originating repo which in this case - is our fork on github.com which we originally
cloned to our local computer. master is the repo branch that you are
currently working on.
**Data Tip:**
Note about branches in Git: We won't cover branches in these tutorials, however,
a Git repo can consist of many branches. You can think about a branch, like
an additional copy of a repo where you can work on changes and updates.
Let's push the changes that we made to the local version of our Git repo to our
fork, in our github.com account.
# check the repo status
$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
# transfer committed changes to the forked repo
git push origin master
Counting objects: 1, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.51 KiB | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To https://github.com/mjones01/DI-NEON-participants.git
5022aca..e3cd622 master -> master
NOTE: You may be asked for your username and password! This is your github.com
username and password.
Understand the output:
Pay attention to the repository URL - the "origin" is the
repository that the commit was pushed to, here https://github.com/mjones01/DI-NEON-participants.git.
Note that because this repo is a fork, your URL will have your GitHub username
in it instead of "mjones01".
**Data Tip:** You can use Git and connect to GitHub
directly in the RStudio interface. If interested, read
this R-bloggers How-To.
View Commits in GitHub
Let’s view our recent commit in our forked repo on GitHub.
Go to github.com and navigate to your forked Data Institute repo - DI-NEON-participants.
Click on the commits link at the top of the page.
Look at the commits - do you see your recent commit message that you typed
into bash on your computer?
Next, click on the <>CODE link which is ABOVE the commits link in github.
Is the Markdown file that you added and committed locally at the command
line on your computer, there in the same directory (participants/pre-institute2-git) that you saved it on your
laptop?
An example .md file located within the
participants/2017-RemoteSensing/pre-institute2-git of a Data Institute repo fork.
Source: National Ecological Observatory Network (NEON)
Is Your File in the NEON Central Repo Yet?
Next, do the following:
Navigate to the NEON central
NEONScience/DI-NEON-participants
repo. (The easiest method to do this is to click the link at the top of the page under your repo name).
Look for your file in the same directory. Is your new file there? If not, why?
Remember the structure of our workflow.
We’ve added changes from our local
repo on our computer and pushed them to our fork on github.com. But this fork
is in our individual user account, not NEONS. This fork is
separate from the central repo. Changes to a fork in our github.com account
do not automatically transfer to the central repo. We need to sync them! We will
learn how to sync these two
repos in the next tutorial
Git 06: Syncing GitHub Repos with Pull Requests .
Summary Workflow - Committing Changes
On your computer, within your local copy of the Git repo:
Create a new markdown file and edit it in your favorite text editor.
On your computer, in shell (at the command line):
git status
git add FileName
git status - make sure everything is added and ready for commit
`git commit -m “messageHere”
git push origin master
On the github.com website:
Check to make sure commit is added.
Check to see if the file that you added is visible online in your Git repo.
Have questions? No problem. Leave your question in the comment box below.
It's likely some of your colleagues have the same question, too! And also
likely someone else knows the answer.
This tutorial covers how to clone a github.com repo to your computer so
that you can work locally on files within the repo.
## Learning Objectives
At the end of this activity, you will be able to:
Be able to use the git clone command to create a local version of a GitHub
repository on your computer.
Additional Resources
Diagram of Git Commands
-- this diagram includes more commands than we will cover in this series but
includes all that we use for our standard workflow.
In the previous tutorial, we used the github.com interface to fork the central NEON repo.
By forking the NEON repo, we created a copy of it in our github.com account.
When you fork a repository on the github.com website, you are creating a
duplicate copy of it in your github.com account. This is useful as a backup
of the material. It also allows you to edit the material without modifying
the original repository.
Source: National Ecological Observatory Network (NEON)
Now we will learn how to create a local version of our forked repo on our
laptop, so that we can efficiently add to and edit repo content.
When you clone a repository to your local computer, you are creating a
copy of that same repo locally on your computer. This
allows you to edit files on your computer. And, of course, it is also yet another
backup of the material!
Source: National Ecological Observatory Network (NEON)
Copy Repo URL
Start from the github.com interface:
Navigate to the repo that you want to clone (copy) to your computer --
this should be YOUR-USER-NAME/DI-NEON-participants.
Click on the Clone or Download dropdown button and copy the URL of the repo.
The clone or download drop down allows you to copy the URL that
you will need to clone a repository. Download allows you to download a .zip file
containing all of the files in the repo.
Source: National Ecological Observatory Network (NEON).
Then on your local computer:
Your computer should already be setup with Git and a bash shell interface.
If not, please refer to the Institute setup materials before continuing.
Open bash on your computer and navigate to the local GitHub directory that
you created using the Set-up Materials.
To do this, at the command prompt, type:
$ cd ~/Documents/GitHub
Note: If you have stored your GitHub directory in a location that is different
i.e. it is not /Documents/GitHub, be sure to adjust the above code to
represent the actual path to the GitHub directory on your computer.
Now use git clone to clone, or create a copy of, the entire repo in the
GitHub directory on your computer.
# clone the forked repo to our computer
$ git clone https://github.com/neon/DI-NEON-participants.git
**Data Tip:**
Are you a Windows user and are having a hard time copying the URL into shell?
You can copy and paste in the shell environment **after** you
have the feature turned on. Right click on your bash shell window (at the top)
and select "properties". Make sure "quick edit" is checked. You should now be
able to copy and paste within the bash environment.
The output shows you what is being cloned to your computer.
Note: The output numbers that you see on your computer, representing the total file
size, etc, may differ from the example provided above.
View the New Repo
Next, let's make sure the repository is created on your
computer in the location where you think it is.
At the command line, type ls to list the contents of the current
directory.
# view directory contents
$ ls
Next, navigate to your copy of the data institute repo using cd or change
directory:
# navigate to the NEON participants repository
$ cd DI-NEON-participants
# view repository contents
$ ls
404.md _includes code
ISSUE_TEMPLATE.md _layouts images
README.md _posts index.md
_config.yml _site institute-materials
_data assets org
Alternatively, we can view the local repo DI-NEON-participants in a finder (Mac)
or Windows Explorer (Windows) window. Simply open your Documents in a window and
navigate to the new local repo.
Using either method, we can see that the file structure of our cloned repo
exactly mirrors the file structure of our forked GitHub repo.
**Thought Question:**
Is the cloned version of this repo that you just created on your laptop, a
direct copy of the NEON central repo -OR- of your forked version of the NEON
central repo?
Summary Workflow -- Create a Local Repo
In the github.com interface:
Copy URL of the repo you want to work on locally
In shell:
git clone URLhere
Note: that you can copy the URL of your repository directly from GitHub.
Got questions? No problem. Leave your question in the comment box below.
It's likely some of your colleagues have the same question, too! And also
likely someone else knows the answer.
In this tutorial, we will fork, or create a copy in your github.com account,
an existing GitHub repository. We will also explore the github.com interface.
## Learning Objectives
At the end of this activity, you will be able to:
Create a GitHub account.
Know how to navigate to and between GitHub repositories.
Create your own fork, or copy, a GitHub repository.
Explain the relationship between your forked repository and the master
repository it was created from.
Additional Resources
Diagram of Git Commands
-- this diagram includes more commands than we will
learn in this series but includes all that we use for our standard workflow.
If you do not already have a GitHub account, go to GitHub and sign up for
your free account. Pick a username that you like! This username is what your
colleagues will see as you work with them in GitHub and Git.
Take a minute to setup your account. If you want to make your account more
recognizable, be sure to add a profile picture to your account!
If you already have a GitHub account, simply sign in.
**Data Tip:** Are you a student? Sign up for the
Student Developer Pack
and get the Git Personal account free (with unlimited private repos and other
discounts/options; normally $7/month).
Navigate GitHub
Repositories, AKA Repos
Let's first discuss the repository or "repo". (The cool kids say repo, so we will
jump on the git cool kid bandwagon) and use "repo" from here on in. According to
the GitHub glossary:
A repository is the most basic element of GitHub. They're easiest to imagine
as a project's folder. A repository contains all of the project files (including
documentation), and stores each file's revision history. Repositories can have
multiple collaborators and can be either public or private.
Once you have found the Data Institute participants repo, take 5 minutes
to explore it.
Git Repo Names
First, get to know the repository naming convention. Repository names all take
the format:
OrganizationName/RepositoryName
So the full name of our repository is:
NEONScience/DI-NEON-participants
Header Tabs
At the top of the page you'll notice a series of tabs. Please focus
on the following 3 for now:
Code: Click here to view structure & contents of the repo.
Issues: Submit discussion topics, or problems that you are having with
the content in the repo, here.
Pull Requests: Submit changes to the repo for review /
acceptance. We will explore pull requests more in the
Git 06 tutorial.
Screenshot of the NEON Data Institute central repository (note,
there has been a slight change in the repo name).
The github.com search bar is at the top of the page. Notice there are 6
"tabs" below the repo name including: Code, Issues, Pull Request, Pulse,
Graphics and Settings. NOTE: Because you are not an administrator for this
repo, you will not see the "Settings" tab in your browser.
Source: National Ecological Observatory Network (NEON)
Other Text Links
A bit further down the page, you'll notice a few other links:
commits: a commit is a saved and documented change to the content
or structure of the repo. The commit history contains all changes that
have been made to that repo. We will discuss commits more in
Git 05: Git Add Changes -- Commits .
Fork a Repository
Next, let's discuss the concept of a fork on the github.com site. A fork is a
copy of the repo that you create in your account. You can fork any repo at
any time by clicking the fork button in the upper right hand corner on github.com.
Click on the "Fork" button to fork any repo. Source:
GitHub Guides.
When we fork a repo in github.com, we are telling Git to create an
exact copy of the repo that we're forking in our own github.com account.
Once the repo is in our own account, we can edit it as we now own that fork.
Note that a fork is just a copy of the repo on github.com.
Source: National Ecological Observatory Network (NEON)
## Activity: Fork the NEON Data Institute Participants Repo
Create your own fork of the DI-NEON-participants now.
**Data Tip:** You can change the name of a forked
repo and it will still be connected to the central repo from which it was forked.
For now, leave it the same.
Check Out Your Data Institute Fork
Now, check out your new fork. Its name should be:
YOUR-USER-NAME/DI-NEON-participants.
It can get confusing sometimes moving between a central repo:
A good way to figure out which repo you are viewing is to look at the name of the
repo. Does it contain your username? Or your colleagues'? Or NEON's?
Your Fork vs the Central Repo
Your fork is an exact copy, or completely in sync with, the NEON central repo.
You could confirm this by comparing your fork to the NEON central repository using
the pull request option. We will learn about pull requests in
Git06: Sync GitHub Repos with Pull Requests.
For now, take our word for it.
The fork will remain in sync with the NEON central repo until:
You begin to make changes to your forked copy of the repo.
The central repository is changed or updated by a collaborator.
If you make changes to your forked repo, the changes will not be added to the
NEON central repo until you sync your fork with the NEON central repo.
Summary Workflow -- Fork a GitHub Repository
On the github.com website:
Navigate to desired repo that you want to fork.
Click Fork button.
Have questions? No problem. Leave your question in the comment box below.
It's likely some of your colleagues have the same question, too! And also
likely someone else knows the answer.
A version control system maintains a record of changes to code and other content.
It also allows us to revert changes to a previous point in time.
Many of us have used the "append a date" to a file name version
of version control at some point in our lives. Source: "Piled Higher and
Deeper" by Jorge Cham www.phdcomics.com
Types of Version control
There are many forms of version control. Some not as good:
Save a document with a new date (we’ve all done it, but it isn’t efficient)
Google Docs "history" function (not bad for some documents, but limited in scope).
Some better:
Mercurial
Subversion
Git - which we’ll be learning much more about in this series.
**Thought Question:** Do you currently implement
any form of version control in your work?
More Resources:
Visit the version control Wikipedia list of version control platforms.
Version control facilitates two important aspects of many scientific workflows:
The ability to save and review or revert to previous versions.
The ability to collaborate on a single project.
This means that you don’t have to worry about a collaborator (or your future self)
overwriting something important. It also allows two people working on the same
document to efficiently combine ideas and changes.
**Thought Questions:** Think of a specific time when
you weren’t using version control that it would have been useful.
Why would version control have been helpful to your project & work flow?
What were the consequences of not having a version control system in place?
How Version Control Systems Works
Simple Version Control Model
A version control system keeps track of what has changed in one or more files
over time. The way this tracking occurs, is slightly different between various
version control tools including git, mercurial and svn. However the
principle is the same.
Version control systems begin with a base version of a document. They then
save the committed changes that you make. You can think of version control
as a tape: if you rewind the tape and start at the base document, then you can
play back each change and end up with your latest version.
A version control system saves changes to a document, sequentially,
as you add and commit them to the system.
Source: Software Carpentry
Once you think of changes as separate from the document itself, you can then
think about “playing back” different sets of changes onto the base document.
You can then retrieve, or revert to, different versions of the document.
The benefit of version control when you are in a collaborative environment is that
two users can make independent changes to the same document.
Different versions of the same document can be saved within a
version control system.
Source: Software Carpentry
If there aren’t conflicts between the users changes (a conflict is an area
where both users modified the same part of the same document in different
ways) you can review two sets of changes on the same base document.
Two sets of changes to the same base document can be reviewed
together, within a version control system if there are no conflicts (areas
where both users modified the same part of the same document in different ways).
Changes submitted by both users can then be merged together.
Source: Software Carpentry
A version control system is a tool that keeps track of these changes for us.
Each version of a file can be viewed and reverted to at any time. That way if you
add something that you end up not liking or delete something that you need, you
can simply go back to a previous version.
Git & GitHub - A Distributed Version Control Model
GitHub uses a distributed version control model. This means that there can be
many copies (or forks in GitHub world) of the repository.
One advantage of a distributed version control system is that there
are many copies of the repository. Thus, if any server or computer dies, any of
the client repositories can be copied and used to restore the data! Every clone
(or fork) is a full backup of all the data.
Source: Pro Git by Scott Chacon & Ben Straub
Have a look at the graphic below. Notice that in the example, there is a "central"
version of our repository. Joe, Sue and Eve are all working together to update
the central repository. Because they are using a distributed system, each user (Joe,
Sue and Eve) has their own copy of the repository and can contribute to the central
copy of the repository at any time.
Distributed version control models allow many users to
contribute to the same central document.
Source: Better Explained
Create A Working Copy of a Git Repo - Fork
There are many different Git and GitHub workflows. In the NEON Data Institute,
we will use a distributed workflow with a Central Repository. This allows
us all (all of the Institute participants) to work independently. We can then
contribute our changes to update the Central (NEON) Repository. Our collaborative workflow goes
like this:
You will create a copy of this repository (known as a fork) in your own GitHub account.
You will then clone (copy) the repository to your local computer. You
will do your work locally on your laptop.
When you are ready to submit your changes to the NEON repository, you will:
Sync your local copy of the repository with NEON's central
repository so you have the most up to date version, and then,
Push the changes you made to your local copy (or fork) of the repository to
NEON's main repository.
Each participant in the institute will be contributing to the NEON central
repository using the same workflow! Pretty cool stuff.
The NEON central repository is the final working version of our
project. You can fork or create a copy of this repository
into your github.com account. You can then copy or clone your
fork, to your local computer where you can make edits. When you are done
working, you can push or transfer those edits back to your local fork. When
you are read to update the NEON central repository, you submit a pull
request. We will walk through the steps of this workflow over the
next few lessons.
Source: National Ecological Observatory Network (NEON)
Let's get some terms straight before we go any further.
Central repository - the central repository is what all participants will
add to. It is the "final working version" of the project.
Your forked repository - is a "personal” working copy of the
central repository stored in your GitHub account. This is called a fork.
When you are happy with your work, you update your repo from the central repo,
then you can update your changes to the central NEON repository.
Your local repository - this is a local version of your fork on your
own computer. You will most often do all of your work locally on your computer.
**Data Tip:** Other Workflows -- There are many other
git workflows.
Read more about other workflows.
This resource mentions Bitbucket, another web-based hosting service like GitHub.
Additional Resources:
Further documentation for and how-to-use direction for Git, is provided by the
Git Pro version 2 book by Scott Chacon and Ben Straub ,
available in print or online. If you enjoy learning from videos, the site hosts
several.
This tutorial builds upon
the previous tutorial,
to work with shapefile attributes in R and explores how to plot multiple
shapefiles using base R graphics. It then covers
how to create a custom legend with colors and symbols that match your plot.
Learning Objectives
After completing this tutorial, you will be able to:
Plot multiple shapefiles using base R graphics.
Apply custom symbology to spatial objects in a plot in R.
Customize a baseplot legend in R.
Things You’ll Need To Complete This Tutorial
You will need the most current version of R and preferably RStudio loaded
on your computer to complete this tutorial.
R Script & Challenge Code: NEON data lessons often contain challenges that reinforce
learned skills. If available, the code for challenge solutions is found in the
downloadable R script of the entire lesson, available in the footer of each lesson page.
Load the Data
To work with vector data in R, we can use the rgdal library. The raster
package also allows us to explore metadata using similar commands for both
raster and vector files.
We will import three shapefiles. The first is our AOI or area of
interest boundary polygon that we worked with in
Open and Plot Shapefiles in R.
The second is a shapefile containing the location of roads and trails within the
field site. The third is a file containing the Harvard Forest Fisher tower
location. These latter two we worked with in the
Explore Shapefile Attributes & Plot Shapefile Objects by Attribute Value in R tutorial.
# load packages
# rgdal: for vector work; sp package should always load with rgdal.
library(rgdal)
# raster: for metadata/attributes- vectors or rasters
library(raster)
# set working directory to data folder
# setwd("pathToDirHere")
# Import a polygon shapefile
aoiBoundary_HARV <- readOGR("NEON-DS-Site-Layout-Files/HARV",
"HarClip_UTMZ18", stringsAsFactors = T)
## OGR data source with driver: ESRI Shapefile
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HarClip_UTMZ18"
## with 1 features
## It has 1 fields
## Integer64 fields read as strings: id
# Import a line shapefile
lines_HARV <- readOGR( "NEON-DS-Site-Layout-Files/HARV", "HARV_roads", stringsAsFactors = T)
## OGR data source with driver: ESRI Shapefile
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HARV_roads"
## with 13 features
## It has 15 fields
# Import a point shapefile
point_HARV <- readOGR("NEON-DS-Site-Layout-Files/HARV",
"HARVtower_UTM18N", stringsAsFactors = T)
## OGR data source with driver: ESRI Shapefile
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HARVtower_UTM18N"
## with 1 features
## It has 14 fields
Plot Data
In the
Explore Shapefile Attributes & Plot Shapefile Objects by Attribute Value in R tutorial
we created a plot where we customized the width of each line in a spatial object
according to a factor level or category. To do this, we create a vector of
colors containing a color value for EACH feature in our spatial object grouped
by factor level or category.
# view the factor levels
levels(lines_HARV$TYPE)
## [1] "boardwalk" "footpath" "stone wall" "woods road"
# create vector of line width values
lineWidth <- c(2,4,3,8)[lines_HARV$TYPE]
# view vector
lineWidth
## [1] 8 4 4 3 3 3 3 3 3 2 8 8 8
# create a color palette of 4 colors - one for each factor level
roadPalette <- c("blue","green","grey","purple")
roadPalette
## [1] "blue" "green" "grey" "purple"
# create a vector of colors - one for each feature in our vector object
# according to its attribute value
roadColors <- c("blue","green","grey","purple")[lines_HARV$TYPE]
roadColors
## [1] "purple" "green" "green" "grey" "grey" "grey" "grey"
## [8] "grey" "grey" "blue" "purple" "purple" "purple"
# create vector of line width values
lineWidth <- c(2,4,3,8)[lines_HARV$TYPE]
# view vector
lineWidth
## [1] 8 4 4 3 3 3 3 3 3 2 8 8 8
# in this case, boardwalk (the first level) is the widest.
plot(lines_HARV,
col=roadColors,
main="NEON Harvard Forest Field Site\n Roads & Trails \nLine Width Varies by Type Attribute Value",
lwd=lineWidth)
**Data Tip:** Given we have a factor with 4 levels,
we can create a vector of numbers, each of which specifies the thickness of each
feature in our `SpatialLinesDataFrame` by factor level (category): `c(6,4,1,2)[lines_HARV$TYPE]`
Add Plot Legend
In the
the previous tutorial,
we also learned how to add a basic legend to our plot.
bottomright: We specify the location of our legend by using a default
keyword. We could also use top, topright, etc.
levels(objectName$attributeName): Label the legend elements using the
categories of levels in an attribute (e.g., levels(lines_HARV$TYPE) means use
the levels boardwalk, footpath, etc).
fill=: apply unique colors to the boxes in our legend. palette() is
the default set of colors that R applies to all plots.
Let's add a legend to our plot.
plot(lines_HARV,
col=roadColors,
main="NEON Harvard Forest Field Site\n Roads & Trails\n Default Legend")
# we can use the color object that we created above to color the legend objects
roadPalette
## [1] "blue" "green" "grey" "purple"
# add a legend to our map
legend("bottomright",
legend=levels(lines_HARV$TYPE),
fill=roadPalette,
bty="n", # turn off the legend border
cex=.8) # decrease the font / legend size
However, what if we want to create a more complex plot with many shapefiles
and unique symbols that need to be represented clearly in a legend?
Plot Multiple Vector Layers
Now, let's create a plot that combines our tower location (point_HARV),
site boundary (aoiBoundary_HARV) and roads (lines_HARV) spatial objects. We
will need to build a custom legend as well.
To begin, create a plot with the site boundary as the first layer. Then layer
the tower location and road data on top using add=TRUE.
# Plot multiple shapefiles
plot(aoiBoundary_HARV,
col = "grey93",
border="grey",
main="NEON Harvard Forest Field Site")
plot(lines_HARV,
col=roadColors,
add = TRUE)
plot(point_HARV,
add = TRUE,
pch = 19,
col = "purple")
# assign plot to an object for easy modification!
plot_HARV<- recordPlot()
Customize Your Legend
Next, let's build a custom legend using the symbology (the colors and symbols)
that we used to create the plot above. To do this, we will need to build three
things:
A list of all "labels" (the text used to describe each element in the legend
to use in the legend.
A list of colors used to color each feature in our plot.
A list of symbols to use in the plot. NOTE: we have a combination of points,
lines and polygons in our plot. So we will need to customize our symbols!
Let's create objects for the labels, colors and symbols so we can easily reuse
them. We will start with the labels.
# create a list of all labels
labels <- c("Tower", "AOI", levels(lines_HARV$TYPE))
labels
## [1] "Tower" "AOI" "boardwalk" "footpath" "stone wall"
## [6] "woods road"
# render plot
plot_HARV
# add a legend to our map
legend("bottomright",
legend=labels,
bty="n", # turn off the legend border
cex=.8) # decrease the font / legend size
Now we have a legend with the labels identified. Let's add colors to each legend
element next. We can use the vectors of colors that we created earlier to do this.
# we have a list of colors that we used above - we can use it in the legend
roadPalette
## [1] "blue" "green" "grey" "purple"
# create a list of colors to use
plotColors <- c("purple", "grey", roadPalette)
plotColors
## [1] "purple" "grey" "blue" "green" "grey" "purple"
# render plot
plot_HARV
# add a legend to our map
legend("bottomright",
legend=labels,
fill=plotColors,
bty="n", # turn off the legend border
cex=.8) # decrease the font / legend size
Great, now we have a legend! However, this legend uses boxes to symbolize each
element in the plot. It might be better if the lines were symbolized as a line
and the points were symbolized as a point symbol. We can customize this using
pch= in our legend: 16 is a point symbol, 15 is a box.
**Data Tip:** To view a short list of `pch` symbols,
type `?pch` into the R console.
# create a list of pch values
# these are the symbols that will be used for each legend value
# ?pch will provide more information on values
plotSym <- c(16,15,15,15,15,15)
plotSym
## [1] 16 15 15 15 15 15
# Plot multiple shapefiles
plot_HARV
# to create a custom legend, we need to fake it
legend("bottomright",
legend=labels,
pch=plotSym,
bty="n",
col=plotColors,
cex=.8)
Now we've added a point symbol to represent our point element in the plot. However
it might be more useful to use line symbols in our legend
rather than squares to represent the line data. We can create line symbols,
using lty = (). We have a total of 6 elements in our legend:
A Tower Location
An Area of Interest (AOI)
and 4 Road types (levels)
The lty list designates, in order, which of those elements should be
designated as a line (1) and which should be designated as a symbol (NA).
Our object will thus look like lty = c(NA,NA,1,1,1,1). This tells R to only use a
line element for the 3-6 elements in our legend.
Once we do this, we still need to modify our pch element. Each line element
(3-6) should be represented by a NA value - this tells R to not use a
symbol, but to instead use a line.
# create line object
lineLegend = c(NA,NA,1,1,1,1)
lineLegend
## [1] NA NA 1 1 1 1
plotSym <- c(16,15,NA,NA,NA,NA)
plotSym
## [1] 16 15 NA NA NA NA
# plot multiple shapefiles
plot_HARV
# build a custom legend
legend("bottomright",
legend=labels,
lty = lineLegend,
pch=plotSym,
bty="n",
col=plotColors,
cex=.8)
### Challenge: Plot Polygon by Attribute
Using the NEON-DS-Site-Layout-Files/HARV/PlotLocations_HARV.shp shapefile,
create a map of study plot locations, with each point colored by the soil type
(soilTypeOr). How many different soil types are there at this particular field
site? Overlay this layer on top of the lines_HARV layer (the roads). Create a
custom legend that applies line symbols to lines and point symbols to the points.
Modify the plot above. Tell R to plot each point, using a different
symbol of pch value. HINT: to do this, create a vector object of symbols by
factor level using the syntax described above for line width:
c(15,17)[lines_HARV$soilTypeOr]. Overlay this on top of the AOI Boundary.
Create a custom legend.
In this tutorial, we will cover the R knitr package that is used to convert
R Markdown into a rendered document (HTML, PDF, etc).
Learning Objectives
At the end of this activity, you will:
Be able to produce (‘knit’) an HTML file from a R Markdown file.
Know how to modify chunk options to change the output in your HTML file.
Things You’ll Need To Complete This Tutorial
You will need the most current version of R and, preferably, RStudio loaded on
your computer to complete this tutorial.
Install R Packages
knitr:install.packages("knitr")
rmarkdown:install.packages("rmarkdown")
Share & Publish Results Directly from Your Code!
The knitr package allow us to:
Publish & share preliminary results with collaborators.
Create professional reports that document our workflow and results directly
from our code, reducing the risk of accidental copy and paste or transcription errors.
Document our workflow to facilitate reproducibility.
Efficiently change code outputs (figures, files) given changes in the data, methods, etc.
Publish from Rmd files with knitr
To complete this tutorial you need:
The R knitr package to complete this tutorial. If you need help installing
packages, visit
the R packages tutorial.
An R Markdown document that contains a YAML header, code chunks and markdown
segments. If you don't have an .Rmd file, visit
the R Markdown tutorial to create one.
**When To Knit**: Knitting is a useful exercise
throughout your scientific workflow. It allows you to see what your outputs
look like and also to test that your code runs without errors.
The time required to knit depends on the length and complexity of the script
and the size of your data.
How to Knit
Location of the knit button in RStudio in Version 0.99.486.
Source: National Ecological Observatory Network (NEON)
To knit in RStudio, click the knit pull down button. You want to use the knit HTML for this lesson.
When you click the Knit HTML button, a window will open in your console
titled R Markdown. This
pane shows the knitting progress. The output (HTML in this case) file will
automatically be saved in the current working directory. If there is an error
in the code, an error message will appear with a line number in the R Console
to help you diagnose the problem.
**Data Tip:** You can run `knitr` from the command prompt
using: `render(“input.Rmd”, “all”)`.
Activity: Knit Script
Knit the .Rmd file that you built in
the last tutorial.
What does it look like?
View the Output
R Markdown (left) and the resultant HTML (right) after knitting.
Source: National Ecological Observatory Network (NEON)
When knitting is complete, the new HTML file produced will automatically open.
Notice that information from the YAML header (title, author, date) is printed
at the top of the HTML document. Then the HTML shows the text, code, and
results of the code that you included in the RMD document.
Data Institute Participants: Complete Week 2 Assignment
Be sure to carefully check your knitr output to make sure it is rendering the
way you think it should!
When you are complete, submit your .Rmd and .html files to the
NEON Institute participants GitHub repository
(NEONScience/DI-NEON-participants).
The files will have automatically saved to your R working directory, you will
need to transfer the files to the /participants/pre-institute3-rmd/
directory and submitted via a pull request.
You will need to have the rmarkdown and knitr
packages installed on your computer prior to completing this tutorial. Refer to
the setup materials to get these installed.
Learning Objectives
At the end of this activity, you will:
Know how to create an R Markdown file in RStudio.
Be able to write a script with text and R code chunks.
Create an R Markdown document ready to be ‘knit’ into an HTML document to
share your code and results.
Things You’ll Need To Complete This Tutorial
You will need the most current version of R and, preferably, RStudio loaded on
your computer to complete this tutorial.
You will want to create a data directory for all the Data Institute teaching
datasets. We suggest the pathway be ~/Documents/data/NEONDI-2016 or
the equivalent for your operating system. Once you've downloaded and unzipped
the dataset, move it to this directory.
The data directory with the teaching data subset. This is the suggested organization for all Data Institute teaching data subsets.
Source: National Ecological Observatory Network (NEON)
Our goal in this series is to document our workflow. We can do this by
Creating an R Markdown (RMD) file in R studio and
Rendering that RMD file to HTML using knitr.
Watch this 6:38 minute video below to learn more about how you can convert an R Markdown
file to HTML (or other formats) using knitr in RStudio.
The text size in the video is small so you may want to watch the video in
full screen mode.
Now that you have a sense of how R Markdown can be used in RStudio, you are
ready to create your own RMD document. Do the following:
Create a new R Markdown file and choose HTML as the desired output format.
Enter a Title (Explore NEON LiDAR Data) and Author Name (your name). Then click OK.
Save the file using the following format: LastName-institute-week3.rmd
NOTE: The document title is not the same as the file name.
Hit the knit button in RStudio (as is done in the video above). What happens?
Location of the knit button in RStudio in Version 0.99.486.
Source: National Ecological Observatory Network (NEON)
If everything went well, you should have an HTML format (web page) output
after you hit the knit button. Note that this HTML output is built from a
combination of code and documentation that was written using markdown syntax.
Next, we'll break down the structure of an R Markdown file.
Understand Structure of an R Markdown file
Screenshot of a new R Markdown document in RStudio. Notice the different
parts of the document.
Source: National Ecological Observatory Network (NEON)
**Data Tip:** Screenshots on this page are
from RStudio with appearance preferences set to `Twilight` with `Monaco` font. You
can change the appearance of your RStudio by **Tools** > **Options**
(or **Global Options** depending on the operating system). For more, see the
Customizing RStudio page.
Let's next review the structure of an R Markdown (.Rmd) file. There are three
main content types:
Header: the text at the top of the document, written in YAML format.
Markdown sections: text that describes your workflow written using markdown syntax.
Code chunks: Chunks of R code that can be run and also can be rendered
using knitr to an output document.
Next let's explore each section type.
Header -- YAML block
An R Markdown file always starts with header written using
YAML syntax.
There are four default elements in the RStudio generated YAML header:
title: the title of your document. Note, this is not the same as the file name.
author: who wrote the document.
date: by default this is the date that the file is created.
output: what format will the output be in. We will use HTML.
A YAML header may be structured differently depending upon how your are using it.
Learn more on the
R Markdown documentation page.
## Activity: R Markdown YAML
Customize the header of your `.Rmd` file as follows:
Title: Provide a title that fits the code that will be in your RMD.
Author: Add your name here.
Output: Leave the default output setting: html_document.
We will be rendering an HTML file.
R Markdown Text/Markdown Blocks
An RMD document contains a mixture of code chunks and markdown blocks where
you can describe aspects of your processing workflow. The markdown blocks use the
same markdown syntax that we learned last week in week 2 materials. In these blocks
you might describe the data that you are using, how it's being processed and
and what the outputs are. You may even add some information that interprets
the outputs.
When you render your document to HTML, this markdown will appear as text on the
output HTML document.
Look closely at the pre-populated markdown and R code chunks in your RMD file.
Does any of the markdown syntax look familiar?
Are any words in bold?
Are any words in italics?
Are any words highlighted as code?
If you are unsure, the answers are at the bottom of this page.
## Activity: R Markdown Text
Remove the template markdown and code chunks added to the RMD file by RStudio.
(Be sure to keep the YAML header!)
At the very top of your RMD document - after the YAML header, add
the bio and short research description that you wrote last week in markdown syntax to
the RMD file.
Between your profile and the research descriptions, add a header that says
About My Project (or something similar).
Add a new header stating R Markdown Activity and text below that explaining
that this page demonstrates using some of the NEON Teakettle LiDAR data products
in R. The wording of this text should clearly describe the code and outputs that
you will be adding the page.
**Data Tip**: You can add code output or an R object
name to markdown segments of an RMD. For more, view this
R Markdown documentation.
Code chunks
Code chunks are where your R code goes. All code chunks start and end with
``` – three backticks or graves. On
your keyboard, the backticks can be found on the same key as the tilde.
Graves are not the same as an apostrophe!
The initial line of a code chunk must appear as:
```{r chunk-name-with-no-spaces}
# code goes here
```
The r part of the chunk header identifies this chunk as an R code chunk and is
mandatory. Next to the {r, there is a chunk name. This name is not required
for basic knitting however, it is good practice to give each chunk a unique
name as it is required for more advanced knitting approaches.
Activity: Add Code Chunks to Your R Markdown File
Continue working on your document. Below the last section that you've just added,
create a code chunk that loads the packages required to work with raster data
in R.
In R scripts, setting the working directory is normally done once near the beginning of your script. In R Markdown files, knit code chunks behave a little differently, and a warning appears upon kitting a chunk that sets a working directory.
```{r code-setwd}
# set working directory to ensure R can find the file we wish to import.
# This will depend on your local environment.
setwd("~/Documents/data/NEONDI-2016/")
```
You changed the working directory to ~/Documents/data/NEONDI-2016/ (probably via setwd()). It will be restored to [directory path of current .rmd file]. See the Note section in ?knitr::knit ?knitr::knit
That's a bad sign if you want to set the working directory in one code chunk, and read or write data in another code chunk. To allow for a working data directory that is different from your Rmd file's current directory, you can store the directory path in a string variable.
```{r code-setwd-stringvariable}
# set working directory as a string variable for use in other code chunks.
# This will depend on your local environment.
wd <- "~/Documents/data/NEONDI-2016/"
setwd(wd)
```
The setwd(wd) line could be at the start of a lengthier code chunk that reads
from and writes to data files. Alternatively, since the variable will be kept in
this document's R environment, it can be used with paste() or paste0() when you
need to refer to a filepath. Proceed to the next step for an example of this.
(For further instruction on setting the working directory, see the NEON Data Skills tutorial
Set A Working Directory in R.)
Let's add another chunk that loads the TEAK_lidarDSM raster file.
```{r load-dsm-raster }
# check for the working directory
getwd()
# In this new chunk, the working directory has reverted to default upon kitting.
# Combining the working directory string variable and
# additional path to the file, import a DSM file.
teak_dsm <- raster(paste0(wd, "NEONdata/D17-California/TEAK/2013/lidar/TEAK_lidarDSM.tif"))
```
Now run the code in this chunk.
You can run code chunks:
Line-by-line: with cursor on current line, Ctrl + Enter (Windows/Linux) or
Command + Enter (Mac OS X).
By chunk: You can run the entire chunk (or multiple chunks) by
clicking on the "Run" button in the upper right corner of the RStudio script
panel and choosing the appropriate option (Run Current Chunk, Run Next Chunk).
Keyboard shortcuts are available for these options.
Code chunk options
You can also add arguments or options to each code chunk. These arguments allow
you to customize how or if you want code to be
processed or appear on the output HTML document. Code chunk arguments are added on
the first line of a code
chunk after the name, within the curly brackets.
The example below, is a code chunk that will not be "run", or evaluated, by R.
The code within the chunk will appear on the output document, however there
will be no outputs from the code.
```{r intro-option, eval=FALSE}
# the code here will not be processed by R
# but it will appear on your output document
1+2
```
We use eval=FALSE often when the chunk is exporting an file that we don't
need to re-export but we want to document the code used to export the file.
Three common code chunk options are:
eval = FALSE: Do not evaluate (or run) this code chunk when
knitting the RMD document. The code in this chunk will still render in our knitted
HTML output, however it will not be evaluated or run by R.
echo = FALSE: Hide the code in the output. The code is
evaluated when the RMD file is knit, however only the output is rendered on the
output document.
results = hide: The code chunk will be evaluated but the results of the code
will not be rendered on the output document. This is useful if you are viewing the
structure of a large object (e.g. outputs of a large data.frame).
Add a new code chunk that plots the TEAK_lidarDSM raster object that you imported above.
Experiment with plot colors and be sure to add a plot title.
Run the code chunk that you just added to your RMD document in R (e.g. run in console, not
knitting). Does it create a plot with a title?
In another new code chunk, import and plot another raster file from the NEON data subset
that you downloaded. The TEAK_lidarCHM is a good raster to plot.
Finally, create histograms for both rasters that you've imported into R.
Be sure to document your steps as you go using both code comments and
markdown syntax in between the code chunks.
For help opening and plotting raster data in R, see the NEON Data Skills tutorial
Plot Raster Data in R.
We will knit this document to HTML in the next tutorial.
Now continue on to the next tutorial
to learn how to knit this document into a HTML file.
## Answers to the Default Text Markdown Syntax Questions
Are any words in bold? - Yes, “Knit” on line 10
Are any words in italics? - No
Are any words highlighted as code? - Yes, “echo = FALSE” on line 22
This tutorial we will work with the knitr and rmarkdown packages within
RStudio to learn how to effectively and efficiently document and publish our
workflows online.
Learning Objectives
At the end of this activity, you will be able to:
Explain why documenting and publishing one's code is important.
Describe two tools that enable ease of publishing code & output: R Markdown and
the knitr package.
This week we will learn about the R Markdown file format (and R package) which
can be used with the knitr package to document and publish (disseminate) your
code and code output.
“R Markdown is an authoring format that enables easy creation of dynamic
documents, presentations, and reports from R. It combines the core syntax of
markdown (an easy to write plain text format) with embedded R code chunks that
are run so their output can be included in the final document. R Markdown
documents are fully reproducible (they can be automatically regenerated whenever
underlying R code or data changes)."
-- RStudio documentation.
We use markdown syntax in R Markdown (.rmd) files to document workflows and
to share data processing, analysis and visualization outputs. We can also use it
to create documents that combine R code, output and text.
There are many advantages to using R Markdown in your work:
Human readable syntax.
Simple syntax - it can be learned quickly.
All components of your work are clearly documented. You don't have to remember
what steps, assumptions, tests were used.
You can easily extend or refine analyses by modifying existing or adding new
code blocks.
Analysis results can be disseminated in various formats including HTML, PDF,
slide shows and more.
Code and data can be shared with a colleague to replicate the workflow.
**Data Tip:**
RPubs
is a quick way to share and publish code.
Knitr
The knitr package for R allows us to create readable documents from R Markdown
files.
R Markdown script (left) and the HTML produced from the knit R
Markdown script (right). Source: National Ecological Observatory Network (NEON)
>The knitr package was designed to be a transparent engine for dynamic report
generation with R --
Yihui Xi -- knitr package creator
In the next tutorial we will learn more about working with the R Markdown format in RStudio.
The primary goal of this tutorial is to explain how to set a working directory
in R. The working directory is where your R session interacts with your hard drive.
This is where you can read data that you want to use, and save new information such
as derived data products, tables, maps, and figures. It is a good practice to store
your information in an organized set of directories, so you will often want to change
your working directory depending on what kinds of information that you need to access.
This tutorial teaches how to download and unzip the data files that accompany many
NEON Data Skills tutorials, and also covers the concept of file paths. You can read
from top to bottom, or use the menu bar at left to navigate to your desired topic.
Learning Objectives
After completing this tutorial, you will be able to:
Be able to download and uncompress NEON Teaching Data Subsets.
Be able to set the R working directory.
Know the difference between full, base and relative paths.
Be able to write out both full and relative paths for a given file or
directory.
Things You’ll Need To Complete This Lesson
To complete this lesson you will need the most current version of R and,
preferably, RStudio loaded on your computer.
Many NEON data tutorials utilize teaching data subsets which are hosted on the
NEON Data Skills figshare repository. If a data subset is required for a
tutorial it can be downloaded at the top of each tutorial in the Download
Data section.
Prior to working with any data in R, we must set the working directory to
the location of the data files. Setting the working directory tells R where
the data files are located on the computer. If the working directory is not
correctly set first, when we try to open a file we will get an error telling us
that R cannot find the file.
**Data Tip:** All NEON Data Skills tutorials are
written assuming the working directory is the parent directory to the
uncompressed .zip file of downloaded data. This allows for multiple data
subsets to be accessed in the tutorial without resetting the working directory.
Generally, these tutorials have a default working directory of **~/Documents/data**.
If you are working on a Mac, we suggest that you save your downloaded datasets
in a directory with the same name and location so that you don't have to edit
the code for the tutorial that you are working on. Most windows machines cannot
use the tilde "~" notation, therefore you must define the working directory
explicitly.
Wondering why we're saying directory instead of folder? See our
discussion of Directory vs. Folder in the middle of this tutorial.
Download & Uncompress the Data
1) Download
First, we will download the data to a location on the computer. To download the
data for this tutorial, click the blue button Download NEON Teaching Data
Subset: Meteorological Data for Harvard Forest within the box at the
top of this page.
Note: In other NEON Data Skills tutorials, download all data subsets in the
Download Data section prior to starting the tutorial. Here, the second
data subset is for those wishing to practice these skills in a Challenge
activity and will be downloaded at that time.
Screenshot of the Download Data button at the top of
NEON Data Skills tutorials. Source: National Ecological Observatory Network
(NEON)
After clicking on the Download Data button, the data will automatically
download to the computer.
2) Locate .zip file
Second, we need to find the downloaded .zip file. Many browsers default to
downloading to the Downloads directory on your computer.
Note: You may have previously specified a specific directory (folder) for files
downloaded from the internet, if so, the .zip file will download there.
Screenshot of the computer's Downloads folder containing the
new NEONDSMetTimeSeries.zip file. Source: National Ecological
Observatory Network (NEON)
3) Move to data directory
Third, we must move the data files to the location we want to work with them.
We recommend moving the .zip to a dedicated data directory within the
Documents directory on your computer. This data directory can
then be a repository for all data subsets you use for the NEON Data Skills
tutorials. Note: If you chose to store your data in a different directory
(e.g., not in ~/Documents/data), modify the directions below with the
appropriate file path to your data directory.
4) Unzip/uncompress
Fourth, we need to unzip/uncompress the file so that the data files can be
accessed. Use your favorite tool that can unpackage/open .zip files (e.g.,
winzip, Archive Utility, etc). The files will now be accessible in a directory
named NEON-DS-Met-Time-Series containing all the subdirectories and files that
make up the dataset or the subdirectories and files will be unzipped directly
into the data directory. If the latter happens, they need to be moved into a
data/NEON-DS-Met-Time-Series directory.
### Challenge: Download and Unzip Teaching Data Subset
Want to make sure you have these steps down! Prepare the
**Site Layout Shapefiles Teaching Data Subset** so that the files
are accessible and ready to be opened in R.
The directory should be the same as in this screenshot (below). Note that
NEON-DS-Site-Lyout-Files directory will only be in your directory if you
completed the challenge above. If you did not, your directory should look the
same but without that directory.
Screenshot of the neon directory with the nested
Documents, data, NEON-DS-Met-Time-Series, and other
directories. Source: National Ecological Observatory Network
(NEON)
Directory vs. Folder
"Directory" and "Folder" both refer to the same thing. Folder makes a lot of
sense when we think of an isolated folder as a "bin" containing many files.
However, the analogy to a physical file folder falters when we start thinking
about the relationship between different folders and how we tell a computer to
find a specific folder. This is why the term directory is often preferred. Any
directory (folder) can hold other directories and/or files. When we set the
working directory, we are telling the computer the location of the directory
(or folder) to start with when looking for other files or directories, or to
save any output to.
Full, Base, and Relative Paths
The data downloaded and unzipped in the previous steps are located within a
nested set of directories:
primary-level/home directory: neon
This directory isn't obvious as we are within this directory once we log
into the computer.
The full path is essentially the complete "directions" for how to find the
desired directory or file. It always starts with the home directory or root
(e.g., /Users/neon/). A full path is the base path when used to set
the working directory to a specific directory. The base path for the
NEON-DS-Met-Time-Series directory would be:
**Data Tip:** File or directory paths and the home
directory will appear slightly different in different operating systems.
Linux will appear as
`/home/neon/`. Windows will be similar to `C:\Documents and Settings\neon\` or
`C:\Users\neon\`. The format varies by Windows version. Make special note of
the direction of the slashes. Mac OS X and Unix format will appear as
`/Users/neon/`. This tutorial will show Mac OS X output unless specifically
noted.
### Challenge: Full File Path
Write out the full path for the `NEON-DS-Site-Layout-Shapefiles` directory. Use
the format of the operating system you are currently using.
Tip: When typing in the Rstudio console or an R script, if you surround your
filepath with quotes you can take advantage of the 'tab-completion' feature.
To use this feature, begin typing your filepath (e.g., "~/" or "C:") and then hit the tab button, which should pop up a list of possible directories and files that you could be pointing to. This method is awesome for avoiding typos in complex or long filepaths!
Bonus Points: Write the path as for one of the other operating systems.
Relative Path
A relative path is a path to a directory or file that starts from the
location determined by the working directory. If our working directory is set
to the data directory,
/Users/neon/Documents/data/
we can then create a relative path for all directories and files within the
data directory.
Screenshot of the data directory containing the both NEON Data
Skills Teaching Subsets. Source: National Ecological Observatory Network
(NEON)
The relative path for the meanNDVI_HARV_2011.csv file would be:
### Challenge: Relative File Path
Use the format of your current operating system:
Write out the full path to for the Boundary-US-State-Mass.shp file.
Write out the relative path for the Boundary-US-State-Mass.shp file
assuming that the working directory is set to /Users/neon/Documents/data/.
Bonus: Write the paths as for one of the other operating systems.
The R Working Directory
In R, the working directory is the directory where R starts when looking for
any file to open (as directed by a file path) and where it saves any output.
Without a working directory all R scripts would need the full file path
written any time we wanted to open or save a file. It is more efficient if we
have a base file path set as our working directory and then all file
paths written in our scripts only consist of the file path relative to that base
path (a relative path).
If you are unfamiliar with the term full path, base path, or
relative path, please see the section below on Full, Base, and Relative Paths
for a more detailed explanation before continuing with this tutorial.
Find a Full Path to a File in Unknown Location
If you are unsure of the path to a specific directory or file, you can
find this information for a particular file/directory of interest by looking in
the:
Windows: Properties, General tab (right click on the file/directory) or
in the file path bar at the top of each window (select versions).
Mac OS X: Get Info (right clicking/control+click on the file/directory)
Mac OS X: /Users/neon/Documents/data/NEON-DS-Met-Time-Series
**Data Tip:** File or directory paths and the home
directory will appear slightly different in different operating systems.
Linux will appear as
`/home/neon/`. Windows will be similar to `C:\Documents and Settings\neon\` or
`C:\Users\neon\`. The format varies by Windows version. Make special note of
the direction of the slashes. Mac OS X and Unix format will appear as
`/Users/neon/`. This tutorial will show Mac OSX output unless specifically
noted.
Determine Current Working Directory
Once we are in the R program, we can view the current working directory
using the code getwd().
# view current working directory
getwd()
[1] "/Users/neon"
The working directory is currently set to the home directory /Users/neon.
Remember, your current working directory will have a different user name and may
appear different based on operating system.
This code can be used at any time to determine the current working directory.
Set the Working Directory
To set our current working directory to the location where our data are located,
we can either set the working directory in the R script or use our current GUI
to select the working directory.
**Data Tip:** All NEON Data Skills tutorials are
written assuming the working directory is the parent directory to the downloaded
data (the **data** directory in this tutorial). This allows for multiple data
subsets to be accessed in the tutorial without resetting the working directory.
We want to set our working directory to the data directory.
Set the Working Directory: Base Path in Script
We can set the working directory using the code setwd("PATH") where PATH is
the full path to the desired directory. You can enter "PATH" as a string (as
shown below), or save the file path as a string to a variable (e.g.,
wd <- "~/Documents/data") and then set the working directory based on
that variable (e.g., setwd(wd)).
This latter method is used in many of the NEON Data Skills tutorials because
of the advantages that this method provides. First, this method is extermely
flexible across different compute environments and formats, including personal
computers, Linux-based servers on 'the cloud' (e.g., AWS, CyVerse), and when using
Rmarkdown (.Rmd) documents. Second this method allows the tutorial's
user to set their working directory once as a string and then to reuse that
string as needed to reference input files, or make output files. For example,
some functions must have a full filepath to an input file (such as when reading
in HDF5 files). Third, this method simplifies the process that NEON uses internally
to create and update these tutorials. All in all, saving the working
directory as a string variable makes the code more explicit and determanistic without
relying on working knowledge of relative filepaths, making your code more
producible and easier for an outsider to interpret.
To practice, use these techniques to set your working directory to the directory where
you have the data saved, and check that you set the working directory correctly.
There is no R output from setwd(). If we want to check
that the working directory is correctly set we can use getwd().
Example Windows File Path
Notice the the backslashes used in Windows paths must be changed to slashes in
R.
# set the working directory to `data` folder
wd <- "C:/Users/neon/Documents/data"
setwd(wd)
# check to ensure path is correct
getwd()
[1] "C:/Users/neon/Documents/data"
Example Mac OS X File Path
# set the working directory to `data` folder
wd <- "/Users/neon/Documents/data"
setwd(wd)
# check to ensure path is correct
getwd()
[1] "/Users/neon/Documents/data"
**Data Tip:** If using RStudio, you can view the
contents of the working directory in the Files pane.
The Files pane in RStudio shows the contents of the current
working directory. Source: National Ecological Observatory Network
(NEON)
Set the Working Directory: Using RStudio GUI
You can also set the working directory using the Rstudio and/or R graphical user interface (GUI).
This method is easy for beginners to learn, but it also makes your code less
reproducible because it relies on a person to follow certain instructions, which
is a process that introduces human error. It may also be impossible for an observer
to determine where your input data are stored, which can make troubleshooting
more difficult as well. Use this method when getting started, or when you will
find it helpful to use a graphical user interface to navigate your files.
Note that this method will run a single line setwd() command in the console
when you select your working directory, so you can copy/paste that line into
your script for future use!
Go to Session in menu bar,
select Select Working Directory,
select Choose Directory,
in the new window that appears, select the appropriate directory.
How to set the working directory using the RStudio GUI.
Source: National Ecological Observatory Network (NEON)
Set the Working Directory: Using R GUI
Windows Operating Systems:
Go to the File menu bar,
select Change dir... or Change Working Directory,
in the new window that appears, select the appropriate directory.
How to set the working directory using the R GUI in Windows.
Source: National Ecological Observatory Network (NEON)
Mac Operating Systems:
Go to the Misc menu,
select Change Working Directory,
in the new window that appears, select the appropriate directory.
How to set the working directory using the R GUI in Mac OS X.
Source: National Ecological Observatory Network (NEON)
This tutorial explores how to import and plot a multiband raster in
R. It also covers how to plot a three-band color image using the plotRGB()
function in R.
Learning Objectives
After completing this tutorial, you will be able to:
Know how to identify a single vs. a multiband raster file.
Be able to import multiband rasters into R using the terra package.
Be able to plot multiband color image rasters in R using plotRGB().
Understand what a NoData value is in a raster.
Things You’ll Need To Complete This Tutorial
You will need the most current version of R and, preferably, RStudio installed on your computer to complete this tutorial.
R Script & Challenge Code: NEON data lessons often contain challenges that reinforce skills. If available, the code for challenge solutions is found in the downloadable R script of the entire lesson, available in the footer of each lesson page.
The Basics of Imagery - About Spectral Remote Sensing Data
A raster can contain one or more bands. We can use the terra `rast` function to import one single band from a single OR multi-band
raster. Source: National Ecological Observatory Network (NEON).
To work with multiband rasters in R, we need to change how we import and plot our data in several ways.
To import multiband raster data we will use the stack() function.
If our multiband data are imagery that we wish to composite, we can use plotRGB() (instead of plot()) to plot a 3 band raster image.
About MultiBand Imagery
One type of multiband raster dataset that is familiar to many of us is a color image. A basic color image consists of three bands: red, green, and blue. Each band represents light reflected from the red, green or blue portions of the electromagnetic spectrum. The pixel brightness for each band, when composited creates the colors that we see in an image.
A color image consists of 3 bands - red, green and blue. When rendered together in a GIS, or even a tool like Photoshop or any other
image software, they create a color image. Source: National Ecological Observatory Network (NEON).
Getting Started with Multi-Band Data in R
To work with multiband raster data we will use the terra package.
# terra package to work with raster data
library(terra)
# package for downloading NEON data
library(neonUtilities)
# package for specifying color palettes
library(RColorBrewer)
# set working directory to ensure R can find the file we wish to import
wd <- "~/data/" # this will depend on your local environment environment
# be sure that the downloaded file is in this directory
setwd(wd)
In this tutorial, the multi-band data that we are working with is imagery collected using the
NEON Airborne Observation Platform
high resolution camera over the NEON Harvard Forest field site. Each RGB image is a 3-band raster. The same steps would apply to working with a multi-spectral image with 4 or more bands - like Landsat imagery, or even hyperspectral imagery (in geotiff format). We can plot each band of a multi-band image individually.
byTileAOP(dpID='DP3.30010.001', # rgb camera data
site='HARV',
year='2022',
easting=732000,
northing=4713500,
check.size=FALSE, # set to TRUE or remove if you want to check the size before downloading
savepath = wd)
## Downloading files totaling approximately 351.004249 MB
## Downloading 1 files
##
# Determine the number of bands
num_bands <- nlyr(RGB_HARV)
# Define colors to plot each
# Define color palettes for each band using RColorBrewer
colors <- list(
brewer.pal(9, "Reds"),
brewer.pal(9, "Greens"),
brewer.pal(9, "Blues")
)
# Plot each band in a loop, with the specified colors
for (i in 1:num_bands) {
plot(RGB_HARV[[i]], main=paste("Band", i), col=colors[[i]])
}
Image Raster Data Attributes
We can display some of the attributes about the raster, as shown below:
# Print dimensions
cat("Dimensions:\n")
## Dimensions:
cat("Number of rows:", nrow(RGB_HARV), "\n")
## Number of rows: 10000
cat("Number of columns:", ncol(RGB_HARV), "\n")
## Number of columns: 10000
cat("Number of layers:", nlyr(RGB_HARV), "\n")
## Number of layers: 3
# Print resolution
resolutions <- res(RGB_HARV)
cat("Resolution:\n")
## Resolution:
cat("X resolution:", resolutions[1], "\n")
## X resolution: 0.1
cat("Y resolution:", resolutions[2], "\n")
## Y resolution: 0.1
# Get the extent of the raster
rgb_extent <- ext(RGB_HARV)
# Convert the extent to a string with rounded values
extent_str <- sprintf("xmin: %d, xmax: %d, ymin: %d, ymax: %d",
round(xmin(rgb_extent)),
round(xmax(rgb_extent)),
round(ymin(rgb_extent)),
round(ymax(rgb_extent)))
# Print the extent string
cat("Extent of the raster: \n")
## Extent of the raster:
cat(extent_str, "\n")
## xmin: 732000, xmax: 733000, ymin: 4713000, ymax: 4714000
Let's take a look at the coordinate reference system, or CRS. You can use the parameters describe=TRUE to display this information more succinctly.
crs(RGB_HARV, describe=TRUE)
## name authority code
## 1 WGS 84 / UTM zone 18N EPSG 32618
## area
## 1 Between 78°W and 72°W, northern hemisphere between equator and 84°N, onshore and offshore. Bahamas. Canada - Nunavut; Ontario; Quebec. Colombia. Cuba. Ecuador. Greenland. Haiti. Jamaica. Panama. Turks and Caicos Islands. United States (USA). Venezuela
## extent
## 1 -78, -72, 84, 0
Let's next examine the raster's minimum and maximum values. What is the range of values for each band?
# Replace Inf and -Inf with NA
values(RGB_HARV)[is.infinite(values(RGB_HARV))] <- NA
# Get min and max values for all bands
min_max_values <- minmax(RGB_HARV)
# Print the results
cat("Min and Max Values for All Bands:\n")
## Min and Max Values for All Bands:
print(min_max_values)
## 2022_HARV_7_732000_4713000_image_1 2022_HARV_7_732000_4713000_image_2 2022_HARV_7_732000_4713000_image_3
## min 0 0 0
## max 255 255 255
This raster contains values between 0 and 255. These values represent the intensity of brightness associated with the image band. In
the case of a RGB image (red, green and blue), band 1 is the red band. When we plot the red band, larger numbers (towards 255) represent
pixels with more red in them (a strong red reflection). Smaller numbers (towards 0) represent pixels with less red in them (less red was reflected).
To plot an RGB image, we mix red + green + blue values into one single color to create a full color image - this is the standard color image a digital camera creates.
Challenge: Making Sense of Single Bands of a Multi-Band Image
Go back to the code chunk where you plotted each band separately. Compare the plots of band 1 (red) and band 2 (green). Is the forested area darker or lighter in band 2 (the green band) compared to band 1 (the red band)?
Other Types of Multi-band Raster Data
Multi-band raster data might also contain:
Time series: the same variable, over the same area, over time.
Multi or hyperspectral imagery: image rasters that have 4 or more (multi-spectral) or more than 10-15 (hyperspectral) bands. Check out the NEON
Data Skills Imaging Spectroscopy HDF5 in R tutorial to learn how to work with hyperspectral data cubes.
The true color image plotted at the beginning of this lesson looks pretty decent. We can explore whether applying a stretch to the image might improve clarity and contrast using stretch="lin" or stretch="hist".
When the range of pixel brightness values is closer to 0, a
darker image is rendered by default. We can stretch the values to extend to
the full 0-255 range of potential values to increase the visual contrast of
the image.
When the range of pixel brightness values is closer to 255, a lighter image is rendered by default. We can stretch the values to extend to the full 0-255 range of potential values to increase the visual contrast of the image.
# What does stretch do?
# Plot the linearly stretched raster
plotRGB(RGB_HARV, stretch="lin")
# Plot the histogram-stretched raster
plotRGB(RGB_HARV, stretch="hist")
In this case, the stretch doesn't enhance the contrast our image significantly given the distribution of reflectance (or brightness) values is distributed well between 0 and 255, and applying a stretch appears to introduce some artificial, almost purple-looking brightness to the image.
Challenge: What Methods Can Be Used on an R Object?
We can view various methods available to call on an R object with methods(class=class(objectNameHere)). Use this to figure out:
What methods can be used to call on the RGB_HARV object?
What methods are available for a single band within RGB_HARV?