DUE: 21 June 2018

During the NEON Data Institute, you will share the code that you create daily with everyone on the NEONScience/DI-NEON-participants repo.

Through this week’s tutorials, you have learned the basic skills needed to successfully share your work at the Institute including how to:

• Create your own GitHub user account,
• Set up Git on your computer (please do this on the computer you will be bringing to the Institute), and
• Create a Markdown file with a biography of yourself and the project you are interested in working on at the Institute. This biography was shared with the group via the Data Institute’s GitHub repo.

Checklist for this week’s Assignment:

You should have completed the following after Pre-institute week 2:

• Fork & clone the NEON-DataSkills/DI-NEON-participants repo.
• Create a .md file in the participants/2018-RemoteSensing/pre-institute2-git directory of the repo. Name the document LastName-FirstName.md.
• Write a biography that introduces yourself to the other participants. Please provide basic information including:
  • name,
  • domain of interest,
  • one goal for the course,
  • an updated version of your Capstone Project idea,
  • and the list of data (NEON or other) to support the project that you created during last week’s materials.
• Push the document from your local computer to your GithHub repo.
• Created a Pull Request to merge this document back into the NEON-DataSkills/DI-NEON-participants repo.

NOTE: The Data Institute repository is a public repository, so all members of the Institute, as well as anyone in the general public who stumbles on the repo, can see the information. If you prefer not to share this information publicly, please submit the same document but use a pseudonym (cartoon character names would work well) and email us with the pseudonym so that we can connect the submitted document to you.

Have questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

This tutorial covers how create and format Markdown files.

What is the .md Format?

Markdown is a human readable syntax for formatting text documents. Markdown can be used to produce nicely formatted documents including pdfs, web pages and more. In fact, this web page that you are reading right now is generated from a markdown document!

In this tutorial, we will create a markdown file that documents both who you are and also the project that you might want to work on at the NEON Data Institute.

Markdown Formatting

Markdown is simple plain text, that is styled using symbols, including:

• #: a header element
• **: bold text
• *: italic text
• `: code blocks

Let's review some basic markdown syntax.

Plain Text

Plain text will appear as text in a Markdown document. You can format that text in different ways.

For example, if we want to highlight a function or some code within a plain text paragraph, we can use one backtick on each side of the text ( ), like this: Here is some code. This is the backtick, or grave; not an apostrophe (on most US keyboards it is on the same key as the tilde).

To add emphasis to other text you can use bold or italics.

Have a look at the markdown below:

  The use of the highlight ( `text` ) will be reserved for denoting code.
To add emphasis to other text use **bold** or *italics*.

Notice that this sentence uses a code highlight "``", bold and italics. As a rendered markdown chunk, it looks like this:

The use of the highlight ( text ) will be reserve for denoting code when used in text. To add emphasis to other text use bold or italics.

Horizontal Lines (rules)

Create a rule:

  ***

Below is the rule rendered:

Section Headings

You can create a heading using the pound (#) sign. For the headers to render properly there must be a space between the # and the header text. Heading one is 1 pound sign, heading two is 2 pound signs, etc as follows:

Heading two

## Heading two

Heading three

### Heading three

Heading four

#### Heading four

For a more thorough list of markdown syntax, please read this GitHub Guide on Markdown.

Got questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

This tutorial covers adding new edits or contents from your forked repo on github.com and a central repo.

We now have done the following:

1. We've forked (made an individual copy of) the NEONScience/DI-NEON-participants repo to our github.com account.
2. We've cloned the forked repo - making a copy of it on our local computers.
3. We've added files and content to our local copy of the repo and committed the changes.
4. We've pushed those changes back up to our forked repo on github.com.

Once you've forked and cloned a repo, you are all setup to work on your project. You won't need to repeat those steps.

In this tutorial, we will learn how to transfer changes from our forked repo in our github.com account to the central NEON Data Institute repo. Adding information from your forked repo to the central repo in GitHub is done using a pull request.

Adding to a Repo Using Pull Requests

Pull Requests in GitHub

Step 1 - Start Pull Request

To start a pull request, click the pull request button on the main repo page.

Alternatively, you can click the Pull requests tab, then on this new page click the "New pull request" button.

Step 2 - Choose Repos to Update

Select your fork to compare with NEON central repo. When you begin a pull request, the head and base will auto-populate as follows:

• base fork: NEONScience/DI-NEON-participants
• head fork: YOUR-USER-NAME/DI-NEON-participants

The above pull request configuration tells Git to sync (or update) the NEON repo with contents from your repo.

Head vs Base

• Base: the repo that will be updated, the changes will be added to this repo.
• Head: the repo from which the changes come.

One way to remember this is that the “head” is always ahead of the base, so we must add from the head to the base.

Step 3 - Verify Changes

When you compare two repos in a pull request page, git will provide an overview of the differences (diffs) between the files (if the file is a binary file, like code. Non-binary files will just show up as a fully new file if it had any changes). Look over the changes and make sure nothing looks surprising.

Step 4 - Create Pull Request

Click the green Create Pull Request button to create the pull request.

Step 5 - Title Pull Request

Give your pull request a title and write a brief description of your changes. When you’re done with your message, click Create pull request!

Check out the repo name up at the top (in your repo and in screenshot above) When creating the pull request you will be automatically transferred to the base repo. Since the central repo was the base, github will automatically transfer you to the central repo landing page.

Step 6 - Merge Pull Request

In this final step, it’s time to merge your changes in the NEONScience/DI-NEON-participants repo.

NOTE 1: You are only able to merge a pull request in a repo that you have permissions to!

NOTE 2: When collaborating, it is generally poor form to merge your own Pull Request, better to tag (@username) a collaborator in the comments so they know you want them to look at it. They can then review and, if acceptable, merge it.

To merge, your (or someone else's PR click the green "Merge Pull Request" button to "accept" or merge the updated commits in the central repo into your repo. Then click Confirm Merge.

We now synced our forked repo with the central NEON Repo. The next step in working in a GitHub workflow is to transfer any changes in the central repository into your local repo so you can work with them.

After completing the pull request to upload your bio markdown file, be sure to continue on to Git 07: Updating Your Repo by Setting Up a Remote to learn how to update your local fork and really begin the cycle of working with Git & GitHub in a collaborative manner.

Workflow Summary

Add updates to Central Repo with Pull Request

On github.com

• Button: Create New Pull Request

• Set base: central Institute repo, set head: your Fork

• Make sure changes are what you want to sync

• Button: Create Pull Request

• Add Pull Request title & comments

• Button: Create Pull Request

• Button: Merge Pull Request - if working collaboratively, poor style to merge your own PR, and you only can if you have contributor permissions

  ---

  Have questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

This tutorial reviews how to add and commit changes to a Git repo.

In the previous lesson, we created a markdown (.md) file in our forked version of the DI-NEON-participants central repo. In order for Git to recognize this new file and track it, we need to:

1. Add the file to the repository using git add.
2. Commit the file to the repository as a set of changes to the repo (in this case, a new document with some text content) using git commit.
3. Push or sync the changes we've made locally with our forked repo hosted on github.com using git push.

Check Repository Status -- git status

Let's first run through some basic commands to get going with Git at the command line. First, it's always a good idea to check the status of your repository. This allows us to see any changes that have occurred.

Do the following:

1. Open bash if it's not already open.
2. Navigate to the DI-NEON-participants repository in bash.
3. Type: git status.

The commands that you type into bash should look like the code below:

# Change directory
# The directory containing the git repo that you wish to work in.
$ cd ~/Documents/GitHub/neon-data-repository-2016

# check the status of the repo
$ git status

Output:

On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git checkout -- <file>..." to discard changes in working directory)

Untracked files:
 (use "git add <file>..." to include in what will be committed)

_posts/ExampleFile.md

Let's make sense of the output of the git status command.

• On branch master: This tells us that we are on the master branch of the repo. Don't worry too much about branches just yet. We will work on the master branch throughout the Data Institute.
• Changes not staged for commit: This lists any file(s) that is/are currently being tracked by Git but have new changes that need to be added for Git to track.
• Untracked file: These are all new files that have never been added to or tracked by Git.

Use git status anytime to view any untracked changes that have occurred, what is being tracked and what is not currently being tracked.

Add a File - git add

Next, let's add the Markdown file containing our bio and short project summary using the command git add FileName.md. Replace FileName.md with the name of your markdown file.

# add a file, so that changes are tracked
$ git add ExampleBioFile.md

# check status again
$ git status

On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
 (use "git reset HEAD <file>..." to unstage)

	new file:  _posts/ExampleBioFile.md

Understand the output:

• Changes to be committed: This lists the new files or files with changes that have been added to the Git tracking system but need to be committed as actual changes in the git repository history.

Commit Changes - git commit

When we add a file in the command line, we are telling Git to recognize that a change has occurred. The file moves to a "staging" area where Git recognizes a change has happened but the change has not yet been formally documented. When we want to permanently document those changes, we commit the change. A single commit will work for all files that are currently added to and in the Git staging area (anything in green when we check the status).

Commit Messages

When we commit a change to the Git version control system, we need to add a commit message. This message describes the changes made in the commit. This commit message is helpful to us when we review commit history to see what has changed over time and when those changes occurred. Be sure that your message covers the change.

# commit changes with message
$ git commit -m “new example file for demonstration”

[master e3cd622] new example file for demonstration
 1 file changed, 56 insertions(+), 4 deletions(-)
 create mode 100644 _posts/ExampleFile.md

Understand the output: Each commit will look slightly different but the important parts include:

• master xxxxxxx this is the unique identifier for this set of changes or this commit. You will always be able to track this specific commit (this specific set of changes) using this identifier.
• _ file change, _ insertions(+), _ deletion (-) this tells us how many files have changed and the number of type of changes made to the files including: insertions, and deletions.

Why Add, then Commit?

To understand what is going on with git add and git commit it is important to understand that Git has a staging area that we add items to with git add. Changes are not actually documented and permanently tracked until we commit them. This allows us to commit specific groups of files at the same time if we wish. For instance, we may decide to add and commit all R scripts together. And Markdown files in another, separate commit.

Transfer Changes (Commits) from a Local Repo to a GitHub Repo - git push

When we are done editing our files and have committed the changes locally, we are ready to transfer or sync these changes to our forked repo on github.com. To do this we need to push our changes from the local Git version control to the remote GitHub repo.

To sync local changes with github.com, we can do the following:

1. Check the status of our repo using git status. Are all of the changes added and committed to the repo?
2. Use git push origin master. origin tells Git to push the files to the originating repo which in this case - is our fork on github.com which we originally cloned to our local computer. master is the repo branch that you are currently working on.

Let's push the changes that we made to the local version of our Git repo to our fork, in our github.com account.

# check the repo status
$ git status

On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

# transfer committed changes to the forked repo
git push origin master

Counting objects: 1, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.51 KiB | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To https://github.com/mjones01/DI-NEON-participants.git
   5022aca..e3cd622  master -> master

NOTE: You may be asked for your username and password! This is your github.com username and password.

Understand the output:

• Pay attention to the repository URL - the "origin" is the repository that the commit was pushed to, here https://github.com/mjones01/DI-NEON-participants.git. Note that because this repo is a fork, your URL will have your GitHub username in it instead of "mjones01".

View Commits in GitHub

Let’s view our recent commit in our forked repo on GitHub.

1. Go to github.com and navigate to your forked Data Institute repo - DI-NEON-participants.
2. Click on the commits link at the top of the page.
3. Look at the commits - do you see your recent commit message that you typed into bash on your computer?
4. Next, click on the <>CODE link which is ABOVE the commits link in github.
5. Is the Markdown file that you added and committed locally at the command line on your computer, there in the same directory (participants/pre-institute2-git) that you saved it on your laptop?

Is Your File in the NEON Central Repo Yet?

Next, do the following:

1. Navigate to the NEON central NEONScience/DI-NEON-participants repo. (The easiest method to do this is to click the link at the top of the page under your repo name).
2. Look for your file in the same directory. Is your new file there? If not, why?

Remember the structure of our workflow.

We’ve added changes from our local repo on our computer and pushed them to our fork on github.com. But this fork is in our individual user account, not NEONS. This fork is separate from the central repo. Changes to a fork in our github.com account do not automatically transfer to the central repo. We need to sync them! We will learn how to sync these two repos in the next tutorial Git 06: Syncing GitHub Repos with Pull Requests .

Summary Workflow - Committing Changes

On your computer, within your local copy of the Git repo:

• Create a new markdown file and edit it in your favorite text editor.

On your computer, in shell (at the command line):

• git status
• git add FileName
• git status - make sure everything is added and ready for commit
• `git commit -m “messageHere”
• git push origin master

On the github.com website:

• Check to make sure commit is added.
• Check to see if the file that you added is visible online in your Git repo.

Have questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

This tutorial covers how to clone a github.com repo to your computer so that you can work locally on files within the repo.

Clone - Copy Repo To Your Computer

In the previous tutorial, we used the github.com interface to fork the central NEON repo. By forking the NEON repo, we created a copy of it in our github.com account.

Now we will learn how to create a local version of our forked repo on our laptop, so that we can efficiently add to and edit repo content.

Copy Repo URL

Start from the github.com interface:

1. Navigate to the repo that you want to clone (copy) to your computer -- this should be YOUR-USER-NAME/DI-NEON-participants.
2. Click on the Clone or Download dropdown button and copy the URL of the repo.

Then on your local computer:

1. Your computer should already be setup with Git and a bash shell interface. If not, please refer to the Institute setup materials before continuing.
2. Open bash on your computer and navigate to the local GitHub directory that you created using the Set-up Materials.

To do this, at the command prompt, type:

$ cd ~/Documents/GitHub

Note: If you have stored your GitHub directory in a location that is different

• i.e. it is not /Documents/GitHub, be sure to adjust the above code to represent the actual path to the GitHub directory on your computer.

Now use git clone to clone, or create a copy of, the entire repo in the GitHub directory on your computer.

# clone the forked repo to our computer
$ git clone https://github.com/neon/DI-NEON-participants.git

The output shows you what is being cloned to your computer.

Cloning into 'DI-NEON-participants.git'...
remote: Counting objects: 3808, done.
remote: Total 3808 (delta 0), reused 0 (delta 0), pack-reused 3808
Receiving objects: 100% (3808/3808), 2.92 MiB | 2.17 MiB/s, done.
Resolving deltas: 100% (2185/2185), done.
Checking connectivity... done.
$

Note: The output numbers that you see on your computer, representing the total file size, etc, may differ from the example provided above.

View the New Repo

Next, let's make sure the repository is created on your computer in the location where you think it is.

At the command line, type ls to list the contents of the current directory.

# view directory contents
$ ls

Next, navigate to your copy of the data institute repo using cd or change directory:

# navigate to the NEON participants repository
$ cd DI-NEON-participants

# view repository contents
$ ls

404.md			_includes		code
ISSUE_TEMPLATE.md	_layouts		images
README.md		_posts			index.md
_config.yml		_site			institute-materials
_data			assets			org

Alternatively, we can view the local repo DI-NEON-participants in a finder (Mac) or Windows Explorer (Windows) window. Simply open your Documents in a window and navigate to the new local repo.

Using either method, we can see that the file structure of our cloned repo exactly mirrors the file structure of our forked GitHub repo.

Summary Workflow -- Create a Local Repo

In the github.com interface:

• Copy URL of the repo you want to work on locally

In shell:

• git clone URLhere

Note: that you can copy the URL of your repository directly from GitHub.

Got questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

In this tutorial, we will fork, or create a copy in your github.com account, an existing GitHub repository. We will also explore the github.com interface.

Create An Account

If you do not already have a GitHub account, go to GitHub and sign up for your free account. Pick a username that you like! This username is what your colleagues will see as you work with them in GitHub and Git.

Take a minute to setup your account. If you want to make your account more recognizable, be sure to add a profile picture to your account!

If you already have a GitHub account, simply sign in.

Navigate GitHub

Repositories, AKA Repos

Let's first discuss the repository or "repo". (The cool kids say repo, so we will jump on the git cool kid bandwagon) and use "repo" from here on in. According to the GitHub glossary:

A repository is the most basic element of GitHub. They're easiest to imagine as a project's folder. A repository contains all of the project files (including documentation), and stores each file's revision history. Repositories can have multiple collaborators and can be either public or private.

In the Data Institute, we will share our work in the DI-NEON-participants repo.

Find an Existing Repo

The first thing that you'll need to do is find the DI-NEON-participants repo. You can find repos in two ways:

1. Type “DI-NEON-participants” in the github.com search bar to find the repository.
2. Use the repository URL if you have it - like so: https://github.com/NEONScience/DI-NEON-participants.

Navigation of a Repo Page

Once you have found the Data Institute participants repo, take 5 minutes to explore it.

Git Repo Names

First, get to know the repository naming convention. Repository names all take the format:

OrganizationName/RepositoryName

So the full name of our repository is:

NEONScience/DI-NEON-participants

Header Tabs

At the top of the page you'll notice a series of tabs. Please focus on the following 3 for now:

• Code: Click here to view structure & contents of the repo.
• Issues: Submit discussion topics, or problems that you are having with the content in the repo, here.
• Pull Requests: Submit changes to the repo for review / acceptance. We will explore pull requests more in the Git 06 tutorial.

Other Text Links

A bit further down the page, you'll notice a few other links:

• commits: a commit is a saved and documented change to the content or structure of the repo. The commit history contains all changes that have been made to that repo. We will discuss commits more in Git 05: Git Add Changes -- Commits .

Fork a Repository

Next, let's discuss the concept of a fork on the github.com site. A fork is a copy of the repo that you create in your account. You can fork any repo at any time by clicking the fork button in the upper right hand corner on github.com.

Check Out Your Data Institute Fork

Now, check out your new fork. Its name should be:

YOUR-USER-NAME/DI-NEON-participants.

It can get confusing sometimes moving between a central repo:

• https://github.com/NEONScience/DI-NEON-participants

and your forked repo:

• https://github.com/YOUR-USER-NAME/DI-NEON-participants

A good way to figure out which repo you are viewing is to look at the name of the repo. Does it contain your username? Or your colleagues'? Or NEON's?

Your Fork vs the Central Repo

Your fork is an exact copy, or completely in sync with, the NEON central repo. You could confirm this by comparing your fork to the NEON central repository using the pull request option. We will learn about pull requests in Git06: Sync GitHub Repos with Pull Requests. For now, take our word for it.

The fork will remain in sync with the NEON central repo until:

1. You begin to make changes to your forked copy of the repo.
2. The central repository is changed or updated by a collaborator.

If you make changes to your forked repo, the changes will not be added to the NEON central repo until you sync your fork with the NEON central repo.

Summary Workflow -- Fork a GitHub Repository

On the github.com website:

• Navigate to desired repo that you want to fork.
• Click Fork button.

Have questions? No problem. Leave your question in the comment box below. It's likely some of your colleagues have the same question, too! And also likely someone else knows the answer.

In this page, you will be introduced to the importance of version control in scientific workflows.

The text and graphics in the first three sections were borrowed, with some modifications, from Software Carpentry's Version Control with Git lessons.

What is Version Control?

A version control system maintains a record of changes to code and other content. It also allows us to revert changes to a previous point in time.

Types of Version control

There are many forms of version control. Some not as good:

• Save a document with a new date (we’ve all done it, but it isn’t efficient)
• Google Docs "history" function (not bad for some documents, but limited in scope).

Some better:

• Mercurial
• Subversion
• Git - which we’ll be learning much more about in this series.

More Resources:

Visit the version control Wikipedia list of version control platforms.

• Read the Git documentation explaining the progression of version control systems.

Why Version Control is Important

Version control facilitates two important aspects of many scientific workflows:

1. The ability to save and review or revert to previous versions.
2. The ability to collaborate on a single project.

This means that you don’t have to worry about a collaborator (or your future self) overwriting something important. It also allows two people working on the same document to efficiently combine ideas and changes.

How Version Control Systems Works

Simple Version Control Model

A version control system keeps track of what has changed in one or more files over time. The way this tracking occurs, is slightly different between various version control tools including git, mercurial and svn. However the principle is the same.

Version control systems begin with a base version of a document. They then save the committed changes that you make. You can think of version control as a tape: if you rewind the tape and start at the base document, then you can play back each change and end up with your latest version.

Once you think of changes as separate from the document itself, you can then think about “playing back” different sets of changes onto the base document. You can then retrieve, or revert to, different versions of the document.

The benefit of version control when you are in a collaborative environment is that two users can make independent changes to the same document.

If there aren’t conflicts between the users changes (a conflict is an area where both users modified the same part of the same document in different ways) you can review two sets of changes on the same base document.

A version control system is a tool that keeps track of these changes for us. Each version of a file can be viewed and reverted to at any time. That way if you add something that you end up not liking or delete something that you need, you can simply go back to a previous version.

Git & GitHub - A Distributed Version Control Model

GitHub uses a distributed version control model. This means that there can be many copies (or forks in GitHub world) of the repository.

Have a look at the graphic below. Notice that in the example, there is a "central" version of our repository. Joe, Sue and Eve are all working together to update the central repository. Because they are using a distributed system, each user (Joe, Sue and Eve) has their own copy of the repository and can contribute to the central copy of the repository at any time.

Create A Working Copy of a Git Repo - Fork

There are many different Git and GitHub workflows. In the NEON Data Institute, we will use a distributed workflow with a Central Repository. This allows us all (all of the Institute participants) to work independently. We can then contribute our changes to update the Central (NEON) Repository. Our collaborative workflow goes like this:

1. NEON "owns" the Central Repository.
2. You will create a copy of this repository (known as a fork) in your own GitHub account.
3. You will then clone (copy) the repository to your local computer. You will do your work locally on your laptop.
4. When you are ready to submit your changes to the NEON repository, you will:
   • Sync your local copy of the repository with NEON's central repository so you have the most up to date version, and then,
   • Push the changes you made to your local copy (or fork) of the repository to NEON's main repository.

Each participant in the institute will be contributing to the NEON central repository using the same workflow! Pretty cool stuff.

Let's get some terms straight before we go any further.

• Central repository - the central repository is what all participants will add to. It is the "final working version" of the project.
• Your forked repository - is a "personal” working copy of the central repository stored in your GitHub account. This is called a fork. When you are happy with your work, you update your repo from the central repo, then you can update your changes to the central NEON repository.
• Your local repository - this is a local version of your fork on your own computer. You will most often do all of your work locally on your computer.

Additional Resources:

Further documentation for and how-to-use direction for Git, is provided by the Git Pro version 2 book by Scott Chacon and Ben Straub , available in print or online. If you enjoy learning from videos, the site hosts several.

This tutorial builds upon the previous tutorial, to work with shapefile attributes in R and explores how to plot multiple shapefiles using base R graphics. It then covers how to create a custom legend with colors and symbols that match your plot.

Load the Data

To work with vector data in R, we can use the rgdal library. The raster package also allows us to explore metadata using similar commands for both raster and vector files.

We will import three shapefiles. The first is our AOI or area of interest boundary polygon that we worked with in Open and Plot Shapefiles in R. The second is a shapefile containing the location of roads and trails within the field site. The third is a file containing the Harvard Forest Fisher tower location. These latter two we worked with in the Explore Shapefile Attributes & Plot Shapefile Objects by Attribute Value in R tutorial.

# load packages
# rgdal: for vector work; sp package should always load with rgdal. 
library(rgdal)  
# raster: for metadata/attributes- vectors or rasters
library(raster)   

# set working directory to data folder
# setwd("pathToDirHere")

# Import a polygon shapefile 
aoiBoundary_HARV <- readOGR("NEON-DS-Site-Layout-Files/HARV",
                            "HarClip_UTMZ18", stringsAsFactors = T)

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HarClip_UTMZ18"
## with 1 features
## It has 1 fields
## Integer64 fields read as strings:  id

# Import a line shapefile
lines_HARV <- readOGR( "NEON-DS-Site-Layout-Files/HARV", "HARV_roads", stringsAsFactors = T)

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HARV_roads"
## with 13 features
## It has 15 fields

# Import a point shapefile 
point_HARV <- readOGR("NEON-DS-Site-Layout-Files/HARV",
                      "HARVtower_UTM18N", stringsAsFactors = T)

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/olearyd/Git/data/NEON-DS-Site-Layout-Files/HARV", layer: "HARVtower_UTM18N"
## with 1 features
## It has 14 fields

Plot Data

In the Explore Shapefile Attributes & Plot Shapefile Objects by Attribute Value in R tutorial we created a plot where we customized the width of each line in a spatial object according to a factor level or category. To do this, we create a vector of colors containing a color value for EACH feature in our spatial object grouped by factor level or category.

# view the factor levels
levels(lines_HARV$TYPE)

## [1] "boardwalk"  "footpath"   "stone wall" "woods road"

# create vector of line width values
lineWidth <- c(2,4,3,8)[lines_HARV$TYPE]
# view vector
lineWidth

##  [1] 8 4 4 3 3 3 3 3 3 2 8 8 8

# create a color palette of 4 colors - one for each factor level
roadPalette <- c("blue","green","grey","purple")
roadPalette

## [1] "blue"   "green"  "grey"   "purple"

# create a vector of colors - one for each feature in our vector object
# according to its attribute value
roadColors <- c("blue","green","grey","purple")[lines_HARV$TYPE]
roadColors

##  [1] "purple" "green"  "green"  "grey"   "grey"   "grey"   "grey"  
##  [8] "grey"   "grey"   "blue"   "purple" "purple" "purple"

# create vector of line width values
lineWidth <- c(2,4,3,8)[lines_HARV$TYPE]
# view vector
lineWidth

##  [1] 8 4 4 3 3 3 3 3 3 2 8 8 8

# in this case, boardwalk (the first level) is the widest.
plot(lines_HARV, 
     col=roadColors,
     main="NEON Harvard Forest Field Site\n Roads & Trails \nLine Width Varies by Type Attribute Value",
     lwd=lineWidth)

Add Plot Legend

In the the previous tutorial, we also learned how to add a basic legend to our plot.

• bottomright: We specify the location of our legend by using a default keyword. We could also use top, topright, etc.
• levels(objectName$attributeName): Label the legend elements using the categories of levels in an attribute (e.g., levels(lines_HARV$TYPE) means use the levels boardwalk, footpath, etc).
• fill=: apply unique colors to the boxes in our legend. palette() is the default set of colors that R applies to all plots.

Let's add a legend to our plot.

plot(lines_HARV, 
     col=roadColors,
     main="NEON Harvard Forest Field Site\n Roads & Trails\n Default Legend")

# we can use the color object that we created above to color the legend objects
roadPalette

## [1] "blue"   "green"  "grey"   "purple"

# add a legend to our map
legend("bottomright", 
       legend=levels(lines_HARV$TYPE), 
       fill=roadPalette, 
       bty="n", # turn off the legend border
       cex=.8) # decrease the font / legend size

However, what if we want to create a more complex plot with many shapefiles and unique symbols that need to be represented clearly in a legend?

Plot Multiple Vector Layers

Now, let's create a plot that combines our tower location (point_HARV), site boundary (aoiBoundary_HARV) and roads (lines_HARV) spatial objects. We will need to build a custom legend as well.

To begin, create a plot with the site boundary as the first layer. Then layer the tower location and road data on top using add=TRUE.

# Plot multiple shapefiles
plot(aoiBoundary_HARV, 
     col = "grey93", 
     border="grey",
     main="NEON Harvard Forest Field Site")

plot(lines_HARV, 
     col=roadColors,
     add = TRUE)

plot(point_HARV, 
     add  = TRUE, 
     pch = 19, 
     col = "purple")

# assign plot to an object for easy modification!
plot_HARV<- recordPlot()

Customize Your Legend

Next, let's build a custom legend using the symbology (the colors and symbols) that we used to create the plot above. To do this, we will need to build three things:

1. A list of all "labels" (the text used to describe each element in the legend to use in the legend.
2. A list of colors used to color each feature in our plot.
3. A list of symbols to use in the plot. NOTE: we have a combination of points, lines and polygons in our plot. So we will need to customize our symbols!

Let's create objects for the labels, colors and symbols so we can easily reuse them. We will start with the labels.

# create a list of all labels
labels <- c("Tower", "AOI", levels(lines_HARV$TYPE))
labels

## [1] "Tower"      "AOI"        "boardwalk"  "footpath"   "stone wall"
## [6] "woods road"

# render plot
plot_HARV

# add a legend to our map
legend("bottomright", 
       legend=labels, 
       bty="n", # turn off the legend border
       cex=.8) # decrease the font / legend size

Now we have a legend with the labels identified. Let's add colors to each legend element next. We can use the vectors of colors that we created earlier to do this.

# we have a list of colors that we used above - we can use it in the legend
roadPalette

## [1] "blue"   "green"  "grey"   "purple"

# create a list of colors to use 
plotColors <- c("purple", "grey", roadPalette)
plotColors

## [1] "purple" "grey"   "blue"   "green"  "grey"   "purple"

# render plot
plot_HARV

# add a legend to our map
legend("bottomright", 
       legend=labels, 
       fill=plotColors,
       bty="n", # turn off the legend border
       cex=.8) # decrease the font / legend size

Great, now we have a legend! However, this legend uses boxes to symbolize each element in the plot. It might be better if the lines were symbolized as a line and the points were symbolized as a point symbol. We can customize this using pch= in our legend: 16 is a point symbol, 15 is a box.

# create a list of pch values
# these are the symbols that will be used for each legend value
# ?pch will provide more information on values
plotSym <- c(16,15,15,15,15,15)
plotSym

## [1] 16 15 15 15 15 15

# Plot multiple shapefiles
plot_HARV

# to create a custom legend, we need to fake it
legend("bottomright", 
       legend=labels,
       pch=plotSym, 
       bty="n", 
       col=plotColors,
       cex=.8)

Now we've added a point symbol to represent our point element in the plot. However it might be more useful to use line symbols in our legend rather than squares to represent the line data. We can create line symbols, using lty = (). We have a total of 6 elements in our legend:

1. A Tower Location
2. An Area of Interest (AOI)
3. and 4 Road types (levels)

The lty list designates, in order, which of those elements should be designated as a line (1) and which should be designated as a symbol (NA). Our object will thus look like lty = c(NA,NA,1,1,1,1). This tells R to only use a line element for the 3-6 elements in our legend.

Once we do this, we still need to modify our pch element. Each line element (3-6) should be represented by a NA value - this tells R to not use a symbol, but to instead use a line.

# create line object
lineLegend = c(NA,NA,1,1,1,1)
lineLegend

## [1] NA NA  1  1  1  1

plotSym <- c(16,15,NA,NA,NA,NA)
plotSym

## [1] 16 15 NA NA NA NA

# plot multiple shapefiles
plot_HARV

# build a custom legend
legend("bottomright", 
       legend=labels, 
       lty = lineLegend,
       pch=plotSym, 
       bty="n", 
       col=plotColors,
       cex=.8)

In this tutorial, we will cover the R knitr package that is used to convert R Markdown into a rendered document (HTML, PDF, etc).

Share & Publish Results Directly from Your Code!

The knitr package allow us to:

• Publish & share preliminary results with collaborators.
• Create professional reports that document our workflow and results directly from our code, reducing the risk of accidental copy and paste or transcription errors.
• Document our workflow to facilitate reproducibility.
• Efficiently change code outputs (figures, files) given changes in the data, methods, etc.

Publish from Rmd files with knitr

To complete this tutorial you need:

1. The R knitr package to complete this tutorial. If you need help installing packages, visit the R packages tutorial.
2. An R Markdown document that contains a YAML header, code chunks and markdown segments. If you don't have an .Rmd file, visit the R Markdown tutorial to create one.

How to Knit

To knit in RStudio, click the knit pull down button. You want to use the
knit HTML for this lesson.

When you click the Knit HTML button, a window will open in your console titled R Markdown. This pane shows the knitting progress. The output (HTML in this case) file will automatically be saved in the current working directory. If there is an error in the code, an error message will appear with a line number in the R Console to help you diagnose the problem.

View the Output

When knitting is complete, the new HTML file produced will automatically open.

Notice that information from the YAML header (title, author, date) is printed at the top of the HTML document. Then the HTML shows the text, code, and results of the code that you included in the RMD document.

You will need to have the rmarkdown and knitr packages installed on your computer prior to completing this tutorial. Refer to the setup materials to get these installed.

Learning Objectives

At the end of this activity, you will:

• Know how to create an R Markdown file in RStudio.
• Be able to write a script with text and R code chunks.
• Create an R Markdown document ready to be ‘knit’ into an HTML document to share your code and results.

Things You’ll Need To Complete This Tutorial

You will need the most current version of R and, preferably, RStudio loaded on your computer to complete this tutorial.

Install R Packages

• knitr: install.packages("knitr")
• rmarkdown: install.packages("rmarkdown")
• raster: install.packages("raster")
• rgdal: install.packages("rgdal")

Download Data

Download NEON Teaching Data Subset: TEAK-Data Institute 2016

The LiDAR and imagery data used to create this raster teaching data subset were collected over the National Ecological Observatory Network's (NEON) Lower Teakettle field site and processed at NEON headquarters. The entire dataset can be accessed by request from the NEON Data Portal.

Download Dataset

You will want to create a data directory for all the Data Institute teaching datasets. We suggest the pathway be ~/Documents/data/NEONDI-2016 or the equivalent for your operating system. Once you've downloaded and unzipped the dataset, move it to this directory.

Additional Resources

• R Markdown Cheatsheet: a very handy reference for using R Markdown
• R Markdown Reference Guide: a more expensive reference for R Markdown
• Introduction to R Markdown by Garrett Grolemund: a tutorial for learning R Markdown

Create an Rmd File

RMarkdown in RStudio Video

Our goal in this series is to document our workflow. We can do this by

1. Creating an R Markdown (RMD) file in R studio and
2. Rendering that RMD file to HTML using knitr.

Watch this 6:38 minute video below to learn more about how you can convert an R Markdown file to HTML (or other formats) using knitr in RStudio. The text size in the video is small so you may want to watch the video in full screen mode.

Now that you have a sense of how R Markdown can be used in RStudio, you are ready to create your own RMD document. Do the following:

1. Create a new R Markdown file and choose HTML as the desired output format.
2. Enter a Title (Explore NEON LiDAR Data) and Author Name (your name). Then click OK.
3. Save the file using the following format: LastName-institute-week3.rmd NOTE: The document title is not the same as the file name.
4. Hit the knit button in RStudio (as is done in the video above). What happens?

If everything went well, you should have an HTML format (web page) output after you hit the knit button. Note that this HTML output is built from a combination of code and documentation that was written using markdown syntax.

Next, we'll break down the structure of an R Markdown file.

Understand Structure of an R Markdown file

Let's next review the structure of an R Markdown (.Rmd) file. There are three main content types:

• Header: the text at the top of the document, written in YAML format.
• Markdown sections: text that describes your workflow written using markdown syntax.
• Code chunks: Chunks of R code that can be run and also can be rendered using knitr to an output document.

Next let's explore each section type.

Header -- YAML block

An R Markdown file always starts with header written using YAML syntax. There are four default elements in the RStudio generated YAML header:

• title: the title of your document. Note, this is not the same as the file name.
• author: who wrote the document.
• date: by default this is the date that the file is created.
• output: what format will the output be in. We will use HTML.

A YAML header may be structured differently depending upon how your are using it. Learn more on the R Markdown documentation page.

R Markdown Text/Markdown Blocks

An RMD document contains a mixture of code chunks and markdown blocks where you can describe aspects of your processing workflow. The markdown blocks use the same markdown syntax that we learned last week in week 2 materials. In these blocks you might describe the data that you are using, how it's being processed and and what the outputs are. You may even add some information that interprets the outputs.

When you render your document to HTML, this markdown will appear as text on the output HTML document.

Learn More about RStudio Markdown Basics

Explore Your R Markdown File

Look closely at the pre-populated markdown and R code chunks in your RMD file.

Does any of the markdown syntax look familiar?

• Are any words in bold?
• Are any words in italics?
• Are any words highlighted as code?

If you are unsure, the answers are at the bottom of this page.

Code chunks

Code chunks are where your R code goes. All code chunks start and end with ``` – three backticks or graves. On your keyboard, the backticks can be found on the same key as the tilde. Graves are not the same as an apostrophe!

The initial line of a code chunk must appear as:

 ```{r chunk-name-with-no-spaces} 
# code goes here
 ```

The r part of the chunk header identifies this chunk as an R code chunk and is mandatory. Next to the {r, there is a chunk name. This name is not required for basic knitting however, it is good practice to give each chunk a unique name as it is required for more advanced knitting approaches.

```{r setup-library }
   
   library(rgdal)
   library(raster)
 
 ```

```{r code-setwd}

   # set working directory to ensure R can find the file we wish to import.
   # This will depend on your local environment.
   setwd("~/Documents/data/NEONDI-2016/") 

```

You changed the working directory to ~/Documents/data/NEONDI-2016/ (probably via setwd()). It will be restored to [directory path of current .rmd file]. See the Note section in ?knitr::knit ?knitr::knit

```{r code-setwd-stringvariable}

	# set working directory as a string variable for use in other code chunks.
	# This will depend on your local environment.
	wd <- "~/Documents/data/NEONDI-2016/"
	setwd(wd)

 ```

```{r load-dsm-raster }
   # check for the working directory
   getwd()
   # In this new chunk, the working directory has reverted to default upon kitting. 
   # Combining the working directory string variable and 
   # additional path to the file, import a DSM file.
   teak_dsm <- raster(paste0(wd, "NEONdata/D17-California/TEAK/2013/lidar/TEAK_lidarDSM.tif"))
   
 ```

Code chunk options

You can also add arguments or options to each code chunk. These arguments allow you to customize how or if you want code to be processed or appear on the output HTML document. Code chunk arguments are added on the first line of a code chunk after the name, within the curly brackets.

The example below, is a code chunk that will not be "run", or evaluated, by R. The code within the chunk will appear on the output document, however there will be no outputs from the code.

```{r intro-option, eval=FALSE}
# the code here will not be processed by R 
# but it will appear on your output document
1+2
 ```

We use eval=FALSE often when the chunk is exporting an file that we don't need to re-export but we want to document the code used to export the file.

Three common code chunk options are:

• eval = FALSE: Do not evaluate (or run) this code chunk when knitting the RMD document. The code in this chunk will still render in our knitted HTML output, however it will not be evaluated or run by R.
• echo = FALSE: Hide the code in the output. The code is evaluated when the RMD file is knit, however only the output is rendered on the output document.
• results = hide: The code chunk will be evaluated but the results of the code will not be rendered on the output document. This is useful if you are viewing the structure of a large object (e.g. outputs of a large data.frame).

Multiple code chunk options can be used for the same chunk. For more on code chunk options, read R Markdown: The Definitive Guide or the knitr documentation.

Now continue on to the next tutorial to learn how to knit this document into a HTML file.