R on plantarum.ca

Spatial Tutorials Update

Fri, 10 May 2024 00:00:00 +0000

A quick update. The spatial analysis libraries in the R Project have undergone a substantial change in the past couple of years. The details are laid out in the R spatial blog, but the crux of the issue is that legacy packages rgdal and rgeos have been retired, and packages that depend on them (such as raster and sp) will have been modified to use new dependencies, or replaced entirely. For the most part, the things we used to do with raster we now do with terra.

The transition was a bit rough, and for a while we needed to translate back and forth between terra and raster in our work. That should now be over, with all current packages using the new terra-based workflow.

I have already updated my quick mapping tutorial, and I’ve just updated my ecospat tutorial, now that ecospat has been fully updated to use terra too. Some of the other spatial tutorials you find here may not work properly, or at all, until I have a chance to review them. If you happen to find anything that isn’t working as expected, please let me know!

Preparing GBIF records for distribution modeling

Thu, 04 Apr 2024 00:00:00 +0000

GBIF.org

The Global Biodiversity Information Facility (GBIF.org) has become the standard open-access online database of occurrence records for all manner of biological organisms. It was initially a clearinghouse for museum records (such as herbarium specimens), but now includes iNaturalist observations (those that are rated ‘research’ grade), survey data, and a growing variety of taxonomic and checklist sources.

While GBIF’s expansion increases the overall value of the database, it also means we need to be more circumspect in how we use the data. When I first encountered GBIF decades ago, I used it as one of several sources for herbarium records. I searched for the species I was looking for, and received a list of museum specimens. Nowadays most, maybe all, online herbarium data is mirrored by GBIF, so I no longer need to chase down multiple websites to round up all the herbarium records I need.

However, I can no longer assume that a GBIF record represents a physical specimen. It could be: a human observation, with or without an associated image; documentation harvested from sequence data submitted to Genbank; an entry from a field survey, which sometimes contain records for both presences and absences. And of course, all of these records are subject to any number of issues: transcription errors, identification errors, georeferencing errors.

All of which to say, we need a way to filter the results of our GBIF query to ensure the data we receive is fit for purpose.

Getting GBIF data

Step one is actually getting the data from GBIF. You can do this from the website, but I now prefer to do this in my R scripts, using the rgbif package.

Finding taxon names

To get started, we need to match our name up with the GBIF taxonomic backbone:

library(rgbif)
conyza <- name_backbone("Conyza canadensis")
erigeron <- name_backbone("Erigeron canadensis")

This snippet searches the GBIF database for “Conyza canadensis” and “Erigeron canadensis” and returns the closest matches. The results can be a bit confusing to interpret. To make sense of it, we need to understand a few terms.

A usageKey is a unique number associated with every taxon in the database, at every level. This includes species, subspecies, genera etc, and importantly, it includes both accepted species and synonyms. Complicating things, GBIF taxonomy tables use the term usageKey, but in the individual observation records the term taxonKey is used instead. They both mean the same thing - you’ll use the usageKey you get from your name_backbone search as the taxonKey in the query you submit (see below).

The acceptedUsageKey (or acceptedTaxonKey) is the number associated with every accepted taxon. For taxa that are synonyms, their acceptedUsageKey is the usageKey of the accepted taxon they belong to.

To illustrate the distinctions, let’s look at the values for the examples above.

Key	Value
scientificName	Conyza canadensis (L.) Cronquist
usageKey	5404801
status	SYNONYM
acceptedUsageKey	3146791

Key	Value
scientificName	Erigeron canadensis L.
usageKey	3146791
status	ACCEPTED

In this case Conyza canadensis is a synonym of Erigeron canadensis. The name Conyza canadensis has it’s own usageKey: 5404801. Its acceptedUsageKey is 3146791, which is the usageKey for Erigeron canadensis. An additional wrinkle is that accepted taxa only have a usageKey, they don’t have an acceptedUsageKey. You can also use the status field to check if a taxon is accepted or a synonym.

Understanding this is important to make sure you get the records you’re after when you query the database. If you request data for taxonKey 5404801 (the usageKey for Conyza canadensis), you’ll get records with that name on them, but not records for Erigeron canadensis. On the other hand, if you search for taxonKey 3146791, you’ll get records for Erigeron canadensis, and also all records for any synonyms of that name, including Conyza canadensis.

In other words, searching for an accepted taxon will return results for that taxon including all its synonyms. Searching for a synonym will return results only for that synonym.

You can search for records by name, without using the usageKey. But it’s safer to look up and use the usageKey, to confirm that the name you asked for matches with something in the GBIF database.

Preparing a query

Once you have a list of one or more taxonKey values, you’re ready to request your data. For large record sets, and especially if you want to request multiple species at once, the rgbif function occ_download_queue is very convenient. Note that you need to have a (free) GBIF account in order to use this.

This is a three step process. You create your query with occ_download_prep, submit the query to GBIF with occ_download_queue, and once the query is done you download the results via occ_download_get.

Starting with occ_download_prep, a basic query only requires your account credentials and one or more taxonKeys:

myQuery <- occ_download_prep(
  pred_in("taxonKey", c(3146791, 3189859)),
  pred("hasCoordinate", TRUE),
  format = "DWCA",
  user = "YourUserName",
  pwd = "YourGBIFPassword",
  email = "your@email.address"
)

SECURITY NOTE You can provide your GBIF username and password in your script as I have done, but there are more secure ways to submit your credentials without listing them in your code. I have mine stored in my ~/.Renviron file, which looks like this:

GBIF_USER="my_user_name"
GBIF_PWD="my_password,"
GBIF_EMAIL="my@email.ca"

See the rgbif documentation for more options.

This will create a query for two taxa, as specified by the provided keys; it will filter the results to include only records with coordinates (i.e., hasCoordinate is TRUE); and the results will be in the Darwin Core Format (i.e., “DWCA”). We reviewed taxon keys above. If we’re mapping our records, and don’t have time or need to do any georeferencing ourselves, we can save time by limiting our results to records that already have coordinates.

The default format is “DWCA”, which includes a lot of columns in the results. You can choose “SIMPLE_CSV” instead, and this will give you a subset of commonly used fields. However, this limits your ability to filter records after download, so I recommend sticking with “DWCA”.

There are many other ways to filter records, documented in ?download_predicate_dsl. Depending on your focus, you might want to restrict the results by basisOfRecord, to select only “HumanObservation” or “PreservedSpecimen”; or by datasetID to select a specific project (i.e., iNaturalist). However, I’ve found that these terms are not applied consistently, so it may be better to download everything and filter after you’ve had a chance to inspect the tables yourself.

Submitting a query

With a query ready, you can now submit it to GBIF via:

out <- occ_download_queue(.list = list(myQuery))

We could have submitted the query directly, by using occ_download above instead of occ_download_prep. The latter method offers two advantages. First, we can submit multiple queries at once. e.g.,

out <- occ_download_queue(.list = list(queryA, queryB,
                                       queryC))

Second, GBIF allows you to submit up to three queries at a time. If you have more, you have to wait until one of the earlier queries is finished. occ_download_queue keeps track of this for you, submitting three requests, and sending additional requests as the first three finish.

Retrieving your query

occ_download_queue returns the details of your submission(s):

$`46c5a45e8f1f2e64fc9eda5318e74972`
<<gbif download>>
  Your download is being processed by GBIF:
  ...
  Check status with
  occ_download_wait('0025322-231120084113126')
  After it finishes, use
  d <- occ_download_get('0025322-231120084113126') %>%
    occ_download_import()
  to retrieve your download.
  ...

Usually it takes a few minutes to process your request. You can check its progress with occ_download_wait('...'), using the details provided. Once the query is done, you can download it via occ_download_get, and read it into R with occ_download_import, as shown.

NOTE: once your query is submitted by occ_download_queue, it will be processed remotely by GBIF. If something should happen in the meantime – your computer crashes, or you cancel the function call for any reason – the query will still be processed. However, should this happen you won’t have the downloadKey you need to retrieve the results. You can get this key by logging into your GBIF account on the website, navigating to the Downloads section of your profile, and clicking on either the DOI or the SHOW links for the download in question. This will take you to a page with the meta data for the query. It doesn’t include the actual downloadKey, but that value is present as the final part of the url, as shown here:

The GBIF download summary page, showing the downloadKey value in the URL

You can also click the Download button on the right side to download the file from your browser, if you prefer.

Cleaning GBIF data

Now that we have our records downloaded, we need to review and clean the data before analysis. Note that in the code below, I’m using a large download to demonstrate my investigations. I haven’t included this data in this tutorial, but you can try the code on your own data.

Filtering based on the data

basisOfRecord

With the data in hand, we can take a closer look at what kind of records they are, and where they came from. Starting with basisofRecord:

d1 <- occ_download_get(out[[1]], path = "./dl/") %>% 
  occ_download_import()
sort(table(d1$basisOfRecord), decreasing = TRUE)

  HUMAN_OBSERVATION  PRESERVED_SPECIMEN          OCCURRENCE 
            4224215              325278              112588 
        OBSERVATION   MATERIAL_CITATION     LIVING_SPECIMEN 
             104335               17703                3123 
    MATERIAL_SAMPLE MACHINE_OBSERVATION     FOSSIL_SPECIMEN 
               1698                 222                  10

In this case, I have nine different kinds of record. The definitions of some of these terms are listed in the Darwin Core Quick Reference Guide. HUMAN_OBSERVATION includes, among other things, iNaturalist records. PRESERVED_SPECIMEN includes mostly (and most) herbarium records. I usually want both of these groups.

OCCURRENCE and OBSERVATION aren’t well defined or consistently used, so require further examination to determine if we want to retain them.

MATERIAL_CITATION, MATERIAL_SAMPLE, and MACHINE_OBSERVATION are a little vague, inconsistently used, and also not used often, so I remove them.

LIVING_SPECIMEN and FOSSIL_SPECIMEN are self-explanatory, and usually not want I want for my work.

iNaturalist and Human Observations

Research-grade iNaturalist records are imported to GBIF every few weeks. We can extract these using the field datasetKey, with the value "50c9509d-22c7-4a22-a47d-8c48425ef4a7":

sum(d1$datasetKey == "50c9509d-22c7-4a22-a47d-8c48425ef4a7")

357863 # iNaturalist Records in my data

You might be tempted to filter on datasetName, since there is a dataset called “iNaturalist research-grade observations”. Unfortunately, this name isn’t used consistently, the name “iNaturalist observations” is also used for some records in the same dataset. In fact, the word iNaturalist appears in a variety of different dataset names:

table(d1$datasetName[grep("iNaturalist", d1$datasetName)])

"Flora of Russia" on iNaturalist: a trusted backlog 
                                                131 
                           iNaturalist observations 
                                                 25 
            iNaturalist research-grade observations 
                                             357838 
               iNaturalist XicotliData observations 
                                                  7

Which leaves us with the unwieldy datasetKey == "50c9509d-22c7-4a22-a47d-8c48425ef4a7" as the most reliable way to get the offical iNaturalist dataset.

All of the iNaturalist records are labelled as HUMAN_OBSERVATION:

table(d1$basisOfRecord[d1$datasetKey ==
                       "50c9509d-22c7-4a22-a47d-8c48425ef4a7"])

HUMAN_OBSERVATION 
           357863

But this is only a small fraction of the 4,224,215 HUMAN_OBSERVATION records in my query. In my data, there are over 1500 different HUMAN_OBSERVATION datasets. This includes a number of surveys, and in some cases these surveys record presences and absences, as recorded in the occurrenceStatus field:

table(d1$occurrenceStatus)

 ABSENT PRESENT 
  27896 4761276

Whether you want to include a heterogeneous collection of surveys in your data depends on the question your asking.

But if you are planning to do distribution modeling, you may not want to include ABSENT records in your training data. It will depend on the scale of your modeling, and the scale of the surveys that contributed their data to GBIF. Fine-scale local surveys may include a mix of PRESENT and ABSENT records within a small area (say a few hundred meters). If your modeling is at the scale of 30 second climate rasters (~1km^2), that’s a problem. A species can be absent in a 10m^2 quadrat, but present in the larger 1km^2 raster grid.

Here are a few examples of filters you might want to use:

herbarium <- d1[, d1$basisOfRecord == "PRESERVED_SPECIMEN"]
iNat <- d1[, d1$datasetKey ==
             "50c9509d-22c7-4a22-a47d-8c48425ef4a7"]
present <- d1[, d1$occurrenceStatus == "PRESENT"
              & d1$basisOfRecord == "HUMAN_OBSERVATION"]

Common location errors

The R package CoordinateCleaner provides some functions for dealing with common problems:

cc_cen : Identifies records close to the centroid of a country. Automated georeferencing will often use the centroid of a country as the coordinates for a specimen that doesn’t have any other location data. This could be 100s/1000s of kilometers away from the actual location.

cc_inst : Identifies records close to the location of museums. Automated georeferencing will often use the location of a museum as the location for specimens it contains, regardless of where the specimen was actually collected.

cc_sea : Identifies records that are in the ocean, which are clearly errors for terrestrial organisms. However, this could also be a consequence of relatively minor errors in record coordinates or the mapping of coastlines.

For a more thorough overview of these kinds of issues, see this post by John Waller on the GBIF data blog.

Identification Errors

Now that we’ve dealt with the more ‘mechanical’ sorts of errors we might find in our data, we can review our records for obvious or suspected identification errors. If you have a large dataset, especially if it includes many species, or species you are not familiar with, this can be a daunting task.

Two important botanical resources can help us with this, at least in North America: the Flora of North America and NatureServe.org. These provide a ‘sanity check’ for our GBIF records, as both websites host carefully curated data on plant distributions.

The Flora of North America includes taxonomic treatments of all plant species that occur outside of cultivation in Canada and the United States¹. The maps aren’t very detailed, showing us only the states or provinces each taxon is found in. But those maps are all based on actual specimens examined by a taxonomist with expertise on that plant. If New York appears on an FNA distribution map, it means there is at least one documented record of that species, confirmed by an expert, in that state. And conversely, if a state or province isn’t included on the map, it means that a thorough search of herbaria has failed to find a single record of the species.

NatureServe is similar, but the distribution maps are based on state/provincial/regional natural heritage programs. These are generally expert field botanists, rather than taxonomists. Their job is to document which plants grow in their jurisdiction. If a NatureServe distribution map includes Ontario, that means an expert field botanist has evidence (which could be a herbarium voucher, or a reliable observation) that the species occurs (or occurred) in Ontario. And again, if a state or province isn’t included on the map, then no such evidence has been found.

NatureServe and FNA distribution maps will usually match fairly closely. NatureServe is more responsive to new information, and those maps will periodically be updated. FNA isn’t yet complete (although it’s getting close), but for the taxa that it covers it provides a comprehensive review of their distribution at the time of publication (i.e., it doesn’t get updated).

Neither source is complete, and plants do move around. But, if you have records in your GBIF dataset from areas beyond the range documented in NatureServe or FNA, these are the ones I’d be most concerned about verifying. Use the map interface on the GBIF website to locate them, and look at the record details for clues to confirm or refute them. If they trace back to iNaturalist records, you may be able to confirm the identification yourself from the photos, or ask the observer for more details.

You won’t likely be able to confirm the identifications of all of your records, but you can often validate or exclude the outlying records that have the greatest potential to distort your analysis.

The FNA project is more properly the “Flora of North America north of Mexico”.↩

Extrapolation Detection (exDet) for SDMs

Tue, 19 Dec 2023 00:00:00 +0000

Identifying Non-Analogous Climate Conditions

A major concern when projecting species distribution models to new contexts (e.g., invaded ranges, or future climates) is establishing whether (and where) the environments in the new context are analogous to those in the training region (see my notes). A common approach is to compare each variable in isolation, and construct a “Multivariate Environmental Similarity Surface”, or MESS (Elith et al., 2010). Areas in the new context that are outside the range of any variable from the training context will have values below 0, with lower values indicating greater departures.

However, the MESS approach does not account for correlations among variables. Perhaps the native range of a species includes areas with hot, wet, summers, but not hot dry regions. The MESS analysis would consider conditions ‘analogous’ as long as they were within the temperature range and precipitation range of the training region. This could result in novel environmental combinations being incorrectly identified as analogous climate:

Visualization of MESS analysis. The green area indicates the reference climate conditions, and the red square shows the climate space MESS will identify as ‘analog’.

Note particularly the upper left corner. This is well outside the range of conditions in the training region (the green area), but by considering each variable separately, MESS will treat this region as if it were analogous.

Extrapolation Detection

Despite the name, MESS is only just barely multivariate: it evaluates multiple variables, but each one is considered in isolation. Extrapolation Detection (Mesgaran et al., 2014), or exDet, provides a more sophisticated approach, based on the Mahalanobis distance. This is a common multivariate distance measure, designed to account for covariation among variables. In this context, our analysis will include the following steps:

Calculate the Mahalanobis distance for every cell in the training region
Scale the distances by dividing by the maximum distance, such that they range between 0 and 1
Calculate the Mahalanobis distance for every cell in the novel region/time period, again dividing by the maximum distance from the training (not the new) region.

This will produce a raster map with cell values ranging from 0 to infinity, which serves as the Novelty Index. Mesgaran et al. (2014) refer to it as NT2, to distinguish it from the MESS index which they call NT1. Cells with NT2 values less than 1 are within the range of conditions present in the training range; cells with NT2 > 1 are outside that range, with higher values indicating greater departure:

Visualization of exDet analysis. The green area indicates the reference climate conditions, and the red ellipse shows the climate space exDet will identify as ‘analog’.

The result more closely captures the (co)variation in conditions present in the training area.

exDet in R

There is an R package that calculates exDet: ntbox. However, it is still using the raster-based workflow, which is now obsolete. Luckily, calculating Mahalanobis distances only requires a few lines of code, so we can do it ourselves.

For this example, I’ll use the Lythrum salicaria data from my previous tutorials, but updated to use the new terra workflow.

Data

I use the same GBIF records previously downloaded in my Ecospat with Terra tutorial.

library(terra)
library(geodata)
library(rgbif) ## not actually necessary, used to download
               ## the occurrence data

library(usdm) ## for vif collinearity test below

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

## convert to a spatial vector:
lsOccs <- vect(lsOccs, geom = c("decimalLongitude",
                                "decimalLatitude"),
               crs = "+proj=longlat +datum=WGS84")

wrld <- world(path = "../data/maps/")

wclim <- worldclim_global(var = "bio", res = 10,
                          path = "../data/")

## North America basemap:
nAmCountries <- c("CAN", "MEX", "USA")
nAm <- gadm(nAmCountries, level = 0,
            path = "../data/maps/",
            resolution = 2)

## Eurasia basemap:
eurasiaCountries <- country_codes("Asia|Europe")

## remove missing countries:
eurasiaCountries <-
  eurasiaCountries[!eurasiaCountries$ISO3 %in%
                    c("HKG", "MAC", "XNC"),] 
eur <- gadm(eurasiaCountries$ISO3, level = 0,
            path = "../data/maps/",
            resolution = 2)

## subset occurrences
lsNA <- lsOccs[nAm]
lsEA <- lsOccs[eur]

Training Region

There are a number of considerations when deciding on the region to use to train our model. For this example, I’ll just use a 500km buffer around occurrences in the native range.

Now we have all the data we need for the analysis:

train <- buffer(lsEA, width = 500000)
train <- aggregate(train)

plot(wrld, mar = c(1.5, 1.5, 0.5, 0.5))
plot(train, col = '#00FF0080', add = TRUE)
plot(lsEA, add = TRUE, cex = 0.5)
plot(lsNA, add = TRUE, cex = 0.5, col = 'blue')

Figure 1: Lythrum salicaria distribution. Green shading shows the training region, black points are occurrences in the native range, and blue points are occurrences in the invaded range.

Note that the training region extends into the ocean, but the underlying worldclim data doesn’t, so we don’t need to worry about cleaning this up.

Mahalanobis Distance

Now we can calculate the NT2 index. We start by clipping the climate data to our training and projection regions, and then selecting a set of variables that aren’t highly collinear (in the training region):

trainWC <- mask(wclim, train)

## 
|---------|---------|---------|---------|
=========================================

trainVal <- values(trainWC)

naWC <- mask(wclim, nAm)

## 
|---------|---------|---------|---------|
=========================================

naVal <- values(naWC)

## screen out collinear variables:
vifSel <- vifstep(data.frame(trainVal), th = 5,
                  size = 20000)

VARS <- vifSel@results$Variables

VARS contains a list of non-collinear variables. For a real analysis you probably should be more deliberate in selecting which variables to retain, in order to consider both their biological interest and statistical properties.

trainMeans <- colMeans(trainVal[, VARS], na.rm = TRUE)
trainVar <- var(trainVal[, VARS], na.rm = TRUE)
trainMah <- mahalanobis(trainVal[, VARS], trainMeans,
                        trainVar, na.rm = TRUE)

trainMah now contains the Mahalanobis distance from each cell in the training region to the climate centroid of the training region. ‘0’ means the point is in the center of the climate space. By the nature of the Mahalanobis distance, this value accounts for covariation among our climate variables.

As a quick check, let’s take a look at the Mahalanobis distances for the training region:

hist(trainMah, main = "Mahalanobis Distances in Eurasia",
     xlab = "Distance")

That’s a bit odd. We have a small number of outliers with very high distances. I think this is likely a consequence of idiosyncracies in our climate rasters: an artifact of the analysis rather than a biologically meaningful value. To clean this up, I’ll identify values above the 95th percentile as outliers and remove them. Then we can proceed with calculating the distances for the invaded range.

thresh <- quantile(trainMah, probs = 0.95, na.rm = TRUE)
trainMah[which(trainMah > thresh)] <- NA
hist(trainMah, main = "Trimmed Mahalanobis Distances in Eurasia",
     xlab = "Distance")

That looks better. Moving on, we can now calculate NT2, which is the Mahalanobis distance divided by the maximum Mahalanobis distance in the training range:

maxMah <- max(trainMah, na.rm = TRUE)
trainMah <- trainMah/maxMah
trainMahR <- trainWC[[1]]
values(trainMahR) <- trainMah

naMah <- mahalanobis(naVal[, VARS], 
                     trainMeans, trainVar, na.rm = TRUE)
naMah <- naMah/maxMah

## Create a new raster map for North America:
naMahR <- naWC[[1]]

## set the values to our Mahalanobis distances:
values(naMahR) <- naMah

Now we have a raster map with the NT2 index values for each cell in the training and invaded ranges. Values between 0 and 1 are within the training range. We can visualize that directly:

COLS <- c("green", "lightgreen", "aquamarine", "blue", 
           "orange", "brown", "red")

BREAKS = c(0, 0.25, 0.5, 1, 2, 4, 8, 16)

plot(trainMahR, breaks = BREAKS, col = COLS,
     mar = c(2, 2, 1, 5))
plot(naMahR, breaks = BREAKS, col = COLS, add = TRUE, legend
     = FALSE)
plot(wrld, border = 'lightgrey', add = TRUE, lwd = 0.5)

This completes main part of the exDet analysis. We can see the training range all falls within the range 0-1, which is by design. Transferring that distribution to North America, we can see most of Canada and the north central US falls within the same range. The high arctic, the SE US and western US are all outside of the training range, with Mexico even more novel. Projections into these areas should be treated with caution, or avoided entirely.

Most Influential Covariate, MIC

While the map above is our primary interest in exDet analysis, we may want to know which variables are most influential in creating novel environments. To do this, we need to calculate the distance map repeatedly, removing one variable at a time. For each cell, we can then identify the variable that contributes the most to its NT2 index, by calculating the difference between the distance for all variables, and the distance with that variable removed.

names(naMahR) <- "all"

## Remove variables one at a time and recalculate distances:

for(V in VARS){
  SEL <- VARS[VARS != V]

  tmpMeans <- colMeans(trainVal[, SEL], na.rm = TRUE)
  tmpVar <- var(trainVal[, SEL], na.rm = TRUE)
  tmpMah <- mahalanobis(trainVal[, SEL], tmpMeans,
                        tmpVar, na.rm = TRUE)

  thresh <- quantile(tmpMah, probs = 0.95, na.rm = TRUE)
  tmpMah[which(tmpMah > thresh)] <- NA
  tmpMax <- max(tmpMah, na.rm = TRUE)

  resMah <- mahalanobis(naVal[, SEL], 
                     tmpMeans, tmpVar, na.rm = TRUE)
  resMah <- resMah/tmpMax
  resMahR <- naWC[[1]]
  
  ## calculate difference from full distance:
  values(resMahR) <- values(naMahR$all) - resMah

  ## add a layer to our NA raster:
  naMahR[[V]] <- resMahR
}

## Find the maximum difference for each cell:
naMaxMah <- max(naMahR[[-1]])
naTest <- naMahR[[-1]] == naMaxMah

## Convert TRUE/FALSE to category numbers:

for(L in seq_along(names(naTest))){
  values(naTest[[L]]) <- values(naTest[[L]]) * L
}

## Collapse layers into a single raster:
naCat <- max(naTest)

## convert to a factor
levs <- data.frame(vals = seq_along(VARS),
                   ## clean up variable names:
                   levels = gsub("wc2.1_10m_", "", VARS))
levels(naCat) <- levs

Now we can visualize which individual variables are contributing most to the NT2 index for each cell:

plot(naCat, xlim = c(-180, -40), ylim = c(10, 90),
     mar = c(2, 2, 4, 5),
     main = "Most Influential Covariate")
plot(wrld, border = 'lightgrey', lwd = 0.5, add = TRUE)

Whether or not that information is useful to you will depend on your research question, of course.

References

Elith, J., M. Kearney, and S. Phillips. 2010. The art of modelling range-shifting species. Methods in Ecology and Evolution 1: 330–342.

Mesgaran, M. B., R. D. Cousens, and B. L. Webber. 2014. Here be dragons: A tool for quantifying novelty due to covariate range and correlation change when projecting species distribution models J. Franklin [ed.], Diversity and Distributions 20: 1147–1159.

Niche Quantification with Ecospat and Terra

Fri, 28 Jul 2023 00:00:00 +0000

Introduction

This is an update of my previous ecospat tutorial. Spatial analysis in R is shifting to terra and sf as the primary packages, so I’ve translated my old, raster-based tutorial to the new workflow. I also took this opportunity to clean up and extend the original tutorial.

See the RSpatial tutorial for a more detailed introduction/overview of using terra for GIS/spatial analysis.

Note this analysis depends on the ecospat package, and as of 2023-07-28 ecospat doesn’t support the spatial objects produced by terra. There are a couple of work-arounds in the code below to account for this. I’ll update the code once ecospat is fully compatible with terra

UPDATE 2024-05-10 ecospat has now been updated to use the terra package for spatial analysis. The code here has been updated to reflect this. Thank you to Lin Lin at Yunan University for bringing this to my attention!

The ecospat package (Cola et al. 2017) provides code to quantify and compare the environmental and geographic niche of two species, or of the same species in different contexts (e.g., in its native and invaded ranges). The included vignette explains how to do such analyses.

However, the vignette assumes you already have a matrix of occurrence records, along with the climate data for each of those records. In our work, we typically have to construct those matrices from observation data (herbarium records, iNaturalist observations, etc) and climate rasters (e.g. Fick and Hijmans 2017). This short tutorial will walk through the steps necessary to do this.

Required Packages

In addition to ecospat, we’ll use terra (Hijmans 2023) to download WorldClim (Fick and Hijmans 2017) rasters, and manipulate the spatial data; rgbif (Chamberlain et al. 2021) to download GBIF records, geodata (Hijmans et al. 2023) to get a world basemap for plots, and ade4 (Thioulouse et al. 2018) to perform the principal components analysis of the climate data.

library(ecospat)
library(terra)
library(rgbif)
library(geodata)
library(ade4)

Getting Data

GBIF Occurrence Data

We’ll start by sourcing our data. For observations, let’s take a look at Purple Loosestrife, a wetland species that is native to Europe, and invasive in North America. For actual research work, I normally download the files directly from GBIF, and examine them carefully to check for errors or missing data. For this demo we’ll use the rgbif package to download the data directly into R, and we’ll assume there are no problems with the data that need to be corrected.

lsGBIF <- occ_search(scientificName = "Lythrum salicaria",
                    limit = 10000,
                    basisOfRecord = "Preserved_Specimen",
                    hasCoordinate = TRUE,
                    fields = c("decimalLatitude",
                               "decimalLongitude", "year",
                               "country", "countryCode"))

save(lsGBIF, file = "../data/2021-07-29-ls-gbif-recs.Rda")

This returned an object with 7969 records. I saved that locally, so that I’m not making GBIF search their database everytime I work on this demo.

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

lsGBIF$data is the table with the actual records in it. That’s what we’ll be working with. The other components of lsGBIF are metadata related to the original GBIF search. That’s useful to have, but not needed for the rest of this example.

Next, we tell R which columns are the coordinates, which allows us to map the observations. This also converts our observation matrix to a SpatVector object.

The crs argument here tells R that our points are in lat/lon (unprojected) coordinates. If all of your data use lat/lon, you don’t need to specify this, but it’s important if you need to reproject your data, or combine data with different projections.

lsOccs <- vect(lsOccs, geom = c("decimalLongitude",
                                "decimalLatitude"),
               crs = "+proj=longlat +datum=WGS84")

Basemap

We’ll also need a world map to use in our plots. the world function from the geodata package will download one for us. The first time you call this function in a directory, it downloads the data from the internet, and saves it locally according to the path argument. Subsequent calls will load your local copy of the data, to speed things up.

In this case I’ve set the path argument to store the downloaded files in a location that is convenient for me. You can set this to anything you like, or leave it out to use the current R working directory.

wrld <- world(path = "../data/maps/")

This is a relatively low resolution map of the continents. For higher resolution maps, see the gadm function.

Now we can plot our data. Note that we set the margins (via mar) inside the plot call, not via the par function used in most base R plots.

plot(wrld, border = "gray80", mar = c(0, 0, 0, 0))
points(lsOccs, col = 2, cex = 0.3)

Climate Data

To get our climate data, we can use geodata’s worldclim_* functions. I’m using the coarsest resolution (10 minutes) to speed things up for this demonstration. The path argument works the same was as for world, storing a local copy of the files. In this instance we’ll use worldclim_global to get the bioclim variables for the world all at once:

wclim <- worldclim_global(var = "bio", res = 10,
                          path = "../data/")

We can take a look at one layer:

plot(wclim$wc2.1_10m_bio_1, main = "bio1",
     mar = c(0.1, 0.1, 2, 4), legend = TRUE,
     axes = FALSE)
par(mar = c(0.1, 0.1, 2, 4))
box()

Sampling Bias

One of the challenges we deal with with herbarium data is that observations tend to be clustered together in non-random ways. Sites near universities, museums, or in well-used field stations will have more records than more remote locations. We can’t completely account for this bias, but we can reduce it by spatial thinning. This is the process of randomly selecting a small number (often just one) of records for each grid cell in our analysis. The result is that those highly-sampled locations will be represented by a single record, meaning they will have a less exaggerated influence on the results.

We will need the full set of records below, so I’ll make a copy of the un-thinned data named lsOccsAll.

lsOccsAll <- lsOccs ## keep this for later

## select one record per cell:
lsOccs <- spatSample(lsOccs, size = 1, strata = wclim)

Linking Climate Data to Species Observations

Next, we need to extract the environmental values from the climate rasters for each of our observation records:

lsOccs <- cbind(lsOccs, extract(wclim, lsOccs))

In the process of extracting wclim values for our observations, we usually end up with a few missing values. This is a consequence of mismatches between the observation coordinates and the climate rasters. In some cases, the observations are placed off the coast in the ocean, or in another area where there is no climate data available. We need to exclude these missing values from our analysis.

lsOccs <- lsOccs[complete.cases(data.frame(lsOccs)), ]

Splitting Records into Native and Introduced Ranges

At this point, all the data we need for the Niche Quantification analysis is in lsOccs and wclim. We need to split this data into native and invasive regions for our comparison. We’ll restrict ourselves to the northern hemisphere, and consider all records from Eurasia as native, and all records from North America as invasive.

We can use GADM to download maps of the continents, and use this to select subsets of the datasets. I will select the country-level borders (level = 0) and low resolution (resolution = 2). For ‘real’ analyses you should use high resolution (resolution = 1).

To speed up analyses further down, I’m just going to download Canada, USA, and Mexico for my “North America” map.

nAmCountries <- c("CAN", "MEX", "USA")
nAm <- gadm(nAmCountries, level = 0,
            path = "../data/maps/",
            resolution = 2)

## select Europe OR Asia:
eurasiaCountries <- country_codes("Asia|Europe")
eur <- gadm(eurasiaCountries$ISO3, level = 0,
            path = "../data/maps/",
            resolution = 2)

lsNA <- lsOccs[nAm]
lsEA <- lsOccs[eur]

plot(wrld, axes = FALSE, xlim = c(-140, 150),
     ylim = c(10, 80), mar = c(0, 0, 0, 0))
points(lsNA, col = 'red', cex = 0.5)
points(lsEA, col = 'darkgreen', cex = 0.5)
par(mar = c(0,0,0,0))
box()

Note that this code will generate some warnings, “this file does not exist”. There are a few countries listed in the country_codes database that don’t have corresonding maps in the GADM repository (e.g. Hong Kong). We’ll ignore this for now.

For the Niche Quantification, we need to have a matrix with the background environment present in the native and invasive ranges, as well as the complete global environmental including the combined extent of the native and introduced environments. After cropping the data, we use values to convert the raster to a dataframe.

## Crop Climate Layers:
naEnvR <- mask(wclim, nAm)

## |---------|---------|---------|---------|=========================================

eaEnvR <- mask(wclim, eur)

## |---------|---------|---------|---------|=========================================

globalEnvR <- mask(wclim, rbind(nAm, eur))

## |---------|---------|---------|---------|=========================================

## Extract values to matrix:
naEnvM <- values(naEnvR)
eaEnvM <- values(eaEnvR)
globalEnvM <- values(globalEnvR)

## Clean out missing values:
naEnvM <- naEnvM[complete.cases(naEnvM), ]
eaEnvM <- eaEnvM[complete.cases(eaEnvM), ]
globalEnvM <- globalEnvM[complete.cases(globalEnvM), ]

Note that for the geographic projection below, it is essential that the globalEnvM be constructed directly from the globalEnvR rasters. If you try to combine naEnvM and eaEnvM to make globalEnvM it will end up scrambling the values in the geographic projection. If you don’t do a geographic projection it doesn’t matter.

Niche Quantification

PCA

The Niche Quantification analysis starts with a Principal Components Analysis of the environmental data. The actual ordination uses the global data, with the observation records and the native and invasive background environment treated as supplemental rows.

pca.clim <- dudi.pca(globalEnvM, center = TRUE,
                    scale = TRUE, scannf = FALSE, nf = 2)
global.scores <- pca.clim$li

nativeLS.scores <-
  suprow(pca.clim,
         data.frame(lsEA)[, colnames(globalEnvM)])$li   
invasiveLS.scores <-
  suprow(pca.clim,
         data.frame(lsNA)[, colnames(globalEnvM)])$li

nativeEnv.scores <- suprow(pca.clim, naEnvM)$li
invasiveEnv.scores <- suprow(pca.clim, eaEnvM)$li

Let’s break that down. dudi.pca does a PCA analysis on globalEnvM, which is a matrix of all the environmental variables over the entire study area. We use that to create a two-dimensional summary of the total environmental variability.

Next, we map our observation data (lsEA and lsNA) into that 2-dimensional ordination, using the suprow function. lsEA and lsNA are SpatialPointsDataFrame objects. Sometimes you can treat them as if they were data.frames, but other times you need to explicity convert them. This is one of those times, hence I’ve wrapped them in data.frame().

Recall that lsEA and lsNA have more columns than the environmental matrix: they also include year, countryCode, country. We only want to include the environmental variables when you project the observations into the ordination. To make sure that we use the same variables as in the original ordination of globalEnvM, in the same order, I select the columns explicitly to match that object:

data.frame(lsEA)[, colnames(globalEnvM)]

The output of dudi.pca and suprow includes a lot of information that we aren’t using here. We only need the li element, so I’ve selected that from each of the function outputs.

Occurrence Density Grids

Finally we’re ready to do the Niche Quantification/Comparisons. We’ll use the PCA scores for the global environment, the native and invasive environments, and the native and invasive occurrence records.

nativeGrid <- ecospat.grid.clim.dyn(global.scores,
                                   nativeEnv.scores,
                                   nativeLS.scores)

invasiveGrid <- ecospat.grid.clim.dyn(global.scores,
                                   invasiveEnv.scores, 
                                   invasiveLS.scores)

ecospat.plot.niche.dyn(nativeGrid, invasiveGrid,
                       quant = 0.05)

The resulting plot shows us the environmental conditions present in Eurasia (inside the green line) and North America (inside the red line). The green area represents environments occupied by Lythrum salicaria in Eurasia, but not in North America, the red area shows environments occupied in North America and not Eurasia, and the blue area shows environments occupied in both ranges. We can also see that there are a few areas in Eurasia with environments not present in North America, and vice versa. However, for the most part, Lythrum salicara doesn’t occur in this environments.

To get the index values we use ecospat.niche.dyn.index:

indexVals <- ecospat.niche.dyn.index(nativeGrid,
                                     invasiveGrid) 
indexVals$dynamic.index.w

##  expansion  stability  unfilling 
## 0.00598794 0.99401206 0.02349507

Projecting Climate Space to Geographic Space

We can take these results and project them onto a geographic map. This will show us which areas of the native and invasive range of Lythrum salicaria fall into each of the three categories.

Note that the function ecospat.niche.dynIndexProjGeo doesn’t yet handle the SpatRaster objects created by the terra package, so we have to convert it to a raster’s stack when we call it. For convenience and consistency with the rest of the code, I convert the results back to a SpatRaster.

geoProj <-
  ecospat.niche.dynIndexProjGeo(nativeGrid,
                                invasiveGrid,
                                env = globalEnvR)

plot(geoProj, legend = FALSE, 
     col = c("grey", "green", "red", "blue"),
     ylim = c(20, 80), axes = FALSE, mar = c(0, 0, 0, 0))
points(lsEA, col = 'darkgreen',
       cex = 0.25, pch = 23, bg = "white")
points(lsNA, col = 'darkred',
       cex = 0.25, pch = 23, bg = "white")
par(mar = c(0,0,0,0))
box()

Geographic Comparisons

You can also apply this analysis to geographic locations, instead of environmental conditions. This won’t make much sense for native vs invaded range comparisons, but it could be useful for comparing different species within the same area.

To demonstrate, let’s compare the distribution of Lythrum salicaria in North America before and after 1950. In this case, I need to go back to the original occurrence data, since I need the thin the records before and after 1950 separately. Otherwise, we won’t accurately capture the locations where Lythrum salicaria was collected in both time periods.

We use geographic coordinates here, so no need for a PCA. We do need to generate the ‘background’ coordinates. I’ll use expand.grid to create the locations for this. I’ve broken up the NA extent into 500 x 500 grids.

lsNA_All <- lsOccsAll[nAm]

lsNAearly <- subset(lsNA_All, lsNA$year <= 1950)
## thin records:
lsNAearly <- spatSample(lsNAearly, size = 1, strata = wclim)

lsNAlate <- subset(lsNA_All, lsNA$year > 1950)
## thin records:
lsNAlate <- spatSample(lsNAlate, size = 1, strata = wclim)

geoGrid <- expand.grid(longitude =
                        seq(-160, -40, length.out = 500),
                      latitude =
                        seq(20, 90, length.out = 500))

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     crds(lsNAearly))

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    crds(lsNAlate))

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)
plot(nAm, add = TRUE)

This looks pretty good. However, ecospat uses a kernel density formula to model the occurence distributions. As a consequence, it projects out into the ocean, which isn’t very realistic. To correct this, we need to mask the analysis to the continental land mass. This requires we have a vector map of the desired area.

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     crds(lsNAearly),
                                     geomask = nAm)

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    crds(lsNAlate),
                                    geomask = nAm)

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)

That gives more reasonable results. The blue area is the range occupied by Lythrum salicara prior to 1950, and the red area is the range it expanded into after that year. The small green areas are regions where it hasn’t been collected since 1950.

Note that this visualization is weighted by the density of points, so there might be a few pre-1950 records in the red area, or a few post-1950 records in the red area. That’s expected.

Summary

This is a fairly quick overview of this workflow. You’ll almost certainly want to consider thinning your observations, among other data cleaning procedures. I’ve also set the study extent very crudely. That might be appropriate for very large scale (global) studies. But you’ll usually want to think a bit more carefully about how you set your extent. The way you process your data will also differ depending on your context.

References

Chamberlain, Scott, Vijay Barve, Dan Mcglinn, Damiano Oldoni, Peter Desmet, Laurens Geffert, and Karthik Ram. 2021. “Rgbif: Interface to the Global Biodiversity Information Facility API.” Manual. https://CRAN.R-project.org/package=rgbif.

Cola, Valeria Di, Olivier Broennimann, Blaise Petitpierre, Frank T. Breiner, Manuela D’Amen, Christophe Randin, Robin Engler, et al. 2017. “Ecospat: An R Package to Support Spatial Analyses and Modeling of Species Niches and Distributions.” Ecography 40 (6): 774–87. https://doi.org/10.1111/ecog.02671.

Fick, Stephen E., and Robert J. Hijmans. 2017. “WorldClim 2: New 1-Km Spatial Resolution Climate Surfaces for Global Land Areas.” International Journal of Climatology 37 (12): 4302–15. https://doi.org/10.1002/joc.5086.

Hijmans, Robert J. 2023. “Terra: Spatial Data Analysis.” Manual. https://CRAN.R-project.org/package=terra.

Hijmans, Robert J., Márcia Barbosa, Aniruddha Ghosh, and Alex Mandel. 2023. “Geodata: Download Geographic Data.” Manual. https://CRAN.R-project.org/package=geodata.

Thioulouse, Jean, Stéphane Dray, Anne-Béatrice Dufour, Aurélie Siberchicot, Thibaut Jombart, and Sandrine Pavoine. 2018. Multivariate Analysis of Ecological Data with Ade4. New York, NY: Springer New York. https://doi.org/10.1007/978-1-4939-8850-1.

Managing Absolute Paths in Reproducible Analyses

Tue, 14 Feb 2023 00:00:00 +0000

In a previous post on reproducible analysis, I explained the importance of using relative paths in your scripts, and organizing your data in a single directory, in order to maintain portability. You want to be able to pack up your analysis in a zip file, or upload it as a single directory to GitHub or Dropbox, in order to share it with colleagues, or transfer it to a new computer.

This is best practice, but you may run into problems achieving this. One challenge is dealing with large data sets that you will use in multiple analyses. For example, we use WorldClim for a lot of our distribution work. Each copy of the global 30s dataset fills nearly 10GB (compressed). That would quickly fill my laptop harddrive if I stored a separate copy for each project.

I have created a separate directory to store such datasets, so that I only need to maintain a single copy on my computer. All analyses that use the WorldClim data will look for it in ~/data/worldclim/ on my laptop. On my workstation, I store the same data in ~/data/enm/worldclim. I could simplify this by using the same absolute path on both machines, but that wouldn’t help anyone else trying to use my script on their own machine.

The approach I’ve come up with for managing this requires a few lines of code to set the paths appropriately:

switch(system2("hostname", stdout = TRUE),
       LAPTOP.HOSTNAME =  ## my laptop:
         {worldClimPath <- "~/data/worldclim/"}, 
       WORKSTATION.HOSTNAME =   ## my workstation:
         {worldClimPath <- "~/data/enm/worldclim/"},
       {worldClimPath <- "dl/worldclim/"}) ## default

First, I check for the machine name with the function system2("hostname", stdout = TRUE). This calls the hostname command on the underlying operating system (which should work on Linux, Windows, and Mac). hostname returns the network hostname for your computer, which should be unique (at least within your organization). The switch function then compares this value to the names for my different machines, which I’ve already looked up. I can then use that information to set the correct path for my shared data.

In the case that I’m not on either of my machines, I set a default, relative path. That will allow other people to use my script without using my hard-coded paths.

In the case of WorldClim, the geodata package provides a convenient way to download the rasters:

library(geodata)
bio <- worldclim_global("bio", res = 10,
                        path = worldClimPath)

The function worldclim_global will check the path argument. If it finds the requested data there, it loads the local copy. If the data isn’t there, it downloads a new copy from the internet, and stores it there.

This makes for a convenient solution: running on my computers, all of my analyses will use the same shared data, and I won’t have to wait for downloads or exhaust my hard drives. But I can also share my code as-is with collaborators, and it will just work, without their having to change any paths.

Simple Maps in R with Terra

Mon, 13 Feb 2023 00:00:00 +0000

Reference

This is an update of my previous mapping tutorial. Spatial analysis in R is shifting to terra and sf as the primary packages, so I’ve translated my old, raster-based tutorial to the new workflow. See the RSpatial tutorial for a more detailed introduction/overview of using terra for GIS/spatial analysis.

The following tutorial walks through some common plotting tasks I use for distribution models.

Basemaps

The geodata package provides several convenient functions for downloading raster and vector maps for use as basemaps and spatial analysis. The first time you use these functions, they will download the requested maps from the internet. It will save the data in your working directory, or in a location specified with the path argument. The next time you request the same data, if it finds them in the local directory (or the specified path), they will be loaded from there, saving the time and bandwith necessary to download them.

library(geodata)

## Loading required package: terra

## terra 1.7.29

us <- gadm(country = "USA", level = 1, resolution = 2,
             path = "../data/maps/")
canada <- gadm(country = "CAN", level = 1, resolution = 2,
               path = "../data/maps")
mexico <- gadm(country = "MX", level = 1, resolution = 2,
               path = "../data/maps")

Countries are specified by their ISO code, which you can find by calling the function country_codes(). The by default, country_codes() returns a table of countries and the various ISO codes they have, as well as the continents they are in. The query argument lets you filter this table on any value. For example, you can get a table of North American countries with:

country_codes("North America")

or, just their ISO codes with:

country_codes("North America")$ISO3

The other arguments allow you to select, national, subnational, or subprovincial borders (level 1-3), the resolution (high: 1, low: 2), and the path to the directory where you want the maps stored.

These maps can be plotted directly with the plot command. If you want to combine them, use the add = TRUE argument to the second plot call:

plot(us, lwd = 2)
plot(canada, add = TRUE, col = 'red')
plot(mexico, add = TRUE, border = "green")

col sets the fill colour, border sets the outline color.

You can also request multiple countries as a single set of polygons, by passing a character vector to the country argument.

NorthAmerica <- gadm(country = country_codes("North America")$ISO3,
                     level = 0, resolution = 2,
                     path = "../data/maps/")

plot(NorthAmerica, xlim = c(-180, -50))

Note that xlim and ylim work as you would expect for plotting.

If you want to combine multiple polygons into a single object, use rbind:

CanUS <- rbind(us, canada)
plot(CanUS, xlim = c(-180, -50))

These maps are ‘unprojected’, meaning they are plotted in latitude/longitude degrees. That makes it easy to set the plot boundaries:

plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60))
plot(NorthAmerica, lwd = 2, add = TRUE)

NB: The size of your plot canvas is fixed, but a map can’t stretch. The x and y dimensions have to maintain the same aspect. That means zooming in one dimension (i.e. latitude only) won’t necessarily change the zoom of your map, if the other dimension fills the canvas. You’ll have to play around with the plot size, and both x and y dimensions together, to tweak your zoom.

It’s handy to have a shapefile of the Great Lakes, for making prettier maps. I created this one in QGIS and use it for plotting:

greatlakes <- vect("../data/maps/greatlakes.shp")

Adding Data

Vectors

You can add points to the plot like a regular scatter plot:

library(scales)  ## for the alpha function below

## 
## Attaching package: 'scales'

## The following object is masked from 'package:terra':
## 
##     rescale

gbif <- read.table("../data/trich-gbif.csv")
## Set the line color to gray to focus on the data points:
plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
     border = "gray")
points(gbif$X, gbif$Y, pch = 16,
       col = alpha("green", 0.2))

You can also convert your points to a spatial vector object, in which case R will know which columns to use for plotting. This is also necessary before we can project our data (see below).

gbif <- vect(gbif, geom = c("X", "Y"))
## plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
##      border = "gray")
## points(gbif, pch = 16, col = alpha("green", 0.2))

Subsetting vectors

You often need to select polygons that contain points, or select only those points that occur within a particular polygon. You can do this with R’s [ subsetting syntax. For instance, <polygons>[<points>, ] will select polygons that contain one or more points:

plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
      border = "gray")
## Select states and provinces where Trichophorum is
## present: 
plot(CanUS[gbif, ], col = 'red', add = TRUE)
points(gbif, pch = 21, col = 'white', bg = 'black')

Conversely, <points>[<polygons>, ] will select points that are within polygons. In this example, we’ll use the fact that our CanUS SpatVector object has a column named NAME_1 that holds the name of the state or province of each polygon:

plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
      border = "gray")
plot(CanUS[gbif, ], col = 'red', add = TRUE)
## Select only points in NY state:
points(gbif[CanUS[CanUS$NAME_1 == "New York", ], ],
       pch = 21, col = 'white', bg = 'black')

Rasters

Similarly, you can plot rasters with plot:

trichPreds <- rast("../data/trichPreds.grd")
plot(trichPreds, xlim = c(-100, -50), ylim = c(30, 60))
plot(CanUS, border = "gray", lwd = 0.5, add = TRUE)

Cells with NA values are transparent. In this case, a species distribution model, low values are displayed in gray. This may be useful for visualizing the extent of the model. However, it looks a bit odd, and makes it hard to see limits of the high-suitability areas. You can tweak this by playing with the color ramp, but it’s also handy to ‘turn off’ the low values entirely (for visualization, not for analysis!!)

trichPredsTrim <- trichPreds
trichPredsTrim[trichPredsTrim <
               quantile(values(trichPreds),
                        probs = 0.75, na.rm = TRUE)] <- NA
plot(trichPredsTrim, xlim = c(-100, -50), ylim = c(30, 60))
plot(CanUS, border = "grey", add = TRUE)

The test I used here, trichPredsTrim < quantile(getValues(trichPreds), probs = 0.75, na.rm = TRUE) identifies all cells in the lower 75% of the suitability scores, which I then set to NA to make them invisible. I decided on 75% after experimenting with different values. In this case, 75% drops most of the grey background (the very lowest values), without eating into the areas that the prediction indicates are suitable.

You could also use an absolute value here, but then you’d need to know the actual distribution of the suitability scores. quantile is easier to tweak.

Alternatively, you can assign colours to the prediction raster based on the value of each pixel, after splitting them into categories:

suitability <- extract(trichPreds, gbif, ID = FALSE)[, 1]

predCat <- (trichPreds > quantile(suitability, 0.25)) + 
  (trichPreds > quantile(suitability, 0.15)) +
  (trichPreds > quantile(suitability, 0.05)) +
  (trichPreds > quantile(suitability, 0.025))

predCols <- c("white", "grey95", "yellow3", "orange", "red") 

plot(predCat, xlim = c(-100, -50), colNA = 'lightblue',
     col = predCols, ylim = c(30, 60), legend = FALSE)
plot(CanUS, border = "grey", add = TRUE)

I find using a small number of categories makes it easier to read the suitability map than trying to interpret the colour gradients usually used.

Projections

Lat/Lon maps look a bit square; we’re more used to seeing maps projected. A common projection for Canada is Lambert Conformal Conic. We can transform our data to this projection to make nicer maps:

## define the projection
canlam <- "+proj=lcc +lat_1=49 +lat_2=77 +lat_0=49 +lon_0=-95 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs"

## project our vector data:
CanUS.lcc <- project(CanUS, canlam)
gl.lcc <- project(greatlakes, canlam)

## Now we to set the projection of our points:
crs(gbif) <- "+proj=longlat +datum=WGS84"

## Finally, we can project our points to LCC:
gbif.lcc <- project(gbif, canlam)

Note that our data needs to be in an object of class Spat*, and it must have a defined coordinate reference system (CRS) before we can project it to a new CRS. The crs function allows us to explicitly set the projection. See the RSpatial tutorial for more details on specifying CRS with terra.

The same function works for rasters (sort of):

rasterLCC <- project(trichPredsTrim, canlam)

This will reproject my raster from lat/lon to Lambert Conformal Conic, but we will unavoidably lose some precision when we do this. This is fine for visualization, but you should avoid projecting raster data used in analysis. For more details, see the terra tutorial.

plot(rasterLCC)
plot(CanUS.lcc, border = "grey", add = TRUE)

The units are no longer Lat/Lon, but meters. We can read them off the plot to improve the zoom:

plot(rasterLCC, xlim = c(0, 2500000),
     ylim = c(-1500000, -400000))
plot(CanUS.lcc, border = "grey", add = TRUE)

Formatting

With the data plotted, we can then turn to making the map a little prettier. Note that when working with Spat* objects, we can’t set the plot margins with par as we usually do. Instead, we use the mar argument within the plot function:

## Make a panel with two plots
par(mfrow = c(1, 2))

## store the plot limits:
my_xlims <- c(0, 2500000) 
my_ylims <- c(-1300000, -200000)

## Plot the points:
plot(CanUS.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", background = "lightblue",
     col = "white", axes = FALSE, mar = c(0.1,0.1,0.1,0))
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
points(gbif.lcc, pch = 16, col = alpha("grey30", 0.2),
       cex = 0.7)
box() 

plot(CanUS.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", background = "lightblue",
     col = "white", mar = c(0.1,0,0.1,0.1), axes = FALSE)
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
plot(rasterLCC, add = TRUE, legend = FALSE, axes = FALSE)

## plotted again to put the border lines on top:
plot(CanUS.lcc, border = "grey", add = TRUE) 
box()

If you want to plot the state/provincial borders on top of the raster, you need to add those layers last. But you can’t set the background colour of the raster layer to “lightblue” (or at least I haven’t figured that out), so the ocean stays white. I get around that by plotting the boundaries twice, first to set the background colour, and then to put the state lines on top of the raster.

Data Management for Reproducible Science

Mon, 17 Oct 2022 00:00:00 +0000

Introduction

Research is reproducible when others can reproduce the results of a scientific study given only the original data, code, and documentation (Alston and Rick, 2021)

Benefits to the Author:

Clear and complete documentation of your work makes it easier to share, write up and extend in future work, including responding to reviewers and developing new projects
Conscientious documentation of your work involves a great deal of error-checking, which is reassuring to you – that you haven’t missed anything, or mis-remembered what you did; and to your readers – that you have conducted your work in a rigorous manner
Reproducible work gets cited more, and developing a data archive creates a new citable product from your research.

Benefits to the Community:

Increases the speed and fidelity with which we can learn and apply new approaches.
Makes it easier to avoid mistakes (through the care and attention required to create the archive), and to detect and correct them if they do happen (by allowing others to critically review your work)

How to make your work reproducible

There are two related goals in producing a reproducible analysis: portability, and reproducibility. Reproducibility is perhaps obvious, but it’s not enough that you can reproduce your analysis on your computer. You are the only person with access to your computer. Your work should be reproducible on anyone’s computer.

Portability

To achieve portability, we need to know which files are needed for an analysis, and to organize them in a way that they can be readily moved from one computer to another. In practice, this means they’ll all be in a single directory, and that directory will only contain files for that particular analysis. You may have several related analyses that share a directory. That’s ok, but take time to organize them in a sensible way.

While you may have related analyses together in a directory, you don’t want to mix unrelated files and data in this directory. That will make it harder to keep track of what is needed and what isn’t, and will waste space on the computers where this work is ultimately archived.

README.txt

There are many ways you can organize your work within this directory. One absolute requirement is there needs to be a clear guide to your files, a “Table of Contents”. This should be a ‘plain text’ file - something that can be opened by any text editor. Your archive might be around for decades, and you don’t know if your readers will be able to find a copy of MSWord97 when they need to read it. We can be reasonably confident that plain text files will be accessible for a long time to come.

By convention, this file is called README.txt, and some data archiving services (DataDryad) require that you include a file with this name. You should probably use this name, unless you have a good reason not to.

One minor exception: if you use markdown, or RMarkdown, you can use these formats for your README. They are both plain text, and even if your audience doesn’t use them, they can open a README.md or README.Rmd file in any text editor. There are other, similar simple markup formats used in different coding communities. As long as they are saved as plain text files they will meet our requirements.

The contents of your README should describe as clearly as possible the contents of your archive. Here is an excerpt from one of mine:

Start with trich.Rmd to read our draft manuscript. See trich-prep.Rmd for the bulk of the code used in generating this manuscript.

File List:

trich.Rmd : main manuscript file

plantarum.json : bibliography (Zotero bibliography)

trich-prep.Rmd : The bulk of the code used in the analysis. Loaded from trich.Rmd to regenerate the figures and tables there.

data/

data/ssr_raw.csv : raw microsatellite data. See trich-prep.Rmd (Loading Data) for code to load and translate this to a genind object

data/survey-pops.csv : coordinates of sampled populations

data/eval.opt.2020-06-24.Rda : the output of the Maxent modeling, saved as a binary R Data object. Load into R with the load() function, so you don’t need to repeat the lengthy Maxent analysis.

data/trich-gbif.csv : GBIF records used in the Maxent analysis

data/trich_soil.csv : Soil analysis for each sampled population

data/maps : Maps (shapefiles and rasters) used in the Maxent analysis, and for some of the manuscript plots

I prefer to use RMarkdown to develop my manuscripts, as this allows me to keep the code for figures, and the resulting images, together with the text that describes the methods and interprets the results. If you prefer to manage your code separately from your writing that’s fine too, but you may end up structuring your archive a little differently.

Directory Organization

If you only have a few files, you may not need to do any further organization of your archive. I find it helpful to use subfolders to keep things organized. Depending on the needs of a project, I use:

data/: unprocessed data used in the analysis
processed/: intermediate data files, generated from data and not stored permanently
downloads/: storage of large external datasets used in the analysis, but not stored permanently in the archive (e.g., WorldClim Climate Data); be sure to include links to the source of any external data you are not archiving yourself!
plots/: images generated by the analysis
code/: code used in the analysis, if it’s not in the top-level directory

These are not hard rules. You can use whatever structure suits your project. Just be sure to explain it in your README.

Reproducibility

File organization gets us most of the way to portability. There are a few things we need to do in our coding to complete this arrangement, and of course to ensure we can reproduce the analysis once we move it to a new computer.

Use Relative Paths

A absolute path is the location of a file on a particular computer. The absolute path to the file I’m editing right now is /home/smithty/blogdown/content/tutorials/2022-10-17-data-management/index.md. That location will only ever exist on a Linux computer with a user named smithty. If I moved it to a Windows computer, it might be located at C:\Users\smithty\Documents\blogdown/content/tutorials/2022-10-17-data-management/index.md. If I try to refer to this file by its absolute path on Linux on the Windows machine, I won’t find it.

On the other hand, from the top directory of my blog, blogdown/, this file will have the same relative path on both machines: content/tutorials/2022-10-17-data-management/index.md. That means links to this file using the relative path will work just fine on both machines.

We should use relative paths in our analyses too.

For example, I could use an absolute path to load my data:

samples <-
  read.csv("/home/smithty/nextcloud/trich/2020-06-25/data/survey-pops.csv")

But this won’t load on anyone else’s computer. If we do this instead:

samples <-
  read.csv("data/survey-pops.csv")

Anyone with our archive can run the code as it is, as long as they have the working directory set properly. For my projects, I do this by running my code from the top directory of the archive. In this case, I have the directory structure:

├── data
│   └── survey-pops.csv
├── README.md
├── trich-prep.Rmd
└── trich.Rmd

When I run code from trich.Rmd, I set the working directory to the location of that file. RStudio manages this for you with its project support. If you don’t use that feature, you can tell R to use that location when it starts. After that, stick to relative file paths, and you don’t need to worry about your code breaking when you move to a different computer.

Structure Your Code to Run On Its Own

Two common problems that make it hard to run your code are mixing your ‘good’ code with non-working code, and writing code that requires you to update it by hand to finish your analysis.

Mixing good and bad code might look like this:

## Load in our data:
myData <- read.data("data/myfile.csv")
myDataScaled <- scale(myData)
myDataScaled <- scale(myData, center = FALSE)

myData <- myData[, -1]
myDataScaled <- scale(myData, scale = FALSE)

It’s not unusual to accumulate various versions of code (should I scale this, center it, both? Do I need the first column?). While you’re actively working on your code, you may find you have multiple versions in the same file. They need to be clearly commented, for your own benefit! But more importantly, when you decide which version you’re going to use, remove the rest. Don’t expect yourself (and certainly don’t expect anyone else) to figure out which lines they should run, and which they should skip.

## Load in our data:
myData <- read.data("data/myfile.csv")

## drop the name columnn
myData <- myData[, colnames(myData) != "name"]

## don't scale, use raw data
## myDataScaled <-scale(myData)

If you have a few lines of code that you might want to revisit, comment them out and leave yourself a note in a comment. If you have large blocks of code you aren’t using, but want to keep a record of, put them in a separate file. The end product should be a file that you can run from start to finish, without deciding which lines to skip and which to run.

A similar problem occurs when you have code that works, but requires you to manually update it in order to complete your analysis. e.g.,

myData <- read.table("data/experiments.csv")

myExperiment <- subset(myExperimentData, exp == 1)

myExperimentResult <- processingCode(myExperiment)

## repeat for exp 1-20

Code structured like this requires you to edit and re-edit the code many times to complete your work. That is tedious, and it’s easy to make a mistake. And when you update processingCode, you need to rerun everything by hand. You can avoid this with loops and lists.

R Programming: Do Not Save Your Workspace!

R offers to save your current workspace when you close the terminal. This is sometimes convenient, but can cause hard to detect problems with your analysis. If you happen to alter one of the objects you are working with in an R session, but don’t capture the code in your script file, you won’t have a record of what you’ve done. If you then save your workspace, the next time you work on your code, you will be able to keep using that modified object. At this point, your code and data are out of sync, with nothing to indicate how they differ.

At best, this is inconvenient. At worst, if undetected, you can waste weeks or months analyzing the wrong data! Better to avoid the risk and set R to never save your workspace.

Metadata: data about your data

If your archive includes data files, these should also be in an open format, such as comma-separated or tab-separated text files (typically with a name like FILE.csv, DATA.txt, or RECORDS.txt. That ensures that anyone can open your data without needing a proprietary program to do so.

Data Tables

In addition, you need to document how your data is coded. This includes things like:

Variable names and descriptions
Definition of codes and classification schemes
Codes of, and reasons for, missing values
Definitions of specialty terminology and acronyms

Be sure to include things like units (meters or feet?), and what your special codes mean (is pet_l the petal length or the petiole length?, what is lf1, lf2 and lf3?). This is also a chance to review your data for consistency - are you using NA and -1 for missing values? Do you have multiple different phrasings for the same thing?

Depending on your project, you may also need to distinguish between absent evidence (you didn’t sample on a day) and evidence of absence (you sampled on a day, but didn’t find any events/individuals). If your analysis will include sampling events that didn’t result in any observations, you’ll need to document these ‘true negatives’ in your data table.

Data Sources

If you’re using data from outside sources, like GBIF.org or WorldClim.og, be sure to record their citation details, including a DOI, if available, when you download them. This will ensure you can properly cite them later, and that your readers will be able to access the same data if they want to reproduce your work. Most data providers have clear policies as to how you should cite them, and what you’re allowed to do with the data they share (i.e., can you share it or archive it yourself). For example, here are the policies for WorldClim and GBIF.

How to get there from here

Now we have an idea of what we want our data archive to look like, how do we get there? Start by setting up your directory structure, and making a README file. If you’re starting from scratch, make it a practice to review your directory regularly, to see what you’ve done, what you’re no longer using, and updating your README to capture that. This kind of regular reflection is useful to track your progress, and helps you keep on top of your archive work so you don’t have a huge mess to wrangle at the end of your project.

If you’re already well-along in your project, it may be easier to create an ‘aspirational’ directory and populate it from your existing work. I do this regularly! I often find I’ve charged into an analysis without thinking about archiving, and after a few weeks I have an unholy mess of files and data to deal with. In that case, I create a new directory, a README, and copy over the main code file I’m using to that directory. When I get to the first data file in that directory, I copy it over to the data/ directory in the new archive, adjust the code to use the relative path, if necessary, and continue. This can be very helpful in clarifying what code and files you actually need and use, and what can be left behind.

This is also a good opportunity to ensure that your analysis is structured in a way that it can run start-to-finish without manual editing

You don’t need to delete anything in the old directory, you can keep it in case you later decide you want to revisit some ideas in there.

What to do with it all

One of the benefits of structuring your code in a single directory is there are lots of tools that you can use to manage it. git and GitHub are popular, and very powerful for managing code, especially tracking different versions of the same files. However, they require a certain amount of discipline to get the full benefit, and they are challenging to learn. RStudio does have good support for GitHub repositories.

You can also keep it simple, and sync your directory to Google Drive, Dropbox, Nextcloud, or many other options. Once your work is published, you can archive it permanently on Zenodo, DataDryad, or other online services.

All of these options will support housing a single directory and its subdirectories. None of these options will be easy to deal with if you have files spread across multiple directories and mixed with files from other projects!

Examples

I’ve been doing some version of this for my own work for years, but have only recently moved to permanent, public archives of my work. Here are three recent examples:

Hayes et al. (2022): The Genetic Diversity of Triploid Celtis pumila and its Diploid Relatives
Nowell et al. (2022): Conservation assessment of a range-edge population of Trichophorum planifolium
Foster et al. (2022): Testing the assumption of environmental equilibrium in an invasive plant species over a 130 year history

I’m still figuring out how best to do this, and my practice will definitely continue to change and evolve. Regardless of the specifics, I have benefited enormously from investing the time needed to make coherent archives of my projects.

References

Alston, J. M., and J. A. Rick. 2021. A Beginner’s Guide to Conducting Reproducible Research. The Bulletin of the Ecological Society of America 102: e01801.

Foster, S. L., H. M. Kharouba, and T. W. Smith. 2022. Testing the assumption of environmental equilibrium in an invasive plant species over a 130 year history. Ecography: e06284.

Hayes, A., S. Wang, A. T. Whittemore, and T. W. Smith. 2022. The Genetic Diversity of Triploid Celtis Pumila and its Diploid Relatives C. Occidentalis and C. Laevigata (Cannabaceae). Systematic Botany 47: 441–451.

Nowell, V. J., S. Wang, and T. W. Smith. 2022. Conservation assessment of a range-edge population of Trichophorum Planifolium (Cyperaceae) reveals range-wide inbreeding and locally divergent environmental conditions. Botany 100: 631–642.

Schoener's D and Study Extent

Thu, 02 Dec 2021 00:00:00 +0000

Background

Schoener’s D was created by Schoener (1968) He was studying the feeding niche of anoles, and needed a way to quantify the overlap in prey items for different species. This is what he came up with:

\[D(p_X, p_X) = 1 - \frac{1}{2} \sum_i \vert p_{X,i} - p_{Y, i} \vert\]

Here, $p_{X,i}$ and $p_{Y,i}$ are the frequencies for species $X$ and $Y$, respectively, for the $i^{th}$ category. For Schoener, the categories were prey sizes. In the context of distribution modeling, they would be regions along an environmental gradient, and the ‘frequencies’ are the fitted values from an SDM, or the density values from an Ecospat dynamic niche grid.

Warren et al. (2008) pointed out some subtle theoretical issues with Schoener’s D in this context, and proposed his own index I, based on the Hellinger distance, to better account for them.

Hellinger’s distance:

\[H(p_X, p_Y = \sqrt{\sum_i(\sqrt{p_{X,i}} - \sqrt{p_{Y,i}})^2}\]

Warren’s I:

\[I(p_X, p_Y) = 1 - \frac{1}{2} H(p_X, p_Y)\]

In application, Schoener’s D suggests that the $p_{X, i}$ values reflect relative use of a particular habitat. However, ENM predictions indicate the relative ‘suitability’ of a cell for occupancy (i.e., presence or absence) by the study species, but do not necessarily reflect density.

However, Warren also noted that despite the potential issues, in practice there is little difference in the qualitative results following from D and I. I think Schoener’s D is more commonly used now, but either or both may show up in distribution modeling studies.

Overlap vs Correlation

Warren (2018) made an interesting contrast between two species’ niche overlap (D), and the correlation between their suitability scores. Schoener’s D quantifies the extent to which a pair of species may interact in the same space (i.e., they’re both likely to be present together in a location). This is important to know, especially in the context of niche-shift studies (e.g. Atwater and Barney, 2021). But while they tell us about where species are found along an environmental gradient, they don’t tell us anything about how they respond to that gradient. In fact, species with perfectly opposite responses to the environment may still have relatively high niche overlap, D.

Let’s revisit the example from Warren (2018). We start with the olaps helper function, which calculates the statistics of interest:

library(ggplot2)
library(grid)

olaps <- function(sp1, sp2){
  ## Calculate Schoener's D, Warren's I, and Spearman
  ## Correlation for sp1 and sp2

  ## sp1 and sp2 are the relative occupancy values for each
  ## species along the same environmental gradient

  ## scale the values for each species 0:1
  sp1 <- sp1/sum(sp1)
  sp2 <- sp2/sum(sp2)
  
  plot.table <- data.frame(
    species = c(rep("sp1", length(sp1)),
                rep("sp2", length(sp2))),
    env = c(seq(1:length(sp1)), seq(1:length(sp2))),
    suitability = c(sp1, sp2))

  D = 1 - sum(abs(sp1 - sp2))/2
  I = 1 - sum((sqrt(sp1) - sqrt(sp2))^2)/2
  cor = cor(sp1, sp2, method = "spearman")

  grob <- grobTree(textGrob(paste("D =", round(D, 2),
                                 "  I =", round(I, 2),
                                 "  Cor =", round(cor, 2)),
                           x = 0.1,  y = 0.95, hjust = 0,
                           gp = gpar(fontsize = 15)))
  

  suitplot = qplot(env, suitability, data = plot.table,
                   col = species, geom = "line") +
    annotation_custom(grob)

  return(list(
    D = D, I = I, cor = cor, suitplot = suitplot
  ))
  
}

Now we can recreate the examples from Warren (2018).

sp1 <- seq(0.1, 1.0, 0.001)
sp2 <- seq(0.1, 1.0, 0.001)

olaps(sp1, sp2)

Figure 1: Identical Species

sp1 <- seq(0.1, 1.0, 0.001)
sp2 <- seq(1.0, 0.1, -0.001)

olaps(sp1, sp2)

Figure 2: Species with Inverse Environmental Response

The point that Warren (2018) was making is that two species may occupy more or less similar locations along an environmental gradient, while having very different responses to that gradient. This isn’t a problem. But it does highlight the importance of clearly articulating the question you are asking in your research, and making sure that the analyses you choose are actually answering that question.

Niche Overlap vs Study Extent

Something else struck me reading Warren’s post. The toy examples he used represent a very narrow slice of an environmental gradient; that is, the portion where both species are present. Applying these analyses to global patterns, as we do when comparing the distribution of invasive species in their native and introduced range, and especially when we apply these analyses to large numbers of species, we can (potentially) include much broader gradients. And this can have significant impact on theses statistics.

Here’s a (ever so slightly) more realistic example to illustrate. We’ll define our gradient over the range 0 to 50

env <- seq(0, 50, by = 0.01)

Then we’ll define two species, with partially overlapping ranges:

sp1 <- dnorm(env, mean = 22.5, 2)
sp2 <- dnorm(env, mean = 27.5, 2)

Now compare the species ‘globally’:

olaps(sp1, sp2)

Figure 3: Global Analysis

At this scale, their response to the gradient appears to be highly correlated, while they have low niche overlap.

If we zoom in a bit, and ‘trim’ off the lowest and highest 1000 values on our gradient, we can emulate a ‘continental’ extent:

## ignore the lowest and highest 1000
## environmental values 
slice <- 1000:4000 

olaps(sp1[slice], sp2[slice])

Figure 4: Continental Analysis

Correlation drops, but niche overlap remains identical. On reflection, this makes sense. Locations where neither species are present get no weight in the calculation of D, so dropping ‘empty’ gradient has no impact. On the other hand, those locations do contribute to inflating correlation.

Now what if we shift our focus, such that the distribution of our species is not equally represented:

slice <- 1000:2500 
olaps(sp1[slice], sp2[slice])

Figure 5: Regional Analysis

Correlation jumps up, as despite both species increase together over most of the sampled gradient. And with this particular slice, our niche overlap is twice the ‘true’ value when we consider the full gradient.

Finally, we can zoom in on the center of the gradient, where both species are equally represented (although with inverse responses):

slice <- 2000:3000
olaps(sp1[slice], sp2[slice])

Figure 6: Contact Zone

Correlation drops again, accurately reflecting the inverse pattern. And D is back down close to the ‘true’ value. That’s ‘lucky’, as my toy species have perfectly symmetrical distributions, so sufficently large, symmetrical regions around the mid-point between the two of them will give reasonably accurate estimates.

Implications

Why does this matter? If you’re interested in comparing the environmental responses of two species, the results (correlation) can vary quite dramatically depending on the extent of your study.

On the other hand, Schoener’s D is robust to data that includes ‘too much’ of a gradient (i.e., extending beyond the region occupied by either species). But it can be sensitive to undersampling a gradient, where the relative occupancy of each species varies depending on how you set set your extent.

In other words, if you’re interested in the ‘underlying models’ that govern species’ comparative distributions along a gradient, you need to be very clear about the scope of the question, and how much of the environmental gradient you sample. But if you want to quantify niche overlap (or relative niche shift), then you want to include environments well beyond the regions actually occupied by your study organisms.

All of which is trivial to do when you get to create the species on the computer, and much trickier when you need to infer the details from museum records and climate rasters!

References

Atwater, D. Z., and J. N. Barney. 2021. Climatic niche shifts in 815 introduced plant species affect their predicted distributions I. Martins [ed.],. Global Ecology and Biogeography 30: 1671–1684.

Schoener, T. W. 1968. The Anolis Lizards of Bimini: Resource Partitioning in a Complex Fauna. Ecology 49: 704–726.

Warren, D. 2018. Species In Space: Why add correlations for suitability scores? Species In Space. Website https://enmtools.blogspot.com/2018/10/why-add-correlations-for-suitability.html [accessed 2 December 2021].

Warren, D. L., R. E. Glor, and M. Turelli. 2008. Environmental Niche Equivalency Versus Conservatism: Quantitative Approaches to Niche Evolution. Evolution 62: 2868–2883.

Thinning Occurrence Records in R

Tue, 26 Oct 2021 00:00:00 +0000

Note that this tutorial refers to the thinning method used in the old version of the rspatial.org tutorial, which used the raster package (along with dismo) for the GIS computations. The terra package will shortly be replacing raster, and all new code should use this instead. The details of spatial thinning with terra are presented in my new ecospat tutorial

A common approach to reducing spatial bias in occurrence records is to randomly select one (or a small number) of samples present in each cell in the landscape. This uses the gridSample function from the package dismo (Hijmans et al. 2017), as described at RSpatial.org.

However, the code presented at RSpatial uses a newly-created raster layer to thin the records. This layer is based on the extent of your occurrence data; even if you set the resolution to match the resolution of the environmental rasters you use, the they won’t necessarily be aligned. That means the cells will be the same size, but the edges won’t line up.

A consequence of this is that you might end up keeping more than one sample, or removing all samples, from a single cell in your environmental data, even after thinning to one sample per cell in your newly-created raster. This lead to some strange behaviour in one of my downstream analyses, where the results for a small data set changed each time I reran the analysis.

I’ll demonstrate using the example from RSpatial:

library(dismo)
library(maptools)
library(sp)
library(raster)

wclim <- getData("worldclim", var = "bio", res = 10,
                path = "../data")

## Warning in getData("worldclim", var = "bio", res = 10, path = "../data"): getData will be removed in a future version of raster
## . Please use the geodata package instead

wc1 <- wclim[[1]]

## crop the climate data to speed up grid creation
wc1crop <- crop(wc1, extent(c(-70, -60),
                           ylim = c(-20, -10)))

data(acaule)
data(wrld_simpl)

acgeo <- subset(acaule, !is.na(lon) & !is.na(lat))
dups2 <- duplicated(acgeo[, c('lon', 'lat')])
acg <- acgeo[!dups2, ]
i <- acg$lon > 0 & acg$lat > 0
acg$lon[i] <- -1 * acg$lon[i]
acg$lat[i] <- -1 * acg$lat[i]
acg <- acg[acg$lon < -50 & acg$lat > -50, ]
coordinates(acg) <- ~lon+lat
crs(acg) <- crs(wrld_simpl)

plot(acg, pch = 20)
plot(wclim[[1]], legend = FALSE, add = TRUE)
points(acg, pch = 20)
plot(wrld_simpl, add=T, border='blue', lwd=2)
box()

This is the same example from RSpatial. See the link above for more details.

Now we want to thin our records, such that we retain only one observation for each cell in the WorldClim climate layer. To track what’s going on here, I’ll zoom in on one of the crowed areas:

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)

Now lets overlay the grid generated by the code from RSpatial:

r <- raster(acg)
# set the resolution of the cells to the same as wclim
res(r) <- res(wclim)
# expand (extend) the extent of the RasterLayer a little
r <- extend(r, extent(r)+1)
p <- rasterToPolygons(r)

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)
plot(p, add = TRUE)

Notice how the climate cells (the coloured squares) are offset from the sampling grid (the black gridlines).

Using this grid for gridSample. I’ll plot a blue ring around the retained occurrences; the records we drop are left as points:

set.seed(1)
acsel <- gridSample(acg, r, n=1)

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)
plot(p, add = TRUE)
points(acsel, col = 'blue', cex = 2)

Take a close look at that last plot. Notice that there are climate cells with no retained observations:

As well as climate cells with multiple observations:

This is fine if you are using the occurrence records in a purely spatial analysis (i.e., without incorporating climate data for each observation). But if you are intending to retain at most one observations for every cell in your climate map, this is not what you were hoping for.

A better way to achieve our desired result is to use the climate layer directly in gridSample:

set.seed(1)
acselClimate <- gridSample(acg, wc1, n=1)

Lets visualize the grid we sampled on:

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)

## using a cropped layer because this is a slow operation:
climGrid <- rasterToPolygons(wc1crop)
plot(climGrid, add = TRUE)
points(acselClimate, col = 'blue', cex = 2)

Everything matches up as we expect now: the sampling grids are perfectly aligned with the climate cells.

This approach will only work if your climate raster covers the full extent of your occurrence records. Which it really should - if it doesn’t, the records that aren’t covered will end up getting dropped from your analysis since there’s no climate data at those locations.

References

Hijmans, Robert J., Steven Phillips, John Leathwick, and Jane Elith. 2017. “Dismo R Package Version 1.1-4.” https://CRAN.R-project.org/package=dismo.

Emacs for Bioinformatics #4: RMarkdown

Sun, 03 Oct 2021 00:00:00 +0000

This is part four in my series of Emacs tutorials aimed at bioinformatics (and other scientific analysis) workflows. See the rest on my tutorials page.

Emacs provides full support for editing RMarkdown documents. RMarkdown has extensive documentation, both at the previous RStudio link, and several free online books by Xie et al. (notably R Markdown: The Definitive Guide, but also several others listed on Yihui Xie’s Bookdown page).

Most of these references assume you are using the RStudio development environment. The purpose of this tutorial is to get you started editing RMarkdown documents in Emacs.

Installation

Prerequisites

You need to have R installed, of course. You will also need Pandoc in order to take full advantage of all the output options available. If you want to create PDF documents, you’ll need LaTeX as well.

All three of these programs are provided in the package repositories for most major Linux distributions. See the links above for instructions for installing on Windows or Apple computers.

You will also need to install the rmarkdown R package. You can do this from within R via:

install.packages("rmarkdown")

This will also install the other R requirements, notably the knitr package.

The bookdown package provides some more advanced citation features. I won’t discuss them in this short tutorial, but in order to use them you need to install that package too:

install.packages("bookdown")

Emacs Packages

We need a few additional Emacs packages to comfortably edit RMarkdown documents. These are:

Markdown Mode: The major mode for editing files in markdown format. This tutorial uses features added after 6 January 2021.
ESS: The collection of modes for editing R code and interacting with the R program.
poly-R (Polymode): polymode is a ‘glue’ mode. The poly-R variant extends markdown mode to allow us to edit embedded code snippets in R (and other languages too)

(poly-R also supports files in .Rnw format, which mix LaTeX and R code. We won’t cover that here)

polymode started out as a collection of modes to support files with different combinations of languages. As it has grown, many of those different modes have been split out into separate packages. When we install poly-R, it will automatically install the core of the polymode system for us.

This tutorial uses features added after 29 September 2021.

As in previous tutorials, (see my blog or the demo on Youtube), we can install all three of these packages from MELPA.

Once we have the required packages installed, no further configuration should be necessary. When we next open a file with a .Rmd extension, Emacs will know to use the poly-markdown+R-mode for these files. If everything is working properly, you’ll see Markdown PM-Rmd in the modeline at the bottom of the window for these files, and Markdown, RMarkdown, and Polymode menus at the top of Emacs frame.

Configuration Note

Depending on how you have installed poly-R, it may be loaded automatically, or you might need to load it yourself in your config. If it isn’t loaded automatically, you might see errors like (void-function poly-gfm+r-mode) when you try to open an Rmarkdown file.

You can fix this with by adding the following line to your Emacs config. The location isn’t critical, but it’s probably most convenient to put it at the beginning of any configuration you use for ESS/R/Markdown.

(require 'poly-R)

Github Flavoured Markdown and Code Blocks

Markdown mode supports several different options for code blocks. To take full advantage of the RMarkdown support provided by the rmarkdown R package, we need to use fenced code blocks, along with language strings wrapped in braces¹. I’ll explain this in more detail below.

This variant of markdown is referred to as “Github Flavoured Markdown”, and the markdown-mode package provides gfm-mode with a few extra features particular to it. Turning on gfm-mode for Rmd files requires the following line in your Emacs configuration to turn it on²:

;; associate the new polymode to Rmd files:
(add-to-list 'auto-mode-alist
             '("\\.[rR]md\\'" . poly-gfm+r-mode))

You will also need the following line, if you want gfm-mode to automatically insert braces for code blocks (described below):

;; uses braces around code block language strings:
(setq markdown-code-block-braces t)

This will switch you from using poly-markdown+R-mode to poly-gfm+r-mode, which shows up in your mode bar as “PM-Rmd(gfm)”. It’s nearly similar, the main differences being the support for fenced code blocks.

RMarkdown

Editing Markdown

Markdown syntax is designed to be easily entered by hand, which means if you’re already familiar with the format you can just get going. Markdown mode will provide you with syntax highlighing automatically:

A Markdown Mode buffer showing syntax highlighting

Of course, there are lots of shortcuts available. You can explore the most frequently used in the Markdown menu. I’ll summarize some of the main ones here to get you started.

Headings

Markdown has two different kinds of headings. The “Atx” style uses # symbols at the beginning of the heading, and, optionally, also at the end:


# Heading Level 1

## Heading Level 2 ##

You can insert these headings with the following commands:

C-c C-s h: insert a heading at the same level as the previous heading.

If the region is active, the contents of the region will be used as the header text. If point is on a line with text, the line will be converted into a header. Otherwise, an empty header will be created.

C-c C-s {1-9}: insert a heading at the specified level. i.e., C-c C-s 3 inserts a third-level heading. region and point can be used to set the heading text as for the previous.

You can manipulate headings with the following commands:

C-c <up> and C-c <down>: move a heading and all of its content up or down in the document.

ie., turn this:

Markdown headings in original order

into this:

Markdown headings with subheading 2 ahead of subheading 1

C-c <left> and C-c <right>: promote or demote a heading.

i.e., turn this:

Markdown headings in hierarchy

into this (and vice versa):

Markdown with subheading 2 demoted

If you prefer asymmetric headings (i.e., with # symbols only at the beginning of the line), you can configue this by setting the variable markdown-asymmetric-header to t:

;; set in your ~/.emacs or ~/.emacs.d/init.el
(setq markdown-asymmetric-header t)

Alternatively, you can do this via M-x customize-variable markdown-asymmetric-header.

Markdown mode also supports the setext style headings:


Heading Level 1
===============

Heading Level 2
---------------

Only two levels are supported, which you can insert automatically with the commands C-c C-s ! (level 1) and C-c C-s @ (level 2).

Heading Visibility

You can hide and show different sections in documents by pressing the <TAB> key with point on a heading. For example, with point on the # Installation heading, when I press <TAB> I move from this:

Markdown buffer with all sections visible

to this:

Markdown buffer with the Installation section hidden

This doesn’t change any of the text in your file, it only hides the parts you don’t want to see. Repeatedly pressing the <TAB> key will toggle through the various levels of hiding and showing.

If you want to toggle all the headings at once, Shift-<TAB> will toggle visibility for all headings at once. You can use this to collapse your entire document to a table of contents:

A Markdown buffer with only headings visible

Links and Images

Inserting links is done with C-c C-l. Emacs will first prompt you for the link URL, followed by the link text, and finally the tooltip text. Only the URL is required. To open a link from Emacs, use C-c C-o, which will take you to the webpage in your browser.

Images are handled similarly, and are inserted with C-c <TAB> or C-c C-i. The URL can be a web resource (e.g., https://my-images.ca/image1.jpg), or a local file (e.g., ./images/image1.jpg). The image will appear as:

![Image Caption](image URL)

You can toggle displaying the actual image in the buffer with C-c C-x C-i.

Tables

To insert a new table, use the command C-c C-s t. You will be prompted for the number of rows and columns, and the alignment you want. When you’re done, you’ll have a proper markdown table ready to edit:

|   |   |   |   |
|---|---|---|---|
|   |   |   |   |
|   |   |   |   |
|   |   |   |   |

With point in any of the cells, you can <TAB> into the next cell, or Shift-<TAB> into the previous cell. Each time you hit tab the cells will resize automatically to accomodate your text.

Additional commands are available for moving, adding and deleting rows and columns; see the Markdown -> Tables menu the options.

Other Markup

Markdown mode also provides shortcuts for other markup elements. See the Markdown menu for some of the options. I find most of the basics (bold, emphasis, unordered lists) are just as fast to type by hand as they are to insert using shortcuts.

Working with R Code

rmarkdown uses fenced code blocks with braces around the language string. i.e.,:

    ```{R code-block-example}
    ## R code goes here!
    1 + 1
    ```

If you’ve set up gfm-mode as described above, you can create one of these code blocks with the command markdown-insert-gfm-code-block, bound to C-c C-s C by default. Alternatively, simply entering three “`” characters at the beginning of a line will call the function for you³. Either way, you’ll be prompted for the language of the code block (which will be R most of the time, but you can use others!). You can also add a label for the code block at the prompt, and any additional options you want to use for the chunk. You can also add options later if you change your mind.⁴

Once you have created a code block, polymode will work its magic. You can continue to edit the markdown portions of your document, with all the features of gfm-mode. But when point is in an R code block, you’ll be editing it in ESS[R] mode. That allows you to use all the features of that package (see plantarum.ca for a quick tutorial/refresher).

Polymode provides some additional conveniences:

Navigation

polymode-next-chunk/polymode-previous-chunk, bound to M-n C-n and M-n C-p: move to the next/previous chunk. i.e., move from an RMarkdown chunk to the next R code chunk.
polymode-next-chunk-same-type/polymode-previous-chunk-same-type, bound to M-n M-C-n and M-n M-C-p: move to the next/previous chunk of the same type. i.e., move from one R code chunk the next R code chunk.
polymode-kill-chunk, bound to M-n M-k: kill the current chunk
polymode-toggle-chunk-narrowing, bound to M-n C-t: toggle narrowing the buffer to display only the current chunk, or to display the entire document

Evaluation

polymode-eval-region-or-chunk, bound to M-n v: evaluate all code chunks in the active region, or the chunk at point if there is no active region
polymode-eval-buffer, bound to M-n b: evaluate all code chunks in the buffer
polymode-eval-buffer-from-beg-to-point/polymode-eval-buffer-from-point-to-end, bound to M-n u or M-n ↑, and M-n d or M-n ↓: evaluate all code chunks from the beginning of the buffer to point (u and ↑), or from point to the end of the buffer (d and ↓)

Exporting RMarkdown

The most important ‘convenience’ of polymode is that it connects Emacs to the programs used to export RMarkdown files to presentation formats (i.e., pdf, html, slides). The main function you need for this is polymode-export, bound to M-n e.

The first time you run this, you’ll be asked which exporter you would like to use. There are two choices, markdown and markdown-ess. markdown means polymode will start a new, self-contained R process and compile your file there. When compilation is finished, the process will be closed.

markdown-ess will use an existing R process, or start a new one if there isn’t an active process available. When compilation is complete, the R process remains active. This allows you to check the values of various objects interactively. This can be useful as you develop a new script.

RMarkdown files can be compiled to produce a variety of output formats. You will be prompted to select which one you want the first time you run the exporter. polymode remembers this setting, so you don’t get prompted a second time. If you want to switch, say from pdf output to html, you can reset the target via C-u M-n e.

Feature added 6 January 2021.↩
Feature added 29 September 2021 ↩
By default; you can turn this feature off by setting the variable markdown-gfm-use-electric-backquote to nil.↩
I’m working on tab-completion for R chunk options, but haven’t decided how best to set it up yet. Watch this space!↩

Evaluating Invasion Stage with SDMs

Wed, 11 Aug 2021 00:00:00 +0000

My attempt to recreate the invasion stage analysis developed by Gallien et al. (2012), inspired by seeing it applied by Eckert et al. (2020). We’ll continue with the Lythrum salicaria data from my tutorial on niche quantification analysis. Specifically, I’ll model how the niche space this species occupies in its invaded range in North America relates to its global niche.

library(ecospat)
library(raster)
library(rgbif)
library(maptools)
library(magrittr)
library(dismo)

Gallien et al. (2012) used an ensemble of SDMs, which is (should be) more robust than applying a single approach. Nevertheless, for this short tutorial, I’ll stick to Maxent. I’m also cutting a lot of corners with respect to variable selection, model validation and other important steps. See my Maxent notebook for pointers.

Niche Models

We start by constructing SDMs for the global and North American distribution of L. salicaria.

Data

We need occurrence data and environmental data, and we’ll need to create background (pseudoabsence) samples.

The occurence data comes from GBIF, with details in my previous post:

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

coordinates(lsOccs) <- c("decimalLongitude",
                        "decimalLatitude") 
  ## Set the projection
crs(lsOccs) <- '+proj=longlat +datum=WGS84'

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

data(wrld_simpl) # load the maptools worldmap

par(mar = c(0,0, 0, 0))
plot(wrld_simpl, border = "gray80")
points(lsOccs, pch = 16, col = 2, cex = 0.3)

We’ll use the same climate data as well, sourced from WorldClim (Fick and Hijmans 2017) and imported using functions provided in the raster (Hijmans 2021) package. Note that I use the path argument to direct the download to a particular location. This is the same location I used in the previous post, and the data is still there, so it doesn’t get downloaded again.

wclim <- getData("worldclim", var = "bio", res = 10,
                path = "../data")

We need to define our study extent for selecting background points. I’ll use a 200 km buffer around our observations. We’re working at the global scale, and Lythrum salicaria is a strong disperser, so a relatively large scale is appropriate here. You’ll need to consider the aims of your own study when setting your extent.

studyExtent <- buffer(lsOccs, 200000, dissolve = TRUE)

## Loading required namespace: rgeos

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files
## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

plot(wrld_simpl, border = "gray80")
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 0.3)

The 200 km buffer creates some isolated pockets in North America. The extent should represent the area the species can access. The buffer I made includes the west coast inwards to Alberta, the east coast inwards to Saskatchewan, with an isolated patch in the center of Canada which looks like it’s at the Alberta/Saskatchewan border, with similar ‘islands’ in the western US:

plot(wrld_simpl, border = "gray80", xlim = c(-135, -90),
     ylim = c(45, 60))
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 1)

Those islands are likely the leading edge of the same invasion, not separate invasions! I’m going to increase our buffer to 300 km to capture the intervening area on the map:

studyExtent <- buffer(lsOccs, 300000, dissolve = TRUE)

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files
## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

plot(wrld_simpl, border = "gray80")
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 0.3)

This is better. I prefer to use ecoregions to set study extent, but for the purposes of this demo I’ll continue with this.

One further issue: our study extent includes the ocean. Let’s trim it back to the land:

land <- aggregate(wrld_simpl) ## dissolve country borders

  ## clip buffer to land:
studyExtent <- intersect(studyExtent, land) 

plot(wrld_simpl, border = "gray80")
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 0.3)

(This generates some warnings, likely related to missing values in my data or issues with the shapefile manipulations. It seems safe to proceed.)

The aggregation is a bit rough, but that should work for my purposes today. Now we can select our background points. I’m using 10000 points, and excluding any cells with a Lythrum salicaria occurrence.

  ## Convert landmass polygon to a raster:
landMask <- rasterize(land, wclim)

  ## sample points from the raster:  
background <- randomPoints(landMask, n = 10000, p = lsOccs)

Global SDM

Now we can fit our Maxent model. To reduce bias, I’ll thin the samples to 5 observations per grid cell (ca. 20 km square). Normally I work on a finer resolution (1 km 2), and thin to 1 observation per cell. Again, the details depend on your study area and goals.

lsThin <- gridSample(lsOccs, wclim, n = 5) %>%
  as.data.frame

coordinates(lsThin) <-
  c("decimalLongitude", "decimalLatitude")
glMax <- maxent(wclim, p = lsThin, a = background)
glPred <- predict(glMax, wclim)

(Warnings here tell us that some of the occurrences are in locations where there is no climate data. That’s normal, and not a problem as long as there are only a few points lost this way. If you are working with small data sets, you’ll want to investigate further to see if you can better match your records with the climate rasters.)

Here’s the model prediction:

plot(glPred)

North America SDM

For the SDM in the invaded range in North America, we need to crop our observations and background. Here I’m repeating the functions I used above to create a raster mask for land, but applying it only to the area of Canada, United States, and Mexico (our species isn’t in the Caribbean). I’m using the pipe (%>%) feature from magrittr, which makes it easier to follow the process.

NA_polygon <- wrld_simpl %>%
  subset(NAME %in%
         c("Canada", "United States", "Mexico")) %>%
  aggregate()

NA_mask <- rasterize(NA_polygon, wclim)

NA_background <-
  randomPoints(NA_mask, n = 10000, p = lsOccs) %>%
  as.data.frame()

coordinates(NA_background) <- c("x", "y")

In the original paper of Gallien et al. (2012), the background points were weighted using the values from the global model. I don’t think Eckert et al. (2020) applied this weighting, and it’s not clear to me how to do so with Maxent. For now I’ll skip it.

For the occurrence records, I’ll take my previously thinned data, and crop it to North America:

NA_polygon <- wrld_simpl %>%
  subset(NAME %in%
         c("Canada", "United States", "Mexico")) %>%
  aggregate()

lsNAThin <- intersect(lsThin, NA_polygon)

Now we can construct the SDM for North America:

naMax <- maxent(wclim, p = lsNAThin, a = NA_background)
naPred <- predict(naMax, wclim)
plot(naPred)

Note that I didn’t crop the WorldClim layer for the North American SDM model fitting. Maxent only uses the data for the presence and background points, so it doesn’t matter if the climate layers cover the whole planet for this step.

Invasion Stage Analysis

Now that we have completed both a global and a local (North America) SDM for L. salicaria, we’re ready to compare the results.

Niche Space

The values we need are the model predictions corresponding to each observation in North America.

globalVals <- extract(glPred, lsNAThin)
naVals <- extract(naPred, lsNAThin)

plot(naVals ~ globalVals, pch = 16, xlim = c(0, 1),
     ylim = c(0, 1),
     xlab = "Global model predictions",
     ylab = "Regional model predictions",
     col = "#00000050")
abline(h = 0.5, lty = 2)
abline(v = 0.5, lty = 2)

This plot compares the default Maxent output, the complementary log-log value. This is an estimate of the probability of presence, which is more appropriate than the other options for this kind of analysis (raw values would be difficult to interpret). However, I’m not sure 50% is the most appropriate value to use in the analysis that follows. Eckert et al. (2020) used optim.thresh from the (now defunct) SDMTools package to determine the best threshold for their study.

Following Gallien et al. (2012), we interpret the four quadrants of this plot as follows:

Upper right: High suitability in both native and global habitat. Observations here are occupying locations that fall within both the global and invaded niche, interpreted as ‘stabilizing.’
Upper left: High suitability in native model, but low suitability in the global model. Observations are occupying locations that are within the invaded niche, but outside the global niche, interpreted as populations demonstrating local adaption
Lower right: High suitability in the global model, but low suitability in the local model. These are interpreted as regional colonizations: the conditions here are within the global niche, but which are only starting to be occupied in the invaded range.
Lower left: Low suitability in both the local and global model. Presumably sink populations (not likely to persist).

Let’s tabulate the results:

tally <- c(stabilizing
          = sum(globalVals >= 0.5 & naVals >= 0.5,
                na.rm = TRUE),
          adapting = sum(globalVals < 0.5 & naVals >= 0.5,
                         na.rm = TRUE),
          sinks = sum(globalVals < 0.5 & naVals < 0.5,
                      na.rm = TRUE),
          colonizing = sum(globalVals >= 0.5 & naVals < 0.5,
                           na.rm = TRUE))

barplot(tally, ylab = "Occurences")

We can plot these regions on the map as well (apologies for the opaque raster algebra; there should be a clearer way to calculate this, but I can’t think of it at the moment).

suitabilityThreshold <- 0.5
na_Niche <- naPred > suitabilityThreshold
gl_Niche <- glPred > suitabilityThreshold

stable_Niche <- (na_Niche + gl_Niche) == 2
expansion_Niche <- ((2 * na_Niche) - gl_Niche) == 2
contraction_Niche <- ((2 * gl_Niche) - na_Niche) == 2

NicheRaster <- stable_Niche + (2 * expansion_Niche) +
  (3 * contraction_Niche)

plot(NicheRaster, xlim = c(-140, -60), ylim = c(30, 70),
     col = c("white", "blue", "red", "green"),
     legend = FALSE)
plot(wrld_simpl, add = TRUE)
points(lsNAThin, pch = 16, cex = 0.5)

In this plot, the blue depicts areas identified as suitable habitat in both the global and regional model. The green is area identified as suitable habitat in the global model, but not the North American model. There are some occurrences in this area, but they aren’t as numerous as the blue regions. Finally, the red areas were identified by the North American model as suitable habitat but they were not part of the global model’s suitable habitat. Following Gallien’s framework, any points in the white areas would be ‘sinks.’ More likely they’re the current leading edge of the invasion front I think.

Obviously, there’s a lot going on here, and each of these steps will warrant careful consideration and additional checks, validations, and optimizations. I hope this simplified outline is enough to get you started.

References

Eckert, Sandra, Amina Hamad, Charles Joseph Kilawe, Theo E. W. Linders, Wai‐Tim Ng, Purity Rima Mbaabu, Hailu Shiferaw, Arne Witt, and Urs Schaffner. 2020. “Niche Change Analysis as a Tool to Inform Management of Two Invasive Species in Eastern Africa.” Ecosphere 11 (2). https://doi.org/10.1002/ecs2.2987.

Gallien, Laure, Rolland Douzet, Steve Pratte, Niklaus E. Zimmermann, and Wilfried Thuiller. 2012. “Invasive Species Distribution Models – How Violating the Equilibrium Assumption Can Create New Insights.” Global Ecology and Biogeography 21 (11): 1126–36. https://doi.org/https://doi.org/10.1111/j.1466-8238.2012.00768.x.

Hijmans, Robert J. 2021. “Raster: Geographic Data Analysis and Modeling.” Manual. https://CRAN.R-project.org/package=raster.

Niche Quantification with Ecospat

Thu, 29 Jul 2021 00:00:00 +0000

Update

NB: this tutorial is now out of date and depends on deprecated versions of R packages! Please refer to the new version of my ecospat tutorial for the current packages and workflow for this analysis.

I’ll leave the old version below in case it’s of any interest, but it won’t work on current versions of R.

Archived Version

Packages

In addition to ecospat, we’ll use raster (Hijmans 2021) to download WorldClim (Fick and Hijmans 2017) rasters, and manipulate the spatial data; rgbif (Chamberlain et al. 2021) to download GBIF records, and maptools (Bivand and Lewin-Koh 2021) to get a world basemap for plots.

library(ecospat)
library(raster)
library(rgbif)
library(maptools)

NB there is a bug in ecospat that prevents us from using the argument geomask (see below). This has been fixed, but as of 2021-07-29, the bug fix has not made it into the released package, currently version 3.2. Consequently, you need to install directly from the development sources:

library(devtools)
install_github(repo = "ecospat/ecospat/ecospat")

Presumably this won’t be necessary for versions 3.3+ or newer (once released).

Getting Data

lsGBIF <- occ_search(scientificName = "Lythrum salicaria",
                    limit = 10000,
                    basisOfRecord = "Preserved_Specimen",
                    hasCoordinate = TRUE,
                    fields = c("decimalLatitude",
                               "decimalLongitude", "year",
                               "country", "countryCode"))

save(lsGBIF, file = "../data/2021-07-29-ls-gbif-recs.Rda")

This returned an object with 7969 records. I saved that locally, so that I’m not making GBIF search their database everytime I work on this demo.

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

Next, we tell R which columns are the coordinates, which allows us to map the observations. This also converts our observation matrix to a SpatialPointsDataFrame object.

coordinates(lsOccs) <- c("decimalLongitude",
                        "decimalLatitude") 
data(wrld_simpl) # load the maptools worldmap

par(mar = c(0,0, 0, 0))
plot(wrld_simpl, border = "gray80")
points(lsOccs, pch = 16, col = 2, cex = 0.3)

To get our climate data, we can use raster’s getData function. The first time you call this function in a directory, it downloads the data from the internet, and saves it locally. Subsequent calls will load your local copy of the data, to speed things up. I’m using the coarsest resolution (10 minutes) to speed things up for this demonstration:

wclim <- getData("worldclim", var = "bio", res = 10,
                path = "../data")

We can take a look at one layer:

par(mar = c(0,0, 3, 1))
plot(wclim[["bio1"]], main = "bio1")

Next, we need to extract the environmental values from the climate rasters for each of our observation records:

lsOccs <- cbind(lsOccs, extract(wclim, lsOccs))

lsOccs <- lsOccs[complete.cases(data.frame(lsOccs)), ]

Splitting Data

At this point, all the data we need for the Niche Quantification analysis is in lsOccs and wclimMat. We need to split this data into native and invasive regions for our comparison. We’ll restrict ourselves to the northern hemisphere north of 20 degrees, and consider all records from Eurasia as native, and all records from North America as invasive.

I’ve created extents to cover the rough outlines of the areas in question. In practice, you could use a more carefully constructed vector map to split your data.

## North America: na
## Longitude from 40 to 180W, Latitude from 20 to 90N
naExt <- extent(c(-180, -40, 20, 90))
lsNA <- crop(lsOccs, naExt)

## Eurasia: ea
## Longitude from 40W to 180E, Latitude from 20 to 90N
eaExt <- extent(c(-40, 180, 20, 90))
lsEA <- crop(lsOccs, eaExt)

par(mar = c(1, 0, 0, 0))
plot(wrld_simpl, ylim = c(20, 80), axes = FALSE)
points(lsNA, pch = 16, col = 'red', cex = 0.5)
points(lsEA, pch = 16, col = 'darkgreen', cex = 0.5)

For the Niche Quantification, we need to have a matrix with the background environment present in the native and invasive ranges, as well as the complete global environmental including the combined extent of the native and introduced environments. After cropping, we use getValues to convert the raster to a dataframe.

## Crop Climate Layers:
naEnvR <- crop(wclim, naExt)
eaEnvR <- crop(wclim, eaExt)

## Extract values to matrix:
naEnvM <- getValues(naEnvR)
eaEnvM <- getValues(eaEnvR)

## Clean out missing values:
naEnvM <- naEnvM[complete.cases(naEnvM), ]
eaEnvM <- eaEnvM[complete.cases(eaEnvM), ]

## Combined global environment:
globalEnvM <- rbind(naEnvM, eaEnvM)

Niche Quantification

PCA

pca.clim <- dudi.pca(globalEnvM, center = TRUE,
                    scale = TRUE, scannf = FALSE, nf = 2)
global.scores <- pca.clim$li

nativeLS.scores <-
  suprow(pca.clim,
         data.frame(lsEA)[, colnames(globalEnvM)])$li   
invasiveLS.scores <-
  suprow(pca.clim,
         data.frame(lsNA)[, colnames(globalEnvM)])$li

nativeEnv.scores <- suprow(pca.clim, naEnvM)$li
invasiveEnv.scores <- suprow(pca.clim, eaEnvM)$li

data.frame(lsEA)[, colnames(globalEnvM)]

The output of dudi.pca and suprow includes a lot of information that we aren’t using here. We only need the li element, so I’ve selected that from each of the function outputs.

Occurence Densities Grid

nativeGrid <- ecospat.grid.clim.dyn(global.scores,
                                   nativeEnv.scores,
                                   nativeLS.scores)

invasiveGrid <- ecospat.grid.clim.dyn(global.scores,
                                   invasiveEnv.scores, 
                                   invasiveLS.scores)

ecospat.plot.niche.dyn(nativeGrid, invasiveGrid,
                       quant = 0.05)

Geographic Comparisons

To demonstrate, let’s compare the distribution of Lythrum salicaria in North America before and after 1950. We use geographic coordinates here, so no need for a PCA. We do need to generate the ‘background’ coordinates. I’ll use expand.grid to create the locations for this. I’ve broken up the NA extent into 500 x 500 grids.

lsNAearly <- subset(lsNA, year <= 1950)
lsNAlate <- subset(lsNA, year > 1950)
geoGrid <- expand.grid(longitude =
                        seq(-160, -40, length.out = 500),
                      latitude =
                        seq(20, 90, length.out = 500))

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     coordinates(lsNAearly))

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    coordinates(lsNAlate))

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)
plot(wrld_simpl, add = TRUE)

naMask <- bind(subset(wrld_simpl, NAME == "Canada"),
              subset(wrld_simpl, NAME == "United States"),
              subset(wrld_simpl, NAME == "Mexico"))

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     coordinates(lsNAearly),
                                     geomask = naMask)

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    coordinates(lsNAlate),
                                    geomask = naMask)

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)
plot(wrld_simpl, add = TRUE)

That gives more reasonable results.

Summary

References

Bivand, Roger, and Nicholas Lewin-Koh. 2021. “Maptools: Tools for Handling Spatial Objects.” Manual. https://CRAN.R-project.org/package=maptools.

Hijmans, Robert J. 2021. “Raster: Geographic Data Analysis and Modeling.” Manual. https://CRAN.R-project.org/package=raster.

GBS Admixture Analysis Workflow

Tue, 01 Jun 2021 00:00:00 +0000

Admixture is a program for completing STRUCTURE-style analyses of large SNP datasets, such as we get with GBS (Elshire et al. 2011). This short tutorial covers getting our SNP data from STACKS (Rochette, Rivera‐Colón, and Catchen 2019) into a format that Admixture will understand, running the analysis, and importing the results into R for further investigation & plotting.

Converting Stacks Output to Admixture Input

Both Stacks and Admixture can process PLINK data. However, there are a few ‘gotchas’ that took a while to sort out. The simplest way I found to bridge the two programs was to export my Stacks data to vcf, clean it up on the command line, and then use the plink program to convert it to a plink file that Admixture could parse.

I’ll start from the vcf file generated by Stacks’ populations program. We expect to have thousands of contigs in a typical GBS dataset, and each of which is numbered in the Stacks output:

head -20 populations.haps.vcf

##fileformat=VCFv4.2
##fileDate=20210531
##source="Stacks v2.3e"
##INFO=<ID=AD,Number=R,Type=Integer,Description="Total Depth for Each Allele">
...
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##INFO=<ID=loc_strand,Number=1,Type=Character,Description="Genomic strand the co
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NIAP1-1011  NIAP1-0342  
26  1   .   TTTCG   TATCG   .   PASS    snp_columns=61,93,136,137,177   GT  0/0 0/0 
46  1   .   AAAGTT  AACATT  .   PASS    snp_columns=13,15,48,73,125,192 GT  0/1 0/1 
103 1   .   CCCACATGATACGCCGC   CCCATATAAGCCGCCGC   .   PASS    snp_columns=11,16   
149 1   .   ACACTGT ACATTGT .   PASS    snp_columns=47,66,84,91,121,126,163 GT  0/0 
271 1   .   CATTGTGCGGAATATGT   TACCTCATTTGCATTAC,CATTGTGCGGAAAATGT .   PASS

This causes a problem when we try to use plink, which won’t accept #CHROM values higher than 21. To fix this, we need to append a letter to the #CHROM numbers:

sed '/^[[:digit:]]/s/^/c/' populations.haps.vcf > popC.haps.vcf

head -20 mingan.haps.vcf

##fileformat=VCFv4.2
##fileDate=20210531
##source="Stacks v2.3e"
##INFO=<ID=AD,Number=R,Type=Integer,Description="Total Depth for Each Allele">
...
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##INFO=<ID=loc_strand,Number=1,Type=Character,Description="Genomic strand the co
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NIAP1-1011  NIAP1-0342  
c26 1   .   TTTCG   TATCG   .   PASS    snp_columns=61,93,136,137,177   GT  0/0 
c46 1   .   AAAGTT  AACATT  .   PASS    snp_columns=13,15,48,73,125,192 GT  0/1 
c103    1   .   CCCACATGATACGCCGC   CCCATATAAGCCGCCGC   .   PASS
c149    1   .   ACACTGT ACATTGT .   PASS    snp_columns=47,66,84,91,121,126,163 GT
c271    1   .   CATTGTGCGGAATATGT   TACCTCATTTGCATTAC,CATTGTGCGGAAAATGT .   PASS

With that small addition, we can now create a plink file with the plink program:

plink --vcf popC.haps.vcf --make-bed --out pop.admix --allow-extra-chr 0

This generates a few files: pop.admix.bed, pop.admix.bim, pop.admix.fam, pop.admix.log, pop.admix.nosex.

Running Admixture

We can now run admixture itself. We need all the files generated by plink together in the same directory. We’ll pass the .bed file as an argument to Admixture, but it will look for the other files when it’s running.

The other key argument for admixture is the number of clusters to look for. We most likely will want to try a range of different values, in order to determine the optimal number (if there is one). We can do this in bash with a loop:

for K in `seq -w 1 20` 
do
    admixture --cv pop.admix.bed $K > ktests/k${K}.out
done

The seq command generates a sequence of numbers, and the -w flag tells it to pad the numbers with zeros (i.e., 01, 02, … 19, 20).

The --cv flag tells admixture to calculate cross-validation error rates, which we will use to determine the optimal K value.

We direct the output to files in the directory ktests. Make sure this directory exists before you start.

Once the loop is finished, we’ll want to examine the results:

grep -h CV ktests/*out

CV error (K=1): 0.39835
CV error (K=2): 0.31327
CV error (K=3): 0.26516
CV error (K=4): 0.19929
CV error (K=5): 0.18499
...

Now we’re ready to move into R to explore the results.

R Plotting

First, we’ll take a look at the CV values. Since they’re scattered in 20 different log files, we’ll use grep to collect them into a single file:

grep -h CV ktests/*out > CV.csv

We can load that into R:

CVs <- read.table("CV.csv", sep = " ")
CVs <- CVs[, 3:4] ## drop the first two columns
## Remove the formatting around the K values:
CVs[, 1] <- gsub(x = CVs[, 1], pattern = "\\(K=",
                replacement = "")
CVs[, 1] <- gsub(x = CVs[, 1], pattern = "\\):",
                replacement = "") 
head(CVs)

##   V3      V4
## 1  1 0.39835
## 2  2 0.31327
## 3  3 0.26516
## 4  4 0.19929
## 5  5 0.18499
## 6  6 0.15408

plot(CVs, xlab = "K", ylab = "CV error")

In our case, there isn’t a real clear optimum. K = 9 is about the bottom of the ‘elbow,’ we’ll use that to make our plot.

ad9 <- read.table("pop.admix.9.Q")
head(ad9)

##         V1    V2    V3    V4    V5       V6    V7    V8    V9
## 1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 2 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 3 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 4 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 5 0.020244 1e-05 1e-05 1e-05 1e-05 0.979686 1e-05 1e-05 1e-05
## 6 0.003441 1e-05 1e-05 1e-05 1e-05 0.996489 1e-05 1e-05 1e-05

We also need a popmap file to annotate our plot. This file lists the population of every sample, and critically, it must be in the same order as the rows in pop.admix.9.Q. In this case, I’ll use the popmap data that I used with the populations program to generate the original vcf files we started wtih, with an additional column added with the population names in it.

popmap <- read.table("popmap.csv")
head(popmap)

##       sample popnum    popname
## 1 NIAP1-1011      1 Niapsikau1
## 2 NIAP1-0342      1 Niapsikau1
## 3 NIAP1-1004      1 Niapsikau1
## 4 NIAP1-1017      1 Niapsikau1
## 5 NIAP1-1014      1 Niapsikau1
## 6 NIAP1-0923      1 Niapsikau1

At this point, the two tables are in the same order. Before I do any manipulations, I’ll combine them. This allows me to sort them in any order, and the names will stay associated with the correct samples.

ad9 <- cbind(popmap, ad9)
head(ad9)

##       sample popnum    popname       V1    V2    V3    V4    V5       V6    V7
## 1 NIAP1-1011      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 2 NIAP1-0342      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 3 NIAP1-1004      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 4 NIAP1-1017      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 5 NIAP1-1014      1 Niapsikau1 0.020244 1e-05 1e-05 1e-05 1e-05 0.979686 1e-05
## 6 NIAP1-0923      1 Niapsikau1 0.003441 1e-05 1e-05 1e-05 1e-05 0.996489 1e-05
##      V8    V9
## 1 1e-05 1e-05
## 2 1e-05 1e-05
## 3 1e-05 1e-05
## 4 1e-05 1e-05
## 5 1e-05 1e-05
## 6 1e-05 1e-05

In my case, the samples aren’t in order, so I need to sort them prior to plotting:

ad9 <- ad9[order(ad9$popnum), ]

Now we’re ready to plot:

barplot(t(as.matrix(ad9[, -1:-3])), col=rainbow(9), 
        space = 0, xlab="Population", ylab = "Ancestry", 
        border=NA, axisnames = FALSE)

Let’s break that down. First, we excluded the first three columns, ad9[, -1:-3], so that we don’t include our labels in the data. Then we transpose the matrix, so that each individual sample is represented by a column of Ancestry proportions. I removed the borders on the bars, and the spaces between them (border = NA, and space = 0), and set the axis labels.

That’s nice, but we’d like to label our original populations on the plot, so we can see how they compare to the clusters produced by admixture. I use the aggregate function for this.

xlabels <- aggregate(1:nrow(ad9),
                    by = list(ad9[, "popname"]),
                    FUN = mean)
xlabels

##      Group.1     x
## 1   Fantome4  58.0
## 2   Fantome5  83.5
## 3     Havre6 107.5
## 4     Havre7 125.5
## 5  Marteau11 149.0
## 6 Niapsikau1   7.0
## 7 Niapsikau2  30.5

Here, I’ve grouped the rows by population name, and then calculated the mean row number for each group. That will be handy, as I can then use that mean value to plot the name of each population centered beneath it.

Similarly, I can find the borders of the groups:

sampleEdges <- aggregate(1:nrow(ad9),
                        by = list(ad9[, "popname"]), 
                        FUN = max)
sampleEdges

##      Group.1   x
## 1   Fantome4  68
## 2   Fantome5  98
## 3     Havre6 116
## 4     Havre7 134
## 5  Marteau11 163
## 6 Niapsikau1  13
## 7 Niapsikau2  47

In this case, I find the highest row for each population, which I’ll use to draw a line between them.

Putting this all together, we get:

barplot(t(as.matrix(ad9[, -1:-3])), col=rainbow(9), 
        space = 0, xlab="Population", ylab = "Ancestry", 
        border=NA, axisnames = FALSE)
abline(v = sampleEdges$x, lwd = 2)
axis(1, at = xlabels$x - 0.5, labels = xlabels$Group.1)

And we’re done!

References

Elshire, Robert J., Jeffrey C. Glaubitz, Qi Sun, Jesse A. Poland, Ken Kawamoto, Edward S. Buckler, and Sharon E. Mitchell. 2011. “A Robust, Simple Genotyping-by-Sequencing (GBS) Approach for High Diversity Species.” PLoS ONE 6 (5). https://doi.org/10.1371/journal.pone.0019379.

Rochette, Nicolas C., Angel G. Rivera‐Colón, and Julian M. Catchen. 2019. “Stacks 2: Analytical Methods for Paired-End Sequencing Improve RADseq-Based Population Genomics.” Molecular Ecology 28 (21): 4737–54. https://doi.org/https://doi.org/10.1111/mec.15253.

Adding Lat/Lon Grids to Maps in R

Mon, 22 Feb 2021 00:00:00 +0000

In a previous post, I outlined my workflow for preparing maps in R. Today I had to add a graticule, a grid of latitude and longitude lines, to my maps. That’s easy enough to do with unprojected maps, as the plot coordinates are latitude and longitude, so your X and Y axes are already graticules. But if you’ve projected your data, the plot coordinates are on a different scale, so you need to do a bit of tuning.

I couldn’t find a direct way to do this in the R sp package. However, sp (sp for ‘spatial’) is slowly being replaced by sf (sf for simple feature), and sf does support graticules. Here are the steps required to add them to your plots:

Importing Data

We can use raster::getData to get our map data again. It’s straightforward to convert objects from sp (Spatial*) and sf (sf*) format and back, with the functions st_as_sf (to convert from a Spatial* to sf*), and as (to convert from sf* to a Spatial* object). As it turns out, getData also supports downloading data directly into sf format:

library(sf)
library(raster)
us <- getData("GADM", country = "USA", level = 1,
             path = "./data/maps/", type = "sf")
canada <- getData("GADM", country = "CAN", level = 1,
                 path = "./data/maps", type = "sf")
mexico <- getData("GADM", country = "MEX", level = 1,
                 path = "./data/maps", type = "sf")

This uses the undocumented type argument, set to sf. Given that it’s not documented, it may change in future, be warned!

You can also use the function st_read to read shapefiles directly:

greatlakes <- st_read("data/maps/greatlakes.shp")

## Reading layer `greatlakes' from data source 
##   `/home/smithty/blogdown/content/tutorials/data/maps/greatlakes.shp' 
##   using driver `ESRI Shapefile'
## Simple feature collection with 2 features and 5 fields
## Geometry type: POLYGON
## Dimension:     XY
## Bounding box:  xmin: -365240.6 ymin: -2741892 xmax: 1888977 ymax: 509590.2
## proj4string:   +proj=laea +lat_0=45 +lon_0=-100 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs

In the previous tutorial, I used bind to combine two Spatial* objects. With sf we need rbind instead:

na <- rbind(us, canada, mexico)

## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()

Plotting complex vector maps like this can be a slow process, especially when you’re constantly tweaking and adjusting them. You can speed this up by simplifying the layers:

na.simp <- st_simplify(na, dTolerance = 0.01)

On my laptop, plotting the original map takes a minute or more, compared to 2 seconds for the simplified vector. I set the tolerance by trial and error. The higher the tolerance, the smoother the map will be. At 0.01, it still looks nearly identical at the scale I’m plotting it, but is much smaller and faster to plot. sf does warn me about not correctly simplifying the data, but since I’m only using this for display that’s not a concern. I wouldn’t simplify a vector if I was going to use it in an analysis.

Plotting Maps

When it comes to plotting, we need to tell R to plot only the geometry. By default it will plot multiple maps, one for each attribute. That’s not what we want here.

plot(st_geometry(na.simp), xlim = c(-130, -70),
     ylim = c(35, 45))

Projections

To project our unprojected data, we need to define a projection, and transform the object.

laea = CRS("+proj=laea +lat_0=30 +lon_0=-95")

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

na.la <- st_transform(na.simp, laea)
plot(st_geometry(na.la), xlim = c(-500000, 2000000),
     ylim = c(-400000, 2100000))

We can add layers just as we did in the previous post:

gl.la <- st_transform(greatlakes, laea)
plot(st_geometry(gl.la), col = 'lightblue', add = TRUE)

You can also mix sf and Spatial* objects on the same plot, as long as they’re in the same projection.

Graticules

Now we have everything we need to add graticules to our map. This includes the map we want to plot, and the CRS data for the graticules we want to overlay. In our case, we’ll use the original, unprojected layer as the source our CRS:

plot(st_geometry(na.la),
     xlim = c(-500000, 2000000), ylim = c(-400000, 2100000),
     graticule = st_crs(na.simp),
     bgc = 'lightblue', ## Background color for the ocean
     col = 'white',
     axes = TRUE)
plot(st_geometry(gl.la), col = 'lightblue', add = TRUE)

If you want to specify the location of the graticules, you can use the arguments lat and lon to specify where you want them.

Emacs for Bioinformatics #3: R and ESS

Wed, 30 Dec 2020 00:00:00 +0000

This is part three in my series of Emacs tutorials aimed at bioinformatics (and other scientific analysis) workflows. See the rest on my tutorials page.

Emacs support for the R programming language is provided by the ESS package (AKA, “Emacs Speaks Statistics”). ESS has been around since at least 1994, and is supported by a very active development team. It provides most or all of the features of the more widely-known RStudio, as well as a great many more. Like all things Emacs, if it doesn’t have a feature you want, it’s likely someone else has written a package that provides it; failing that, the motivated hacker you can create their own customizations using the built-in scripting language, elisp.

However, lets not let all that potential scare us off. Getting up and running with ESS doesn’t require much effort at all.

Installation

Prerequisites

You need to have R installed in order to use ESS!

Installing ESS from MELPA

The easiest way to install it is to use the MELPA package repository. MELPA hosts Emacs packages provided by hackers who are not part of the Emacs development team. (“packages” here has roughly the same meaning as “plugins” or “extensions” in other software systems).

If you aren’t already using MELPA, you need to add it to your configuration file (typically ~/.emacs.d/init.el, or ~/.emacs):

(require 'package)
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

Once this code is evaluated, you can view the complete list of packages available on MELPA via M-x package-list-packages. It’s a big list, and it will take a few seconds for Emacs to get the latest version from the server (you need an internet connection for this).

Search down to the entry for ESS, select it by pressing i, and then install it by pressing x. ESS is one of the larger packages, so it may take a few seconds to download and install all the files.

Once this is done, you have ESS, and don’t need to return to package-list-packages until you want to update to a new version (or add some other packages).

Getting Started

ESS comes with a comprehensive manual. That will be your canonical reference for learning about this package. However, you can get started with just a few commands.

Interactive R Session

From within Emacs, start R with the command M-x R. You will be prompted for the project starting directory. Select whatever you like and press enter. You will then be presented with an R shell:

You can enter code and view results here, just as you would in the terminal in RStudio, or with R running on the command line. ESS uses the same code to manage this as for shell mode. That means we can use the same keybindings here:

<tab>: with the cursor at the active prompt, tab will complete function and variable names, as well as the arguments for functions
<enter>: with the cursor at the active prompt, send the command on the prompt to R for evaluation
<enter>: With the cursor on a previous command, re-enter that command
M-p and M-n: Move through your command history at the active prompt
C-c C-p and C-c C-n: Move the cursor to the previous and next prompts
C-c <enter>: With the cursor on a previous command, copy that command the to active prompt, but don’t enter it. This allows you to edit a previous command before sending a new variation to R for evaluation
C-c C-o: Delete the output from the previous command
C-c C-v: Opens a prompt to select a help file, which will be displayed in Emacs (you can also open help files from the prompt via ?<function>)

These few commands cover most general interactions. There are a lot more features available. Check the iESS menu item on the toolbar for some of them; see the manual for the details.

Plots

Calling plotting commands will create a new window (frame) for your figure. There isn’t a dedicated pane in Emacs to display them, like in RStudio, and you can’t scroll forward and backward through your history of images. You can, however, create multiple image windows, and view them side by side.

To create and manipulate new image windows, you’ll need the following commands:

dev.new() ## Create new plot window, and make it the
          ## active window
dev.set() ## If more than one plot window is open,
          ## set the next window to be the active
          ##window

See ?dev page for more details.

Writing R Scripts

After ESS is installed, anytime you open a file with a .r or .R extension, it will be in ESS[R] mode. You can enter text as usual, and additionally have the following helpful commands available:

C-c C-n or C-c <enter>: send the current line to the R process and step to the next
C-c C-r: send the current region to the R process
C-c C-f: send the current function to the R process
C-c C-c: send the current region, paragraph, or function to the R process
C-c C-b: send the entire buffer to the R process
C-c C-v: prompt for a help file to open
M-tab: tab completion of objects (functions, variables, file names) and function arguments

If there is no R process running when you try to send code, you will be prompted for a working directory in which to start a new process. In addition, you can manage R processes with the following commands:

C-c C-z: switch from the script buffer to the process buffer (and vice versa)
C-c C-s: change the process linked to the current script buffer (e.g., if you want to run multiple R processes at once, with different scripts in each process)

Next Steps

This may well be all you need, and if that’s the case, you’re all done. However, there is a lot more available to you, including support for writing documentation, package development, managing git repositories, editing on remote servers, and more.

My advice is to start slowly. The pointers on this page will get you up and running. When you find yourself repeating something tedious multiple times, it may be time to investigate if there’s a shortcut available to make your life easier. I recommend skimming the manual, to get a sense of all that’s available, and if something catches your eye see about incorporating it into your workflow.

Plotting Simple Maps in R

Fri, 30 Oct 2020 00:00:00 +0000

NOTE: This tutorial uses older R packages that are scheduled to be deprecated at the end of 2023. I have updated this tutorial using the new packages. Unless you need to use older code, you should use the new Terra-based approach instead of this!

Reference

See the RSpatial tutorial for a more detailed introduction/overview of using R for GIS/spatial analysis. The following tutorial walks through some common plotting tasks I use for distribution models.

Basemaps

The raster package provides the function getData, which is a handy way to download basemaps for plotting. (You can also use it to get WorldClim data, see the man page). The first time you call it, it will download the requested maps from the internet. It will save the data in your working directory, or in a location specified with the path argument. The next time you request the same map from getData, if it finds it in the local directory it will load it from there, rather than downloading it again.

library(raster)

## Loading required package: sp

## Loading required package: methods

## Warning: multiple methods tables found for 'metadata'

us <- getData("GADM", country = "USA", level = 1,
             path = "./data/maps/")

## Warning in getData("GADM", country = "USA", level = 1, path = "./data/maps/"): getData will be removed in a future version of raster
## . Please use the geodata package instead

canada <- getData("GADM", country = "CAN", level = 1,
                 path = "./data/maps")

## Warning in getData("GADM", country = "CAN", level = 1, path = "./data/maps"): getData will be removed in a future version of raster
## . Please use the geodata package instead

These maps can be plotted directly with the plot command. If you want to combine them, use the add = TRUE argument to the second plot call:

plot(us)
plot(canada, add = TRUE)

You can combine multiple vector maps into a single map with bind:

na <- bind(us, canada)

These maps are ‘unprojected’, meaning they are plotted in latitude/longitude degrees. That makes it easy to set the plot boundaries:

plot(na, xlim = c(-100, -50), ylim = c(30, 60))

It’s handy to have a shapefile of the Great Lakes, for making prettier maps. I created this one in QGIS and use it for plotting:

greatlakes <- shapefile("data/maps/greatlakes.shp")

Adding Data

You can add points to the plot like a regular scatter plot:

library(scales)  ## for the alpha function below
gbif <- read.table("data/trich-gbif.csv")
## Set the line color to gray to focus on the data points:
plot(na, xlim = c(-100, -50), ylim = c(30, 60),
     border = "gray")
points(gbif$X, gbif$Y, pch = 16,
       col = alpha("green", 0.2))

You can also convert your points to a spatial points object, in which case R will know which columns to use for plotting. This is also necessary before we can project our data (see below).

coordinates(gbif) <- ~X+Y
## plot(na, xlim = c(-100, -50), ylim = c(30, 60),
##      border = "gray")
## points(gbif, pch = 16, col = alpha("green", 0.2))

Rasters

Similarly, you can plot rasters with plot:

trichPreds <- raster("./data/trichPreds")
plot(trichPreds, xlim = c(-100, -50), ylim = c(30, 60))
plot(na, border = "gray", lwd = 0.5, add = TRUE)

trichPredsTrim <- trichPreds
trichPredsTrim[trichPredsTrim <
               quantile(getValues(trichPreds),
                        probs = 0.75, na.rm = TRUE)] <- NA
plot(trichPredsTrim, xlim = c(-100, -50), ylim = c(30, 60))
plot(na, border = "grey", add = TRUE)

You could also use an absolute value here, but then you’d need to know the actual distribution of the suitability scores. quantile is easier to tweak.

Projections

Lat/Lon maps look a bit square; we’re more used to seeing maps projected. A common projection for Canada is Lambert Conformal Conic. We can transform our data to this projection to make nicer maps:

## define the projection
canlam <- CRS("+proj=lcc +lat_1=49 +lat_2=77 +lat_0=49 +lon_0=-95 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs")

## project our vector data:
na.lcc <- spTransform(na, canlam)
gl.lcc <- spTransform(greatlakes, canlam)

## Warning: PROJ support is provided by the sf and terra packages among others

## We already convereted gbif to spatial points object above!
## Now we to set the projection of our points:
crs(gbif) <- CRS("+proj=longlat +datum=WGS84")

## Finally, we can project our points to LCC:
gbif.lcc <- spTransform(gbif, canlam)

Note that our data needs to be in an object of class Spatial*, and it must have a defined coordinate reference system (CRS) before we can project it to a new CRS. Setting the coordinates of our points via the coordinates function creates a Spatial* object. The crs function allows us to explicitly set the projection. We need to know the EPSG code for the projection to use this. The function make_EPSG in the package rgdal is helpful for finding this information. See the RSpatial tutorial for details.

There are a few more steps for raster layers:

rasterLCC <- projectExtent(trichPredsTrim, canlam)
res(rasterLCC) <- 10000 ## set the cell size to 10km
predLCC <- projectRaster(trichPredsTrim, rasterLCC)

Note that I set the resolution to 10km here. That’s the size of the raster cells. The original raster cells, in the lat/lon projection, were at 30 second resolution, which is about 1km. I could have set a smaller cell size here. However, since I’m only using this map for visualization, 10km is plenty big enough for my plot, and will run faster (and take less memory) than a map with 1km cell size.

Now we can plot our data in the Lambert Conformal Projection:

plot(predLCC)
plot(na.lcc, border = "grey", add = TRUE)

The units are no longer Lat/Lon, but meters. We can read them off the plot to improve the zoom:

plot(predLCC, xlim = c(0, 2500000),
     ylim = c(-1500000, -400000))
plot(na.lcc, border = "grey", add = TRUE)

Formatting

With the data plotted, we can then turn to making the map a little prettier:

## Make a panel with two plots, set the right margin tight:
par(mar = c(0.1,0.1,0.1,0), mfrow = c(1, 2))

## store the plot limits:
my_xlims <- c(0, 2500000) 
my_ylims <- c(-1300000, -200000)

## Plot the points:
plot(na.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", bg = "lightblue", col = "white")
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
points(gbif.lcc, pch = 16, col = alpha("grey30", 0.2),
       cex = 0.7)
box() 

## tighten up the left margin:
par(mar = c(0.1,0,0.1,0.1))
plot(na.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", bg = "lightblue", col = "white")
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
plot(predLCC, add = TRUE, legend = FALSE)

## plotted again to put the border lines on top:
plot(na.lcc, border = "grey", add = TRUE) 
box()

Medium Performance Cluster Computing

Tue, 19 Aug 2014 00:00:00 +0000

I recently ran into a crunch getting some memory-intensive GIS analysis completed. My work laptop has 2 CPUs and 4GB RAM, and running one instance of the GRASS GIS r.horizon command on a 16GB map was gobbling up 8GB of virtual RAM, which temporarily ground my machine to a crawl before the process was killed.

GRASS is not yet installed on the high performance cluster at work, so I decided to try setting up my own medium performance cluster on a Digital Ocean¹ VPS (which they refer to as ‘droplets’).

Why?

Price: Not only are their prices competitively low, they charge by the hour. This means you can have your own virtual machine with 64GB RAM and 20 CPUs for less than $1/hour. What’s more, the minimum period is one hour. Meaning, in theory, you could spin up a droplet, get 20 hours of processing done, and then shut it all down again with no further commitment.
Quick & Easy Installation: They offer a variety of GNU/Linux distributions, and once you’ve chosen you’ll be able to log on to your VPS in 60 seconds. This assumes you’re comfortable working on the command line. But if you’re doing cluster computing, if that’s not already true you’ll need to learn anyways.
No Queues: No need to worry about submitting batch jobs to a queue or waiting your turn. It’s your VPS, not shared with anyone else.
Reuse Your Installation: You pay for the time your droplet is available. However, you can save a ‘snapshot’, which is stored in your account. This allows you to destroy your droplet when you don’t need it. Then, simply reload it from the saved snapshot when you next need to crunch some numbers.
Nerdy fun: I have to admit, I was motivated in part by sheer, unbridled nerdy curiosity. Who wouldn’t want to ssh in to their very own server?

Why not?

This won’t be a practical solution in all cases. The longer your job will take to run, the more practical a real cluster becomes. It’s also worth noting that the CPUs aren’t particularly high-powered. So processes won’t run faster than on a recent laptop, assuming memory isn’t limiting. Another thing to consider is how much data you have to upload. This is not a viable approach for true ‘big data’ projects! Sending gigabytes over the open internet can be a very slow process, which is another point in favour of using a local HPC cluster.

How-To

With that in mind, here’s how I set up my temporary cluster:

Purchase the droplet

Browse over to Digital Ocean and sign up. Once you’re logged in, click ‘create’ and fill in your details:

Hostname: it’s your server, call it what you like
Size: they offer everything from 512MB/1 CPU up to 64GB/20 CPUs, with prices varying accordingly. From my project I selected 32GB/12 CPUs for $0.476/hour
Region: pick something close, particularly if you’ll be up/downloading a lot of data. In my case, that’s New York.
Linux Distribution: Choices include Ubuntu, Fedora, Debian, CentOS. I picked Debian, as that’s been my OS for the past decade. Regardless of the distribution, if you’re going to be working with large files, you will definitely want to select the 64bit version of your OS.

Security

Your root password and dedicated IP address will be emailed to you. Which means the NSA will know it before you do. So immediately log in and change your password. Actually, as soon as you log in you will be required to change your password in any case.

ssh root@123.45.67.89

You’ll probably also want a regular-strength user for non-administrative work, so add that next:

adduser tyler

At this point, you can log in as root or as your regular user. A more secure option is to authenticate via rsa keys. If you haven’t done this before, generate the key on your laptop/local computer:

ssh-keygen -t rsa

Note that you can use a blank passphrase here. Doing so will allow you to log in to the server without entering a password from now on. It also means that anyone that has physical access to your laptop also be able to log in to the server without a password. If you’ve lost control of your laptop, this is likely the least of your worries.

Next, transfer the key to the server:

ssh-copy-id tyler@123.45.67.89

You’ll be asked for your password again here. Now try logging into the server again:

ssh tyler@123.45.67.89

If everything is working correctly, you should be logged in to your droplet without entering a password. If that is the case, we can proceed to shore up our security. su to root user, and edit the ssh config files:

su
nano /etc/ssh/sshd_config

Look for and modify the following lines, then save the file. You need to remove the comment character (#)from the beginning of the line, if it’s there, and make sure they say ‘no’, not ‘yes’. You don’t need to modify any other lines.

PermitRootLogin no
PasswordAuthentication no

The second line will prevent anyone from logging in using a password - you’ll only be able to login if you have the correct RSA key on your computer. This prevents ne’er-do-wells from trying to crack your password.

The first line will prevent anyone (including you!) from logging in directly as root. Meaning a potential cracker will have to get your RSA key in order to log on to the machine, and then they’ll have to crack the root password in order to do anything really nasty.

Note that if you want to access the server from another computer, you’ll have to log in from each computer via password, or at least ssh-copy-id the RSA key, before your set PasswordAuthentication no. Or, afterwards, simply set it back to PasswordAuthentication yes briefly from the first computer long enough for the second computer to log on and ssh-copy-id their RSA key.

One final configuration detail: if you’re going to use an X server (to view graphical windows of any kind), you need to modify /etc/ssh/ssh_config. Make sure it includes the following uncommented line:

ForwardX11 yes

Now you need to reload the modified configuration. Still as root:

/etc/init.d/ssh reload

Given that we won’t have any outward facing servers on this machine, that should do for our security for now. If you do want to put some servers on here, you’ll definitely want to look into getting at least a firewall, and probably some intrusion detection software on here. The DigitalOcean tutorials are quite good in this area.

Install Software

Now that we have a passably secured machine, it’s time to install the software you’ll want. Given this is Debian:

run aptitude as root
update
install any security updates
select and install your desired programs

In my case, to run GRASS, I needed the following:

aptitude install grass emacs screen htop avce00 \
  e00compr git mercurial xorg

(yes, you can run aptitude from the command line just like apt-get if you like!)

Setup GRASS

GRASS will require you to create your database directory on the server (not as root!):

mkdir ~/grassdata

Next you need to transfer any files you need from your local machine:

rsync -az --progress --compresslevel=9 --partial \
  grassdata/location tyler@123.45.67.89:grassdata/

The options here include a, which among other things will transfer directories recursively, z, which will compress files prior to transfer (which dramatically reduces upload times), compresslevel=9, which uses the greatest amount of compression, and partial which allows rsync to pick up where it left off in case the connection is interrupted.

Running GRASS Inside Screen

Finally, we begin. Now we’re ready to use X windows, so when you log back in you’ll want to use the X and C flags to ssh:

ssh -XC tyler@123.45.67.89

After logging back into your server, run screen. You’ll be transported from a regular terminal window into a screen window. It will look almost exactly the same. But it gives you some super powers, as we’ll see shortly.

Next, start GRASS. In order to conveniently run multiple processes at once, I prefer to use text mode, hence I start with: grass -text. Navigate through the charmingly archaic text windows until you’re at the familiar GRASS text prompt.

One very helpful thing I discovered about GRASS is that each command is really a stand-alone program. Which has the lovely side-effect of giving us quick access to parallel programming. So long as any command foo does not require the output of command bar to run, and vice versa, you can run them both concurrently. Which means, in my case, I can do things like this:

r.slope.aspect elevin=dem slope=myslope aspect=myaspect &
r.horizon elevin=elevation horizonstep=30 \
    bufferzone=200 horizon=horangle \
    maxdistance=2000 &

You might find yourself getting carried away, starting process after process. You might want to check on the load your server is under, before you max out your RAM or CPUs. Here is where screen comes in handy.

Type Ctrl-a c, and you’ll have a new terminal window to work in. Your GRASS session is still working away in the background, and you can check on how many processes you’ve spun out by calling htop. To go back to the GRASS session, Ctrl-a " brings up a list of all the windows available inside your screen instance, which you can select from with the arrow keys.

Finally, you may need to shut down your laptop at some point while the GRASS session is still running. To do this, we detach the screen session, with Ctrl-a d. This tucks the session away out of sight, but it continues to run. It will continue to run even after we log out of the server. When you want to reconnect to the session, simply enter screen -r at the command line and you’re back in charge.

screen can do a lot more than this, check out the docs for details!

Goodbye, but not Farewell

When you’ve completed all the work you need to do, you can save a snapshot of your server to use later on. Log in to your Digital Ocean account, select your droplet, and follow the links to create a snapshot. Once that’s done, you can safely destroy the droplet. A destroyed droplet will no longer accrue charges (and obviously it won’t be doing any processing either). To reinstate your droplet, follow the same steps you used above to create a droplet, but instead of selecting a Linux Distribution, select your snapshot from the My Images tab.

Comments or questions? Ping me on mastodon, or send me an email (address in sidebar)

Full disclaimer: this link uses my referral code, so if you sign up through here I’ll get a small kick-back from Digital Ocean. I hope this won’t lower your confidence in what some random guy on the internet has written. ↩︎

Publication Quality R Figures

Wed, 19 Feb 2014 00:00:00 +0000

Figure 1: A. Iris Sepal Size by Species. B. Iris Petal Width

Introduction

Learning Objectives

At the end of this lesson, you should be able to:

Customize plots produced with the R base graphics system
Design multi-panel plots
Design plots to suit the publication requirements of a journal
Save your plots as high-resolution raster or vector image files as required by your publisher

Pre-requisites

You will need:

A recent version of R installed on your computer
Familiarity editing R scripts and passing commands from a script to the R interpreter
Note that RStudio is not ideal for this lesson, due to limitations in how it processes plotting commands; the default RGui installed on Windows or Mac will work better
These notes!
Optionally, some of your own data to work with during the exercises

Motivation

You have several options for plotting with R. The simplest is the built-in or base graphics package. Base graphics are less powerful than newer alternatives like lattice or ggplot2. On the other hand, it’s much easier to customize base graphics than the others. For this reason, I prefer to use the built-in functions when preparing single-panel plots.

ggplot2 is definitely worth investigating, especially if you want to produce complex multi-panel faceted plots. The official website has all the documentation. Roger Peng has also posted a very nice introductory video on YouTube.

Building Our Plot

For this example, we’ll use the guidelines provided by the American Journal of Botany. AJB accepts figures 3.5 inches (1 column), 5-6 inches (1.5 columns), or 7.25 inches wide (2 columns). The height can be up to 9 inches. We’ll start with a one-column plot, so the dimensions should be 3.5 inches wide.

The figure we’ll plot is from the built-in iris data set. We’ll do a simple scatterplot of Sepal.Length against Sepal.Width.

Size

Let’s start with a square. If we need more height, we can increase the size as necessary. Similarly, if we decide we need to stretch our figure over two columns, we can change later.

Note that RStudio isn’t the best environment for this exercise. Unfortunately, it’s not possible to create new plot windows in RStudio, so you can’t specify the dimensions of the figure for the on-screen display. Consequently, when you save your figure to an image file, you won’t necessarily get exactly what you see on the screen. In many cases, this may well be fine. But double-check the image file to make sure that you get what you expected. If you didn’t, it may be because the on-screen display and the saved image were not close enough to the same dimensions.

To set up the canvas for our plot, start a new device:

dev.new(height = 3.5, width = 3.5)

If you are using RStudio, dev.new() won’t work. Instead, drag the edges of the plot window to get as close to a 3.5" square as possible.

Content

Now that our canvas is ready, we can start placing our graphics. Let’s start with the default plot.

plot(Sepal.Length ~ Sepal.Width, data = iris)

Plot Symbols

The default plot uses the same symbol for each point. However, our data frame includes samples from three different species:

str(iris)

## 'data.frame':    150 obs. of  5 variables:
##  $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##  $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##  $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##  $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##  $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

Recall that a factor is just a vector of integers, with each integer having it’s own label. We can display the underlying numbers by converting from factor to numeric:

as.numeric(iris$Species)

##   [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
##  [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
##  [75] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [149] 3 3

This is a very useful feature. It means we can set a different plot symbol for each species:

plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris)

Margins

Now we can see what we’re working with, but the layout isn’t ideal. In particular, the plot is very small relative to the size of the figure. We can fix this with the mar parameter. mar takes a vector of four integers, which set the width of the margin on the bottom, left, top and right sides respectively (remember clockwise from the bottom!). These numbers refer to the width of each margin in lines — i.e., the width required for a single line of text. The default is c(5, 4, 4, 2) + 0.1. The top margin in particular is usually too wide. We will very rarely add a title to a published figure, so we don’t need to set aside space for it.

Set the value of mar with the par function.

par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris)

That’s better. But we’ve lost our axis labels. They aren’t actually lost, but they are plotted outside of the margins we’ve set, so they are no longer visible. I find the defaults that R uses for the axes to be larger than we need. Better to turn off the axes entirely and replot them ourselves.

Note that once the margins are set with par(), they will keep their value until we open a new plot window, or reset them.

Axes

par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris, 
     ann = FALSE,      # turn off axis labels
     axes = FALSE)     # turn off axis ticks

Notice that axes = FALSE has turned of the box around our plot. We can put it back easily:

box()

Now we can explicitly add each axis with their size and placement specified:

axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8)

What just happened? Let’s breakdown the arguments to axis:

side:: which side of the plot, clockwise from bottom, same as for mar above
tcl:: length of the ticks. Negative values indicate extending outwards from plot, positive values extend inward. The default is -0.5, which I find a bit too long.
mgp:: margin line, a vector of three numbers, which indicate the position of the axis title, axis labels, and axis line, respectively. The values are the number of lines away from the plot border to place each item, with 0 indicating the margin of the plot area. Note that title doesn’t matter here, since we aren’t using an axis title (yet).
cex.axis:: axis character expansion. Scale the size of the tick labels. < 1 reduces the size, > 1 increases the size.

Now we can add our axis titles back in:

mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)

Here, we can use line to adjust the distance between the label text and the axis.

The finished plot

Putting this altogether gives us the following plot.

par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris, 
     ann = FALSE,      # turn off axis labels
     axes = FALSE)     # turn off axis ticks
box()
axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)

Exercise 1: adding a legend

We now have a complete figure. We could provide an explanation of the symbols in the caption, but it might be nicer to have a legend plotted on the figure. This is easily done with the legend() function.

legend(legend = levels(iris$Species), x="topleft",
       pch = 1:3)

Since we set the pch argument in our plots using the factor iris$Species, we can use levels() function to extract the labels for the legend. pch indicates the actual symbols to use, and x is the location of the legend.

This is clearly not “publication quality.” Our plot needs a bit more space for the legend. See if you can make an attractive plot. The following options might be helpful:

dev.new():: width, height
plot():: xlim, ylim, cex
legend():: x, y, bty, horiz, cex, pt.cex, text.width

If you need a hint, take a look at the next section.

Additional Customization

Selecting Plot Symbols

If you want to select different symbols, it’s easy to do using R’s subsetting syntax. By default, for three levels of our Species factor, we get symbols 1, 2, and 3. If instead we wanted to use symbols¹ 19, 5, and 3, we could do this:

par(mar = c(3, 3, 0.5, 0.5))
mysymbols <- c(19, 5, 3)
plot(Sepal.Length ~ Sepal.Width,
     pch = mysymbols[as.numeric(Species)], data = iris,
     ylim = c(4.0, 8.5), ann = FALSE, axes = FALSE)
box()
axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)
legend(legend = levels(iris$Species), x = "top",
       pch = mysymbols, horiz = TRUE, bty = 'n',
       cex = 0.9, text.width = c(0.6, 0.7, 0.6))

That’s an important application of R’s subsetting commands, so make sure you follow what happened — we subset the mysymbols vector with the longer as.numeric(iris$Species) vector, which converted the values from (1, 2, 3) to (19, 5, 3):

mysymbols

## [1] 19  5  3

as.numeric(iris$Species)

##   [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
##  [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
##  [75] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [149] 3 3

mysymbols[as.numeric(iris$Species)]

##   [1] 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19
##  [26] 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19
##  [51]  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5
##  [76]  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5
## [101]  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3
## [126]  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3

Panels

ggplot2 provides a very sophisticated system for producing multi-panel plots. But it’s easy enough to create a simple panel using the base graphics. For this example, let’s do a two-plot horizontal panel, with our scatter plot in the first position, and a boxplot of petal widths in the second position. A two-column plot in AJB is 7.25 inches wide:

dev.new(width = 7.25, height = 3.5)

Next we need, to inform R that we’re splitting the figure into two panels:

par(mfrow = c(1, 2))

mfrow sets the graphics device for rows and columns, in this case one row, two columns. We can now put our first plot in the first spot:

After dividing a plot device into panels with mfrow, the first high-level plot (i.e., plot, boxplot etc.) command will be placed in the first panel. All subsequent low-level plotting commands (i.e., legend, axis, mtext etc.) will be added to this same panel. When the next high-level command is called, it will be placed in the next panel, and focus shifts with it. So we can now add our boxplot to the second panel:

boxplot(Petal.Width ~ Species, data = iris)

Note that the margins we set for the first panel are still in effect. Consequently, we’ve lost the axis labels on our second plot. It’s going to need some attention to make it look right. We’ll leave that for the next exercise.

In the meantime, we have one more requirement to meet. On multi-figure panels, AJB requires an uppercase letter (A, B, etc) to label each plot. This label should go in the upper-left corner of each panel. This is easy to do with the text command. At the moment, we don’t have space in the upper-left corner, so we’ll put the labels in the lower-right temporarily. For example:

## For the first panel:
text("A", x = 4.2, y = 4.5, cex = 2)

## For the second panel:
text("B", x = 3.2, y = 0.24, cex = 2)

Unfortunately, each figure is plotted on different scales, so placing the letters in the same position is not straightforward. Luckily, R provides a function for getting “universal” coordinates for every plot.

## For the first panel:
text("A", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)

## For the second panel:
text("B", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)

The functions grconvertX and grconvertY convert between different coordinate systems. npc is “normalized plot coordinates.” In this system, (0, 0) is the lower left corner of the plot, and (1, 1) is the upper right corner. user, on the other hand, is the coordinate system in effect for the actual plotted data. Which, for our Panel A, means the lower right corner is ca. (2.0, 4.0) and the upper left corner is ca. (4.5, 8.5). So grconvertX(0.9, from = "npc", to = "user") returns the X coordinate to plot our text 90% of the way to the left side of the plot, regardless of the scale used in that plot. With this addition, we have the following code, and the generated panels:

par(mfrow = c(1, 2))
par(mar = c(3, 3, 0.5, 0.5))
mysymbols <- c(19, 5, 3)
plot(Sepal.Length ~ Sepal.Width,
     pch = mysymbols[as.numeric(Species)], data = iris,
     ylim = c(4.0, 8.5), ann = FALSE, axes = FALSE)
box()
axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)
legend(legend = levels(iris$Species), x = "top",
       pch = mysymbols, horiz = TRUE, bty = 'n',
       cex = 0.9, text.width = c(0.6, 0.7, 0.6))
## For the first panel:
text("A", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)
boxplot(Petal.Width ~ Species, data = iris)
## For the second panel:
text("B", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)

Exercise 2: Completing the Panel

There are still a few problems with our panel:

The title of the Y axis on the second panel is not visible
The panel labels (A and B) are in the wrong positions — they should be in the top left corners
Fixing the panel labels will require moving the legend for the first figure

If you change ylim, you can put the legend on the bottom, and make space for the label at the top. Go ahead and see what you can do with this. You can use my example, Figure 1 at the top of this article as a model. There is a trick to formatting the x-axis of boxplots. When calling the function axis, you’ll have to set the at argument to indicate where to plot the labels (which should be c(1, 2, 3), and you’ll have to set the labels argument to indicate what the labels should be.

Alternatively, try formatting your own data according to the requirements of a journal in your field.

Image Formats

R can save graphics to a variety of formats, including anything your target journal might require. In general, you can store your images in one of two classes of file format:

raster:: images are stored as a matrix of values, with each value indicating the color of a single pixel in the grid. Best used for photographs. Examples: jpg, tiff, png.
vector:: images are stored as a series of mathematical instructions for re-creating the display: lines, polygons, text etc. Best used for line drawings. Examples: eps, svg.

Raster Images

Raster images are stored as a grid of numbers called pixels. Each number records the colour of a single pixel in the image. As a consequence, the image resolution is limited by the number of pixels recorded in the file. In our example, we need a figure 3.5 inches wide. AJB requires a resolution of 1000 dots per inch (DPI) for line drawings, which means we need a source image 3500 pixels in the x and y dimension. We don’t actually need to do these calculations, though, R will handle it for us. We just need to pick a format and set the final resolution.

AJB prefers TIFF format for raster files. To generate one we will use the tiff() function. Note that this function only sets the file details for our plot; we need to add the plotting code after we open the file, and close it when we’re done with dev.off().

tiff(filename = "iris.tiff", width = 3.5, height = 3.5,
     units="in", res = 1000, compression = "lzw")
par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris, 
     ann = FALSE,      # turn off axis labels
     axes = FALSE)     # turn off axis ticks
## Insert additional plot code here
dev.off()

width, height and units set the size of the image, res sets the resolution in points per inch. compression reduces the size of the file. The lzw options is only available for tiff files. It’s lossless, which means the compressed image is just as good as the original, so there’s no reason not to use it. In this case, it reduces the file size from 36Mb to 366K — a 99% reduction!

To create the same image as a jpg with the same resolution we’d use:

jpeg(filename = "iris.jpg", width = 3.5, height = 3.5,
     units = "in", res = 1000, quality = 85)
## insert plot code here!
dev.off()

jpg files are always compressed, and they use a lossy compression. That means there is some degradation of the image quality associated with the compression. The quality argument determines how aggressively the image is compressed. Higher values produce larger, less-degraded images. As a rule of thumb, 85 usually produces fine images at a reasonable size. In this case, the file is 550K, so a little larger than the compressed TIFF file.

Vector Images

Vector images are stored as a list of instructions: ‘draw a line from here to here, put a circle at this coordinate’ etc. As a consequence, they don’t have an inherent resolution; rather, they can be printed at any resolution necessary. So we don’t worry about the resolution when creating them, just the size and width. There are other options we need to be concerned with here:

paper:: "special" indicates that we are making a single image, not a full-page
onefile:: FALSE indicates that we are making a new file for each image (probably not necessary with a single image)

horizontal: :FALSE indicates we don’t want a landscape-orientation

To create an eps file:

postscript("iris.eps", height = 3.5, width = 3.5,
           paper = "special", onefile = FALSE,
           horizontal = FALSE) 
## insert plot code here!
dev.off()

This file is only 11K, and can be printed at any resolution. That makes eps a very convenient format to use. However, you may run into issues with fonts. By default, eps files produced by R don’t include the fonts, just the position of the letters to place on the image. If you need to embed the fonts, you need to explicitly request this:

embedFonts("iris.eps", outfile="iris-embed.eps")

This command creates a new file, iris-embed.eps, that has the font information embedded in the file. Fonts can be tricky, and specific details vary between Windows, Mac and Linux. It’s easiest to stick to the default font settings, and only dive into custom fonts and settings if you are required by the publisher.

Note that, pdf files are more common than eps. You can create pdf image files directly from R as well, using:

pdf("iris.pdf", height = 3.5, width = 3.5,
    paper = "special", onefile = FALSE) 
## insert plot code here!
dev.off()

How did I pick 19, 5 and 3? You can see all 25 symbols with plot(1:25, pch = 1:25).↩︎