Tutorials on plantarum.ca

Plant Metabarcoding with the Minion

Mon, 16 Mar 2026 00:00:00 +0000

Introduction

This is my bespoke pipeline for processing MinION metabarcoding data, focused specifically on ITS2 in vascular plants. It hasn’t yet been reviewed by anyone with any special expertise in this area, but you may nevertheless find it useful.

I find the documentation of bioinformatics tools and pipelines generally lacking, or at least targeted at people with more background knowledge than I have. To compensate for this, I move very slowly through a protocol, and examine intermediate outputs with standard text-processing tools. This is essential for my learning process, but may be tedious if you do have the understanding the bioinformaticians expect you to have.

In any case, what follows is a very detailed walk through of my pipeline. Now that it works I’ll convert it to a single script to run through all the bits below.

Library Preparation

The wet lab protocol we’re using is very similar to that used by ONTBarcoder (Srivathsan et al., 2024), except we’re using ITS2 instead of COI. The ONTBarcoder pipeline isn’t limited to COI, but it does expect the target amplicon to be a consistent length. That’s not the case with ITS2, so I put together the following pipeline.

Our wet lab protocol consists of:

DNA extraction
amplification with ITS2-S2F and ITS4, both primers with a 9bp index
PCR products further processed with the Nanopore ligation kit SQK-LSK114

The completed library is:

LSK adapter - Forward Index - ITS2-S2F - amplicon - 
        - ITS4 - Reverse Index - LSK adapter

We expect the length of individual products to be 218-378 bp. This accounts for documented variation in IT2 length (160-320bp) plus primers and adapters. Adding a bit of buffer, I’ll consider sequences in the range 160-500bp as a potentially ‘valid’ read.

The LSK Adapter sequence is:

TTTTTTTTCCTGTACTTCGTTCAGTTACGTATTGCT
        GGACATGAAGCAAGTCAATGCATAACG

Primers:

ITS-S2F: ATGCGATACTTGGTGTGAAT
ITS4: TCCTCCGCTTATTGATATGC

Our indexes are 9 bp long:

Forward	Reverse
GGAGAAGAA	CTCACAAGG
CAGAGAGAA	AAGAGCAGG
ACTCCAGAA	TAACTGCGA
TCCAAGGAA	GTCATCCGA
…	…

Dorado will trim the LSK adapters, either as part of basecalling (which is the default) or in a separate step. However, in my early trials this seems to leave an inconsistent residue at the ends of the reads which was (initially at least) kind of confusing. (this turned out to be a non-issue with the approach I took).

Another issue is a small proportion of reads are concatenated into chimeras during sequencing. i.e., two sequences are read as one long one, containing two sets of amplicons along with their flanking primers and indexes. ONTBarcoder deals with this by splitting long sequences into two, and then treating each half as a separate read. In our first run only 1.1 percent of reads were long enough to be chimeras, so I’m just going to filter out these long reads and get on with my life.

With all that in mind, our pipeline will be:

basecalling via Dorado
extracting amplicons + flanking primers/indices via seqkit amplicon
length filtering via seqkit seq
demultiplex via demultiplex demux
trim primers/indices via seqkit amplicon
dereplicate via vsearch –fastx-uniques
match to reference via vsearch –usearch-global
further processing in R

Basecalling

In the end, it doesn’t really matter if we trim the adapters or not. We’re going to excise the amplicons in the next step, which will remove everything external to the indices.

In addition to the Dorado program, we need the basecalling models. We can download them with Dorado:

dorado download --model all

This downloads a whole load of models. The one we want will be in a directory named dna_r10.4.1_e8.2_400bps_sup@v5.2.0. The particulars of the model are encoded in the directory name:

dna: DNA sequence
r10.4.1: FLO-MIN114 flow cell
e8.2: chemistry type, Kit 14
400bps: translocation speed
sup: super accurate
v5.2.0: model version

We will always use dna_r10.4.1_e8.2_400bps_sup, as these parameters are set by the hardware we’re using. We may want to update to the newer model versions as they are released.

dorado basecaller \
       ./path/to/model/dna_r10.4.1_e8.2_400bps_sup@v5.2.0 \
       ./path/to/pod5 \
       --device cuda:all --emit-fastq --no-trim \
       > output_dir/basecalls.fastq

NB: --device cuda:all for specifying a GPU.

This takes about 20 minutes on the cluster (all times will depend on the size of your read files of course; I include mine here to give you an idea of the relative time required at each step).

Check the number of reads:

  wc -l < output_dir/basecalls.fastq

Remember, four lines per read, so total reads = 1,090,727.

Amplicon Selection

In initial trials, this worked best if I select the 9 bp external to each primer. i.e., exactly matching the expected size of the barcodes. This means we’ll drop any reads where there was an indel in the index. Which is probably a good thing to do in any case.

seqkit amplicon -F ATGCGATACTTGGTGTGAAT \
       -R TCCTCCGCTTATTGATATGC -r -9:9 -f \
       output_dir/basecalls.fastq -m 4 > \
       output_dir/amplicons_9m4.fastq

-F and -R: primer sequences
-r -9:9 -f: extract the sequence starting 9 bp before the forward primer and extending 9 bp after the reverse primer
-m 4: allow a maximum of four mismatches in primer sites

Takes about 2 minutes.

As a quick sanity check, look at the first read in the output:

head -2 amplicons_9m4.fastq

@d2b8520e-173a-441d-ab34-b9f8ea53d679	qs:f:9.27552 [ ... ]
TTCCAACAAATGCGATACTTGGTGTGAATTGCAGAATCCCGTGAACCATCGAGTCTTTGAACGCAAGTTGCGCCCGAGGCCTCCTGGTCGAGGGCACGTCTGCCTGGGTGTCACGCATCGTCGCCCCCACTCCCCTCGGCTCACGAGGGCGGGGGCGGATACTGGTCTCCCGCGCGCTCCCGCCCGTGGTTGGCCTAAAATCGAGTCCTCGGCGACGGTCGCCACGACAAGCGGTGGTTGAGATAGCTCGATGGTCGGTGTGTGTCGTTGCCGCCCTGGGGAACTCCCGGTACCGCCGAGCATTTGGCTTCAGGGATGCTCGCTGGGACCCCAGCGTGGCAGGACCCTCTAAGCATATCAATAAGCGGAGGATGTCGTGTT

And compare it to the raw read:

grep -A 1 @d2b8520e basecalls_2026.fastq

@d2b8520e-173a-441d-ab34-b9f8ea53d679	qs:f:9.27552 [ ... ]
> TTAAGTTGTAACCTACTCGACTTTCAGTTACGTATTGCTAACACGACATCCTCCGCTTATTGATATGCTTAGAGGGTCCTGCCACGCTGGGGTCCCAGCGAGCATCCCTGAAGCCAAATGCTCGGCGGTACCGGGAGTTCCCCAGGGCGGCAACGACACACACCGACCATCGAGCTATCTCAACCACCGCTTGTCGTGGCGACCGTCGCCGAGGACTCGATTTTAGGCCAACCACGGGCGGGAGCGCGCGGGAGACCAGTATCCGCCCCCGCCCTCGTGAGCCGAGGGGAGTGGGGGCGACGATGCGTGACACCCAGGCAGACGTGCCCTCGACCAGGAGGCCTCGGGCGCAACTTGCGTTCAAAGACTCGATGGTTCACGGGATTCTGCAATTCACACCAAGTATCGCATTTGTTGGAAAGCAATGCGTT

I visually matched the raw read (reverse complimented in this case, the top line) to the output from amplicon (the middle line):

AACGCATTGCTTTCCAACAA ATGCGATACTTGGTGTGAAT TGCAGAATCC [ ... ]
           TTCCAACAA ATGCGATACTTGGTGTGAAT TGCAGAATCC [ ... ]
           INDEX     Forward Primer       AMPLICON

[ ... ] GTGGCAGGACCCTCTAA GCATATCAATAAGCGGAGGA TGTCGTGTT AGCAATACG [ ... ]
[ ... ] GTGGCAGGACCCTCTAA GCATATCAATAAGCGGAGGA TGTCGTGTT
[ ... ] AMPLICON          Reverse Primer (rc)  INDEX

All is as expected, so we can proceed with the analysis.

Length Filtering

Extract read lengths:

awk 'NR % 4 == 2 { print length }' \
    output_dir/amplicons_9m4.fastq > \
    output_dir/9m4_lengths.txt

Review read lengths in R:

reads <- read.table("output_dir/9m4_lengths.txt",
                    sep = "\t")
readsTrim <- reads[reads <= 1000, ]
hist(readsTrim, breaks = 50, xlab = "Read Length",
       ylab = "Number of Reads", main = "")

Note in this case I’m not plotting reads longer than 1000 bp, of which there are 160 (from 848,255 total, less than 0.01%).

Filtering my reads to 200-500 bp drops ~10% of the total:

seqkit seq -m 200 -M 500 output_dir/amplicons_9m4.fastq > \
       output_dir/amp9M4_500.fastq

wc -l < amp9M4_500.fastq

757,998 reads retained. Takes 10 seconds.

Demultiplexing

Note: demultiplex appends output to existing files. So we need to remove any files from a previous run before we re-run it on the same files (e.g., when tweaking parameters).

for_indexes.tsv:

F1	GGAGAAGAA
F2	CAGAGAGAA
F3	ACTCCAGAA
F4	TCCAAGGAA
...

rev_indexes.tsv:

R1	GAGTACGGA
R2	AGAACCGGA
R3	TAACTGCGA
R4	GTCATCCGA
...

rm -r output_dir/demux9/*
demultiplex demux -p output_dir/demux9 -s 1 -e 9 \
            for_indexes.tsv output_dir/amp9M4_500.fastq

-s 1 -e 9: match indexes against base pairs [S]tarting at bp 1 and [E]nding at bp 9.

Takes one minute.

I’m not sure how to tell demultiplex to look in the last 9 bp for the barcodes. So I’m going to reverse compliment the sequences here. This is very quick and can be done on the head node:

for FILE in output_dir/demux9/*fastq
do
    OUT=$(basename $FILE .fastq)_rc.fastq
    seqkit seq -r -p -t dna $FILE > output_dir/demux9/$OUT
done

Now we have the sequences in the right order, proceed to demultiplex a second time with the reverse index:

rm output_dir/demux9/demux9b/*
for FILE in output_dir/demux9/*_rc.fastq
do
    demultiplex demux -p output_dir/demux9/demux9b \
                -s 1 -e 9 rev_indexes.tsv $FILE
done

Takes one minute.

Checking one sample:

wc -l < output_dir/demux9/demux9b/amp9M4_500_F8_rc_R11.fastq

i.e., 26,311 reads for this sample.

Trimming

Return to seqkit amplicon to trim our primer sites:

rm output_dir/trim/*
for FILE in output_dir/demux9/demux9b/*_rc_R*
do
    OUT=$(basename \$FILE .fastq)
    OUT=${OUT/_rc_/_}_trim.fastq
    seqkit amplicon -F ATGCGATACTTGGTGTGAAT \
           -R TCCTCCGCTTATTGATATGC \
           -r 20:-20 $FILE -m 4 > output_dir/trim/${OUT}
done

-r 20:-20: Extract the first and last 20 base pairs from each amplicon (i.e., the primer sites)
-m 4: maximum four mismatches in primer sites

Dereplication

We are expecting a small number of distinct sequences, each of which may be present in very high numbers. We don’t want to match each of the duplicate sequences to the database, so the next step is to extract a single representative of each unique sequence.

for FILE in output_dir/trim/*trim.fastq
do
    NAME=$(basename $FILE)
    OUT=${NAME%trim.fastq}

    vsearch --fastx_uniques $FILE \
            --sizeout --quiet --minuniquesize 2 \
            --fastqout output_dir/derep/${OUT}dr.fq
done

--fastqout: output fastq formats
--sizeout: include number of replicates in output
--minuniquesize 2: Discard singleton sequences, as they are likely to be sequencing errors

Takes a few seconds.

Note that as we’ve requested --sizeout, every retained read will have the string size= in its header line. Counting these tells us how many unique sequences were found in a sample.

grep -c "size=" output_dir/amp9M4_500_F8_R11_dr.fq

Summing the values listed for size tells us how many of the original reads we retained (i.e., after excluding singletons).

awk 'BEGIN { FS = "size=" }
       /size/ { tot = tot + $2}
       END { print tot}' output_dir/amp9M4_500_F8_R11_dr.fq

Recall we had 26,311 after dereplication, so 20k reads were singletons.

Taxon Matching

globaldb.fasta is from (Quaresma et al., 2024). It’s comprehensive, based on a lightly cleaned set of sequences from GenBank. A locally curated, validated database will be better whenever available. If it’s not, be sure to examine the individual reference sequences matched.

There are quite a few output options. We can get everything we need from --biomout or --uc.

for FILE in ${output_dir}/derep/*dr.fq
do
    NAME=$(basename $FILE)
    OUT=$(basename $NAME _dr.fq)
    vsearch --usearch_global ${output_dir}/derep/$NAME \
            --db globaldb.fasta \
            --uc ${outout_dir}/taxon/$OUT.uc \
            --biomout ${output_dir}/taxon/$OUT.biom \
            --maxaccepts 100 --id 0.98 --sizeout
done

--maxaccepts 100: by default, the search stops when the first target is found that passes the matching criterion (i.e., > 0.98 identity). This may not be the best match! Set to 0 to search the entire database, or another number N to force the search to consider at least the first N references that pass the criterion before deciding which one is the best.
--id 0.98: reject a sequence match if the pairwise identify is less than the 0.98
--sizeout: add abundance annotations to –uc file

Takes about an hour.

We can check the --uc output from the command line. Each query sequence has a line in the file. The first column indicates Hits (matches) or No matches. The ninth column includes the query sequence ID and the number of replicates (i.e., size=). The 10th column indicates the taxon the query matched to, in case of matches.

Unique query sequences:

wc -l < output_dir/amp9M4_500_F8_R11.uc

Matching queries:

grep -c '^H' output_dir/amp9M4_500_F8_R11.uc

Total sequences matched:

grep '^H' output_dir/amp9M4_500_F8_R11.uc | cut -f9 | \
    awk 'BEGIN { FS = "size=" }
                 { tot = tot + $2}
           END { print tot}'

Our reference has multiple sequences for many taxa. We can sum those up by species:

grep '^H' output_dir/amp9M4_500_F8_R11.uc | cut -f10 | \
    sed 's/^.*s://' | sort | uniq -c | sort -h

   128 Cucumis_sativus;
    83 Trichophorum_pumilum;
    53 Rhododendron_tomentosum;
    39 Capsicum_frutescens;
    15 Brassica_napus;
     9 Solanum_lycopersicum;
     2 Rhododendron_groenlandicum;
     1 Rhododendron_ferrugineum;

At this point the analysis moves from the realm of bioinformatics and back into the realm of botany. Further assessment is easiest (for me) to do in R. I’ll leave that for another post.

References

Quaresma, A., M. J. Ankenbrand, C. A. Y. Garcia, J. Rufino, M. Honrado, J. Amaral, R. Brodschneider, et al. 2024. Semi-automated sequence curation for reliable reference datasets in ITS2 vascular plant DNA (meta-)barcoding. Scientific Data 11: 129.

Srivathsan, A., V. Feng, D. Suárez, B. Emerson, and R. Meier. 2024. ONTbarcoder 2.0: Rapid species discovery and identification with real-time barcoding facilitated by Oxford Nanopore R10.4. Cladistics 40: 192–203.

Parallelizing Bash loops

Thu, 18 Sep 2025 00:00:00 +0000

Routine data maintenance

Processing large numbers of (large) files is a common task when working on a cluster. For operations that take only take a few seconds, maybe a few minutes total for the full set of files, you can do this right on the head node. If you need to, you can run a job on the head node for a few hours. But anything that takes more than 15 or 30 minutes is worth considering for submission as a job for the scheduler. Among the many advantages, this allows you to use multiple CPUs to speed up your work.

A simple Bash loop

For processing files, this often involves loops. A previous post explains how to use array jobs to automate submitting each cycle through a loop as a separate job.

Another option, especially handy for Bash scripts, is to execute each command as a background process. For example, lets start with this loop:

for FILE in raw_data/*
do
    FILE=$(basename $FILE)
    echo ${FILE}
    processor all_files/${FILE} > clean_data/${FILE}
done

This loops over all files in the raw_data directory, runs the program processor on them, and makes a copy in the clean_data directory. That should work, but it only processes one file at a time.

Running commands in the background

If we direct the program to run the processor in the background, we can process multiple files at once:

for FILE in raw_data/*
do
    (
    FILE=$(basename \$FILE)
    echo ${FILE}
    processor all_files/${FILE} > clean_data/${FILE}
    ) &
done

The & tells the cluster to run the previous expression in the background, and immediately proceed to the next command. In this case, that means the next trip through the loop.

Controlling the number of background jobs

That’s an improvement, but if we have 1000 files to process, we probably don’t have enough CPUs to actually run them all at once. The job may fail, or the cluster may try to manage the competing processes itself. This will be slow.

Better to specifically request the number of CPUs we want, and then to limit our loop to never use more than that many at a time. We can do this with the following:

N=8

for FILE in raw_data/*
do
    (
    FILE=$(basename \$FILE)
    echo ${FILE}
    processor all_files/${FILE} > clean_data/${FILE}
    ) &

    if [[ $(jobs -r -p | wc -l) -ge $N ]]; then
        wait -n
    fi

done

The jobs program reports the process ID (via the -p flag) of the running jobs (the -r flag), one ID per line. The wc -l program tells us how many lines (i.e., how many running jobs) there are in the output. -ge $N checks if that number is greater than or equal to the value of N, which we set to 8 on the first line of the script.

If that’s true (there are 8 jobs running), the wait command will pause all further commands until any of the jobs finishes (the -n flag).

That’s exactly what we needed. Now we can ask for the number of CPUs we want in our SLURM script:

#SBATCH --cpus-per-task=8

And make sure that number matches the value we set for N in our Bash loop. The higher the number, the more files will get processed at once. But it may also take longer for your job to get scheduled, depending on the size of your cluster and how busy it is.

References

I found this approach on StackOverflow.

Replacing Loops with Array Jobs

Fri, 24 Jan 2025 00:00:00 +0000

Loops in Bioinformatic Pipelines

A common requirement of many bioinformatics pipelines is completing the same analysis on a list of files. This is simple to do with a bit of Bash scripting:

#!/bin/bash
#SBATCH --job-name=ustacks
#SBATCH --output=ustacks.log
#SBATCH --time=360:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=8G
source ~/miniconda3/etc/profile.d/conda.sh
conda activate stacksM

M=4

reads_dir=./prort/concat/
out_dir=ustacks
threads=8

while read SAMPLE; do
    echo "$SAMPLE"
    ustacks -t gzfastq -f ${reads_dir}${SAMPLE}.1.fq.gz \
            -o ${out_dir} -m 3 \
            --name ${SAMPLE} -M $M -p ${threads}
done <SAMPLES.txt

This script reads each line of the file SAMPLES.txt, passes that value to the program ustacks.

That’s fine, but it can require a lot of cluster time. In this case, I have 96 files listed in SAMPLES.txt, and each of them will take three or four hours to process. That’s 15 days. In reality, it may take much longer, as the cluster may not be able to schedule my job immediately given the length (at least on the cluster I use!).

Replacing Loops with Array Jobs

However, each of these 96 jobs can be processed at the same time. There’s no interaction between them. If instead of asking for a single 15 day job, I ask for 96 six-hour jobs, it’s easier for the cluster to schedule them, and my total run time drops to something closer to the length of a single job (maybe 6-12 hours, depending on how quickly they get scheduled).

This is what array jobs do. I can replace the script above with this:

#!/bin/bash
#SBATCH --job-name=ustacks%a
#SBATCH --output=ustacks_%a.log
#SBATCH --time=6:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=8G
source ~/miniconda3/etc/profile.d/conda.sh
conda activate stacksM

M=4

reads_dir=./prort/concat/
out_dir=ustacks
threads=8

SAMPLE=$(sed "${SLURM_ARRAY_TASK_ID}q;d" SAMPLES.txt)

echo "$SAMPLE"
ustacks -t gzfastq -f ${reads_dir}${SAMPLE}.1.fq.gz \
        -o ${out_dir} -m 3 \
        --name ${SAMPLE} -M $M -p ${threads}

It’s nearly the same as the original. The key difference is I’ve replaced the while loop with a single variable, set using a line of sed:

SAMPLE=$(sed "${SLURM_ARRAY_TASK_ID}q;d" SAMPLES.txt)

Sed is a very powerful tool for processing text. In this case, I’ve asked it to search through the file SAMPLES.txt, skip forward to the line ${SLURM_ARRAY_TASK_ID}, print that line, and then quit.

Where does SLURM_ARRAY_TASK_ID come from? That’s variable is set for us automatically when we submit our script as an array job. Instead of the usual invocation of Slurm:

sbatch my_script.sh

We’ll do this:

sbatch --array=1-96 my_script.sh

When we do this, it tells Slurm that we want to submit 96 jobs. Each job will be run with identical code, except for one thing: Slurm will insert the variable SLURM_ARRAY_TASK_ID with a different value into each script. That value is also inserted into my job name and log file, using the %a placeholder:

#SBATCH --job-name=ustacks%a
#SBATCH --output=ustacks_%a.log

With that little change, instead of waiting a month for my 15 day job to run, I wait a day for 96 six-hour jobs to run.

Selectively Repeating Failed jobs

Another benefit of this approach is that it allows you to easily resubmit only a subset of your jobs. In the example above I’ve asked for six hours for each file. If some of those files happen to require more time than that, they will fail with a timeout. You’lll see this in your job logs. Here’s an example from an array where I requested 36 hours per job:

date
sacct --jobs=3069231 --format=jobid,jobname,state,elapsed | grep -v 'batch\|extern'

: Fri 24 Jan 2025 12:10:04 PM EST
: JobID           JobName      State    Elapsed 
: ------------ ---------- ---------- ---------- 
: 3069231_1    ustacks23+  COMPLETED   19:14:14 
: 3069231_2    ustacks23+  COMPLETED   17:51:58 
: 3069231_3    ustacks23+  COMPLETED   13:20:56 
: 3069231_4    ustacks23+  COMPLETED   23:43:24 
: 3069231_5    ustacks23+    TIMEOUT 1-12:00:01 
: 3069231_6    ustacks23+  COMPLETED   17:43:17 
: 3069231_7    ustacks23+  COMPLETED   17:07:07

You can see that that wasn’t enough time for the fifth job. I can rerun just that job (after increasing the requested time) with:

sbatch --array=5 my_script.sh

Or if there are a group of failed jobs, I can combine them like this:

sbatch --array=5,7,20-23 my_script.sh

Much easier than repeating the analysis of all 96 samples just because one or two of them needed more time.

Tips for Using Array jobs

Automating Making Lists to use in Array Jobs

You can create the lists of samples or files you want to cycle over by hand. That’s probably the fastest way to do it, if you only need to do it once. But as soon as you start repeating an analysis on different files, you might benefit from some automation.

Bash scripting provides a huge variety of useful tools for creating lists of files or samples you want to cycle through with SLURM_ARRAY_TASK_ID. I won’t provide a full tutorial, but a few examples may be useful in showing you why it might be worth investing some time to learn more about this.

Filtering Filenames with Bash Pipes

I have a directory containing a list of files like this:

B12A10.1.fq.gz	    BWA11.rem.2.fq.gz  HHA19.rem.1.fq.gz  RBB8.1.fq.gz
B12A10.2.fq.gz	    BWA1.2.fq.gz       HHA19.rem.2.fq.gz  RBB8.2.fq.gz
B12A10.rem.1.fq.gz  BWA13.1.fq.gz      HHA1.rem.1.fq.gz   RBB8.rem.1.fq.gz
B12A10.rem.2.fq.gz  BWA13.2.fq.gz      HHA1.rem.2.fq.gz   RBB8.rem.2.fq.gz

Each sample has a file of matched forward (.1.fq.gz) and reverse reads (.2.fq.gz), and a file of forward and reverse unmatched reads (.rem.1.fq.gz and .rem.2.fq.gz). In my processing I’ll want to loop over each individual sample, not over all four files for each sample. I do this with the following line:

ls reads/*.1.fq.gz | grep -v rem | \
    xargs -n 1  basename -s .1.fq.gz > 2023_list.txt

The first clause, ls reads/*.1.fq.gz lists all the forward read files. This will include both B12A10.1.fq.gz and B12A10.rem.1.fq.gz.

Next, I use grep -v rem. That tells grep to filter the values we give it for the string rem. The -v argument tells grep to ‘invert’ the search. In other words, remove all the values that include rem. At this point, we have a single file name per sample, e.g. B12A10.1.fq.gz and RBB8.1.fq.gz.

The next step is a combination. First, we pass the values from grep to the program xargs with the argument -n 1. That tells xargs to pass the values it recieves one at a time to the following command. That command is basename, which will strip off the suffix we identify with the -s argument: .1.fg.gz.

Finally, the results are stored in a new file, 2023_list.txt, which will look like this:

B12A10
BWA11
HHA19
RBB8
...

If you’d rather not have another file cluttering up your directory, you can include the entire pipeline in your Slurm script

#!/bin/bash
#SBATCH --job-name=ustacks%a
#SBATCH --output=ustacks_%a.log
#SBATCH --time=6:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=8G
source ~/miniconda3/etc/profile.d/conda.sh
conda activate stacksM

M=4

reads_dir=./prort/concat/
out_dir=ustacks
threads=8

SAMPLE=$(ls reads/*.1.fq.gz | grep -v rem | \
    xargs -n 1  basename -s .1.fq.gz | \
    sed "${SLURM_ARRAY_TASK_ID}q;d")

echo "$SAMPLE"
ustacks -t gzfastq -f ${reads_dir}${SAMPLE}.1.fq.gz \
        -o ${out_dir} -m 3 \
        --name ${SAMPLE} -M $M -p ${threads}

Parameter Substitutions

Another set of useful tools are shell parameter substitutions. They can be a convenient way of trimming prefixes and suffices off of names, particularly if you use (or received) concistently formatted file names. For example, if my samples are coded as population_individual.suffix, I can isolate the various bits like this:

FILE=NS01_01.1.fq.gz
SAMPLE=${FILE%.1.fq.gz}
POP=${FILE%_*}
IND=${SAMPLE#${POP}_}

echo File: ${FILE}
echo   Sample: ${SAMPLE}, Pop: ${POP}, Ind: ${IND}

File: NS01_01.1.fq.gz
Sample: NS01_01, Pop: NS01, Ind: 01

Tips for Emacs Users

I’ve written the above for use by anyone working on a cluster that uses Slurm for job submission. However, as I explained in my previous post, Emacs snippets provide an extra layer of convenience for managing Slurm scripts. Following the approach I outline there, I write my Slurm scripts as code blocks in https://orgmode.org/. In this context, an array job will look something like this:

#+begin_src bash :results output
  date
  sbatch --array=1-96 <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=my_job_%a
  #SBATCH --output=my_job_%a.log
  #SBATCH --time=1-00:00:00
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=4
  #SBATCH --mem-per-cpu=96GB

  source ~/miniconda3/etc/profile.d/conda.sh
  conda activate stacksM

  THREADS=8

  FILE=\$(ls stacks23_out/*alleles.tsv.gz | grep -v catalog | sed -n "\${SLURM_ARRAY_TASK_ID}p")
  DIR=\$(dirname \$FILE)
  SAMPLE=\$(basename -s .alleles.tsv.gz \$FILE)

  echo "\$DIR : \$SAMPLE"

  cd \${DIR}
  sstacks -c ../\${DIR} -s ../\${DIR}/\$SAMPLE -p \$THREADS

  SUBMITSCRIPT              
#+end_src

Taming Slurm with Emacs' Yasnippet

Fri, 10 Jan 2025 00:00:00 +0000

In a previous post I explained how to set up Emacs Org-mode for working on a remote cluster. If your cluster uses Slurm to manage jobs, you will need to specify a set of options for submission. This isn’t difficult, but it’s tedious. We can automate away most of the tedium with Emacs, specifically the YASnippet package. (YASnippet stands for “yet another snippet package”, by the way). The animation above shows what we’re aiming for.

Install YASnippet

The homepage includes instructions on how to install it. The easiest uses the Emacs package manager.

If you want to use it by default, you should add the following to your config:

(yas-global-mode 1)

Alternatively, you can set it for each mode you want it applied to. For example, the following turns it on for org-mode:

(add-hook ‘org-mode-hook #‘yas-minor-mode)

You may also need the following line to ensure all your previously defined templates are loaded:

(yas-reload-all)

Writing Snippets

A snippet is a bit of text that you can automatically insert into a file. It can include:

text, inserted ‘as-is’;
tab stop fields, which you can navigate through and enter text (with or without default values) when the template is inserted;
more complex variations combining fields, transformations, mirrors and elisp code

To get started, call the command M-x yas-new-snippet. This will open a new buffer with the following text:

# -*- mode: snippet -*-
# name: 
# key: 
# --

The cursor will be on the name row. This value is for our benefit, so something descriptive is appropriate. I’ll use the phrase ‘Slurm header’.

This buffer is actually a snippet itself, so we can use the TAB key to jump to the next tab stop, which is on the next line.

The key is a combination of one or more letters that we’ll use to insert the template. It can’t contain a space. It should be memorable, but not something we’ll type in other contexts. In this case, slrm will do. Hit TAB again, and point will move down into the body of the snippet.

This is where we put the text for our submission script.

For starters, we can include any text we want to appear ‘as-is’:

# -*- mode: snippet -*-
# name: Slurm header
# key: slrm
# --
#+BEGIN_SRC bash :results output
  date
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=MYJOB
  #SBATCH --output=MYJOB.log
  #SBATCH --open-mode=truncate
  #SBATCH --partition=standard
  #SBATCH --time=24:00:00
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=1

  $0

  SUBMITSCRIPT
#+END_SRC

I start and end with the org-mode block header/footer, with the language set to bash and the :results output option. These never change.

I start the code chunk with the date command, so it will capture the time and date I submitted the script, which will be recorded in the org file.

<<SUBMITSCRIPT
...
SUBMITSCRIPT

Defines the script that is passed to Slurm, as discussed in my previous.

After that we find the actual Slurm directives with default values for each line.

There is one special field here, $0. This is where the cursor will be after the template is inserted.

That’s fine, but we still need to go back and fill in the actual values for the directives. We can improve this with by adding tab stops, along with default values and mirrors:

# -*- mode: snippet -*-
# name: Slurm header
# expand-env: ((yas-indent-line 'fixed))
# key: slrm
# --
#+BEGIN_SRC bash :results output
  date
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=${1:NAME}
  #SBATCH --output=$1.log
  #SBATCH --open-mode=truncate
  #SBATCH --partition=standard
  #SBATCH --time=${2:24:00:00}
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=${3:1}

  $0

  SUBMITSCRIPT
#+END_SRC

When we insert this template, the cursor starts at the --job-name directive where the ${1:NAME} field is. The 1 indicates this is the first tab stop. The value NAME is the default for this field. Whatever value we enter here will be mirrored to the line below with the suffix log, i.e., --output=NAME.log.

After we’ve added the name, pressing <tab> takes us to the second tab stop, for the time directive. My default is 24:00:00, but I can change that to whatever I need. The third tab stop is for cpus-per-task. After that we tab to stop zero, where we can enter the body of our script.

One final tweak: I want yasnippet to leave the indentation of this template exactly as I’ve written it (i.e., ‘fixed’). To accomplish this, I’ve set the expand-env options in the header to use fixed indentation.

# expand-env: ((yas-indent-line 'fixed))

You can test out your snippet by calling M-x yas-try-snippet, which is bound to C-c C-t. This will open a temporary buffer with the body of your snippet inserted. You can then enter text and tab through the stops to see how it works. C-x k will kill the buffer when you’re done.

When you’re done, you can install the snippet with C-c C-c, (or M-x yas-load-snippet-buffer-and-close). Emacs will ask you what mode you want to use the snippet in. In this case, it’s org-mode.

Using Snippets

With that out of the way, you can insert your snippet in an org file with the key sequence slurm<tab>. Yasnippet provides a lot of additional features. If you’re interested, see the online manual.

Interacting with Slurm

Once you’ve composed your Slurm script, you’ll want to submit it to the cluster. If you’ve set up your org file to point to the cluster, as I described in my previous post, all you need to do is type C-c C-c, while the cursor (point) is in your Slurm code block.

It may take a few moments to connect, depending on your network. Once it does, you should see something like this appear in your org file:

#+RESULTS:
: Tue 27 Aug 2024 12:17:15 PM EDT
: Submitted batch job 2931015

We could log into the server to check on the status of our job. But with another snippet, we can have Emacs do that for us.

I use the following snippet for this purpose:

# name: sacct
# key: sss
# expand-env: ((yas-indent-line 'fixed))
# --
#+BEGIN_SRC bash :results output 
date
sacct --jobs=`(save-excursion
                   (re-search-backward "^: Submitted batch job \\([[:digit:]]+\\)")
                   (buffer-substring-no-properties
                     (match-beginning 1) (match-end 1)))`$0
#+END_SRC

I’ve used a bit of elisp code to do some work here. Yasnippet will process any text between back ticks ("`") as elisp code. Let’s step through that:

`(save-excursion 
   (re-search-backward "^: Submitted batch job \\([[:digit:]]+\\)")
   (buffer-substring-no-properties (match-beginning 1) 
     (match-end 1)))`

We start with save-excursion. That tells Emacs that we want to come back to the spot we started when the code is finished.

Next we use re-search-backward to find a line that starts with (^) the string “: Submitted batch job”, followed by a number. The number is wrapped in \$ and \$: these symbols tell Emacs to record the number as a ‘match group’.

Finally, we return the number with the buffer-substring-no-properties call. This returns the substring (text) in the current buffer, starting with (match-beginning 1) (which is the first digit in our number), and ending with (match-end 2), which is the last digit in our number.

This results in the following text being inserted in the buffer:

#+BEGIN_SRC bash :results output 
date
sacct --jobs=3472303
#+END_SRC

As written, this bit of code assumes the job we want to see the status of is somewhere above the point that we call this snippet. It’s not very robust if this isn’t true, but in my limited use-case it works well enough.

The snippet leaves the cursor inside the code block. If you type C-c C-c at that point, it will submit the command to the cluster, and after a moment or two you should see something like this:

#+RESULTS:
: Fri 10 Jan 2025 05:28:53 PM EST
: JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
: ------------ ---------- ---------- ---------- ---------- ---------- -------- 
: 3472303          my_job   standard grdi_gena+          1  COMPLETED      0:0 
: 3472303.bat+      batch            grdi_gena+          1  COMPLETED      0:0 
: 3472303.ext+     extern            grdi_gena+          1  COMPLETED      0:0

Viewing Job Output

The preceding covers most of my needs for interacting with my cluster. One last trick that can be handy is linking to log files or program output.

You can insert a link to a file in org mode with C-c C-l. You will be prompted for the file location. This can include remote files with the syntax: /ssh:user@host:path/to/file; this can be shortened to /ssh:host:path/to/file if you’ve set the appropriate options in .ssh/config as I describe in a previous post. Next you’re prompted for the name of the link, which can be anything you like. Once the link is complete, Emacs will colour it to let you know it has a special power: you can view the file by pressing enter on the link.

I don’t do this often enough to have made a snippet for it.

Spatial Tutorials Update

Fri, 10 May 2024 00:00:00 +0000

A quick update. The spatial analysis libraries in the R Project have undergone a substantial change in the past couple of years. The details are laid out in the R spatial blog, but the crux of the issue is that legacy packages rgdal and rgeos have been retired, and packages that depend on them (such as raster and sp) will have been modified to use new dependencies, or replaced entirely. For the most part, the things we used to do with raster we now do with terra.

The transition was a bit rough, and for a while we needed to translate back and forth between terra and raster in our work. That should now be over, with all current packages using the new terra-based workflow.

I have already updated my quick mapping tutorial, and I’ve just updated my ecospat tutorial, now that ecospat has been fully updated to use terra too. Some of the other spatial tutorials you find here may not work properly, or at all, until I have a chance to review them. If you happen to find anything that isn’t working as expected, please let me know!

Preparing GBIF records for distribution modeling

Thu, 04 Apr 2024 00:00:00 +0000

GBIF.org

The Global Biodiversity Information Facility (GBIF.org) has become the standard open-access online database of occurrence records for all manner of biological organisms. It was initially a clearinghouse for museum records (such as herbarium specimens), but now includes iNaturalist observations (those that are rated ‘research’ grade), survey data, and a growing variety of taxonomic and checklist sources.

While GBIF’s expansion increases the overall value of the database, it also means we need to be more circumspect in how we use the data. When I first encountered GBIF decades ago, I used it as one of several sources for herbarium records. I searched for the species I was looking for, and received a list of museum specimens. Nowadays most, maybe all, online herbarium data is mirrored by GBIF, so I no longer need to chase down multiple websites to round up all the herbarium records I need.

However, I can no longer assume that a GBIF record represents a physical specimen. It could be: a human observation, with or without an associated image; documentation harvested from sequence data submitted to Genbank; an entry from a field survey, which sometimes contain records for both presences and absences. And of course, all of these records are subject to any number of issues: transcription errors, identification errors, georeferencing errors.

All of which to say, we need a way to filter the results of our GBIF query to ensure the data we receive is fit for purpose.

Getting GBIF data

Step one is actually getting the data from GBIF. You can do this from the website, but I now prefer to do this in my R scripts, using the rgbif package.

Finding taxon names

To get started, we need to match our name up with the GBIF taxonomic backbone:

library(rgbif)
conyza <- name_backbone("Conyza canadensis")
erigeron <- name_backbone("Erigeron canadensis")

This snippet searches the GBIF database for “Conyza canadensis” and “Erigeron canadensis” and returns the closest matches. The results can be a bit confusing to interpret. To make sense of it, we need to understand a few terms.

A usageKey is a unique number associated with every taxon in the database, at every level. This includes species, subspecies, genera etc, and importantly, it includes both accepted species and synonyms. Complicating things, GBIF taxonomy tables use the term usageKey, but in the individual observation records the term taxonKey is used instead. They both mean the same thing - you’ll use the usageKey you get from your name_backbone search as the taxonKey in the query you submit (see below).

The acceptedUsageKey (or acceptedTaxonKey) is the number associated with every accepted taxon. For taxa that are synonyms, their acceptedUsageKey is the usageKey of the accepted taxon they belong to.

To illustrate the distinctions, let’s look at the values for the examples above.

Key	Value
scientificName	Conyza canadensis (L.) Cronquist
usageKey	5404801
status	SYNONYM
acceptedUsageKey	3146791

Key	Value
scientificName	Erigeron canadensis L.
usageKey	3146791
status	ACCEPTED

In this case Conyza canadensis is a synonym of Erigeron canadensis. The name Conyza canadensis has it’s own usageKey: 5404801. Its acceptedUsageKey is 3146791, which is the usageKey for Erigeron canadensis. An additional wrinkle is that accepted taxa only have a usageKey, they don’t have an acceptedUsageKey. You can also use the status field to check if a taxon is accepted or a synonym.

Understanding this is important to make sure you get the records you’re after when you query the database. If you request data for taxonKey 5404801 (the usageKey for Conyza canadensis), you’ll get records with that name on them, but not records for Erigeron canadensis. On the other hand, if you search for taxonKey 3146791, you’ll get records for Erigeron canadensis, and also all records for any synonyms of that name, including Conyza canadensis.

In other words, searching for an accepted taxon will return results for that taxon including all its synonyms. Searching for a synonym will return results only for that synonym.

You can search for records by name, without using the usageKey. But it’s safer to look up and use the usageKey, to confirm that the name you asked for matches with something in the GBIF database.

Preparing a query

Once you have a list of one or more taxonKey values, you’re ready to request your data. For large record sets, and especially if you want to request multiple species at once, the rgbif function occ_download_queue is very convenient. Note that you need to have a (free) GBIF account in order to use this.

This is a three step process. You create your query with occ_download_prep, submit the query to GBIF with occ_download_queue, and once the query is done you download the results via occ_download_get.

Starting with occ_download_prep, a basic query only requires your account credentials and one or more taxonKeys:

myQuery <- occ_download_prep(
  pred_in("taxonKey", c(3146791, 3189859)),
  pred("hasCoordinate", TRUE),
  format = "DWCA",
  user = "YourUserName",
  pwd = "YourGBIFPassword",
  email = "your@email.address"
)

SECURITY NOTE You can provide your GBIF username and password in your script as I have done, but there are more secure ways to submit your credentials without listing them in your code. I have mine stored in my ~/.Renviron file, which looks like this:

GBIF_USER="my_user_name"
GBIF_PWD="my_password,"
GBIF_EMAIL="my@email.ca"

See the rgbif documentation for more options.

This will create a query for two taxa, as specified by the provided keys; it will filter the results to include only records with coordinates (i.e., hasCoordinate is TRUE); and the results will be in the Darwin Core Format (i.e., “DWCA”). We reviewed taxon keys above. If we’re mapping our records, and don’t have time or need to do any georeferencing ourselves, we can save time by limiting our results to records that already have coordinates.

The default format is “DWCA”, which includes a lot of columns in the results. You can choose “SIMPLE_CSV” instead, and this will give you a subset of commonly used fields. However, this limits your ability to filter records after download, so I recommend sticking with “DWCA”.

There are many other ways to filter records, documented in ?download_predicate_dsl. Depending on your focus, you might want to restrict the results by basisOfRecord, to select only “HumanObservation” or “PreservedSpecimen”; or by datasetID to select a specific project (i.e., iNaturalist). However, I’ve found that these terms are not applied consistently, so it may be better to download everything and filter after you’ve had a chance to inspect the tables yourself.

Submitting a query

With a query ready, you can now submit it to GBIF via:

out <- occ_download_queue(.list = list(myQuery))

We could have submitted the query directly, by using occ_download above instead of occ_download_prep. The latter method offers two advantages. First, we can submit multiple queries at once. e.g.,

out <- occ_download_queue(.list = list(queryA, queryB,
                                       queryC))

Second, GBIF allows you to submit up to three queries at a time. If you have more, you have to wait until one of the earlier queries is finished. occ_download_queue keeps track of this for you, submitting three requests, and sending additional requests as the first three finish.

Retrieving your query

occ_download_queue returns the details of your submission(s):

$`46c5a45e8f1f2e64fc9eda5318e74972`
<<gbif download>>
  Your download is being processed by GBIF:
  ...
  Check status with
  occ_download_wait('0025322-231120084113126')
  After it finishes, use
  d <- occ_download_get('0025322-231120084113126') %>%
    occ_download_import()
  to retrieve your download.
  ...

Usually it takes a few minutes to process your request. You can check its progress with occ_download_wait('...'), using the details provided. Once the query is done, you can download it via occ_download_get, and read it into R with occ_download_import, as shown.

NOTE: once your query is submitted by occ_download_queue, it will be processed remotely by GBIF. If something should happen in the meantime – your computer crashes, or you cancel the function call for any reason – the query will still be processed. However, should this happen you won’t have the downloadKey you need to retrieve the results. You can get this key by logging into your GBIF account on the website, navigating to the Downloads section of your profile, and clicking on either the DOI or the SHOW links for the download in question. This will take you to a page with the meta data for the query. It doesn’t include the actual downloadKey, but that value is present as the final part of the url, as shown here:

The GBIF download summary page, showing the downloadKey value in the URL

You can also click the Download button on the right side to download the file from your browser, if you prefer.

Cleaning GBIF data

Now that we have our records downloaded, we need to review and clean the data before analysis. Note that in the code below, I’m using a large download to demonstrate my investigations. I haven’t included this data in this tutorial, but you can try the code on your own data.

Filtering based on the data

basisOfRecord

With the data in hand, we can take a closer look at what kind of records they are, and where they came from. Starting with basisofRecord:

d1 <- occ_download_get(out[[1]], path = "./dl/") %>% 
  occ_download_import()
sort(table(d1$basisOfRecord), decreasing = TRUE)

  HUMAN_OBSERVATION  PRESERVED_SPECIMEN          OCCURRENCE 
            4224215              325278              112588 
        OBSERVATION   MATERIAL_CITATION     LIVING_SPECIMEN 
             104335               17703                3123 
    MATERIAL_SAMPLE MACHINE_OBSERVATION     FOSSIL_SPECIMEN 
               1698                 222                  10

In this case, I have nine different kinds of record. The definitions of some of these terms are listed in the Darwin Core Quick Reference Guide. HUMAN_OBSERVATION includes, among other things, iNaturalist records. PRESERVED_SPECIMEN includes mostly (and most) herbarium records. I usually want both of these groups.

OCCURRENCE and OBSERVATION aren’t well defined or consistently used, so require further examination to determine if we want to retain them.

MATERIAL_CITATION, MATERIAL_SAMPLE, and MACHINE_OBSERVATION are a little vague, inconsistently used, and also not used often, so I remove them.

LIVING_SPECIMEN and FOSSIL_SPECIMEN are self-explanatory, and usually not want I want for my work.

iNaturalist and Human Observations

Research-grade iNaturalist records are imported to GBIF every few weeks. We can extract these using the field datasetKey, with the value "50c9509d-22c7-4a22-a47d-8c48425ef4a7":

sum(d1$datasetKey == "50c9509d-22c7-4a22-a47d-8c48425ef4a7")

357863 # iNaturalist Records in my data

You might be tempted to filter on datasetName, since there is a dataset called “iNaturalist research-grade observations”. Unfortunately, this name isn’t used consistently, the name “iNaturalist observations” is also used for some records in the same dataset. In fact, the word iNaturalist appears in a variety of different dataset names:

table(d1$datasetName[grep("iNaturalist", d1$datasetName)])

"Flora of Russia" on iNaturalist: a trusted backlog 
                                                131 
                           iNaturalist observations 
                                                 25 
            iNaturalist research-grade observations 
                                             357838 
               iNaturalist XicotliData observations 
                                                  7

Which leaves us with the unwieldy datasetKey == "50c9509d-22c7-4a22-a47d-8c48425ef4a7" as the most reliable way to get the offical iNaturalist dataset.

All of the iNaturalist records are labelled as HUMAN_OBSERVATION:

table(d1$basisOfRecord[d1$datasetKey ==
                       "50c9509d-22c7-4a22-a47d-8c48425ef4a7"])

HUMAN_OBSERVATION 
           357863

But this is only a small fraction of the 4,224,215 HUMAN_OBSERVATION records in my query. In my data, there are over 1500 different HUMAN_OBSERVATION datasets. This includes a number of surveys, and in some cases these surveys record presences and absences, as recorded in the occurrenceStatus field:

table(d1$occurrenceStatus)

 ABSENT PRESENT 
  27896 4761276

Whether you want to include a heterogeneous collection of surveys in your data depends on the question your asking.

But if you are planning to do distribution modeling, you may not want to include ABSENT records in your training data. It will depend on the scale of your modeling, and the scale of the surveys that contributed their data to GBIF. Fine-scale local surveys may include a mix of PRESENT and ABSENT records within a small area (say a few hundred meters). If your modeling is at the scale of 30 second climate rasters (~1km^2), that’s a problem. A species can be absent in a 10m^2 quadrat, but present in the larger 1km^2 raster grid.

Here are a few examples of filters you might want to use:

herbarium <- d1[, d1$basisOfRecord == "PRESERVED_SPECIMEN"]
iNat <- d1[, d1$datasetKey ==
             "50c9509d-22c7-4a22-a47d-8c48425ef4a7"]
present <- d1[, d1$occurrenceStatus == "PRESENT"
              & d1$basisOfRecord == "HUMAN_OBSERVATION"]

Common location errors

The R package CoordinateCleaner provides some functions for dealing with common problems:

cc_cen : Identifies records close to the centroid of a country. Automated georeferencing will often use the centroid of a country as the coordinates for a specimen that doesn’t have any other location data. This could be 100s/1000s of kilometers away from the actual location.

cc_inst : Identifies records close to the location of museums. Automated georeferencing will often use the location of a museum as the location for specimens it contains, regardless of where the specimen was actually collected.

cc_sea : Identifies records that are in the ocean, which are clearly errors for terrestrial organisms. However, this could also be a consequence of relatively minor errors in record coordinates or the mapping of coastlines.

For a more thorough overview of these kinds of issues, see this post by John Waller on the GBIF data blog.

Identification Errors

Now that we’ve dealt with the more ‘mechanical’ sorts of errors we might find in our data, we can review our records for obvious or suspected identification errors. If you have a large dataset, especially if it includes many species, or species you are not familiar with, this can be a daunting task.

Two important botanical resources can help us with this, at least in North America: the Flora of North America and NatureServe.org. These provide a ‘sanity check’ for our GBIF records, as both websites host carefully curated data on plant distributions.

The Flora of North America includes taxonomic treatments of all plant species that occur outside of cultivation in Canada and the United States¹. The maps aren’t very detailed, showing us only the states or provinces each taxon is found in. But those maps are all based on actual specimens examined by a taxonomist with expertise on that plant. If New York appears on an FNA distribution map, it means there is at least one documented record of that species, confirmed by an expert, in that state. And conversely, if a state or province isn’t included on the map, it means that a thorough search of herbaria has failed to find a single record of the species.

NatureServe is similar, but the distribution maps are based on state/provincial/regional natural heritage programs. These are generally expert field botanists, rather than taxonomists. Their job is to document which plants grow in their jurisdiction. If a NatureServe distribution map includes Ontario, that means an expert field botanist has evidence (which could be a herbarium voucher, or a reliable observation) that the species occurs (or occurred) in Ontario. And again, if a state or province isn’t included on the map, then no such evidence has been found.

NatureServe and FNA distribution maps will usually match fairly closely. NatureServe is more responsive to new information, and those maps will periodically be updated. FNA isn’t yet complete (although it’s getting close), but for the taxa that it covers it provides a comprehensive review of their distribution at the time of publication (i.e., it doesn’t get updated).

Neither source is complete, and plants do move around. But, if you have records in your GBIF dataset from areas beyond the range documented in NatureServe or FNA, these are the ones I’d be most concerned about verifying. Use the map interface on the GBIF website to locate them, and look at the record details for clues to confirm or refute them. If they trace back to iNaturalist records, you may be able to confirm the identification yourself from the photos, or ask the observer for more details.

You won’t likely be able to confirm the identifications of all of your records, but you can often validate or exclude the outlying records that have the greatest potential to distort your analysis.

The FNA project is more properly the “Flora of North America north of Mexico”.↩

Introduction to Emacs' Org Mode for cluster computing

Thu, 28 Mar 2024 00:00:00 +0000

Getting Started

This tutorial builds on my previous post Emacs posts (ie., introduction, Orgmode, R and ESS). If you’re not familiar with Emacs, you might want to look through those first, particularly Orgmode.

In this post, I will show you how to setup an Org mode (or Orgmode, org-mode) file on your local machine (laptop or office desktop) to manage and run a cluster computing project.

While not absolutely necessary, working this way is much easier if you’ve configured your machine to provide keywordless access to the server. I’ve talked about setting up your .ssh/config file to manage usernames and hostnames/addresses previously, as well as setting up RSA keys so you can securely access remote servers without a password.

In addition, within Emacs you’ll likely want to configure the variable org-confirm-babel-evaluate to nil, so that you don’t get asked for confirmation each time you try to evalute a code block, and also to configure which languages you would like to use in code blocks, via the Org Babel Load Languages setting. I step through this in my previous post.

Org file headers

First off, we want to tell Org Mode where we want to run our code. Putting this information in the file header ensures it will apply to all the code blocks in the file (unless we explicitly change it for a particular block).

My file template includes the following two lines as the first two lines of my .org file:

# -*- org-export-babel-evaluate: nil -*-
#+PROPERTY: header-args:bash :results output :dir /ssh:gpsc:./path/to/my/project

The first line tells Orgmode not to evaluate code blocks if/when we export our source file (i.e., if we want to create a pdf or html report). If we don’t do this, then every time we export the document it will resubmit all of our cluster jobs, which is almost certainly not what we want.

The second line sets the default properties for bash code blocks. :results output means we want the text printed out by the code in our code blocks inserted into the file. :dir /ssh:gpsc:./path/to/my/project means we want these code blocks to be run on the remote host gpsc, which we access through ssh, and in the directory ./path/to/my/project (relative to my home directory).

When this is properly configured, we can edit the org file on our local machine, and send all the code to the server to run!

If you make a mistake in the header lines, or just want to change the path or other details, be sure to update the settings by pressing C-c C-c with the cursor on the header line, or, in the menus, select ‘Org’ -> ‘Refresh/Reload’ -> ‘Refresh setup current buffer’.

Executing code on the server

Now we’re ready to run some code. To create a block, enter the following text in the file:

#+begin_src bash
  date
  ls
#+end_src

The lines #+begin_src ... and #+end_src denote the code block, with the argument bash indicating that this is the language for this block.

With the cursor (point) anywhere in the block, hit C-c C-c to evaluate the code. The results will show up in a few moments:

#+RESULTS:
: Thu 28 Mar 2024 09:31:21 PM UTC
: hostname.tld.ca
: bin  data  man	miniconda3  ovbin  rubus  slurm_sample.sh  srcs  wp2

This is the contents of my home directory on hostname.tld.ca - success!

Write & submit job requests (with slurm)

Now we can run bash scripts on the head node. That’s fine for installing packages with conda/mamba or whatever system you’re using. But for real work, we’ll need to submit a job request. In my case we use the slurm scheduler. I use two different methods to run these scripts.

Heredocs

In most cases, I include the entire script as a heredoc:

#+begin_src bash :results output
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=myjob
  #SBATCH --output=myjob.log
  #SBATCH --time=24:00:00
  #SBATCH --cpus-per-task=1
  #SBATCH --mem-per-cpu=1G

  VARIABLE="My variable"

  source ~/miniconda3/etc/profile.d/conda.sh
  conda activate bowtie2

  echo \$VARIABLE

  bowtie2 --help

  SUBMITSCRIPT
#+end_src

(Of course the actual details will depend on how your server is set up, with slurm, qsub, or whatever is set up).

Note that when you include bash variables in scripts submitted as heredocs, you need to escape the $ when you reference them (e.g., \$VARIABLE).

Again, we can submit this script by pressing C-c C-c when the cursor is anywhere in the code block. In this case, the output we get is from sbatch:

#+RESULTS:
: Submitted batch job 2075188

Having a record of the job number allows us to track its progress with another code block:

#+BEGIN_SRC bash
sacct --jobs=2075188 --format=jobid,jobname,state,elapsed,ReqMem,MaxRSS
#+END_SRC

Running this (C-c C-c again) returns the status of my job:

#+RESULTS:
: JobID           JobName      State    Elapsed     ReqMem     MaxRSS 
: ------------ ---------- ---------- ---------- ---------- ---------- 
: 2075188         ustacks    RUNNING   00:00:07        64G            
: 2075188.bat+      batch    RUNNING   00:00:07                       
: 2075188.ext+     extern    RUNNING   00:00:07

Now we’re getting somewhere! In this one file we can include our freehand notes and metadata, along side the scripts we used in the analysis. One file per project, so we don’t need to keep track of different script versions for each step in a long workflow.

Standalone scripts

Most job scripts are pretty simple, so the heredoc approach is fine, and keeps all the details together in the org file. However, if you have a more involved script, you may want to keep it in a file of its own. That will give you access to the language-specific editing features of Emacs, and may help keep your main org file to a manageable length.

If you choose this approach, you can include a link to the external script with the format:

[[file:/ssh:servername:path/to/script.sh][script.sh]]

You’ll want this file to be in the same directory you specified in the org file header.

Using this link format, pressing enter with the cursor on the link text will open that file for you, allowing you to read and edit it with ease.

To submit the script, we can use a much smaller code block:

sbatch script.sh

This would be the way I would submit jobs in other languages as well, if you use Perl, Python, or R in your analyses.

Streamlining all the text entry

This system works very well, and being based on plain text files it’s pretty robust and portable. However, there are a lot of bits of text to enter, and remembering all the syntax can be tedious. orgmode provides some conveniences to simplify this.

To enter a new code block, you can use the shortcut C-c C-, to open the block menu, and select source code with s. Then add the header bash and you’re ready to go. You can do the same thing via the menus with ‘Org’ -> ‘Editing’ -> ‘Add block structure’.

The headers for slurm scripts can be quite involved as well. I use the yasnippet extension to insert templates for these. I’ll leave the details for another post.

What about R?

You can run R scripts in batch mode on a server just like any other script. But you might want to run an interactive process for exploring the results of your scripts. I do this by passing the output from code blocks run on the server to a local instance of R. I’ll explain that in a future post as well.

Extrapolation Detection (exDet) for SDMs

Tue, 19 Dec 2023 00:00:00 +0000

Identifying Non-Analogous Climate Conditions

A major concern when projecting species distribution models to new contexts (e.g., invaded ranges, or future climates) is establishing whether (and where) the environments in the new context are analogous to those in the training region (see my notes). A common approach is to compare each variable in isolation, and construct a “Multivariate Environmental Similarity Surface”, or MESS (Elith et al., 2010). Areas in the new context that are outside the range of any variable from the training context will have values below 0, with lower values indicating greater departures.

However, the MESS approach does not account for correlations among variables. Perhaps the native range of a species includes areas with hot, wet, summers, but not hot dry regions. The MESS analysis would consider conditions ‘analogous’ as long as they were within the temperature range and precipitation range of the training region. This could result in novel environmental combinations being incorrectly identified as analogous climate:

Visualization of MESS analysis. The green area indicates the reference climate conditions, and the red square shows the climate space MESS will identify as ‘analog’.

Note particularly the upper left corner. This is well outside the range of conditions in the training region (the green area), but by considering each variable separately, MESS will treat this region as if it were analogous.

Extrapolation Detection

Despite the name, MESS is only just barely multivariate: it evaluates multiple variables, but each one is considered in isolation. Extrapolation Detection (Mesgaran et al., 2014), or exDet, provides a more sophisticated approach, based on the Mahalanobis distance. This is a common multivariate distance measure, designed to account for covariation among variables. In this context, our analysis will include the following steps:

Calculate the Mahalanobis distance for every cell in the training region
Scale the distances by dividing by the maximum distance, such that they range between 0 and 1
Calculate the Mahalanobis distance for every cell in the novel region/time period, again dividing by the maximum distance from the training (not the new) region.

This will produce a raster map with cell values ranging from 0 to infinity, which serves as the Novelty Index. Mesgaran et al. (2014) refer to it as NT2, to distinguish it from the MESS index which they call NT1. Cells with NT2 values less than 1 are within the range of conditions present in the training range; cells with NT2 > 1 are outside that range, with higher values indicating greater departure:

Visualization of exDet analysis. The green area indicates the reference climate conditions, and the red ellipse shows the climate space exDet will identify as ‘analog’.

The result more closely captures the (co)variation in conditions present in the training area.

exDet in R

There is an R package that calculates exDet: ntbox. However, it is still using the raster-based workflow, which is now obsolete. Luckily, calculating Mahalanobis distances only requires a few lines of code, so we can do it ourselves.

For this example, I’ll use the Lythrum salicaria data from my previous tutorials, but updated to use the new terra workflow.

Data

I use the same GBIF records previously downloaded in my Ecospat with Terra tutorial.

library(terra)
library(geodata)
library(rgbif) ## not actually necessary, used to download
               ## the occurrence data

library(usdm) ## for vif collinearity test below

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

## convert to a spatial vector:
lsOccs <- vect(lsOccs, geom = c("decimalLongitude",
                                "decimalLatitude"),
               crs = "+proj=longlat +datum=WGS84")

wrld <- world(path = "../data/maps/")

wclim <- worldclim_global(var = "bio", res = 10,
                          path = "../data/")

## North America basemap:
nAmCountries <- c("CAN", "MEX", "USA")
nAm <- gadm(nAmCountries, level = 0,
            path = "../data/maps/",
            resolution = 2)

## Eurasia basemap:
eurasiaCountries <- country_codes("Asia|Europe")

## remove missing countries:
eurasiaCountries <-
  eurasiaCountries[!eurasiaCountries$ISO3 %in%
                    c("HKG", "MAC", "XNC"),] 
eur <- gadm(eurasiaCountries$ISO3, level = 0,
            path = "../data/maps/",
            resolution = 2)

## subset occurrences
lsNA <- lsOccs[nAm]
lsEA <- lsOccs[eur]

Training Region

There are a number of considerations when deciding on the region to use to train our model. For this example, I’ll just use a 500km buffer around occurrences in the native range.

Now we have all the data we need for the analysis:

train <- buffer(lsEA, width = 500000)
train <- aggregate(train)

plot(wrld, mar = c(1.5, 1.5, 0.5, 0.5))
plot(train, col = '#00FF0080', add = TRUE)
plot(lsEA, add = TRUE, cex = 0.5)
plot(lsNA, add = TRUE, cex = 0.5, col = 'blue')

Figure 1: Lythrum salicaria distribution. Green shading shows the training region, black points are occurrences in the native range, and blue points are occurrences in the invaded range.

Note that the training region extends into the ocean, but the underlying worldclim data doesn’t, so we don’t need to worry about cleaning this up.

Mahalanobis Distance

Now we can calculate the NT2 index. We start by clipping the climate data to our training and projection regions, and then selecting a set of variables that aren’t highly collinear (in the training region):

trainWC <- mask(wclim, train)

## 
|---------|---------|---------|---------|
=========================================

trainVal <- values(trainWC)

naWC <- mask(wclim, nAm)

## 
|---------|---------|---------|---------|
=========================================

naVal <- values(naWC)

## screen out collinear variables:
vifSel <- vifstep(data.frame(trainVal), th = 5,
                  size = 20000)

VARS <- vifSel@results$Variables

VARS contains a list of non-collinear variables. For a real analysis you probably should be more deliberate in selecting which variables to retain, in order to consider both their biological interest and statistical properties.

trainMeans <- colMeans(trainVal[, VARS], na.rm = TRUE)
trainVar <- var(trainVal[, VARS], na.rm = TRUE)
trainMah <- mahalanobis(trainVal[, VARS], trainMeans,
                        trainVar, na.rm = TRUE)

trainMah now contains the Mahalanobis distance from each cell in the training region to the climate centroid of the training region. ‘0’ means the point is in the center of the climate space. By the nature of the Mahalanobis distance, this value accounts for covariation among our climate variables.

As a quick check, let’s take a look at the Mahalanobis distances for the training region:

hist(trainMah, main = "Mahalanobis Distances in Eurasia",
     xlab = "Distance")

That’s a bit odd. We have a small number of outliers with very high distances. I think this is likely a consequence of idiosyncracies in our climate rasters: an artifact of the analysis rather than a biologically meaningful value. To clean this up, I’ll identify values above the 95th percentile as outliers and remove them. Then we can proceed with calculating the distances for the invaded range.

thresh <- quantile(trainMah, probs = 0.95, na.rm = TRUE)
trainMah[which(trainMah > thresh)] <- NA
hist(trainMah, main = "Trimmed Mahalanobis Distances in Eurasia",
     xlab = "Distance")

That looks better. Moving on, we can now calculate NT2, which is the Mahalanobis distance divided by the maximum Mahalanobis distance in the training range:

maxMah <- max(trainMah, na.rm = TRUE)
trainMah <- trainMah/maxMah
trainMahR <- trainWC[[1]]
values(trainMahR) <- trainMah

naMah <- mahalanobis(naVal[, VARS], 
                     trainMeans, trainVar, na.rm = TRUE)
naMah <- naMah/maxMah

## Create a new raster map for North America:
naMahR <- naWC[[1]]

## set the values to our Mahalanobis distances:
values(naMahR) <- naMah

Now we have a raster map with the NT2 index values for each cell in the training and invaded ranges. Values between 0 and 1 are within the training range. We can visualize that directly:

COLS <- c("green", "lightgreen", "aquamarine", "blue", 
           "orange", "brown", "red")

BREAKS = c(0, 0.25, 0.5, 1, 2, 4, 8, 16)

plot(trainMahR, breaks = BREAKS, col = COLS,
     mar = c(2, 2, 1, 5))
plot(naMahR, breaks = BREAKS, col = COLS, add = TRUE, legend
     = FALSE)
plot(wrld, border = 'lightgrey', add = TRUE, lwd = 0.5)

This completes main part of the exDet analysis. We can see the training range all falls within the range 0-1, which is by design. Transferring that distribution to North America, we can see most of Canada and the north central US falls within the same range. The high arctic, the SE US and western US are all outside of the training range, with Mexico even more novel. Projections into these areas should be treated with caution, or avoided entirely.

Most Influential Covariate, MIC

While the map above is our primary interest in exDet analysis, we may want to know which variables are most influential in creating novel environments. To do this, we need to calculate the distance map repeatedly, removing one variable at a time. For each cell, we can then identify the variable that contributes the most to its NT2 index, by calculating the difference between the distance for all variables, and the distance with that variable removed.

names(naMahR) <- "all"

## Remove variables one at a time and recalculate distances:

for(V in VARS){
  SEL <- VARS[VARS != V]

  tmpMeans <- colMeans(trainVal[, SEL], na.rm = TRUE)
  tmpVar <- var(trainVal[, SEL], na.rm = TRUE)
  tmpMah <- mahalanobis(trainVal[, SEL], tmpMeans,
                        tmpVar, na.rm = TRUE)

  thresh <- quantile(tmpMah, probs = 0.95, na.rm = TRUE)
  tmpMah[which(tmpMah > thresh)] <- NA
  tmpMax <- max(tmpMah, na.rm = TRUE)

  resMah <- mahalanobis(naVal[, SEL], 
                     tmpMeans, tmpVar, na.rm = TRUE)
  resMah <- resMah/tmpMax
  resMahR <- naWC[[1]]
  
  ## calculate difference from full distance:
  values(resMahR) <- values(naMahR$all) - resMah

  ## add a layer to our NA raster:
  naMahR[[V]] <- resMahR
}

## Find the maximum difference for each cell:
naMaxMah <- max(naMahR[[-1]])
naTest <- naMahR[[-1]] == naMaxMah

## Convert TRUE/FALSE to category numbers:

for(L in seq_along(names(naTest))){
  values(naTest[[L]]) <- values(naTest[[L]]) * L
}

## Collapse layers into a single raster:
naCat <- max(naTest)

## convert to a factor
levs <- data.frame(vals = seq_along(VARS),
                   ## clean up variable names:
                   levels = gsub("wc2.1_10m_", "", VARS))
levels(naCat) <- levs

Now we can visualize which individual variables are contributing most to the NT2 index for each cell:

plot(naCat, xlim = c(-180, -40), ylim = c(10, 90),
     mar = c(2, 2, 4, 5),
     main = "Most Influential Covariate")
plot(wrld, border = 'lightgrey', lwd = 0.5, add = TRUE)

Whether or not that information is useful to you will depend on your research question, of course.

References

Elith, J., M. Kearney, and S. Phillips. 2010. The art of modelling range-shifting species. Methods in Ecology and Evolution 1: 330–342.

Mesgaran, M. B., R. D. Cousens, and B. L. Webber. 2014. Here be dragons: A tool for quantifying novelty due to covariate range and correlation change when projecting species distribution models J. Franklin [ed.], Diversity and Distributions 20: 1147–1159.

Niche Quantification with Ecospat and Terra

Fri, 28 Jul 2023 00:00:00 +0000

Introduction

This is an update of my previous ecospat tutorial. Spatial analysis in R is shifting to terra and sf as the primary packages, so I’ve translated my old, raster-based tutorial to the new workflow. I also took this opportunity to clean up and extend the original tutorial.

See the RSpatial tutorial for a more detailed introduction/overview of using terra for GIS/spatial analysis.

Note this analysis depends on the ecospat package, and as of 2023-07-28 ecospat doesn’t support the spatial objects produced by terra. There are a couple of work-arounds in the code below to account for this. I’ll update the code once ecospat is fully compatible with terra

UPDATE 2024-05-10 ecospat has now been updated to use the terra package for spatial analysis. The code here has been updated to reflect this. Thank you to Lin Lin at Yunan University for bringing this to my attention!

The ecospat package (Cola et al. 2017) provides code to quantify and compare the environmental and geographic niche of two species, or of the same species in different contexts (e.g., in its native and invaded ranges). The included vignette explains how to do such analyses.

However, the vignette assumes you already have a matrix of occurrence records, along with the climate data for each of those records. In our work, we typically have to construct those matrices from observation data (herbarium records, iNaturalist observations, etc) and climate rasters (e.g. Fick and Hijmans 2017). This short tutorial will walk through the steps necessary to do this.

Required Packages

In addition to ecospat, we’ll use terra (Hijmans 2023) to download WorldClim (Fick and Hijmans 2017) rasters, and manipulate the spatial data; rgbif (Chamberlain et al. 2021) to download GBIF records, geodata (Hijmans et al. 2023) to get a world basemap for plots, and ade4 (Thioulouse et al. 2018) to perform the principal components analysis of the climate data.

library(ecospat)
library(terra)
library(rgbif)
library(geodata)
library(ade4)

Getting Data

GBIF Occurrence Data

We’ll start by sourcing our data. For observations, let’s take a look at Purple Loosestrife, a wetland species that is native to Europe, and invasive in North America. For actual research work, I normally download the files directly from GBIF, and examine them carefully to check for errors or missing data. For this demo we’ll use the rgbif package to download the data directly into R, and we’ll assume there are no problems with the data that need to be corrected.

lsGBIF <- occ_search(scientificName = "Lythrum salicaria",
                    limit = 10000,
                    basisOfRecord = "Preserved_Specimen",
                    hasCoordinate = TRUE,
                    fields = c("decimalLatitude",
                               "decimalLongitude", "year",
                               "country", "countryCode"))

save(lsGBIF, file = "../data/2021-07-29-ls-gbif-recs.Rda")

This returned an object with 7969 records. I saved that locally, so that I’m not making GBIF search their database everytime I work on this demo.

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

lsGBIF$data is the table with the actual records in it. That’s what we’ll be working with. The other components of lsGBIF are metadata related to the original GBIF search. That’s useful to have, but not needed for the rest of this example.

Next, we tell R which columns are the coordinates, which allows us to map the observations. This also converts our observation matrix to a SpatVector object.

The crs argument here tells R that our points are in lat/lon (unprojected) coordinates. If all of your data use lat/lon, you don’t need to specify this, but it’s important if you need to reproject your data, or combine data with different projections.

lsOccs <- vect(lsOccs, geom = c("decimalLongitude",
                                "decimalLatitude"),
               crs = "+proj=longlat +datum=WGS84")

Basemap

We’ll also need a world map to use in our plots. the world function from the geodata package will download one for us. The first time you call this function in a directory, it downloads the data from the internet, and saves it locally according to the path argument. Subsequent calls will load your local copy of the data, to speed things up.

In this case I’ve set the path argument to store the downloaded files in a location that is convenient for me. You can set this to anything you like, or leave it out to use the current R working directory.

wrld <- world(path = "../data/maps/")

This is a relatively low resolution map of the continents. For higher resolution maps, see the gadm function.

Now we can plot our data. Note that we set the margins (via mar) inside the plot call, not via the par function used in most base R plots.

plot(wrld, border = "gray80", mar = c(0, 0, 0, 0))
points(lsOccs, col = 2, cex = 0.3)

Climate Data

To get our climate data, we can use geodata’s worldclim_* functions. I’m using the coarsest resolution (10 minutes) to speed things up for this demonstration. The path argument works the same was as for world, storing a local copy of the files. In this instance we’ll use worldclim_global to get the bioclim variables for the world all at once:

wclim <- worldclim_global(var = "bio", res = 10,
                          path = "../data/")

We can take a look at one layer:

plot(wclim$wc2.1_10m_bio_1, main = "bio1",
     mar = c(0.1, 0.1, 2, 4), legend = TRUE,
     axes = FALSE)
par(mar = c(0.1, 0.1, 2, 4))
box()

Sampling Bias

One of the challenges we deal with with herbarium data is that observations tend to be clustered together in non-random ways. Sites near universities, museums, or in well-used field stations will have more records than more remote locations. We can’t completely account for this bias, but we can reduce it by spatial thinning. This is the process of randomly selecting a small number (often just one) of records for each grid cell in our analysis. The result is that those highly-sampled locations will be represented by a single record, meaning they will have a less exaggerated influence on the results.

We will need the full set of records below, so I’ll make a copy of the un-thinned data named lsOccsAll.

lsOccsAll <- lsOccs ## keep this for later

## select one record per cell:
lsOccs <- spatSample(lsOccs, size = 1, strata = wclim)

Linking Climate Data to Species Observations

Next, we need to extract the environmental values from the climate rasters for each of our observation records:

lsOccs <- cbind(lsOccs, extract(wclim, lsOccs))

In the process of extracting wclim values for our observations, we usually end up with a few missing values. This is a consequence of mismatches between the observation coordinates and the climate rasters. In some cases, the observations are placed off the coast in the ocean, or in another area where there is no climate data available. We need to exclude these missing values from our analysis.

lsOccs <- lsOccs[complete.cases(data.frame(lsOccs)), ]

Splitting Records into Native and Introduced Ranges

At this point, all the data we need for the Niche Quantification analysis is in lsOccs and wclim. We need to split this data into native and invasive regions for our comparison. We’ll restrict ourselves to the northern hemisphere, and consider all records from Eurasia as native, and all records from North America as invasive.

We can use GADM to download maps of the continents, and use this to select subsets of the datasets. I will select the country-level borders (level = 0) and low resolution (resolution = 2). For ‘real’ analyses you should use high resolution (resolution = 1).

To speed up analyses further down, I’m just going to download Canada, USA, and Mexico for my “North America” map.

nAmCountries <- c("CAN", "MEX", "USA")
nAm <- gadm(nAmCountries, level = 0,
            path = "../data/maps/",
            resolution = 2)

## select Europe OR Asia:
eurasiaCountries <- country_codes("Asia|Europe")
eur <- gadm(eurasiaCountries$ISO3, level = 0,
            path = "../data/maps/",
            resolution = 2)

lsNA <- lsOccs[nAm]
lsEA <- lsOccs[eur]

plot(wrld, axes = FALSE, xlim = c(-140, 150),
     ylim = c(10, 80), mar = c(0, 0, 0, 0))
points(lsNA, col = 'red', cex = 0.5)
points(lsEA, col = 'darkgreen', cex = 0.5)
par(mar = c(0,0,0,0))
box()

Note that this code will generate some warnings, “this file does not exist”. There are a few countries listed in the country_codes database that don’t have corresonding maps in the GADM repository (e.g. Hong Kong). We’ll ignore this for now.

For the Niche Quantification, we need to have a matrix with the background environment present in the native and invasive ranges, as well as the complete global environmental including the combined extent of the native and introduced environments. After cropping the data, we use values to convert the raster to a dataframe.

## Crop Climate Layers:
naEnvR <- mask(wclim, nAm)

## |---------|---------|---------|---------|=========================================

eaEnvR <- mask(wclim, eur)

## |---------|---------|---------|---------|=========================================

globalEnvR <- mask(wclim, rbind(nAm, eur))

## |---------|---------|---------|---------|=========================================

## Extract values to matrix:
naEnvM <- values(naEnvR)
eaEnvM <- values(eaEnvR)
globalEnvM <- values(globalEnvR)

## Clean out missing values:
naEnvM <- naEnvM[complete.cases(naEnvM), ]
eaEnvM <- eaEnvM[complete.cases(eaEnvM), ]
globalEnvM <- globalEnvM[complete.cases(globalEnvM), ]

Note that for the geographic projection below, it is essential that the globalEnvM be constructed directly from the globalEnvR rasters. If you try to combine naEnvM and eaEnvM to make globalEnvM it will end up scrambling the values in the geographic projection. If you don’t do a geographic projection it doesn’t matter.

Niche Quantification

PCA

The Niche Quantification analysis starts with a Principal Components Analysis of the environmental data. The actual ordination uses the global data, with the observation records and the native and invasive background environment treated as supplemental rows.

pca.clim <- dudi.pca(globalEnvM, center = TRUE,
                    scale = TRUE, scannf = FALSE, nf = 2)
global.scores <- pca.clim$li

nativeLS.scores <-
  suprow(pca.clim,
         data.frame(lsEA)[, colnames(globalEnvM)])$li   
invasiveLS.scores <-
  suprow(pca.clim,
         data.frame(lsNA)[, colnames(globalEnvM)])$li

nativeEnv.scores <- suprow(pca.clim, naEnvM)$li
invasiveEnv.scores <- suprow(pca.clim, eaEnvM)$li

Let’s break that down. dudi.pca does a PCA analysis on globalEnvM, which is a matrix of all the environmental variables over the entire study area. We use that to create a two-dimensional summary of the total environmental variability.

Next, we map our observation data (lsEA and lsNA) into that 2-dimensional ordination, using the suprow function. lsEA and lsNA are SpatialPointsDataFrame objects. Sometimes you can treat them as if they were data.frames, but other times you need to explicity convert them. This is one of those times, hence I’ve wrapped them in data.frame().

Recall that lsEA and lsNA have more columns than the environmental matrix: they also include year, countryCode, country. We only want to include the environmental variables when you project the observations into the ordination. To make sure that we use the same variables as in the original ordination of globalEnvM, in the same order, I select the columns explicitly to match that object:

data.frame(lsEA)[, colnames(globalEnvM)]

The output of dudi.pca and suprow includes a lot of information that we aren’t using here. We only need the li element, so I’ve selected that from each of the function outputs.

Occurrence Density Grids

Finally we’re ready to do the Niche Quantification/Comparisons. We’ll use the PCA scores for the global environment, the native and invasive environments, and the native and invasive occurrence records.

nativeGrid <- ecospat.grid.clim.dyn(global.scores,
                                   nativeEnv.scores,
                                   nativeLS.scores)

invasiveGrid <- ecospat.grid.clim.dyn(global.scores,
                                   invasiveEnv.scores, 
                                   invasiveLS.scores)

ecospat.plot.niche.dyn(nativeGrid, invasiveGrid,
                       quant = 0.05)

The resulting plot shows us the environmental conditions present in Eurasia (inside the green line) and North America (inside the red line). The green area represents environments occupied by Lythrum salicaria in Eurasia, but not in North America, the red area shows environments occupied in North America and not Eurasia, and the blue area shows environments occupied in both ranges. We can also see that there are a few areas in Eurasia with environments not present in North America, and vice versa. However, for the most part, Lythrum salicara doesn’t occur in this environments.

To get the index values we use ecospat.niche.dyn.index:

indexVals <- ecospat.niche.dyn.index(nativeGrid,
                                     invasiveGrid) 
indexVals$dynamic.index.w

##  expansion  stability  unfilling 
## 0.00598794 0.99401206 0.02349507

Projecting Climate Space to Geographic Space

We can take these results and project them onto a geographic map. This will show us which areas of the native and invasive range of Lythrum salicaria fall into each of the three categories.

Note that the function ecospat.niche.dynIndexProjGeo doesn’t yet handle the SpatRaster objects created by the terra package, so we have to convert it to a raster’s stack when we call it. For convenience and consistency with the rest of the code, I convert the results back to a SpatRaster.

geoProj <-
  ecospat.niche.dynIndexProjGeo(nativeGrid,
                                invasiveGrid,
                                env = globalEnvR)

plot(geoProj, legend = FALSE, 
     col = c("grey", "green", "red", "blue"),
     ylim = c(20, 80), axes = FALSE, mar = c(0, 0, 0, 0))
points(lsEA, col = 'darkgreen',
       cex = 0.25, pch = 23, bg = "white")
points(lsNA, col = 'darkred',
       cex = 0.25, pch = 23, bg = "white")
par(mar = c(0,0,0,0))
box()

Geographic Comparisons

You can also apply this analysis to geographic locations, instead of environmental conditions. This won’t make much sense for native vs invaded range comparisons, but it could be useful for comparing different species within the same area.

To demonstrate, let’s compare the distribution of Lythrum salicaria in North America before and after 1950. In this case, I need to go back to the original occurrence data, since I need the thin the records before and after 1950 separately. Otherwise, we won’t accurately capture the locations where Lythrum salicaria was collected in both time periods.

We use geographic coordinates here, so no need for a PCA. We do need to generate the ‘background’ coordinates. I’ll use expand.grid to create the locations for this. I’ve broken up the NA extent into 500 x 500 grids.

lsNA_All <- lsOccsAll[nAm]

lsNAearly <- subset(lsNA_All, lsNA$year <= 1950)
## thin records:
lsNAearly <- spatSample(lsNAearly, size = 1, strata = wclim)

lsNAlate <- subset(lsNA_All, lsNA$year > 1950)
## thin records:
lsNAlate <- spatSample(lsNAlate, size = 1, strata = wclim)

geoGrid <- expand.grid(longitude =
                        seq(-160, -40, length.out = 500),
                      latitude =
                        seq(20, 90, length.out = 500))

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     crds(lsNAearly))

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    crds(lsNAlate))

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)
plot(nAm, add = TRUE)

This looks pretty good. However, ecospat uses a kernel density formula to model the occurence distributions. As a consequence, it projects out into the ocean, which isn’t very realistic. To correct this, we need to mask the analysis to the continental land mass. This requires we have a vector map of the desired area.

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     crds(lsNAearly),
                                     geomask = nAm)

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    crds(lsNAlate),
                                    geomask = nAm)

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)

That gives more reasonable results. The blue area is the range occupied by Lythrum salicara prior to 1950, and the red area is the range it expanded into after that year. The small green areas are regions where it hasn’t been collected since 1950.

Note that this visualization is weighted by the density of points, so there might be a few pre-1950 records in the red area, or a few post-1950 records in the red area. That’s expected.

Summary

This is a fairly quick overview of this workflow. You’ll almost certainly want to consider thinning your observations, among other data cleaning procedures. I’ve also set the study extent very crudely. That might be appropriate for very large scale (global) studies. But you’ll usually want to think a bit more carefully about how you set your extent. The way you process your data will also differ depending on your context.

References

Chamberlain, Scott, Vijay Barve, Dan Mcglinn, Damiano Oldoni, Peter Desmet, Laurens Geffert, and Karthik Ram. 2021. “Rgbif: Interface to the Global Biodiversity Information Facility API.” Manual. https://CRAN.R-project.org/package=rgbif.

Cola, Valeria Di, Olivier Broennimann, Blaise Petitpierre, Frank T. Breiner, Manuela D’Amen, Christophe Randin, Robin Engler, et al. 2017. “Ecospat: An R Package to Support Spatial Analyses and Modeling of Species Niches and Distributions.” Ecography 40 (6): 774–87. https://doi.org/10.1111/ecog.02671.

Fick, Stephen E., and Robert J. Hijmans. 2017. “WorldClim 2: New 1-Km Spatial Resolution Climate Surfaces for Global Land Areas.” International Journal of Climatology 37 (12): 4302–15. https://doi.org/10.1002/joc.5086.

Hijmans, Robert J. 2023. “Terra: Spatial Data Analysis.” Manual. https://CRAN.R-project.org/package=terra.

Hijmans, Robert J., Márcia Barbosa, Aniruddha Ghosh, and Alex Mandel. 2023. “Geodata: Download Geographic Data.” Manual. https://CRAN.R-project.org/package=geodata.

Thioulouse, Jean, Stéphane Dray, Anne-Béatrice Dufour, Aurélie Siberchicot, Thibaut Jombart, and Sandrine Pavoine. 2018. Multivariate Analysis of Ecological Data with Ade4. New York, NY: Springer New York. https://doi.org/10.1007/978-1-4939-8850-1.

Managing Absolute Paths in Reproducible Analyses

Tue, 14 Feb 2023 00:00:00 +0000

In a previous post on reproducible analysis, I explained the importance of using relative paths in your scripts, and organizing your data in a single directory, in order to maintain portability. You want to be able to pack up your analysis in a zip file, or upload it as a single directory to GitHub or Dropbox, in order to share it with colleagues, or transfer it to a new computer.

This is best practice, but you may run into problems achieving this. One challenge is dealing with large data sets that you will use in multiple analyses. For example, we use WorldClim for a lot of our distribution work. Each copy of the global 30s dataset fills nearly 10GB (compressed). That would quickly fill my laptop harddrive if I stored a separate copy for each project.

I have created a separate directory to store such datasets, so that I only need to maintain a single copy on my computer. All analyses that use the WorldClim data will look for it in ~/data/worldclim/ on my laptop. On my workstation, I store the same data in ~/data/enm/worldclim. I could simplify this by using the same absolute path on both machines, but that wouldn’t help anyone else trying to use my script on their own machine.

The approach I’ve come up with for managing this requires a few lines of code to set the paths appropriately:

switch(system2("hostname", stdout = TRUE),
       LAPTOP.HOSTNAME =  ## my laptop:
         {worldClimPath <- "~/data/worldclim/"}, 
       WORKSTATION.HOSTNAME =   ## my workstation:
         {worldClimPath <- "~/data/enm/worldclim/"},
       {worldClimPath <- "dl/worldclim/"}) ## default

First, I check for the machine name with the function system2("hostname", stdout = TRUE). This calls the hostname command on the underlying operating system (which should work on Linux, Windows, and Mac). hostname returns the network hostname for your computer, which should be unique (at least within your organization). The switch function then compares this value to the names for my different machines, which I’ve already looked up. I can then use that information to set the correct path for my shared data.

In the case that I’m not on either of my machines, I set a default, relative path. That will allow other people to use my script without using my hard-coded paths.

In the case of WorldClim, the geodata package provides a convenient way to download the rasters:

library(geodata)
bio <- worldclim_global("bio", res = 10,
                        path = worldClimPath)

The function worldclim_global will check the path argument. If it finds the requested data there, it loads the local copy. If the data isn’t there, it downloads a new copy from the internet, and stores it there.

This makes for a convenient solution: running on my computers, all of my analyses will use the same shared data, and I won’t have to wait for downloads or exhaust my hard drives. But I can also share my code as-is with collaborators, and it will just work, without their having to change any paths.

Simple Maps in R with Terra

Mon, 13 Feb 2023 00:00:00 +0000

Reference

This is an update of my previous mapping tutorial. Spatial analysis in R is shifting to terra and sf as the primary packages, so I’ve translated my old, raster-based tutorial to the new workflow. See the RSpatial tutorial for a more detailed introduction/overview of using terra for GIS/spatial analysis.

The following tutorial walks through some common plotting tasks I use for distribution models.

Basemaps

The geodata package provides several convenient functions for downloading raster and vector maps for use as basemaps and spatial analysis. The first time you use these functions, they will download the requested maps from the internet. It will save the data in your working directory, or in a location specified with the path argument. The next time you request the same data, if it finds them in the local directory (or the specified path), they will be loaded from there, saving the time and bandwith necessary to download them.

library(geodata)

## Loading required package: terra

## terra 1.7.29

us <- gadm(country = "USA", level = 1, resolution = 2,
             path = "../data/maps/")
canada <- gadm(country = "CAN", level = 1, resolution = 2,
               path = "../data/maps")
mexico <- gadm(country = "MX", level = 1, resolution = 2,
               path = "../data/maps")

Countries are specified by their ISO code, which you can find by calling the function country_codes(). The by default, country_codes() returns a table of countries and the various ISO codes they have, as well as the continents they are in. The query argument lets you filter this table on any value. For example, you can get a table of North American countries with:

country_codes("North America")

or, just their ISO codes with:

country_codes("North America")$ISO3

The other arguments allow you to select, national, subnational, or subprovincial borders (level 1-3), the resolution (high: 1, low: 2), and the path to the directory where you want the maps stored.

These maps can be plotted directly with the plot command. If you want to combine them, use the add = TRUE argument to the second plot call:

plot(us, lwd = 2)
plot(canada, add = TRUE, col = 'red')
plot(mexico, add = TRUE, border = "green")

col sets the fill colour, border sets the outline color.

You can also request multiple countries as a single set of polygons, by passing a character vector to the country argument.

NorthAmerica <- gadm(country = country_codes("North America")$ISO3,
                     level = 0, resolution = 2,
                     path = "../data/maps/")

plot(NorthAmerica, xlim = c(-180, -50))

Note that xlim and ylim work as you would expect for plotting.

If you want to combine multiple polygons into a single object, use rbind:

CanUS <- rbind(us, canada)
plot(CanUS, xlim = c(-180, -50))

These maps are ‘unprojected’, meaning they are plotted in latitude/longitude degrees. That makes it easy to set the plot boundaries:

plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60))
plot(NorthAmerica, lwd = 2, add = TRUE)

NB: The size of your plot canvas is fixed, but a map can’t stretch. The x and y dimensions have to maintain the same aspect. That means zooming in one dimension (i.e. latitude only) won’t necessarily change the zoom of your map, if the other dimension fills the canvas. You’ll have to play around with the plot size, and both x and y dimensions together, to tweak your zoom.

It’s handy to have a shapefile of the Great Lakes, for making prettier maps. I created this one in QGIS and use it for plotting:

greatlakes <- vect("../data/maps/greatlakes.shp")

Adding Data

Vectors

You can add points to the plot like a regular scatter plot:

library(scales)  ## for the alpha function below

## 
## Attaching package: 'scales'

## The following object is masked from 'package:terra':
## 
##     rescale

gbif <- read.table("../data/trich-gbif.csv")
## Set the line color to gray to focus on the data points:
plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
     border = "gray")
points(gbif$X, gbif$Y, pch = 16,
       col = alpha("green", 0.2))

You can also convert your points to a spatial vector object, in which case R will know which columns to use for plotting. This is also necessary before we can project our data (see below).

gbif <- vect(gbif, geom = c("X", "Y"))
## plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
##      border = "gray")
## points(gbif, pch = 16, col = alpha("green", 0.2))

Subsetting vectors

You often need to select polygons that contain points, or select only those points that occur within a particular polygon. You can do this with R’s [ subsetting syntax. For instance, <polygons>[<points>, ] will select polygons that contain one or more points:

plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
      border = "gray")
## Select states and provinces where Trichophorum is
## present: 
plot(CanUS[gbif, ], col = 'red', add = TRUE)
points(gbif, pch = 21, col = 'white', bg = 'black')

Conversely, <points>[<polygons>, ] will select points that are within polygons. In this example, we’ll use the fact that our CanUS SpatVector object has a column named NAME_1 that holds the name of the state or province of each polygon:

plot(CanUS, xlim = c(-100, -50), ylim = c(30, 60),
      border = "gray")
plot(CanUS[gbif, ], col = 'red', add = TRUE)
## Select only points in NY state:
points(gbif[CanUS[CanUS$NAME_1 == "New York", ], ],
       pch = 21, col = 'white', bg = 'black')

Rasters

Similarly, you can plot rasters with plot:

trichPreds <- rast("../data/trichPreds.grd")
plot(trichPreds, xlim = c(-100, -50), ylim = c(30, 60))
plot(CanUS, border = "gray", lwd = 0.5, add = TRUE)

Cells with NA values are transparent. In this case, a species distribution model, low values are displayed in gray. This may be useful for visualizing the extent of the model. However, it looks a bit odd, and makes it hard to see limits of the high-suitability areas. You can tweak this by playing with the color ramp, but it’s also handy to ‘turn off’ the low values entirely (for visualization, not for analysis!!)

trichPredsTrim <- trichPreds
trichPredsTrim[trichPredsTrim <
               quantile(values(trichPreds),
                        probs = 0.75, na.rm = TRUE)] <- NA
plot(trichPredsTrim, xlim = c(-100, -50), ylim = c(30, 60))
plot(CanUS, border = "grey", add = TRUE)

The test I used here, trichPredsTrim < quantile(getValues(trichPreds), probs = 0.75, na.rm = TRUE) identifies all cells in the lower 75% of the suitability scores, which I then set to NA to make them invisible. I decided on 75% after experimenting with different values. In this case, 75% drops most of the grey background (the very lowest values), without eating into the areas that the prediction indicates are suitable.

You could also use an absolute value here, but then you’d need to know the actual distribution of the suitability scores. quantile is easier to tweak.

Alternatively, you can assign colours to the prediction raster based on the value of each pixel, after splitting them into categories:

suitability <- extract(trichPreds, gbif, ID = FALSE)[, 1]

predCat <- (trichPreds > quantile(suitability, 0.25)) + 
  (trichPreds > quantile(suitability, 0.15)) +
  (trichPreds > quantile(suitability, 0.05)) +
  (trichPreds > quantile(suitability, 0.025))

predCols <- c("white", "grey95", "yellow3", "orange", "red") 

plot(predCat, xlim = c(-100, -50), colNA = 'lightblue',
     col = predCols, ylim = c(30, 60), legend = FALSE)
plot(CanUS, border = "grey", add = TRUE)

I find using a small number of categories makes it easier to read the suitability map than trying to interpret the colour gradients usually used.

Projections

Lat/Lon maps look a bit square; we’re more used to seeing maps projected. A common projection for Canada is Lambert Conformal Conic. We can transform our data to this projection to make nicer maps:

## define the projection
canlam <- "+proj=lcc +lat_1=49 +lat_2=77 +lat_0=49 +lon_0=-95 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs"

## project our vector data:
CanUS.lcc <- project(CanUS, canlam)
gl.lcc <- project(greatlakes, canlam)

## Now we to set the projection of our points:
crs(gbif) <- "+proj=longlat +datum=WGS84"

## Finally, we can project our points to LCC:
gbif.lcc <- project(gbif, canlam)

Note that our data needs to be in an object of class Spat*, and it must have a defined coordinate reference system (CRS) before we can project it to a new CRS. The crs function allows us to explicitly set the projection. See the RSpatial tutorial for more details on specifying CRS with terra.

The same function works for rasters (sort of):

rasterLCC <- project(trichPredsTrim, canlam)

This will reproject my raster from lat/lon to Lambert Conformal Conic, but we will unavoidably lose some precision when we do this. This is fine for visualization, but you should avoid projecting raster data used in analysis. For more details, see the terra tutorial.

plot(rasterLCC)
plot(CanUS.lcc, border = "grey", add = TRUE)

The units are no longer Lat/Lon, but meters. We can read them off the plot to improve the zoom:

plot(rasterLCC, xlim = c(0, 2500000),
     ylim = c(-1500000, -400000))
plot(CanUS.lcc, border = "grey", add = TRUE)

Formatting

With the data plotted, we can then turn to making the map a little prettier. Note that when working with Spat* objects, we can’t set the plot margins with par as we usually do. Instead, we use the mar argument within the plot function:

## Make a panel with two plots
par(mfrow = c(1, 2))

## store the plot limits:
my_xlims <- c(0, 2500000) 
my_ylims <- c(-1300000, -200000)

## Plot the points:
plot(CanUS.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", background = "lightblue",
     col = "white", axes = FALSE, mar = c(0.1,0.1,0.1,0))
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
points(gbif.lcc, pch = 16, col = alpha("grey30", 0.2),
       cex = 0.7)
box() 

plot(CanUS.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", background = "lightblue",
     col = "white", mar = c(0.1,0,0.1,0.1), axes = FALSE)
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
plot(rasterLCC, add = TRUE, legend = FALSE, axes = FALSE)

## plotted again to put the border lines on top:
plot(CanUS.lcc, border = "grey", add = TRUE) 
box()

If you want to plot the state/provincial borders on top of the raster, you need to add those layers last. But you can’t set the background colour of the raster layer to “lightblue” (or at least I haven’t figured that out), so the ocean stays white. I get around that by plotting the boundaries twice, first to set the background colour, and then to put the state lines on top of the raster.

Data Management for Reproducible Science

Mon, 17 Oct 2022 00:00:00 +0000

Introduction

Research is reproducible when others can reproduce the results of a scientific study given only the original data, code, and documentation (Alston and Rick, 2021)

Benefits to the Author:

Clear and complete documentation of your work makes it easier to share, write up and extend in future work, including responding to reviewers and developing new projects
Conscientious documentation of your work involves a great deal of error-checking, which is reassuring to you – that you haven’t missed anything, or mis-remembered what you did; and to your readers – that you have conducted your work in a rigorous manner
Reproducible work gets cited more, and developing a data archive creates a new citable product from your research.

Benefits to the Community:

Increases the speed and fidelity with which we can learn and apply new approaches.
Makes it easier to avoid mistakes (through the care and attention required to create the archive), and to detect and correct them if they do happen (by allowing others to critically review your work)

How to make your work reproducible

There are two related goals in producing a reproducible analysis: portability, and reproducibility. Reproducibility is perhaps obvious, but it’s not enough that you can reproduce your analysis on your computer. You are the only person with access to your computer. Your work should be reproducible on anyone’s computer.

Portability

To achieve portability, we need to know which files are needed for an analysis, and to organize them in a way that they can be readily moved from one computer to another. In practice, this means they’ll all be in a single directory, and that directory will only contain files for that particular analysis. You may have several related analyses that share a directory. That’s ok, but take time to organize them in a sensible way.

While you may have related analyses together in a directory, you don’t want to mix unrelated files and data in this directory. That will make it harder to keep track of what is needed and what isn’t, and will waste space on the computers where this work is ultimately archived.

README.txt

There are many ways you can organize your work within this directory. One absolute requirement is there needs to be a clear guide to your files, a “Table of Contents”. This should be a ‘plain text’ file - something that can be opened by any text editor. Your archive might be around for decades, and you don’t know if your readers will be able to find a copy of MSWord97 when they need to read it. We can be reasonably confident that plain text files will be accessible for a long time to come.

By convention, this file is called README.txt, and some data archiving services (DataDryad) require that you include a file with this name. You should probably use this name, unless you have a good reason not to.

One minor exception: if you use markdown, or RMarkdown, you can use these formats for your README. They are both plain text, and even if your audience doesn’t use them, they can open a README.md or README.Rmd file in any text editor. There are other, similar simple markup formats used in different coding communities. As long as they are saved as plain text files they will meet our requirements.

The contents of your README should describe as clearly as possible the contents of your archive. Here is an excerpt from one of mine:

Start with trich.Rmd to read our draft manuscript. See trich-prep.Rmd for the bulk of the code used in generating this manuscript.

File List:

trich.Rmd : main manuscript file

plantarum.json : bibliography (Zotero bibliography)

trich-prep.Rmd : The bulk of the code used in the analysis. Loaded from trich.Rmd to regenerate the figures and tables there.

data/

data/ssr_raw.csv : raw microsatellite data. See trich-prep.Rmd (Loading Data) for code to load and translate this to a genind object

data/survey-pops.csv : coordinates of sampled populations

data/eval.opt.2020-06-24.Rda : the output of the Maxent modeling, saved as a binary R Data object. Load into R with the load() function, so you don’t need to repeat the lengthy Maxent analysis.

data/trich-gbif.csv : GBIF records used in the Maxent analysis

data/trich_soil.csv : Soil analysis for each sampled population

data/maps : Maps (shapefiles and rasters) used in the Maxent analysis, and for some of the manuscript plots

I prefer to use RMarkdown to develop my manuscripts, as this allows me to keep the code for figures, and the resulting images, together with the text that describes the methods and interprets the results. If you prefer to manage your code separately from your writing that’s fine too, but you may end up structuring your archive a little differently.

Directory Organization

If you only have a few files, you may not need to do any further organization of your archive. I find it helpful to use subfolders to keep things organized. Depending on the needs of a project, I use:

data/: unprocessed data used in the analysis
processed/: intermediate data files, generated from data and not stored permanently
downloads/: storage of large external datasets used in the analysis, but not stored permanently in the archive (e.g., WorldClim Climate Data); be sure to include links to the source of any external data you are not archiving yourself!
plots/: images generated by the analysis
code/: code used in the analysis, if it’s not in the top-level directory

These are not hard rules. You can use whatever structure suits your project. Just be sure to explain it in your README.

Reproducibility

File organization gets us most of the way to portability. There are a few things we need to do in our coding to complete this arrangement, and of course to ensure we can reproduce the analysis once we move it to a new computer.

Use Relative Paths

A absolute path is the location of a file on a particular computer. The absolute path to the file I’m editing right now is /home/smithty/blogdown/content/tutorials/2022-10-17-data-management/index.md. That location will only ever exist on a Linux computer with a user named smithty. If I moved it to a Windows computer, it might be located at C:\Users\smithty\Documents\blogdown/content/tutorials/2022-10-17-data-management/index.md. If I try to refer to this file by its absolute path on Linux on the Windows machine, I won’t find it.

On the other hand, from the top directory of my blog, blogdown/, this file will have the same relative path on both machines: content/tutorials/2022-10-17-data-management/index.md. That means links to this file using the relative path will work just fine on both machines.

We should use relative paths in our analyses too.

For example, I could use an absolute path to load my data:

samples <-
  read.csv("/home/smithty/nextcloud/trich/2020-06-25/data/survey-pops.csv")

But this won’t load on anyone else’s computer. If we do this instead:

samples <-
  read.csv("data/survey-pops.csv")

Anyone with our archive can run the code as it is, as long as they have the working directory set properly. For my projects, I do this by running my code from the top directory of the archive. In this case, I have the directory structure:

├── data
│   └── survey-pops.csv
├── README.md
├── trich-prep.Rmd
└── trich.Rmd

When I run code from trich.Rmd, I set the working directory to the location of that file. RStudio manages this for you with its project support. If you don’t use that feature, you can tell R to use that location when it starts. After that, stick to relative file paths, and you don’t need to worry about your code breaking when you move to a different computer.

Structure Your Code to Run On Its Own

Two common problems that make it hard to run your code are mixing your ‘good’ code with non-working code, and writing code that requires you to update it by hand to finish your analysis.

Mixing good and bad code might look like this:

## Load in our data:
myData <- read.data("data/myfile.csv")
myDataScaled <- scale(myData)
myDataScaled <- scale(myData, center = FALSE)

myData <- myData[, -1]
myDataScaled <- scale(myData, scale = FALSE)

It’s not unusual to accumulate various versions of code (should I scale this, center it, both? Do I need the first column?). While you’re actively working on your code, you may find you have multiple versions in the same file. They need to be clearly commented, for your own benefit! But more importantly, when you decide which version you’re going to use, remove the rest. Don’t expect yourself (and certainly don’t expect anyone else) to figure out which lines they should run, and which they should skip.

## Load in our data:
myData <- read.data("data/myfile.csv")

## drop the name columnn
myData <- myData[, colnames(myData) != "name"]

## don't scale, use raw data
## myDataScaled <-scale(myData)

If you have a few lines of code that you might want to revisit, comment them out and leave yourself a note in a comment. If you have large blocks of code you aren’t using, but want to keep a record of, put them in a separate file. The end product should be a file that you can run from start to finish, without deciding which lines to skip and which to run.

A similar problem occurs when you have code that works, but requires you to manually update it in order to complete your analysis. e.g.,

myData <- read.table("data/experiments.csv")

myExperiment <- subset(myExperimentData, exp == 1)

myExperimentResult <- processingCode(myExperiment)

## repeat for exp 1-20

Code structured like this requires you to edit and re-edit the code many times to complete your work. That is tedious, and it’s easy to make a mistake. And when you update processingCode, you need to rerun everything by hand. You can avoid this with loops and lists.

R Programming: Do Not Save Your Workspace!

R offers to save your current workspace when you close the terminal. This is sometimes convenient, but can cause hard to detect problems with your analysis. If you happen to alter one of the objects you are working with in an R session, but don’t capture the code in your script file, you won’t have a record of what you’ve done. If you then save your workspace, the next time you work on your code, you will be able to keep using that modified object. At this point, your code and data are out of sync, with nothing to indicate how they differ.

At best, this is inconvenient. At worst, if undetected, you can waste weeks or months analyzing the wrong data! Better to avoid the risk and set R to never save your workspace.

Metadata: data about your data

If your archive includes data files, these should also be in an open format, such as comma-separated or tab-separated text files (typically with a name like FILE.csv, DATA.txt, or RECORDS.txt. That ensures that anyone can open your data without needing a proprietary program to do so.

Data Tables

In addition, you need to document how your data is coded. This includes things like:

Variable names and descriptions
Definition of codes and classification schemes
Codes of, and reasons for, missing values
Definitions of specialty terminology and acronyms

Be sure to include things like units (meters or feet?), and what your special codes mean (is pet_l the petal length or the petiole length?, what is lf1, lf2 and lf3?). This is also a chance to review your data for consistency - are you using NA and -1 for missing values? Do you have multiple different phrasings for the same thing?

Depending on your project, you may also need to distinguish between absent evidence (you didn’t sample on a day) and evidence of absence (you sampled on a day, but didn’t find any events/individuals). If your analysis will include sampling events that didn’t result in any observations, you’ll need to document these ‘true negatives’ in your data table.

Data Sources

If you’re using data from outside sources, like GBIF.org or WorldClim.og, be sure to record their citation details, including a DOI, if available, when you download them. This will ensure you can properly cite them later, and that your readers will be able to access the same data if they want to reproduce your work. Most data providers have clear policies as to how you should cite them, and what you’re allowed to do with the data they share (i.e., can you share it or archive it yourself). For example, here are the policies for WorldClim and GBIF.

How to get there from here

Now we have an idea of what we want our data archive to look like, how do we get there? Start by setting up your directory structure, and making a README file. If you’re starting from scratch, make it a practice to review your directory regularly, to see what you’ve done, what you’re no longer using, and updating your README to capture that. This kind of regular reflection is useful to track your progress, and helps you keep on top of your archive work so you don’t have a huge mess to wrangle at the end of your project.

If you’re already well-along in your project, it may be easier to create an ‘aspirational’ directory and populate it from your existing work. I do this regularly! I often find I’ve charged into an analysis without thinking about archiving, and after a few weeks I have an unholy mess of files and data to deal with. In that case, I create a new directory, a README, and copy over the main code file I’m using to that directory. When I get to the first data file in that directory, I copy it over to the data/ directory in the new archive, adjust the code to use the relative path, if necessary, and continue. This can be very helpful in clarifying what code and files you actually need and use, and what can be left behind.

This is also a good opportunity to ensure that your analysis is structured in a way that it can run start-to-finish without manual editing

You don’t need to delete anything in the old directory, you can keep it in case you later decide you want to revisit some ideas in there.

What to do with it all

One of the benefits of structuring your code in a single directory is there are lots of tools that you can use to manage it. git and GitHub are popular, and very powerful for managing code, especially tracking different versions of the same files. However, they require a certain amount of discipline to get the full benefit, and they are challenging to learn. RStudio does have good support for GitHub repositories.

You can also keep it simple, and sync your directory to Google Drive, Dropbox, Nextcloud, or many other options. Once your work is published, you can archive it permanently on Zenodo, DataDryad, or other online services.

All of these options will support housing a single directory and its subdirectories. None of these options will be easy to deal with if you have files spread across multiple directories and mixed with files from other projects!

Examples

I’ve been doing some version of this for my own work for years, but have only recently moved to permanent, public archives of my work. Here are three recent examples:

Hayes et al. (2022): The Genetic Diversity of Triploid Celtis pumila and its Diploid Relatives
Nowell et al. (2022): Conservation assessment of a range-edge population of Trichophorum planifolium
Foster et al. (2022): Testing the assumption of environmental equilibrium in an invasive plant species over a 130 year history

I’m still figuring out how best to do this, and my practice will definitely continue to change and evolve. Regardless of the specifics, I have benefited enormously from investing the time needed to make coherent archives of my projects.

References

Alston, J. M., and J. A. Rick. 2021. A Beginner’s Guide to Conducting Reproducible Research. The Bulletin of the Ecological Society of America 102: e01801.

Foster, S. L., H. M. Kharouba, and T. W. Smith. 2022. Testing the assumption of environmental equilibrium in an invasive plant species over a 130 year history. Ecography: e06284.

Hayes, A., S. Wang, A. T. Whittemore, and T. W. Smith. 2022. The Genetic Diversity of Triploid Celtis Pumila and its Diploid Relatives C. Occidentalis and C. Laevigata (Cannabaceae). Systematic Botany 47: 441–451.

Nowell, V. J., S. Wang, and T. W. Smith. 2022. Conservation assessment of a range-edge population of Trichophorum Planifolium (Cyperaceae) reveals range-wide inbreeding and locally divergent environmental conditions. Botany 100: 631–642.

Georeferencing Notes

Wed, 23 Mar 2022 00:00:00 +0000

GBIF Data

GBIF is the main online clearing house for occurence data in the world. It includes most (but not all) online herbarium databases. It also includes iNaturalist records, as well as a number of other survey repositories. I’m not familiar with all of the sources included.

The main page presents a search bar. Enter your species name (e.g., Rubus chamaemorus) in the box, and select the ‘occurrences’ tab. You’ll be taken to a list of results that match your species (2.9e6 results for R. chamaemorus). This may include records for other species, presumably because the name of the species you are searching for is recorded in the metadata for those records. We don’t usually want these.

To fix this, look for the prompt: “Your search matches a Species: ‘Rubus chamaemorus L.’. Do you wish to limit your search to this taxon only?”, in the left-hand tool bar. Select yes to restrict the results to only records for your species (i.e, excluding records where your species is mentioned in the data). For R. chamaemorus we drop from 2.9 e6 records to 87 e3.

Basis of Record

In the right hand toolbar, look for the option: Basis of Record. Expanding that tab, you see a list of options, along with the number of records for each option. For R. chamaemorus I see:

Observation (6)
Machine Observation (46)
Human Observation (79686)
Material Sample (516)
Material Citation (1)
Preserved Specimens (6032)
Fossil Specimen (133)
Living Specimen (28)
Occurrence (1222)

The Basis of Record field is no longer recommended, and is deprecated in the DarwinCore system. Looking through the various categories, that’s probably a good idea. I’m not sure what the difference is between ‘Observation’, ‘Machine Observation’, ‘Occurrence’ etc, and it looks like the groups are applied very inconsistently.

‘Human Observation’ does include iNaturalist records, but it also includes a lot of other data. The iNaturalist records are updated periodically, but it’s also possible to download them directly from iNaturalist. That’s my preference, as then I don’t need to concern myself with what else might be included under ‘Human Observation’.

To restrict ourselves to herbarium records, we’ll select ‘Preserved Specimens’ and move on. For R. chamaemorus that brings us down to 5.9 e3 records.

Geographic Filters

With our data filtered to just herbarium records, we can see where they are from by clicking on the ‘map’ tab at the top of the table. Doing that I see that of the 5900 records available, 2700 have coordinates already.

The rectangle and pentagon symbols in the upper right provide tools for further filtering the results by rectangle or polygon, respectively. You can also use the buttons on the upper left to zoom in and out, and to examine record details for an area (the arrow pointing at a circle). In my case, I want all records for the planet, so I’ll skip those options.

Downloading

Now I can download the records. I think GBIF requires that you have a (free) account for this. That helps them track use. The ‘simple’ format is sufficient, the DarwinCore format includes a lot of extra columns.

VERY IMPORTANT!

Record the doi for your download! You will need this to properly cite the data you use in a publication. If you have a GBIF account, you may be able to recover it after the fact. You can still cite GBIF in your paper without a DOI for the download, but it’s much less useful.

For my R. chamaemorus data, GBIF provides the following citation:

GBIF.org (23 March 2022) GBIF Occurrence Download https://doi.org/10.15468/dl.uhqfg3

For large queries, it may take several minutes or longer for the data to be ready. You’ll get an email when it’s available.

Data Processing

Uploading

Once you have your data, we’ll upload it to our shared Google Drive for processing. From the shared drive, select New -> File upload, and select the unzipped .csv file from your GBIF download. Double-click on the file, and select ‘Open With Google Sheets’ at the top of the page.

The filename is listed in the top left corner. Change that to the species name, possibly with any additional details to distinguish it from other files with the same species. For most cases, when we have a single global file for the species, just the name is fine.

However, we do want to link our data to the GBIF DOI. I do this by renaming the sheet to the DOI url: https://doi.org/10.15468/dl.uhqfg3. If we need to maintain separate datasets for the same species, they can be added as new sheets. This hasn’t come up yet.

Prepping the Spreadsheet

Even the ‘simple’ export has a lot of extra columns we don’t need. These can be ‘hidden’ from view to make it easier to work with the file. The main columns we use are:

countryCode
locality
stateProvince
decimalLatitude
decimalLongitude
coordinateUncertaintyInMeters
georeferencedBy
georeferencedDate
georeferenceSources
georeferenceProtocol

We do need to add several columns for our georeferencing. Find the column coordinateUncertainty, and add five columns to the right. The names of these columns are:

georeferencedBy
georeferencedDate
georeferenceSources
georeferenceProtocol
georeferenceRemarks

Data Entry

For records that already have coordinates, don’t change them or add any additional notes.

For records that you locate:

add your name to georeferencedBy. Use the same spelling/abbreviation every time
Add the date in yyyy-mm-dd format (i.e., 2022-03-23) to georeferencedDate
Add the coordinates to decimalLatitude and decimalLongitude
Add the uncertainty in meters to coordinateUncertaintyInMeters
Add the location source to georeferenceSources (usually this will be “GoogleMaps” or “GeoLocate”)

georeferenceRemarks is for comments regarding your efforts to locate the record. Some possible values include:

AttemptedGoogleMaps: I tried to find the location using google maps, and could not
AttemptedGetty: I tried to find the location using Getty Thesaurus of Geographic Names
AttemptedGNIS = I tried to find the location using Geographic Names Information System (GNIS)
AttemptedNRC = I tried to find the location using Natural Resources Canada, geographical place/feature names of Canada
LocationNotFound = I could not find this location / I could not georeference it
ContradictingLocation = I could not georeference the location as there are two locations mentioned which are not close by each other
Duplicate = duplicate georeference; other georeferences share the same lat/long and uncertainty

Problems

Inconsistencies in Source Data

We don’t normally need to modify the original data. However, if you see the following known errors they can be corrected as indicated.

0, 0 latitude and longitude is an error. Can be overwritten, do so as you are working through the data.
coordinateUncertaintyInMeters without latitude and longitude coordinates is an error. Can be overwritten, do so as you are working through the data
Data with no country code: If no other locality data given, cannot be given a georeference; If locality data given, can add the country code and province information.
Localities with no province should be corrected as you are working through the data. Either add the province that is listed in the locality to the province column, or add the province that applies given the country and locality to the province column.

Finding Coordinates

Start by sorting the spreadsheet by country, state within country, and locality within state. This will save time, as you’ll be processing records from the same location together. In many cases you’ll have multiple records in the same state, and possibly in the same town. If you’re lucky, there will be records with coordinates for the same locaion as records without coordinates. Sorting allows you to take advantage of this.

The final step is actually finding the coordinates. Hopefully there’s enough detail in the locality field that you can search for landmarks in GoogleMaps, and use the satellite view to get close to the actual location. It really depends on the record what will be possible.

Google Maps

Google Maps has a very straightforward interface, and is the simplest option. The GeoLocate web client provides a slightly more sophisticated search tool, which allows you to use administrative boundaries or editable circles to set uncertainty values.

GeoLocate

GeoLocate also supports batch processing. This allows you to upload a file and process a group of records together, automatically. You will still need to review the results to make sure they make sense, but you can manually edit them directly in the map viewer. When you’re done you can export the results as a csv file. To use this approach, you need to prepare the spreadsheet according to the instructions, and we need a simple approach for exporting from Google Sheets, and reimporting back into Google Sheets when we’re done.

Record coordinates to five decimal places. At the equator, 1 degree is about 100 km, so 0.00001 is about 1 m. We don’t need sub-meter accuracy, and none of these records are that precise.

Uncertainty

Estimating uncertainty is a judgement call. If the locality data is very precise (“the corner of Main and Second Avenue in Springfield”) you might have a very small uncertainty. If all you have is a state, your coordinate will be the center of the state, and the uncertainty will be the diameter of the state. Don’t worry about being precise: 1m, 100m, 1000m, 10000m (or bigger) are fine.

For vague/approximate locations, do what you can. Try to think like a field biologist. How far from a town do you need to be before you stop describing your location as “South of X”, and start describing it as “between X and Y”?

Priorities

Some records take a lot of work to locate, and some are impossible. We’re most interested in filling in gaps. If we have a thousand records from Ottawa, we don’t need to waste a lot of energy tracking down the coordinates for one more record. But if we only have one record for China, it will be worth spending some time trying to find it. Similarly, earlier records are rarer and more valuable: the 100th record from New York (in 2021) doesn’t add as much value as the first one (from 1830).

Tips For Deciphering Labels

Compiled by Laura Kostyniuk.

Abbreviations seen in data

Latin

opp. = likely oppidum, town in english
fl. = likely flumen, river in english
urb. = likely urbs/urbem, city in english

Other languages

VC or v.c. in Great Britain codes likely stands for vice county. Not sure if applies to other countries
Arred. / Arred. de in Portugal is abbreviation for Arredores; meaning ‘surroundings of’
Locations in Sweden starting with W often do not show up in Google maps. Sometimes can be found if W is changed for V (ex: Wrigstad = no hits, Vrigstad = village in right general location).

Other Sources

For valuable records you can’t locate in GoogleMaps or GeoLocate. Compiled by Laura Kostyniuk and Shannon Ascensio.

Place/feature names, gazetteers

Coordinate converters

Tool-online Coordinate converter: search by country or by world if the type of coordinate system you are looking to convert is available
Converter for Swiss coordinates: If numbers are written as 123.4/567.8, 567.8 is 567 800 as a Y coordinate, and 123 400 as an X coordinate

Useful maps

Europe in the XIX century
Poland: ATPOL map
US: PLSS system of township/section/range
Ontario: Historic County/Township maps
Ontario: Lot/Concession Maps
Quebec: historic counties (1920)
OldMapsonline Old map searching tool; search by location, year, and add filters to search.
MapCarta: Good general map with more locations than Google; sometimes have to manually search the map though, locations will not always show up when searched in search bar.
FindLatitudeLongitude Another general mapping / searchable tool.
Spanish grid map
Scandinavia

Schoener's D and Study Extent

Thu, 02 Dec 2021 00:00:00 +0000

Background

Schoener’s D was created by Schoener (1968) He was studying the feeding niche of anoles, and needed a way to quantify the overlap in prey items for different species. This is what he came up with:

\[D(p_X, p_X) = 1 - \frac{1}{2} \sum_i \vert p_{X,i} - p_{Y, i} \vert\]

Here, $p_{X,i}$ and $p_{Y,i}$ are the frequencies for species $X$ and $Y$, respectively, for the $i^{th}$ category. For Schoener, the categories were prey sizes. In the context of distribution modeling, they would be regions along an environmental gradient, and the ‘frequencies’ are the fitted values from an SDM, or the density values from an Ecospat dynamic niche grid.

Warren et al. (2008) pointed out some subtle theoretical issues with Schoener’s D in this context, and proposed his own index I, based on the Hellinger distance, to better account for them.

Hellinger’s distance:

\[H(p_X, p_Y = \sqrt{\sum_i(\sqrt{p_{X,i}} - \sqrt{p_{Y,i}})^2}\]

Warren’s I:

\[I(p_X, p_Y) = 1 - \frac{1}{2} H(p_X, p_Y)\]

In application, Schoener’s D suggests that the $p_{X, i}$ values reflect relative use of a particular habitat. However, ENM predictions indicate the relative ‘suitability’ of a cell for occupancy (i.e., presence or absence) by the study species, but do not necessarily reflect density.

However, Warren also noted that despite the potential issues, in practice there is little difference in the qualitative results following from D and I. I think Schoener’s D is more commonly used now, but either or both may show up in distribution modeling studies.

Overlap vs Correlation

Warren (2018) made an interesting contrast between two species’ niche overlap (D), and the correlation between their suitability scores. Schoener’s D quantifies the extent to which a pair of species may interact in the same space (i.e., they’re both likely to be present together in a location). This is important to know, especially in the context of niche-shift studies (e.g. Atwater and Barney, 2021). But while they tell us about where species are found along an environmental gradient, they don’t tell us anything about how they respond to that gradient. In fact, species with perfectly opposite responses to the environment may still have relatively high niche overlap, D.

Let’s revisit the example from Warren (2018). We start with the olaps helper function, which calculates the statistics of interest:

library(ggplot2)
library(grid)

olaps <- function(sp1, sp2){
  ## Calculate Schoener's D, Warren's I, and Spearman
  ## Correlation for sp1 and sp2

  ## sp1 and sp2 are the relative occupancy values for each
  ## species along the same environmental gradient

  ## scale the values for each species 0:1
  sp1 <- sp1/sum(sp1)
  sp2 <- sp2/sum(sp2)
  
  plot.table <- data.frame(
    species = c(rep("sp1", length(sp1)),
                rep("sp2", length(sp2))),
    env = c(seq(1:length(sp1)), seq(1:length(sp2))),
    suitability = c(sp1, sp2))

  D = 1 - sum(abs(sp1 - sp2))/2
  I = 1 - sum((sqrt(sp1) - sqrt(sp2))^2)/2
  cor = cor(sp1, sp2, method = "spearman")

  grob <- grobTree(textGrob(paste("D =", round(D, 2),
                                 "  I =", round(I, 2),
                                 "  Cor =", round(cor, 2)),
                           x = 0.1,  y = 0.95, hjust = 0,
                           gp = gpar(fontsize = 15)))
  

  suitplot = qplot(env, suitability, data = plot.table,
                   col = species, geom = "line") +
    annotation_custom(grob)

  return(list(
    D = D, I = I, cor = cor, suitplot = suitplot
  ))
  
}

Now we can recreate the examples from Warren (2018).

sp1 <- seq(0.1, 1.0, 0.001)
sp2 <- seq(0.1, 1.0, 0.001)

olaps(sp1, sp2)

Figure 1: Identical Species

sp1 <- seq(0.1, 1.0, 0.001)
sp2 <- seq(1.0, 0.1, -0.001)

olaps(sp1, sp2)

Figure 2: Species with Inverse Environmental Response

The point that Warren (2018) was making is that two species may occupy more or less similar locations along an environmental gradient, while having very different responses to that gradient. This isn’t a problem. But it does highlight the importance of clearly articulating the question you are asking in your research, and making sure that the analyses you choose are actually answering that question.

Niche Overlap vs Study Extent

Something else struck me reading Warren’s post. The toy examples he used represent a very narrow slice of an environmental gradient; that is, the portion where both species are present. Applying these analyses to global patterns, as we do when comparing the distribution of invasive species in their native and introduced range, and especially when we apply these analyses to large numbers of species, we can (potentially) include much broader gradients. And this can have significant impact on theses statistics.

Here’s a (ever so slightly) more realistic example to illustrate. We’ll define our gradient over the range 0 to 50

env <- seq(0, 50, by = 0.01)

Then we’ll define two species, with partially overlapping ranges:

sp1 <- dnorm(env, mean = 22.5, 2)
sp2 <- dnorm(env, mean = 27.5, 2)

Now compare the species ‘globally’:

olaps(sp1, sp2)

Figure 3: Global Analysis

At this scale, their response to the gradient appears to be highly correlated, while they have low niche overlap.

If we zoom in a bit, and ‘trim’ off the lowest and highest 1000 values on our gradient, we can emulate a ‘continental’ extent:

## ignore the lowest and highest 1000
## environmental values 
slice <- 1000:4000 

olaps(sp1[slice], sp2[slice])

Figure 4: Continental Analysis

Correlation drops, but niche overlap remains identical. On reflection, this makes sense. Locations where neither species are present get no weight in the calculation of D, so dropping ‘empty’ gradient has no impact. On the other hand, those locations do contribute to inflating correlation.

Now what if we shift our focus, such that the distribution of our species is not equally represented:

slice <- 1000:2500 
olaps(sp1[slice], sp2[slice])

Figure 5: Regional Analysis

Correlation jumps up, as despite both species increase together over most of the sampled gradient. And with this particular slice, our niche overlap is twice the ‘true’ value when we consider the full gradient.

Finally, we can zoom in on the center of the gradient, where both species are equally represented (although with inverse responses):

slice <- 2000:3000
olaps(sp1[slice], sp2[slice])

Figure 6: Contact Zone

Correlation drops again, accurately reflecting the inverse pattern. And D is back down close to the ‘true’ value. That’s ‘lucky’, as my toy species have perfectly symmetrical distributions, so sufficently large, symmetrical regions around the mid-point between the two of them will give reasonably accurate estimates.

Implications

Why does this matter? If you’re interested in comparing the environmental responses of two species, the results (correlation) can vary quite dramatically depending on the extent of your study.

On the other hand, Schoener’s D is robust to data that includes ‘too much’ of a gradient (i.e., extending beyond the region occupied by either species). But it can be sensitive to undersampling a gradient, where the relative occupancy of each species varies depending on how you set set your extent.

In other words, if you’re interested in the ‘underlying models’ that govern species’ comparative distributions along a gradient, you need to be very clear about the scope of the question, and how much of the environmental gradient you sample. But if you want to quantify niche overlap (or relative niche shift), then you want to include environments well beyond the regions actually occupied by your study organisms.

All of which is trivial to do when you get to create the species on the computer, and much trickier when you need to infer the details from museum records and climate rasters!

References

Atwater, D. Z., and J. N. Barney. 2021. Climatic niche shifts in 815 introduced plant species affect their predicted distributions I. Martins [ed.],. Global Ecology and Biogeography 30: 1671–1684.

Schoener, T. W. 1968. The Anolis Lizards of Bimini: Resource Partitioning in a Complex Fauna. Ecology 49: 704–726.

Warren, D. 2018. Species In Space: Why add correlations for suitability scores? Species In Space. Website https://enmtools.blogspot.com/2018/10/why-add-correlations-for-suitability.html [accessed 2 December 2021].

Warren, D. L., R. E. Glor, and M. Turelli. 2008. Environmental Niche Equivalency Versus Conservatism: Quantitative Approaches to Niche Evolution. Evolution 62: 2868–2883.

Thinning Occurrence Records in R

Tue, 26 Oct 2021 00:00:00 +0000

Note that this tutorial refers to the thinning method used in the old version of the rspatial.org tutorial, which used the raster package (along with dismo) for the GIS computations. The terra package will shortly be replacing raster, and all new code should use this instead. The details of spatial thinning with terra are presented in my new ecospat tutorial

A common approach to reducing spatial bias in occurrence records is to randomly select one (or a small number) of samples present in each cell in the landscape. This uses the gridSample function from the package dismo (Hijmans et al. 2017), as described at RSpatial.org.

However, the code presented at RSpatial uses a newly-created raster layer to thin the records. This layer is based on the extent of your occurrence data; even if you set the resolution to match the resolution of the environmental rasters you use, the they won’t necessarily be aligned. That means the cells will be the same size, but the edges won’t line up.

A consequence of this is that you might end up keeping more than one sample, or removing all samples, from a single cell in your environmental data, even after thinning to one sample per cell in your newly-created raster. This lead to some strange behaviour in one of my downstream analyses, where the results for a small data set changed each time I reran the analysis.

I’ll demonstrate using the example from RSpatial:

library(dismo)
library(maptools)
library(sp)
library(raster)

wclim <- getData("worldclim", var = "bio", res = 10,
                path = "../data")

## Warning in getData("worldclim", var = "bio", res = 10, path = "../data"): getData will be removed in a future version of raster
## . Please use the geodata package instead

wc1 <- wclim[[1]]

## crop the climate data to speed up grid creation
wc1crop <- crop(wc1, extent(c(-70, -60),
                           ylim = c(-20, -10)))

data(acaule)
data(wrld_simpl)

acgeo <- subset(acaule, !is.na(lon) & !is.na(lat))
dups2 <- duplicated(acgeo[, c('lon', 'lat')])
acg <- acgeo[!dups2, ]
i <- acg$lon > 0 & acg$lat > 0
acg$lon[i] <- -1 * acg$lon[i]
acg$lat[i] <- -1 * acg$lat[i]
acg <- acg[acg$lon < -50 & acg$lat > -50, ]
coordinates(acg) <- ~lon+lat
crs(acg) <- crs(wrld_simpl)

plot(acg, pch = 20)
plot(wclim[[1]], legend = FALSE, add = TRUE)
points(acg, pch = 20)
plot(wrld_simpl, add=T, border='blue', lwd=2)
box()

This is the same example from RSpatial. See the link above for more details.

Now we want to thin our records, such that we retain only one observation for each cell in the WorldClim climate layer. To track what’s going on here, I’ll zoom in on one of the crowed areas:

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)

Now lets overlay the grid generated by the code from RSpatial:

r <- raster(acg)
# set the resolution of the cells to the same as wclim
res(r) <- res(wclim)
# expand (extend) the extent of the RasterLayer a little
r <- extend(r, extent(r)+1)
p <- rasterToPolygons(r)

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)
plot(p, add = TRUE)

Notice how the climate cells (the coloured squares) are offset from the sampling grid (the black gridlines).

Using this grid for gridSample. I’ll plot a blue ring around the retained occurrences; the records we drop are left as points:

set.seed(1)
acsel <- gridSample(acg, r, n=1)

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)
plot(p, add = TRUE)
points(acsel, col = 'blue', cex = 2)

Take a close look at that last plot. Notice that there are climate cells with no retained observations:

As well as climate cells with multiple observations:

This is fine if you are using the occurrence records in a purely spatial analysis (i.e., without incorporating climate data for each observation). But if you are intending to retain at most one observations for every cell in your climate map, this is not what you were hoping for.

A better way to achieve our desired result is to use the climate layer directly in gridSample:

set.seed(1)
acselClimate <- gridSample(acg, wc1, n=1)

Lets visualize the grid we sampled on:

plot(acg, pch = 20, xlim = c(-68.5, -67),
     ylim = c(-17.5, -16.5))
plot(wc1crop, legend = FALSE, add = TRUE)
points(acg, pch = 20)

## using a cropped layer because this is a slow operation:
climGrid <- rasterToPolygons(wc1crop)
plot(climGrid, add = TRUE)
points(acselClimate, col = 'blue', cex = 2)

Everything matches up as we expect now: the sampling grids are perfectly aligned with the climate cells.

This approach will only work if your climate raster covers the full extent of your occurrence records. Which it really should - if it doesn’t, the records that aren’t covered will end up getting dropped from your analysis since there’s no climate data at those locations.

References

Hijmans, Robert J., Steven Phillips, John Leathwick, and Jane Elith. 2017. “Dismo R Package Version 1.1-4.” https://CRAN.R-project.org/package=dismo.

Emacs for Bioinformatics #4: RMarkdown

Sun, 03 Oct 2021 00:00:00 +0000

This is part four in my series of Emacs tutorials aimed at bioinformatics (and other scientific analysis) workflows. See the rest on my tutorials page.

Emacs provides full support for editing RMarkdown documents. RMarkdown has extensive documentation, both at the previous RStudio link, and several free online books by Xie et al. (notably R Markdown: The Definitive Guide, but also several others listed on Yihui Xie’s Bookdown page).

Most of these references assume you are using the RStudio development environment. The purpose of this tutorial is to get you started editing RMarkdown documents in Emacs.

Installation

Prerequisites

You need to have R installed, of course. You will also need Pandoc in order to take full advantage of all the output options available. If you want to create PDF documents, you’ll need LaTeX as well.

All three of these programs are provided in the package repositories for most major Linux distributions. See the links above for instructions for installing on Windows or Apple computers.

You will also need to install the rmarkdown R package. You can do this from within R via:

install.packages("rmarkdown")

This will also install the other R requirements, notably the knitr package.

The bookdown package provides some more advanced citation features. I won’t discuss them in this short tutorial, but in order to use them you need to install that package too:

install.packages("bookdown")

Emacs Packages

We need a few additional Emacs packages to comfortably edit RMarkdown documents. These are:

Markdown Mode: The major mode for editing files in markdown format. This tutorial uses features added after 6 January 2021.
ESS: The collection of modes for editing R code and interacting with the R program.
poly-R (Polymode): polymode is a ‘glue’ mode. The poly-R variant extends markdown mode to allow us to edit embedded code snippets in R (and other languages too)

(poly-R also supports files in .Rnw format, which mix LaTeX and R code. We won’t cover that here)

polymode started out as a collection of modes to support files with different combinations of languages. As it has grown, many of those different modes have been split out into separate packages. When we install poly-R, it will automatically install the core of the polymode system for us.

This tutorial uses features added after 29 September 2021.

As in previous tutorials, (see my blog or the demo on Youtube), we can install all three of these packages from MELPA.

Once we have the required packages installed, no further configuration should be necessary. When we next open a file with a .Rmd extension, Emacs will know to use the poly-markdown+R-mode for these files. If everything is working properly, you’ll see Markdown PM-Rmd in the modeline at the bottom of the window for these files, and Markdown, RMarkdown, and Polymode menus at the top of Emacs frame.

Configuration Note

Depending on how you have installed poly-R, it may be loaded automatically, or you might need to load it yourself in your config. If it isn’t loaded automatically, you might see errors like (void-function poly-gfm+r-mode) when you try to open an Rmarkdown file.

You can fix this with by adding the following line to your Emacs config. The location isn’t critical, but it’s probably most convenient to put it at the beginning of any configuration you use for ESS/R/Markdown.

(require 'poly-R)

Github Flavoured Markdown and Code Blocks

Markdown mode supports several different options for code blocks. To take full advantage of the RMarkdown support provided by the rmarkdown R package, we need to use fenced code blocks, along with language strings wrapped in braces¹. I’ll explain this in more detail below.

This variant of markdown is referred to as “Github Flavoured Markdown”, and the markdown-mode package provides gfm-mode with a few extra features particular to it. Turning on gfm-mode for Rmd files requires the following line in your Emacs configuration to turn it on²:

;; associate the new polymode to Rmd files:
(add-to-list 'auto-mode-alist
             '("\\.[rR]md\\'" . poly-gfm+r-mode))

You will also need the following line, if you want gfm-mode to automatically insert braces for code blocks (described below):

;; uses braces around code block language strings:
(setq markdown-code-block-braces t)

This will switch you from using poly-markdown+R-mode to poly-gfm+r-mode, which shows up in your mode bar as “PM-Rmd(gfm)”. It’s nearly similar, the main differences being the support for fenced code blocks.

RMarkdown

Editing Markdown

Markdown syntax is designed to be easily entered by hand, which means if you’re already familiar with the format you can just get going. Markdown mode will provide you with syntax highlighing automatically:

A Markdown Mode buffer showing syntax highlighting

Of course, there are lots of shortcuts available. You can explore the most frequently used in the Markdown menu. I’ll summarize some of the main ones here to get you started.

Headings

Markdown has two different kinds of headings. The “Atx” style uses # symbols at the beginning of the heading, and, optionally, also at the end:


# Heading Level 1

## Heading Level 2 ##

You can insert these headings with the following commands:

C-c C-s h: insert a heading at the same level as the previous heading.

If the region is active, the contents of the region will be used as the header text. If point is on a line with text, the line will be converted into a header. Otherwise, an empty header will be created.

C-c C-s {1-9}: insert a heading at the specified level. i.e., C-c C-s 3 inserts a third-level heading. region and point can be used to set the heading text as for the previous.

You can manipulate headings with the following commands:

C-c <up> and C-c <down>: move a heading and all of its content up or down in the document.

ie., turn this:

Markdown headings in original order

into this:

Markdown headings with subheading 2 ahead of subheading 1

C-c <left> and C-c <right>: promote or demote a heading.

i.e., turn this:

Markdown headings in hierarchy

into this (and vice versa):

Markdown with subheading 2 demoted

If you prefer asymmetric headings (i.e., with # symbols only at the beginning of the line), you can configue this by setting the variable markdown-asymmetric-header to t:

;; set in your ~/.emacs or ~/.emacs.d/init.el
(setq markdown-asymmetric-header t)

Alternatively, you can do this via M-x customize-variable markdown-asymmetric-header.

Markdown mode also supports the setext style headings:


Heading Level 1
===============

Heading Level 2
---------------

Only two levels are supported, which you can insert automatically with the commands C-c C-s ! (level 1) and C-c C-s @ (level 2).

Heading Visibility

You can hide and show different sections in documents by pressing the <TAB> key with point on a heading. For example, with point on the # Installation heading, when I press <TAB> I move from this:

Markdown buffer with all sections visible

to this:

Markdown buffer with the Installation section hidden

This doesn’t change any of the text in your file, it only hides the parts you don’t want to see. Repeatedly pressing the <TAB> key will toggle through the various levels of hiding and showing.

If you want to toggle all the headings at once, Shift-<TAB> will toggle visibility for all headings at once. You can use this to collapse your entire document to a table of contents:

A Markdown buffer with only headings visible

Links and Images

Inserting links is done with C-c C-l. Emacs will first prompt you for the link URL, followed by the link text, and finally the tooltip text. Only the URL is required. To open a link from Emacs, use C-c C-o, which will take you to the webpage in your browser.

Images are handled similarly, and are inserted with C-c <TAB> or C-c C-i. The URL can be a web resource (e.g., https://my-images.ca/image1.jpg), or a local file (e.g., ./images/image1.jpg). The image will appear as:

![Image Caption](image URL)

You can toggle displaying the actual image in the buffer with C-c C-x C-i.

Tables

To insert a new table, use the command C-c C-s t. You will be prompted for the number of rows and columns, and the alignment you want. When you’re done, you’ll have a proper markdown table ready to edit:

|   |   |   |   |
|---|---|---|---|
|   |   |   |   |
|   |   |   |   |
|   |   |   |   |

With point in any of the cells, you can <TAB> into the next cell, or Shift-<TAB> into the previous cell. Each time you hit tab the cells will resize automatically to accomodate your text.

Additional commands are available for moving, adding and deleting rows and columns; see the Markdown -> Tables menu the options.

Other Markup

Markdown mode also provides shortcuts for other markup elements. See the Markdown menu for some of the options. I find most of the basics (bold, emphasis, unordered lists) are just as fast to type by hand as they are to insert using shortcuts.

Working with R Code

rmarkdown uses fenced code blocks with braces around the language string. i.e.,:

    ```{R code-block-example}
    ## R code goes here!
    1 + 1
    ```

If you’ve set up gfm-mode as described above, you can create one of these code blocks with the command markdown-insert-gfm-code-block, bound to C-c C-s C by default. Alternatively, simply entering three “`” characters at the beginning of a line will call the function for you³. Either way, you’ll be prompted for the language of the code block (which will be R most of the time, but you can use others!). You can also add a label for the code block at the prompt, and any additional options you want to use for the chunk. You can also add options later if you change your mind.⁴

Once you have created a code block, polymode will work its magic. You can continue to edit the markdown portions of your document, with all the features of gfm-mode. But when point is in an R code block, you’ll be editing it in ESS[R] mode. That allows you to use all the features of that package (see plantarum.ca for a quick tutorial/refresher).

Polymode provides some additional conveniences:

Navigation

polymode-next-chunk/polymode-previous-chunk, bound to M-n C-n and M-n C-p: move to the next/previous chunk. i.e., move from an RMarkdown chunk to the next R code chunk.
polymode-next-chunk-same-type/polymode-previous-chunk-same-type, bound to M-n M-C-n and M-n M-C-p: move to the next/previous chunk of the same type. i.e., move from one R code chunk the next R code chunk.
polymode-kill-chunk, bound to M-n M-k: kill the current chunk
polymode-toggle-chunk-narrowing, bound to M-n C-t: toggle narrowing the buffer to display only the current chunk, or to display the entire document

Evaluation

polymode-eval-region-or-chunk, bound to M-n v: evaluate all code chunks in the active region, or the chunk at point if there is no active region
polymode-eval-buffer, bound to M-n b: evaluate all code chunks in the buffer
polymode-eval-buffer-from-beg-to-point/polymode-eval-buffer-from-point-to-end, bound to M-n u or M-n ↑, and M-n d or M-n ↓: evaluate all code chunks from the beginning of the buffer to point (u and ↑), or from point to the end of the buffer (d and ↓)

Exporting RMarkdown

The most important ‘convenience’ of polymode is that it connects Emacs to the programs used to export RMarkdown files to presentation formats (i.e., pdf, html, slides). The main function you need for this is polymode-export, bound to M-n e.

The first time you run this, you’ll be asked which exporter you would like to use. There are two choices, markdown and markdown-ess. markdown means polymode will start a new, self-contained R process and compile your file there. When compilation is finished, the process will be closed.

markdown-ess will use an existing R process, or start a new one if there isn’t an active process available. When compilation is complete, the R process remains active. This allows you to check the values of various objects interactively. This can be useful as you develop a new script.

RMarkdown files can be compiled to produce a variety of output formats. You will be prompted to select which one you want the first time you run the exporter. polymode remembers this setting, so you don’t get prompted a second time. If you want to switch, say from pdf output to html, you can reset the target via C-u M-n e.

Feature added 6 January 2021.↩
Feature added 29 September 2021 ↩
By default; you can turn this feature off by setting the variable markdown-gfm-use-electric-backquote to nil.↩
I’m working on tab-completion for R chunk options, but haven’t decided how best to set it up yet. Watch this space!↩

Evaluating Invasion Stage with SDMs

Wed, 11 Aug 2021 00:00:00 +0000

My attempt to recreate the invasion stage analysis developed by Gallien et al. (2012), inspired by seeing it applied by Eckert et al. (2020). We’ll continue with the Lythrum salicaria data from my tutorial on niche quantification analysis. Specifically, I’ll model how the niche space this species occupies in its invaded range in North America relates to its global niche.

library(ecospat)
library(raster)
library(rgbif)
library(maptools)
library(magrittr)
library(dismo)

Gallien et al. (2012) used an ensemble of SDMs, which is (should be) more robust than applying a single approach. Nevertheless, for this short tutorial, I’ll stick to Maxent. I’m also cutting a lot of corners with respect to variable selection, model validation and other important steps. See my Maxent notebook for pointers.

Niche Models

We start by constructing SDMs for the global and North American distribution of L. salicaria.

Data

We need occurrence data and environmental data, and we’ll need to create background (pseudoabsence) samples.

The occurence data comes from GBIF, with details in my previous post:

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

coordinates(lsOccs) <- c("decimalLongitude",
                        "decimalLatitude") 
  ## Set the projection
crs(lsOccs) <- '+proj=longlat +datum=WGS84'

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

data(wrld_simpl) # load the maptools worldmap

par(mar = c(0,0, 0, 0))
plot(wrld_simpl, border = "gray80")
points(lsOccs, pch = 16, col = 2, cex = 0.3)

We’ll use the same climate data as well, sourced from WorldClim (Fick and Hijmans 2017) and imported using functions provided in the raster (Hijmans 2021) package. Note that I use the path argument to direct the download to a particular location. This is the same location I used in the previous post, and the data is still there, so it doesn’t get downloaded again.

wclim <- getData("worldclim", var = "bio", res = 10,
                path = "../data")

We need to define our study extent for selecting background points. I’ll use a 200 km buffer around our observations. We’re working at the global scale, and Lythrum salicaria is a strong disperser, so a relatively large scale is appropriate here. You’ll need to consider the aims of your own study when setting your extent.

studyExtent <- buffer(lsOccs, 200000, dissolve = TRUE)

## Loading required namespace: rgeos

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files
## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

plot(wrld_simpl, border = "gray80")
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 0.3)

The 200 km buffer creates some isolated pockets in North America. The extent should represent the area the species can access. The buffer I made includes the west coast inwards to Alberta, the east coast inwards to Saskatchewan, with an isolated patch in the center of Canada which looks like it’s at the Alberta/Saskatchewan border, with similar ‘islands’ in the western US:

plot(wrld_simpl, border = "gray80", xlim = c(-135, -90),
     ylim = c(45, 60))
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 1)

Those islands are likely the leading edge of the same invasion, not separate invasions! I’m going to increase our buffer to 300 km to capture the intervening area on the map:

studyExtent <- buffer(lsOccs, 300000, dissolve = TRUE)

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files
## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

plot(wrld_simpl, border = "gray80")
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 0.3)

This is better. I prefer to use ecoregions to set study extent, but for the purposes of this demo I’ll continue with this.

One further issue: our study extent includes the ocean. Let’s trim it back to the land:

land <- aggregate(wrld_simpl) ## dissolve country borders

  ## clip buffer to land:
studyExtent <- intersect(studyExtent, land) 

plot(wrld_simpl, border = "gray80")
plot(studyExtent, col = 'lightgreen', add = TRUE)
points(lsOccs, pch = 16, col = 2, cex = 0.3)

(This generates some warnings, likely related to missing values in my data or issues with the shapefile manipulations. It seems safe to proceed.)

The aggregation is a bit rough, but that should work for my purposes today. Now we can select our background points. I’m using 10000 points, and excluding any cells with a Lythrum salicaria occurrence.

  ## Convert landmass polygon to a raster:
landMask <- rasterize(land, wclim)

  ## sample points from the raster:  
background <- randomPoints(landMask, n = 10000, p = lsOccs)

Global SDM

Now we can fit our Maxent model. To reduce bias, I’ll thin the samples to 5 observations per grid cell (ca. 20 km square). Normally I work on a finer resolution (1 km 2), and thin to 1 observation per cell. Again, the details depend on your study area and goals.

lsThin <- gridSample(lsOccs, wclim, n = 5) %>%
  as.data.frame

coordinates(lsThin) <-
  c("decimalLongitude", "decimalLatitude")
glMax <- maxent(wclim, p = lsThin, a = background)
glPred <- predict(glMax, wclim)

(Warnings here tell us that some of the occurrences are in locations where there is no climate data. That’s normal, and not a problem as long as there are only a few points lost this way. If you are working with small data sets, you’ll want to investigate further to see if you can better match your records with the climate rasters.)

Here’s the model prediction:

plot(glPred)

North America SDM

For the SDM in the invaded range in North America, we need to crop our observations and background. Here I’m repeating the functions I used above to create a raster mask for land, but applying it only to the area of Canada, United States, and Mexico (our species isn’t in the Caribbean). I’m using the pipe (%>%) feature from magrittr, which makes it easier to follow the process.

NA_polygon <- wrld_simpl %>%
  subset(NAME %in%
         c("Canada", "United States", "Mexico")) %>%
  aggregate()

NA_mask <- rasterize(NA_polygon, wclim)

NA_background <-
  randomPoints(NA_mask, n = 10000, p = lsOccs) %>%
  as.data.frame()

coordinates(NA_background) <- c("x", "y")

In the original paper of Gallien et al. (2012), the background points were weighted using the values from the global model. I don’t think Eckert et al. (2020) applied this weighting, and it’s not clear to me how to do so with Maxent. For now I’ll skip it.

For the occurrence records, I’ll take my previously thinned data, and crop it to North America:

NA_polygon <- wrld_simpl %>%
  subset(NAME %in%
         c("Canada", "United States", "Mexico")) %>%
  aggregate()

lsNAThin <- intersect(lsThin, NA_polygon)

Now we can construct the SDM for North America:

naMax <- maxent(wclim, p = lsNAThin, a = NA_background)
naPred <- predict(naMax, wclim)
plot(naPred)

Note that I didn’t crop the WorldClim layer for the North American SDM model fitting. Maxent only uses the data for the presence and background points, so it doesn’t matter if the climate layers cover the whole planet for this step.

Invasion Stage Analysis

Now that we have completed both a global and a local (North America) SDM for L. salicaria, we’re ready to compare the results.

Niche Space

The values we need are the model predictions corresponding to each observation in North America.

globalVals <- extract(glPred, lsNAThin)
naVals <- extract(naPred, lsNAThin)

plot(naVals ~ globalVals, pch = 16, xlim = c(0, 1),
     ylim = c(0, 1),
     xlab = "Global model predictions",
     ylab = "Regional model predictions",
     col = "#00000050")
abline(h = 0.5, lty = 2)
abline(v = 0.5, lty = 2)

This plot compares the default Maxent output, the complementary log-log value. This is an estimate of the probability of presence, which is more appropriate than the other options for this kind of analysis (raw values would be difficult to interpret). However, I’m not sure 50% is the most appropriate value to use in the analysis that follows. Eckert et al. (2020) used optim.thresh from the (now defunct) SDMTools package to determine the best threshold for their study.

Following Gallien et al. (2012), we interpret the four quadrants of this plot as follows:

Upper right: High suitability in both native and global habitat. Observations here are occupying locations that fall within both the global and invaded niche, interpreted as ‘stabilizing.’
Upper left: High suitability in native model, but low suitability in the global model. Observations are occupying locations that are within the invaded niche, but outside the global niche, interpreted as populations demonstrating local adaption
Lower right: High suitability in the global model, but low suitability in the local model. These are interpreted as regional colonizations: the conditions here are within the global niche, but which are only starting to be occupied in the invaded range.
Lower left: Low suitability in both the local and global model. Presumably sink populations (not likely to persist).

Let’s tabulate the results:

tally <- c(stabilizing
          = sum(globalVals >= 0.5 & naVals >= 0.5,
                na.rm = TRUE),
          adapting = sum(globalVals < 0.5 & naVals >= 0.5,
                         na.rm = TRUE),
          sinks = sum(globalVals < 0.5 & naVals < 0.5,
                      na.rm = TRUE),
          colonizing = sum(globalVals >= 0.5 & naVals < 0.5,
                           na.rm = TRUE))

barplot(tally, ylab = "Occurences")

We can plot these regions on the map as well (apologies for the opaque raster algebra; there should be a clearer way to calculate this, but I can’t think of it at the moment).

suitabilityThreshold <- 0.5
na_Niche <- naPred > suitabilityThreshold
gl_Niche <- glPred > suitabilityThreshold

stable_Niche <- (na_Niche + gl_Niche) == 2
expansion_Niche <- ((2 * na_Niche) - gl_Niche) == 2
contraction_Niche <- ((2 * gl_Niche) - na_Niche) == 2

NicheRaster <- stable_Niche + (2 * expansion_Niche) +
  (3 * contraction_Niche)

plot(NicheRaster, xlim = c(-140, -60), ylim = c(30, 70),
     col = c("white", "blue", "red", "green"),
     legend = FALSE)
plot(wrld_simpl, add = TRUE)
points(lsNAThin, pch = 16, cex = 0.5)

In this plot, the blue depicts areas identified as suitable habitat in both the global and regional model. The green is area identified as suitable habitat in the global model, but not the North American model. There are some occurrences in this area, but they aren’t as numerous as the blue regions. Finally, the red areas were identified by the North American model as suitable habitat but they were not part of the global model’s suitable habitat. Following Gallien’s framework, any points in the white areas would be ‘sinks.’ More likely they’re the current leading edge of the invasion front I think.

Obviously, there’s a lot going on here, and each of these steps will warrant careful consideration and additional checks, validations, and optimizations. I hope this simplified outline is enough to get you started.

References

Eckert, Sandra, Amina Hamad, Charles Joseph Kilawe, Theo E. W. Linders, Wai‐Tim Ng, Purity Rima Mbaabu, Hailu Shiferaw, Arne Witt, and Urs Schaffner. 2020. “Niche Change Analysis as a Tool to Inform Management of Two Invasive Species in Eastern Africa.” Ecosphere 11 (2). https://doi.org/10.1002/ecs2.2987.

Gallien, Laure, Rolland Douzet, Steve Pratte, Niklaus E. Zimmermann, and Wilfried Thuiller. 2012. “Invasive Species Distribution Models – How Violating the Equilibrium Assumption Can Create New Insights.” Global Ecology and Biogeography 21 (11): 1126–36. https://doi.org/https://doi.org/10.1111/j.1466-8238.2012.00768.x.

Hijmans, Robert J. 2021. “Raster: Geographic Data Analysis and Modeling.” Manual. https://CRAN.R-project.org/package=raster.

Niche Quantification with Ecospat

Thu, 29 Jul 2021 00:00:00 +0000

Update

NB: this tutorial is now out of date and depends on deprecated versions of R packages! Please refer to the new version of my ecospat tutorial for the current packages and workflow for this analysis.

I’ll leave the old version below in case it’s of any interest, but it won’t work on current versions of R.

Archived Version

Packages

In addition to ecospat, we’ll use raster (Hijmans 2021) to download WorldClim (Fick and Hijmans 2017) rasters, and manipulate the spatial data; rgbif (Chamberlain et al. 2021) to download GBIF records, and maptools (Bivand and Lewin-Koh 2021) to get a world basemap for plots.

library(ecospat)
library(raster)
library(rgbif)
library(maptools)

NB there is a bug in ecospat that prevents us from using the argument geomask (see below). This has been fixed, but as of 2021-07-29, the bug fix has not made it into the released package, currently version 3.2. Consequently, you need to install directly from the development sources:

library(devtools)
install_github(repo = "ecospat/ecospat/ecospat")

Presumably this won’t be necessary for versions 3.3+ or newer (once released).

Getting Data

lsGBIF <- occ_search(scientificName = "Lythrum salicaria",
                    limit = 10000,
                    basisOfRecord = "Preserved_Specimen",
                    hasCoordinate = TRUE,
                    fields = c("decimalLatitude",
                               "decimalLongitude", "year",
                               "country", "countryCode"))

save(lsGBIF, file = "../data/2021-07-29-ls-gbif-recs.Rda")

This returned an object with 7969 records. I saved that locally, so that I’m not making GBIF search their database everytime I work on this demo.

load("../data/2021-07-29-ls-gbif-recs.Rda")
lsOccs <- lsGBIF$data

Next, we tell R which columns are the coordinates, which allows us to map the observations. This also converts our observation matrix to a SpatialPointsDataFrame object.

coordinates(lsOccs) <- c("decimalLongitude",
                        "decimalLatitude") 
data(wrld_simpl) # load the maptools worldmap

par(mar = c(0,0, 0, 0))
plot(wrld_simpl, border = "gray80")
points(lsOccs, pch = 16, col = 2, cex = 0.3)

To get our climate data, we can use raster’s getData function. The first time you call this function in a directory, it downloads the data from the internet, and saves it locally. Subsequent calls will load your local copy of the data, to speed things up. I’m using the coarsest resolution (10 minutes) to speed things up for this demonstration:

wclim <- getData("worldclim", var = "bio", res = 10,
                path = "../data")

We can take a look at one layer:

par(mar = c(0,0, 3, 1))
plot(wclim[["bio1"]], main = "bio1")

Next, we need to extract the environmental values from the climate rasters for each of our observation records:

lsOccs <- cbind(lsOccs, extract(wclim, lsOccs))

lsOccs <- lsOccs[complete.cases(data.frame(lsOccs)), ]

Splitting Data

At this point, all the data we need for the Niche Quantification analysis is in lsOccs and wclimMat. We need to split this data into native and invasive regions for our comparison. We’ll restrict ourselves to the northern hemisphere north of 20 degrees, and consider all records from Eurasia as native, and all records from North America as invasive.

I’ve created extents to cover the rough outlines of the areas in question. In practice, you could use a more carefully constructed vector map to split your data.

## North America: na
## Longitude from 40 to 180W, Latitude from 20 to 90N
naExt <- extent(c(-180, -40, 20, 90))
lsNA <- crop(lsOccs, naExt)

## Eurasia: ea
## Longitude from 40W to 180E, Latitude from 20 to 90N
eaExt <- extent(c(-40, 180, 20, 90))
lsEA <- crop(lsOccs, eaExt)

par(mar = c(1, 0, 0, 0))
plot(wrld_simpl, ylim = c(20, 80), axes = FALSE)
points(lsNA, pch = 16, col = 'red', cex = 0.5)
points(lsEA, pch = 16, col = 'darkgreen', cex = 0.5)

For the Niche Quantification, we need to have a matrix with the background environment present in the native and invasive ranges, as well as the complete global environmental including the combined extent of the native and introduced environments. After cropping, we use getValues to convert the raster to a dataframe.

## Crop Climate Layers:
naEnvR <- crop(wclim, naExt)
eaEnvR <- crop(wclim, eaExt)

## Extract values to matrix:
naEnvM <- getValues(naEnvR)
eaEnvM <- getValues(eaEnvR)

## Clean out missing values:
naEnvM <- naEnvM[complete.cases(naEnvM), ]
eaEnvM <- eaEnvM[complete.cases(eaEnvM), ]

## Combined global environment:
globalEnvM <- rbind(naEnvM, eaEnvM)

Niche Quantification

PCA

pca.clim <- dudi.pca(globalEnvM, center = TRUE,
                    scale = TRUE, scannf = FALSE, nf = 2)
global.scores <- pca.clim$li

nativeLS.scores <-
  suprow(pca.clim,
         data.frame(lsEA)[, colnames(globalEnvM)])$li   
invasiveLS.scores <-
  suprow(pca.clim,
         data.frame(lsNA)[, colnames(globalEnvM)])$li

nativeEnv.scores <- suprow(pca.clim, naEnvM)$li
invasiveEnv.scores <- suprow(pca.clim, eaEnvM)$li

data.frame(lsEA)[, colnames(globalEnvM)]

The output of dudi.pca and suprow includes a lot of information that we aren’t using here. We only need the li element, so I’ve selected that from each of the function outputs.

Occurence Densities Grid

nativeGrid <- ecospat.grid.clim.dyn(global.scores,
                                   nativeEnv.scores,
                                   nativeLS.scores)

invasiveGrid <- ecospat.grid.clim.dyn(global.scores,
                                   invasiveEnv.scores, 
                                   invasiveLS.scores)

ecospat.plot.niche.dyn(nativeGrid, invasiveGrid,
                       quant = 0.05)

Geographic Comparisons

To demonstrate, let’s compare the distribution of Lythrum salicaria in North America before and after 1950. We use geographic coordinates here, so no need for a PCA. We do need to generate the ‘background’ coordinates. I’ll use expand.grid to create the locations for this. I’ve broken up the NA extent into 500 x 500 grids.

lsNAearly <- subset(lsNA, year <= 1950)
lsNAlate <- subset(lsNA, year > 1950)
geoGrid <- expand.grid(longitude =
                        seq(-160, -40, length.out = 500),
                      latitude =
                        seq(20, 90, length.out = 500))

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     coordinates(lsNAearly))

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    coordinates(lsNAlate))

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)
plot(wrld_simpl, add = TRUE)

naMask <- bind(subset(wrld_simpl, NAME == "Canada"),
              subset(wrld_simpl, NAME == "United States"),
              subset(wrld_simpl, NAME == "Mexico"))

earlyGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                     coordinates(lsNAearly),
                                     geomask = naMask)

lateGeoGrid <- ecospat.grid.clim.dyn(geoGrid, geoGrid,
                                    coordinates(lsNAlate),
                                    geomask = naMask)

ecospat.plot.niche.dyn(earlyGeoGrid, lateGeoGrid, quant = 0)
plot(wrld_simpl, add = TRUE)

That gives more reasonable results.

Summary

References

Bivand, Roger, and Nicholas Lewin-Koh. 2021. “Maptools: Tools for Handling Spatial Objects.” Manual. https://CRAN.R-project.org/package=maptools.

Hijmans, Robert J. 2021. “Raster: Geographic Data Analysis and Modeling.” Manual. https://CRAN.R-project.org/package=raster.

GBS Admixture Analysis Workflow

Tue, 01 Jun 2021 00:00:00 +0000

Admixture is a program for completing STRUCTURE-style analyses of large SNP datasets, such as we get with GBS (Elshire et al. 2011). This short tutorial covers getting our SNP data from STACKS (Rochette, Rivera‐Colón, and Catchen 2019) into a format that Admixture will understand, running the analysis, and importing the results into R for further investigation & plotting.

Converting Stacks Output to Admixture Input

Both Stacks and Admixture can process PLINK data. However, there are a few ‘gotchas’ that took a while to sort out. The simplest way I found to bridge the two programs was to export my Stacks data to vcf, clean it up on the command line, and then use the plink program to convert it to a plink file that Admixture could parse.

I’ll start from the vcf file generated by Stacks’ populations program. We expect to have thousands of contigs in a typical GBS dataset, and each of which is numbered in the Stacks output:

head -20 populations.haps.vcf

##fileformat=VCFv4.2
##fileDate=20210531
##source="Stacks v2.3e"
##INFO=<ID=AD,Number=R,Type=Integer,Description="Total Depth for Each Allele">
...
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##INFO=<ID=loc_strand,Number=1,Type=Character,Description="Genomic strand the co
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NIAP1-1011  NIAP1-0342  
26  1   .   TTTCG   TATCG   .   PASS    snp_columns=61,93,136,137,177   GT  0/0 0/0 
46  1   .   AAAGTT  AACATT  .   PASS    snp_columns=13,15,48,73,125,192 GT  0/1 0/1 
103 1   .   CCCACATGATACGCCGC   CCCATATAAGCCGCCGC   .   PASS    snp_columns=11,16   
149 1   .   ACACTGT ACATTGT .   PASS    snp_columns=47,66,84,91,121,126,163 GT  0/0 
271 1   .   CATTGTGCGGAATATGT   TACCTCATTTGCATTAC,CATTGTGCGGAAAATGT .   PASS

This causes a problem when we try to use plink, which won’t accept #CHROM values higher than 21. To fix this, we need to append a letter to the #CHROM numbers:

sed '/^[[:digit:]]/s/^/c/' populations.haps.vcf > popC.haps.vcf

head -20 mingan.haps.vcf

##fileformat=VCFv4.2
##fileDate=20210531
##source="Stacks v2.3e"
##INFO=<ID=AD,Number=R,Type=Integer,Description="Total Depth for Each Allele">
...
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##INFO=<ID=loc_strand,Number=1,Type=Character,Description="Genomic strand the co
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  NIAP1-1011  NIAP1-0342  
c26 1   .   TTTCG   TATCG   .   PASS    snp_columns=61,93,136,137,177   GT  0/0 
c46 1   .   AAAGTT  AACATT  .   PASS    snp_columns=13,15,48,73,125,192 GT  0/1 
c103    1   .   CCCACATGATACGCCGC   CCCATATAAGCCGCCGC   .   PASS
c149    1   .   ACACTGT ACATTGT .   PASS    snp_columns=47,66,84,91,121,126,163 GT
c271    1   .   CATTGTGCGGAATATGT   TACCTCATTTGCATTAC,CATTGTGCGGAAAATGT .   PASS

With that small addition, we can now create a plink file with the plink program:

plink --vcf popC.haps.vcf --make-bed --out pop.admix --allow-extra-chr 0

This generates a few files: pop.admix.bed, pop.admix.bim, pop.admix.fam, pop.admix.log, pop.admix.nosex.

Running Admixture

We can now run admixture itself. We need all the files generated by plink together in the same directory. We’ll pass the .bed file as an argument to Admixture, but it will look for the other files when it’s running.

The other key argument for admixture is the number of clusters to look for. We most likely will want to try a range of different values, in order to determine the optimal number (if there is one). We can do this in bash with a loop:

for K in `seq -w 1 20` 
do
    admixture --cv pop.admix.bed $K > ktests/k${K}.out
done

The seq command generates a sequence of numbers, and the -w flag tells it to pad the numbers with zeros (i.e., 01, 02, … 19, 20).

The --cv flag tells admixture to calculate cross-validation error rates, which we will use to determine the optimal K value.

We direct the output to files in the directory ktests. Make sure this directory exists before you start.

Once the loop is finished, we’ll want to examine the results:

grep -h CV ktests/*out

CV error (K=1): 0.39835
CV error (K=2): 0.31327
CV error (K=3): 0.26516
CV error (K=4): 0.19929
CV error (K=5): 0.18499
...

Now we’re ready to move into R to explore the results.

R Plotting

First, we’ll take a look at the CV values. Since they’re scattered in 20 different log files, we’ll use grep to collect them into a single file:

grep -h CV ktests/*out > CV.csv

We can load that into R:

CVs <- read.table("CV.csv", sep = " ")
CVs <- CVs[, 3:4] ## drop the first two columns
## Remove the formatting around the K values:
CVs[, 1] <- gsub(x = CVs[, 1], pattern = "\\(K=",
                replacement = "")
CVs[, 1] <- gsub(x = CVs[, 1], pattern = "\\):",
                replacement = "") 
head(CVs)

##   V3      V4
## 1  1 0.39835
## 2  2 0.31327
## 3  3 0.26516
## 4  4 0.19929
## 5  5 0.18499
## 6  6 0.15408

plot(CVs, xlab = "K", ylab = "CV error")

In our case, there isn’t a real clear optimum. K = 9 is about the bottom of the ‘elbow,’ we’ll use that to make our plot.

ad9 <- read.table("pop.admix.9.Q")
head(ad9)

##         V1    V2    V3    V4    V5       V6    V7    V8    V9
## 1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 2 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 3 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 4 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05 1e-05 1e-05
## 5 0.020244 1e-05 1e-05 1e-05 1e-05 0.979686 1e-05 1e-05 1e-05
## 6 0.003441 1e-05 1e-05 1e-05 1e-05 0.996489 1e-05 1e-05 1e-05

We also need a popmap file to annotate our plot. This file lists the population of every sample, and critically, it must be in the same order as the rows in pop.admix.9.Q. In this case, I’ll use the popmap data that I used with the populations program to generate the original vcf files we started wtih, with an additional column added with the population names in it.

popmap <- read.table("popmap.csv")
head(popmap)

##       sample popnum    popname
## 1 NIAP1-1011      1 Niapsikau1
## 2 NIAP1-0342      1 Niapsikau1
## 3 NIAP1-1004      1 Niapsikau1
## 4 NIAP1-1017      1 Niapsikau1
## 5 NIAP1-1014      1 Niapsikau1
## 6 NIAP1-0923      1 Niapsikau1

At this point, the two tables are in the same order. Before I do any manipulations, I’ll combine them. This allows me to sort them in any order, and the names will stay associated with the correct samples.

ad9 <- cbind(popmap, ad9)
head(ad9)

##       sample popnum    popname       V1    V2    V3    V4    V5       V6    V7
## 1 NIAP1-1011      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 2 NIAP1-0342      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 3 NIAP1-1004      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 4 NIAP1-1017      1 Niapsikau1 0.000010 1e-05 1e-05 1e-05 1e-05 0.999920 1e-05
## 5 NIAP1-1014      1 Niapsikau1 0.020244 1e-05 1e-05 1e-05 1e-05 0.979686 1e-05
## 6 NIAP1-0923      1 Niapsikau1 0.003441 1e-05 1e-05 1e-05 1e-05 0.996489 1e-05
##      V8    V9
## 1 1e-05 1e-05
## 2 1e-05 1e-05
## 3 1e-05 1e-05
## 4 1e-05 1e-05
## 5 1e-05 1e-05
## 6 1e-05 1e-05

In my case, the samples aren’t in order, so I need to sort them prior to plotting:

ad9 <- ad9[order(ad9$popnum), ]

Now we’re ready to plot:

barplot(t(as.matrix(ad9[, -1:-3])), col=rainbow(9), 
        space = 0, xlab="Population", ylab = "Ancestry", 
        border=NA, axisnames = FALSE)

Let’s break that down. First, we excluded the first three columns, ad9[, -1:-3], so that we don’t include our labels in the data. Then we transpose the matrix, so that each individual sample is represented by a column of Ancestry proportions. I removed the borders on the bars, and the spaces between them (border = NA, and space = 0), and set the axis labels.

That’s nice, but we’d like to label our original populations on the plot, so we can see how they compare to the clusters produced by admixture. I use the aggregate function for this.

xlabels <- aggregate(1:nrow(ad9),
                    by = list(ad9[, "popname"]),
                    FUN = mean)
xlabels

##      Group.1     x
## 1   Fantome4  58.0
## 2   Fantome5  83.5
## 3     Havre6 107.5
## 4     Havre7 125.5
## 5  Marteau11 149.0
## 6 Niapsikau1   7.0
## 7 Niapsikau2  30.5

Here, I’ve grouped the rows by population name, and then calculated the mean row number for each group. That will be handy, as I can then use that mean value to plot the name of each population centered beneath it.

Similarly, I can find the borders of the groups:

sampleEdges <- aggregate(1:nrow(ad9),
                        by = list(ad9[, "popname"]), 
                        FUN = max)
sampleEdges

##      Group.1   x
## 1   Fantome4  68
## 2   Fantome5  98
## 3     Havre6 116
## 4     Havre7 134
## 5  Marteau11 163
## 6 Niapsikau1  13
## 7 Niapsikau2  47

In this case, I find the highest row for each population, which I’ll use to draw a line between them.

Putting this all together, we get:

barplot(t(as.matrix(ad9[, -1:-3])), col=rainbow(9), 
        space = 0, xlab="Population", ylab = "Ancestry", 
        border=NA, axisnames = FALSE)
abline(v = sampleEdges$x, lwd = 2)
axis(1, at = xlabels$x - 0.5, labels = xlabels$Group.1)

And we’re done!

References

Elshire, Robert J., Jeffrey C. Glaubitz, Qi Sun, Jesse A. Poland, Ken Kawamoto, Edward S. Buckler, and Sharon E. Mitchell. 2011. “A Robust, Simple Genotyping-by-Sequencing (GBS) Approach for High Diversity Species.” PLoS ONE 6 (5). https://doi.org/10.1371/journal.pone.0019379.

Rochette, Nicolas C., Angel G. Rivera‐Colón, and Julian M. Catchen. 2019. “Stacks 2: Analytical Methods for Paired-End Sequencing Improve RADseq-Based Population Genomics.” Molecular Ecology 28 (21): 4737–54. https://doi.org/https://doi.org/10.1111/mec.15253.

Adding Lat/Lon Grids to Maps in R

Mon, 22 Feb 2021 00:00:00 +0000

In a previous post, I outlined my workflow for preparing maps in R. Today I had to add a graticule, a grid of latitude and longitude lines, to my maps. That’s easy enough to do with unprojected maps, as the plot coordinates are latitude and longitude, so your X and Y axes are already graticules. But if you’ve projected your data, the plot coordinates are on a different scale, so you need to do a bit of tuning.

I couldn’t find a direct way to do this in the R sp package. However, sp (sp for ‘spatial’) is slowly being replaced by sf (sf for simple feature), and sf does support graticules. Here are the steps required to add them to your plots:

Importing Data

We can use raster::getData to get our map data again. It’s straightforward to convert objects from sp (Spatial*) and sf (sf*) format and back, with the functions st_as_sf (to convert from a Spatial* to sf*), and as (to convert from sf* to a Spatial* object). As it turns out, getData also supports downloading data directly into sf format:

library(sf)
library(raster)
us <- getData("GADM", country = "USA", level = 1,
             path = "./data/maps/", type = "sf")
canada <- getData("GADM", country = "CAN", level = 1,
                 path = "./data/maps", type = "sf")
mexico <- getData("GADM", country = "MEX", level = 1,
                 path = "./data/maps", type = "sf")

This uses the undocumented type argument, set to sf. Given that it’s not documented, it may change in future, be warned!

You can also use the function st_read to read shapefiles directly:

greatlakes <- st_read("data/maps/greatlakes.shp")

## Reading layer `greatlakes' from data source 
##   `/home/smithty/blogdown/content/tutorials/data/maps/greatlakes.shp' 
##   using driver `ESRI Shapefile'
## Simple feature collection with 2 features and 5 fields
## Geometry type: POLYGON
## Dimension:     XY
## Bounding box:  xmin: -365240.6 ymin: -2741892 xmax: 1888977 ymax: 509590.2
## proj4string:   +proj=laea +lat_0=45 +lon_0=-100 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs

In the previous tutorial, I used bind to combine two Spatial* objects. With sf we need rbind instead:

na <- rbind(us, canada, mexico)

## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()
## old-style crs object detected; please recreate object with a recent sf::st_crs()

Plotting complex vector maps like this can be a slow process, especially when you’re constantly tweaking and adjusting them. You can speed this up by simplifying the layers:

na.simp <- st_simplify(na, dTolerance = 0.01)

On my laptop, plotting the original map takes a minute or more, compared to 2 seconds for the simplified vector. I set the tolerance by trial and error. The higher the tolerance, the smoother the map will be. At 0.01, it still looks nearly identical at the scale I’m plotting it, but is much smaller and faster to plot. sf does warn me about not correctly simplifying the data, but since I’m only using this for display that’s not a concern. I wouldn’t simplify a vector if I was going to use it in an analysis.

Plotting Maps

When it comes to plotting, we need to tell R to plot only the geometry. By default it will plot multiple maps, one for each attribute. That’s not what we want here.

plot(st_geometry(na.simp), xlim = c(-130, -70),
     ylim = c(35, 45))

Projections

To project our unprojected data, we need to define a projection, and transform the object.

laea = CRS("+proj=laea +lat_0=30 +lon_0=-95")

## NOTE: rgdal::checkCRSArgs: no proj_defs.dat in PROJ.4 shared files

na.la <- st_transform(na.simp, laea)
plot(st_geometry(na.la), xlim = c(-500000, 2000000),
     ylim = c(-400000, 2100000))

We can add layers just as we did in the previous post:

gl.la <- st_transform(greatlakes, laea)
plot(st_geometry(gl.la), col = 'lightblue', add = TRUE)

You can also mix sf and Spatial* objects on the same plot, as long as they’re in the same projection.

Graticules

Now we have everything we need to add graticules to our map. This includes the map we want to plot, and the CRS data for the graticules we want to overlay. In our case, we’ll use the original, unprojected layer as the source our CRS:

plot(st_geometry(na.la),
     xlim = c(-500000, 2000000), ylim = c(-400000, 2100000),
     graticule = st_crs(na.simp),
     bgc = 'lightblue', ## Background color for the ocean
     col = 'white',
     axes = TRUE)
plot(st_geometry(gl.la), col = 'lightblue', add = TRUE)

If you want to specify the location of the graticules, you can use the arguments lat and lon to specify where you want them.

Emacs for Bioinformatics #3: R and ESS

Wed, 30 Dec 2020 00:00:00 +0000

This is part three in my series of Emacs tutorials aimed at bioinformatics (and other scientific analysis) workflows. See the rest on my tutorials page.

Emacs support for the R programming language is provided by the ESS package (AKA, “Emacs Speaks Statistics”). ESS has been around since at least 1994, and is supported by a very active development team. It provides most or all of the features of the more widely-known RStudio, as well as a great many more. Like all things Emacs, if it doesn’t have a feature you want, it’s likely someone else has written a package that provides it; failing that, the motivated hacker you can create their own customizations using the built-in scripting language, elisp.

However, lets not let all that potential scare us off. Getting up and running with ESS doesn’t require much effort at all.

Installation

Prerequisites

You need to have R installed in order to use ESS!

Installing ESS from MELPA

The easiest way to install it is to use the MELPA package repository. MELPA hosts Emacs packages provided by hackers who are not part of the Emacs development team. (“packages” here has roughly the same meaning as “plugins” or “extensions” in other software systems).

If you aren’t already using MELPA, you need to add it to your configuration file (typically ~/.emacs.d/init.el, or ~/.emacs):

(require 'package)
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

Once this code is evaluated, you can view the complete list of packages available on MELPA via M-x package-list-packages. It’s a big list, and it will take a few seconds for Emacs to get the latest version from the server (you need an internet connection for this).

Search down to the entry for ESS, select it by pressing i, and then install it by pressing x. ESS is one of the larger packages, so it may take a few seconds to download and install all the files.

Once this is done, you have ESS, and don’t need to return to package-list-packages until you want to update to a new version (or add some other packages).

Getting Started

ESS comes with a comprehensive manual. That will be your canonical reference for learning about this package. However, you can get started with just a few commands.

Interactive R Session

From within Emacs, start R with the command M-x R. You will be prompted for the project starting directory. Select whatever you like and press enter. You will then be presented with an R shell:

You can enter code and view results here, just as you would in the terminal in RStudio, or with R running on the command line. ESS uses the same code to manage this as for shell mode. That means we can use the same keybindings here:

<tab>: with the cursor at the active prompt, tab will complete function and variable names, as well as the arguments for functions
<enter>: with the cursor at the active prompt, send the command on the prompt to R for evaluation
<enter>: With the cursor on a previous command, re-enter that command
M-p and M-n: Move through your command history at the active prompt
C-c C-p and C-c C-n: Move the cursor to the previous and next prompts
C-c <enter>: With the cursor on a previous command, copy that command the to active prompt, but don’t enter it. This allows you to edit a previous command before sending a new variation to R for evaluation
C-c C-o: Delete the output from the previous command
C-c C-v: Opens a prompt to select a help file, which will be displayed in Emacs (you can also open help files from the prompt via ?<function>)

These few commands cover most general interactions. There are a lot more features available. Check the iESS menu item on the toolbar for some of them; see the manual for the details.

Plots

Calling plotting commands will create a new window (frame) for your figure. There isn’t a dedicated pane in Emacs to display them, like in RStudio, and you can’t scroll forward and backward through your history of images. You can, however, create multiple image windows, and view them side by side.

To create and manipulate new image windows, you’ll need the following commands:

dev.new() ## Create new plot window, and make it the
          ## active window
dev.set() ## If more than one plot window is open,
          ## set the next window to be the active
          ##window

See ?dev page for more details.

Writing R Scripts

After ESS is installed, anytime you open a file with a .r or .R extension, it will be in ESS[R] mode. You can enter text as usual, and additionally have the following helpful commands available:

C-c C-n or C-c <enter>: send the current line to the R process and step to the next
C-c C-r: send the current region to the R process
C-c C-f: send the current function to the R process
C-c C-c: send the current region, paragraph, or function to the R process
C-c C-b: send the entire buffer to the R process
C-c C-v: prompt for a help file to open
M-tab: tab completion of objects (functions, variables, file names) and function arguments

If there is no R process running when you try to send code, you will be prompted for a working directory in which to start a new process. In addition, you can manage R processes with the following commands:

C-c C-z: switch from the script buffer to the process buffer (and vice versa)
C-c C-s: change the process linked to the current script buffer (e.g., if you want to run multiple R processes at once, with different scripts in each process)

Next Steps

This may well be all you need, and if that’s the case, you’re all done. However, there is a lot more available to you, including support for writing documentation, package development, managing git repositories, editing on remote servers, and more.

My advice is to start slowly. The pointers on this page will get you up and running. When you find yourself repeating something tedious multiple times, it may be time to investigate if there’s a shortcut available to make your life easier. I recommend skimming the manual, to get a sense of all that’s available, and if something catches your eye see about incorporating it into your workflow.

Plotting Simple Maps in R

Fri, 30 Oct 2020 00:00:00 +0000

NOTE: This tutorial uses older R packages that are scheduled to be deprecated at the end of 2023. I have updated this tutorial using the new packages. Unless you need to use older code, you should use the new Terra-based approach instead of this!

Reference

See the RSpatial tutorial for a more detailed introduction/overview of using R for GIS/spatial analysis. The following tutorial walks through some common plotting tasks I use for distribution models.

Basemaps

The raster package provides the function getData, which is a handy way to download basemaps for plotting. (You can also use it to get WorldClim data, see the man page). The first time you call it, it will download the requested maps from the internet. It will save the data in your working directory, or in a location specified with the path argument. The next time you request the same map from getData, if it finds it in the local directory it will load it from there, rather than downloading it again.

library(raster)

## Loading required package: sp

## Loading required package: methods

## Warning: multiple methods tables found for 'metadata'

us <- getData("GADM", country = "USA", level = 1,
             path = "./data/maps/")

## Warning in getData("GADM", country = "USA", level = 1, path = "./data/maps/"): getData will be removed in a future version of raster
## . Please use the geodata package instead

canada <- getData("GADM", country = "CAN", level = 1,
                 path = "./data/maps")

## Warning in getData("GADM", country = "CAN", level = 1, path = "./data/maps"): getData will be removed in a future version of raster
## . Please use the geodata package instead

These maps can be plotted directly with the plot command. If you want to combine them, use the add = TRUE argument to the second plot call:

plot(us)
plot(canada, add = TRUE)

You can combine multiple vector maps into a single map with bind:

na <- bind(us, canada)

These maps are ‘unprojected’, meaning they are plotted in latitude/longitude degrees. That makes it easy to set the plot boundaries:

plot(na, xlim = c(-100, -50), ylim = c(30, 60))

It’s handy to have a shapefile of the Great Lakes, for making prettier maps. I created this one in QGIS and use it for plotting:

greatlakes <- shapefile("data/maps/greatlakes.shp")

Adding Data

You can add points to the plot like a regular scatter plot:

library(scales)  ## for the alpha function below
gbif <- read.table("data/trich-gbif.csv")
## Set the line color to gray to focus on the data points:
plot(na, xlim = c(-100, -50), ylim = c(30, 60),
     border = "gray")
points(gbif$X, gbif$Y, pch = 16,
       col = alpha("green", 0.2))

You can also convert your points to a spatial points object, in which case R will know which columns to use for plotting. This is also necessary before we can project our data (see below).

coordinates(gbif) <- ~X+Y
## plot(na, xlim = c(-100, -50), ylim = c(30, 60),
##      border = "gray")
## points(gbif, pch = 16, col = alpha("green", 0.2))

Rasters

Similarly, you can plot rasters with plot:

trichPreds <- raster("./data/trichPreds")
plot(trichPreds, xlim = c(-100, -50), ylim = c(30, 60))
plot(na, border = "gray", lwd = 0.5, add = TRUE)

trichPredsTrim <- trichPreds
trichPredsTrim[trichPredsTrim <
               quantile(getValues(trichPreds),
                        probs = 0.75, na.rm = TRUE)] <- NA
plot(trichPredsTrim, xlim = c(-100, -50), ylim = c(30, 60))
plot(na, border = "grey", add = TRUE)

You could also use an absolute value here, but then you’d need to know the actual distribution of the suitability scores. quantile is easier to tweak.

Projections

Lat/Lon maps look a bit square; we’re more used to seeing maps projected. A common projection for Canada is Lambert Conformal Conic. We can transform our data to this projection to make nicer maps:

## define the projection
canlam <- CRS("+proj=lcc +lat_1=49 +lat_2=77 +lat_0=49 +lon_0=-95 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs")

## project our vector data:
na.lcc <- spTransform(na, canlam)
gl.lcc <- spTransform(greatlakes, canlam)

## Warning: PROJ support is provided by the sf and terra packages among others

## We already convereted gbif to spatial points object above!
## Now we to set the projection of our points:
crs(gbif) <- CRS("+proj=longlat +datum=WGS84")

## Finally, we can project our points to LCC:
gbif.lcc <- spTransform(gbif, canlam)

Note that our data needs to be in an object of class Spatial*, and it must have a defined coordinate reference system (CRS) before we can project it to a new CRS. Setting the coordinates of our points via the coordinates function creates a Spatial* object. The crs function allows us to explicitly set the projection. We need to know the EPSG code for the projection to use this. The function make_EPSG in the package rgdal is helpful for finding this information. See the RSpatial tutorial for details.

There are a few more steps for raster layers:

rasterLCC <- projectExtent(trichPredsTrim, canlam)
res(rasterLCC) <- 10000 ## set the cell size to 10km
predLCC <- projectRaster(trichPredsTrim, rasterLCC)

Note that I set the resolution to 10km here. That’s the size of the raster cells. The original raster cells, in the lat/lon projection, were at 30 second resolution, which is about 1km. I could have set a smaller cell size here. However, since I’m only using this map for visualization, 10km is plenty big enough for my plot, and will run faster (and take less memory) than a map with 1km cell size.

Now we can plot our data in the Lambert Conformal Projection:

plot(predLCC)
plot(na.lcc, border = "grey", add = TRUE)

The units are no longer Lat/Lon, but meters. We can read them off the plot to improve the zoom:

plot(predLCC, xlim = c(0, 2500000),
     ylim = c(-1500000, -400000))
plot(na.lcc, border = "grey", add = TRUE)

Formatting

With the data plotted, we can then turn to making the map a little prettier:

## Make a panel with two plots, set the right margin tight:
par(mar = c(0.1,0.1,0.1,0), mfrow = c(1, 2))

## store the plot limits:
my_xlims <- c(0, 2500000) 
my_ylims <- c(-1300000, -200000)

## Plot the points:
plot(na.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", bg = "lightblue", col = "white")
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
points(gbif.lcc, pch = 16, col = alpha("grey30", 0.2),
       cex = 0.7)
box() 

## tighten up the left margin:
par(mar = c(0.1,0,0.1,0.1))
plot(na.lcc, xlim = my_xlims , ylim = my_ylims,
     border = "grey", bg = "lightblue", col = "white")
plot(gl.lcc, add = TRUE, border = "grey", col = "lightblue")
plot(predLCC, add = TRUE, legend = FALSE)

## plotted again to put the border lines on top:
plot(na.lcc, border = "grey", add = TRUE) 
box()

Emacs for Bioinformatics #2: Orgmode

Wed, 17 Jun 2020 00:00:00 +0000

In the previous post we took a first look at Emacs, including creating and editing a script file, and passing commands from the file to the shell terminal. At the end of that post, I recommended you check out the built-in tutorial (accessible via C-h t from within Emacs). In this post I assume you’ve done so, although I won’t expect you’ve understood everything you found there.

Orgmode

Last time, I promised a better way to integrate scripts, output, and notes in a single file. The better way is provided by orgmode, which comes bundled with Emacs. Orgmode evolved from a simple task-manager, to a full-fledged information management system, especially for people whose work includes computer code. This lesson will focus on getting started with orgmode, and using it to help explore some Illumina data (you don’t need to undertand Illumina data to follow along).

Objectives

Create a new org file with:
- text notes
- Bash code blocks
- the results of executing the code blocks

Setting Up

First off, we need to create a new file. We do this from the menu “File - Visit New File” option. Name it gbs.org, and put it in the directory we created last time: ~/gbs-analysis.

Emacs will recognize this file as an org file, based on the .org suffix, and will turn on orgmode for us. The file is still a normal plain text file, but Emacs will look for special tags that identify which parts are code, what is a heading, which text to make bold etc.

We need to customize one of the orgmode options before we start. For security, by default orgmode won’t allow you to execute code in programming languages other than Emacs’ built-in language elisp. We need to add bash to the list of permitted languages, so we can use it for our scripts.

We can find the options in the menu “Org - Customize”. At first, there are only two options here. Select the “Expand this Menu” option, then open the “Org - Customize” menu again. Now you have a second menu item named “Customize”. This will lead you to a long list of options. We’ll ignore them, and pick the “Babel” option from up near the top:

This opens up the Babel customize window:

Scroll down until you find the Org Babel Load Languages option. Click on the triangle to reveal the current settings:

To begin with, there’s only one entry, for Emacs Lisp. Press the INS button to insert a new option. The Value Menu will show “Awk”, which we need to change. Click on the “Value Menu” button, and enter “Shell Script” and press enter.

Now press the “Save” button in the tool bar to set all options, and press the q key to close the window.

Writing our Script

We can now enter whatever we like in the file: introductory notes, comments about the code we will create, and the code itself. This is just a plain text file, so there is no restriction.

However, if we use special tags, we can insert “code blocks” that we can run directly in this file. A shell code block looks like this:

#+begin_src bash
ls
#+end_src

With the cursor anywhere in this code block, we can run the code by pressing C-c C-c. Note that Emacs will ask us to confirm we want to run the code, and then it runs it:

Emacs has taken our ls command, run it through a shell interpreter, and inserted the results back in our file. Again, the file is still just plain text, so we can add any comments we like, although it’s best to keep our comments out of the code blocks, and not put anything between the code block and the results it generates:

It’s often handy to have basic commands like ls included in your scripts, to confirm the files you think you are working with are in fact where you want them to be.

I use code blocks to build templates for my analyses, along with useful notes and sanity checks. The following template includes two ‘checks’ before and after the command process_radtags:

Examine the contents of our sequencing files:
#+begin_src bash
zcat data/Vaccinium3-*.fastq.gz | head -4
#+end_src

Demultiplex: expect this to take about 10 minutes:
#+begin_src bash
process_radtags -p ./data/ -b ./data/barcodes.csv \
                -i gzfastq -o ./output -q -r \
                --inline-null -e pstI
#+end_src

Demultiplexed sequencing reads:
#+begin_src bash
zcat output/samp-001.fq.gz | head -4 
#+end_src

Note that the process_radtags command is longer than a single line. Put a single \ symbol as the very last character on a line to make sure the computer treats it as a single line.

I can step through this sequence, pressing C-c C-c on each block, to work through the analysis. My notes remind me that I’ll have time to get a cup of tea while I wait for process_radtags to finish.

After I run the code, my file looks like this:

Note that the results for the process_radtags chunk are empty. That’s good, that means process_radtags completed without issuing any warnings or errors. The files it produced are saved on my hard-drive, and I confirm that by peaking inside one of them in the third chunk.

This is still a simple example, but I hope that now you can start to see some of the potential for using orgmode to structure your analysis.

Conveniences

If you do think you’d like to use orgmode in your analyses, you might think entering all those #+BEGIN tags will get tedious. That’s true. There are keyboard shortcuts to help though. On older versions of orgmode, up to version 9.1, you can enter the following shortcut. Starting at the beginning of a line, type

<s

and then press the TAB key. The <s characters will be replaced by:

#+begin_src
#+end_src

You can then add the bash keyword at the end of the first line, and start your code between the two lines.

If that doesn’t work, your orgmode is probably one of the newer releases. Starting with orgmode version 9.2, instead of the above, you can press C-c C-, (i.e, control-c, control-comma), then select s for source code, and you’ll get the begin and end tags you need.

One other shortcut: you’ll probably get tired of Emacs asking you if you really want to execute code each time you run a code chunk. You can turn off that check in the customize menu: “Org - Customize - Customize - Babel”, scroll down to find “Org Confirm Babel Evaluate”, click the toggle button to “off”, and click the “Save” button on the menu. You’ll never be asked again, so be careful!

Emacs for Bioinformatics: Getting Started

Tue, 16 Jun 2020 00:00:00 +0000

Emacs for Bioinformatics

GNU Emacs is likely one of the oldest pieces of software still in active development. It is also one of the most powerful systems for editing code, built by and for hackers. However, it does have a reputation for unwieldy complexity. I think this is largely undeserved. While it would take years of study to understand all its nooks and crannies, if you focus on just those features that you actually need, you can get going fairly quickly.

The purpose of this series of posts is to introduce new coders to the benefits of Emacs. I’m specifically targeting biologists, but hopefully the content will be generally useful.

You may have heard of orgmode. This is one of Emacs’ killer featues, and will feature prominently in future posts. However, for this first post we’ll stick to more straightforward features, and a simple task: developing a Bash script.

One last caveat: I work on Linux, and so the examples here will assume you are too. Most of Emacs’ features work the same on Windows and Macs, but interacting with external processes, like a Bash shell, may require additional configuration on those systems.

Objectives

Create a new bash shell script:
- write the code
- run the code in a shell
- save the results

More importantly, by the end of this quick example, I hope you’ll see that Emacs, while a little weird, isn’t that different from more conventional programs. And if I can convince you of that, then we’ll be ready to explore some of the more useful features it has for us.

Getting Started

You’ll need to install Emacs if it isn’t already. On Ubuntu, you can do this directly from the Software Center. Other distributions will undoubtedly have it in their repositories, and you can get the latest release directly from GNU for Windows and Mac (and Linux)¹.

Start by opening Emacs from the launcher, or with the command emacs on the command line. This should open up a new graphical window that looks something like this:

NB: Emacs uses a lot of keyboard shortcuts. I’ll be introducing them slowly, to keep things simple. However, it’s possible you’ll accidentally hit one, and something strange will happen. Most of the time, you can fix this by quitting, which you can do with the keyboard shortcut C-g (that is, hold the Control key down while pressing g). It may also be helpful to know there’s an undo option in the “Edit” menu, if you accidentally change a bunch of text.

First off, we’ll create a new file: click on the “File” menu, and select “Visit New File.” You’ll see a file browser. We use the browser to create a new folder gbs-analysis, and open a new file script.sh in that folder.

Now we have an empty file for our script. We’ll add a few commands to set up our project:

mkdir data ## for our raw data
mkdir output ## analysis results
cp ~/dl/Vaccinium3-SingleRead300_S1_L001_R1_001.fastq.gz data

To save the file, we can use the menu “File - Save.” We’ll do this a lot, so we can use a keyboard shortcut instead: C-x C-s. That is, hold the Control key down and press x, then s.

Now we’re ready to run the code in a shell. We can start a shell from the menu: “Tools - Shell Commands - Run Shell Interactively.”

This opens a new shell terminal inside Emacs. In my case, it opens below the script window. Depending on the size and shape of your screen, it might open beside your script instead:

This terminal is almost normal: you can enter commands at the prompt and view the output, just as you would with a regular terminal. To do that, you need to move the cursor to the prompt. You can do that with your mouse. However, moving back and forth between different windows in Emacs is something that we’ll do alot, so there’s a keyboard shortcut for that too: C-x o. That moves the cursor from one window to the other. Try that a few times.

Now we have a script window, and a terminal, and we’d like to run a few lines from our script in the terminal. First, we move the cursor back to our script window (C-x o), and then move it to the beginning of the first line (you can use the arrow keys).

Here we encounter one of Emacs quirks: it doesn’t use the usual C-c/C-v copy and paste convention. Emacs was already 10 or 15 years old when this was developed, and it already had its own way: “killing” and “yanking.” We kill with C-k, and yank with C-y.

So to copy the first line, we first kill it with C-k. And it’s gone!

We can get it back by “yanking” it with C-y. Now we’re back where we started, except that the contents of the line are stored in the “kill-ring.” Now switch back to the shell prompt and yank again:

Now the line we killed is on the command prompt and ready to run. Hit enter and we’ll create our new directory. Repeat with the next line. The third line won’t work of course, because you don’t have the same file on your computer, but I’ll use it here to round out my example.

We can enter commands directly in the terminal. I’ll use ls to check that the new files and directories are all where we put them:

Remember I said the terminal is ‘almost’ normal. One of the things that’s not normal about it is that you can move around in it with the arrow keys, and kill and yank text if you like. You can even enter additional text in the transcript if you like. That might be useful if you want annotate something; just be careful not to hit enter when you do this, or the text will get entered at the prompt, which you usually don’t want to do.

You can also save the text in the terminal window as a text file, again using the File - Save dialoag, or the keyboard shortcut C-x C-s. That’s not very useful for this toy example. But if you had spent hours on a bioinformatic pipeline, you could then save a record of everything you’d done as a text file.

So… what?

That may have been a little underwhelming, if you’ve ever listened to an Emacs zealot ramble on about how mind-blowing this program is. What I hope you got from this short introduction is:

Emacs has some quirks, but it’s basically just a text editor that you can use like any other editor, with a few concessions
Having your script file and terminal in the same program is handy for developing scripts

There are some obvious deficiencies here:

transfering text from the script to the shell is a bit clunky
maintaining a script file and a separate file with the shell output will get confusing quickly

It would be much better if we could:

write our script in a single file
have the commands sent automatically to the shell
have the results pasted back into the script file at the appropriate location

That would be a big improvement. Emacs can do that and more:

mix different languages, including Bash, R, Python and more in one file
include human language, including sections, formatting (bold, italics), and links to other files and websites – all in the same source file

That’s where orgmode comes in, and it really is a killer feature. But before we get there, we need to get a bit more familiar with the basics of Emacs. For this, I strongly recommend the built-in tutorial. You can start it with C-h t from Emacs. It takes 20 or 30 minutes, and explains the basics of getting around in the program. Try that out, and we’ll look at orgmode in the next post.

There are several customized Emacs distributions that provide improved default settings. Some of them are quite good, but I’m going to stick to standard Emacs to minimize distractions.↩︎

Publication Quality R Figures

Wed, 19 Feb 2014 00:00:00 +0000

Figure 1: A. Iris Sepal Size by Species. B. Iris Petal Width

Introduction

Learning Objectives

At the end of this lesson, you should be able to:

Customize plots produced with the R base graphics system
Design multi-panel plots
Design plots to suit the publication requirements of a journal
Save your plots as high-resolution raster or vector image files as required by your publisher

Pre-requisites

You will need:

A recent version of R installed on your computer
Familiarity editing R scripts and passing commands from a script to the R interpreter
Note that RStudio is not ideal for this lesson, due to limitations in how it processes plotting commands; the default RGui installed on Windows or Mac will work better
These notes!
Optionally, some of your own data to work with during the exercises

Motivation

You have several options for plotting with R. The simplest is the built-in or base graphics package. Base graphics are less powerful than newer alternatives like lattice or ggplot2. On the other hand, it’s much easier to customize base graphics than the others. For this reason, I prefer to use the built-in functions when preparing single-panel plots.

ggplot2 is definitely worth investigating, especially if you want to produce complex multi-panel faceted plots. The official website has all the documentation. Roger Peng has also posted a very nice introductory video on YouTube.

Building Our Plot

For this example, we’ll use the guidelines provided by the American Journal of Botany. AJB accepts figures 3.5 inches (1 column), 5-6 inches (1.5 columns), or 7.25 inches wide (2 columns). The height can be up to 9 inches. We’ll start with a one-column plot, so the dimensions should be 3.5 inches wide.

The figure we’ll plot is from the built-in iris data set. We’ll do a simple scatterplot of Sepal.Length against Sepal.Width.

Size

Let’s start with a square. If we need more height, we can increase the size as necessary. Similarly, if we decide we need to stretch our figure over two columns, we can change later.

Note that RStudio isn’t the best environment for this exercise. Unfortunately, it’s not possible to create new plot windows in RStudio, so you can’t specify the dimensions of the figure for the on-screen display. Consequently, when you save your figure to an image file, you won’t necessarily get exactly what you see on the screen. In many cases, this may well be fine. But double-check the image file to make sure that you get what you expected. If you didn’t, it may be because the on-screen display and the saved image were not close enough to the same dimensions.

To set up the canvas for our plot, start a new device:

dev.new(height = 3.5, width = 3.5)

If you are using RStudio, dev.new() won’t work. Instead, drag the edges of the plot window to get as close to a 3.5" square as possible.

Content

Now that our canvas is ready, we can start placing our graphics. Let’s start with the default plot.

plot(Sepal.Length ~ Sepal.Width, data = iris)

Plot Symbols

The default plot uses the same symbol for each point. However, our data frame includes samples from three different species:

str(iris)

## 'data.frame':    150 obs. of  5 variables:
##  $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##  $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##  $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##  $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##  $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

Recall that a factor is just a vector of integers, with each integer having it’s own label. We can display the underlying numbers by converting from factor to numeric:

as.numeric(iris$Species)

##   [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
##  [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
##  [75] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [149] 3 3

This is a very useful feature. It means we can set a different plot symbol for each species:

plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris)

Margins

Now we can see what we’re working with, but the layout isn’t ideal. In particular, the plot is very small relative to the size of the figure. We can fix this with the mar parameter. mar takes a vector of four integers, which set the width of the margin on the bottom, left, top and right sides respectively (remember clockwise from the bottom!). These numbers refer to the width of each margin in lines — i.e., the width required for a single line of text. The default is c(5, 4, 4, 2) + 0.1. The top margin in particular is usually too wide. We will very rarely add a title to a published figure, so we don’t need to set aside space for it.

Set the value of mar with the par function.

par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris)

That’s better. But we’ve lost our axis labels. They aren’t actually lost, but they are plotted outside of the margins we’ve set, so they are no longer visible. I find the defaults that R uses for the axes to be larger than we need. Better to turn off the axes entirely and replot them ourselves.

Note that once the margins are set with par(), they will keep their value until we open a new plot window, or reset them.

Axes

par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris, 
     ann = FALSE,      # turn off axis labels
     axes = FALSE)     # turn off axis ticks

Notice that axes = FALSE has turned of the box around our plot. We can put it back easily:

box()

Now we can explicitly add each axis with their size and placement specified:

axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8)

What just happened? Let’s breakdown the arguments to axis:

side:: which side of the plot, clockwise from bottom, same as for mar above
tcl:: length of the ticks. Negative values indicate extending outwards from plot, positive values extend inward. The default is -0.5, which I find a bit too long.
mgp:: margin line, a vector of three numbers, which indicate the position of the axis title, axis labels, and axis line, respectively. The values are the number of lines away from the plot border to place each item, with 0 indicating the margin of the plot area. Note that title doesn’t matter here, since we aren’t using an axis title (yet).
cex.axis:: axis character expansion. Scale the size of the tick labels. < 1 reduces the size, > 1 increases the size.

Now we can add our axis titles back in:

mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)

Here, we can use line to adjust the distance between the label text and the axis.

The finished plot

Putting this altogether gives us the following plot.

par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris, 
     ann = FALSE,      # turn off axis labels
     axes = FALSE)     # turn off axis ticks
box()
axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)

Exercise 1: adding a legend

We now have a complete figure. We could provide an explanation of the symbols in the caption, but it might be nicer to have a legend plotted on the figure. This is easily done with the legend() function.

legend(legend = levels(iris$Species), x="topleft",
       pch = 1:3)

Since we set the pch argument in our plots using the factor iris$Species, we can use levels() function to extract the labels for the legend. pch indicates the actual symbols to use, and x is the location of the legend.

This is clearly not “publication quality.” Our plot needs a bit more space for the legend. See if you can make an attractive plot. The following options might be helpful:

dev.new():: width, height
plot():: xlim, ylim, cex
legend():: x, y, bty, horiz, cex, pt.cex, text.width

If you need a hint, take a look at the next section.

Additional Customization

Selecting Plot Symbols

If you want to select different symbols, it’s easy to do using R’s subsetting syntax. By default, for three levels of our Species factor, we get symbols 1, 2, and 3. If instead we wanted to use symbols¹ 19, 5, and 3, we could do this:

par(mar = c(3, 3, 0.5, 0.5))
mysymbols <- c(19, 5, 3)
plot(Sepal.Length ~ Sepal.Width,
     pch = mysymbols[as.numeric(Species)], data = iris,
     ylim = c(4.0, 8.5), ann = FALSE, axes = FALSE)
box()
axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)
legend(legend = levels(iris$Species), x = "top",
       pch = mysymbols, horiz = TRUE, bty = 'n',
       cex = 0.9, text.width = c(0.6, 0.7, 0.6))

That’s an important application of R’s subsetting commands, so make sure you follow what happened — we subset the mysymbols vector with the longer as.numeric(iris$Species) vector, which converted the values from (1, 2, 3) to (19, 5, 3):

mysymbols

## [1] 19  5  3

as.numeric(iris$Species)

##   [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
##  [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
##  [75] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3
## [112] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
## [149] 3 3

mysymbols[as.numeric(iris$Species)]

##   [1] 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19
##  [26] 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19
##  [51]  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5
##  [76]  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5  5
## [101]  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3
## [126]  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3  3

Panels

ggplot2 provides a very sophisticated system for producing multi-panel plots. But it’s easy enough to create a simple panel using the base graphics. For this example, let’s do a two-plot horizontal panel, with our scatter plot in the first position, and a boxplot of petal widths in the second position. A two-column plot in AJB is 7.25 inches wide:

dev.new(width = 7.25, height = 3.5)

Next we need, to inform R that we’re splitting the figure into two panels:

par(mfrow = c(1, 2))

mfrow sets the graphics device for rows and columns, in this case one row, two columns. We can now put our first plot in the first spot:

After dividing a plot device into panels with mfrow, the first high-level plot (i.e., plot, boxplot etc.) command will be placed in the first panel. All subsequent low-level plotting commands (i.e., legend, axis, mtext etc.) will be added to this same panel. When the next high-level command is called, it will be placed in the next panel, and focus shifts with it. So we can now add our boxplot to the second panel:

boxplot(Petal.Width ~ Species, data = iris)

Note that the margins we set for the first panel are still in effect. Consequently, we’ve lost the axis labels on our second plot. It’s going to need some attention to make it look right. We’ll leave that for the next exercise.

In the meantime, we have one more requirement to meet. On multi-figure panels, AJB requires an uppercase letter (A, B, etc) to label each plot. This label should go in the upper-left corner of each panel. This is easy to do with the text command. At the moment, we don’t have space in the upper-left corner, so we’ll put the labels in the lower-right temporarily. For example:

## For the first panel:
text("A", x = 4.2, y = 4.5, cex = 2)

## For the second panel:
text("B", x = 3.2, y = 0.24, cex = 2)

Unfortunately, each figure is plotted on different scales, so placing the letters in the same position is not straightforward. Luckily, R provides a function for getting “universal” coordinates for every plot.

## For the first panel:
text("A", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)

## For the second panel:
text("B", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)

The functions grconvertX and grconvertY convert between different coordinate systems. npc is “normalized plot coordinates.” In this system, (0, 0) is the lower left corner of the plot, and (1, 1) is the upper right corner. user, on the other hand, is the coordinate system in effect for the actual plotted data. Which, for our Panel A, means the lower right corner is ca. (2.0, 4.0) and the upper left corner is ca. (4.5, 8.5). So grconvertX(0.9, from = "npc", to = "user") returns the X coordinate to plot our text 90% of the way to the left side of the plot, regardless of the scale used in that plot. With this addition, we have the following code, and the generated panels:

par(mfrow = c(1, 2))
par(mar = c(3, 3, 0.5, 0.5))
mysymbols <- c(19, 5, 3)
plot(Sepal.Length ~ Sepal.Width,
     pch = mysymbols[as.numeric(Species)], data = iris,
     ylim = c(4.0, 8.5), ann = FALSE, axes = FALSE)
box()
axis(side = 1, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
axis(side = 2, tcl = -0.2, mgp = c(3, 0.3, 0),
     cex.axis = 0.8) 
mtext("Sepal Width", side = 1, line = 1.5)
mtext("Sepal Length", side = 2, line = 1.5)
legend(legend = levels(iris$Species), x = "top",
       pch = mysymbols, horiz = TRUE, bty = 'n',
       cex = 0.9, text.width = c(0.6, 0.7, 0.6))
## For the first panel:
text("A", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)
boxplot(Petal.Width ~ Species, data = iris)
## For the second panel:
text("B", x = grconvertX(0.9, from="npc", to="user"), 
     y = grconvertY(0.1, from = "npc", to="user"), cex = 2)

Exercise 2: Completing the Panel

There are still a few problems with our panel:

The title of the Y axis on the second panel is not visible
The panel labels (A and B) are in the wrong positions — they should be in the top left corners
Fixing the panel labels will require moving the legend for the first figure

If you change ylim, you can put the legend on the bottom, and make space for the label at the top. Go ahead and see what you can do with this. You can use my example, Figure 1 at the top of this article as a model. There is a trick to formatting the x-axis of boxplots. When calling the function axis, you’ll have to set the at argument to indicate where to plot the labels (which should be c(1, 2, 3), and you’ll have to set the labels argument to indicate what the labels should be.

Alternatively, try formatting your own data according to the requirements of a journal in your field.

Image Formats

R can save graphics to a variety of formats, including anything your target journal might require. In general, you can store your images in one of two classes of file format:

raster:: images are stored as a matrix of values, with each value indicating the color of a single pixel in the grid. Best used for photographs. Examples: jpg, tiff, png.
vector:: images are stored as a series of mathematical instructions for re-creating the display: lines, polygons, text etc. Best used for line drawings. Examples: eps, svg.

Raster Images

Raster images are stored as a grid of numbers called pixels. Each number records the colour of a single pixel in the image. As a consequence, the image resolution is limited by the number of pixels recorded in the file. In our example, we need a figure 3.5 inches wide. AJB requires a resolution of 1000 dots per inch (DPI) for line drawings, which means we need a source image 3500 pixels in the x and y dimension. We don’t actually need to do these calculations, though, R will handle it for us. We just need to pick a format and set the final resolution.

AJB prefers TIFF format for raster files. To generate one we will use the tiff() function. Note that this function only sets the file details for our plot; we need to add the plotting code after we open the file, and close it when we’re done with dev.off().

tiff(filename = "iris.tiff", width = 3.5, height = 3.5,
     units="in", res = 1000, compression = "lzw")
par(mar = c(3, 3, 0.5, 0.5))
plot(Sepal.Length ~ Sepal.Width, pch = as.numeric(Species),
     data = iris, 
     ann = FALSE,      # turn off axis labels
     axes = FALSE)     # turn off axis ticks
## Insert additional plot code here
dev.off()

width, height and units set the size of the image, res sets the resolution in points per inch. compression reduces the size of the file. The lzw options is only available for tiff files. It’s lossless, which means the compressed image is just as good as the original, so there’s no reason not to use it. In this case, it reduces the file size from 36Mb to 366K — a 99% reduction!

To create the same image as a jpg with the same resolution we’d use:

jpeg(filename = "iris.jpg", width = 3.5, height = 3.5,
     units = "in", res = 1000, quality = 85)
## insert plot code here!
dev.off()

jpg files are always compressed, and they use a lossy compression. That means there is some degradation of the image quality associated with the compression. The quality argument determines how aggressively the image is compressed. Higher values produce larger, less-degraded images. As a rule of thumb, 85 usually produces fine images at a reasonable size. In this case, the file is 550K, so a little larger than the compressed TIFF file.

Vector Images

Vector images are stored as a list of instructions: ‘draw a line from here to here, put a circle at this coordinate’ etc. As a consequence, they don’t have an inherent resolution; rather, they can be printed at any resolution necessary. So we don’t worry about the resolution when creating them, just the size and width. There are other options we need to be concerned with here:

paper:: "special" indicates that we are making a single image, not a full-page
onefile:: FALSE indicates that we are making a new file for each image (probably not necessary with a single image)

horizontal: :FALSE indicates we don’t want a landscape-orientation

To create an eps file:

postscript("iris.eps", height = 3.5, width = 3.5,
           paper = "special", onefile = FALSE,
           horizontal = FALSE) 
## insert plot code here!
dev.off()

This file is only 11K, and can be printed at any resolution. That makes eps a very convenient format to use. However, you may run into issues with fonts. By default, eps files produced by R don’t include the fonts, just the position of the letters to place on the image. If you need to embed the fonts, you need to explicitly request this:

embedFonts("iris.eps", outfile="iris-embed.eps")

This command creates a new file, iris-embed.eps, that has the font information embedded in the file. Fonts can be tricky, and specific details vary between Windows, Mac and Linux. It’s easiest to stick to the default font settings, and only dive into custom fonts and settings if you are required by the publisher.

Note that, pdf files are more common than eps. You can create pdf image files directly from R as well, using:

pdf("iris.pdf", height = 3.5, width = 3.5,
    paper = "special", onefile = FALSE) 
## insert plot code here!
dev.off()

How did I pick 19, 5 and 3? You can see all 25 symbols with plot(1:25, pch = 1:25).↩︎

Preparing Rubus samples for herbarium study

Sun, 25 Aug 2013 00:00:00 +0000

Collecting blackberries and their relatives (Rubus spp.) for herbarium study is particularly challenging, and even experienced field-botanists may not appreciate everything that is involved. More than in other vascular plant groups, to make a good Rubus specimens, you need to understand a bit about the their life-cycle.

A single Rubus allegheniensis specimen, with the first-year primocane on the right, the second-year floricane on the left, and my expert Rubus presser Charlotte in the middle. Note that the primocane is unbranched, while the floricane has many flowering branches

Blackberries, and most other Rubus, are perennial shrubs. However, their stems, called canes, are biennial. In the first year of growth a cane remains in a vegetative state, and does not flower. Usually, it doesn’t even form any side branches. These canes are referred to as primocanes. At the end of the season, buds form in the leaf axils and at the shoot tip.

In the second year of growth, flowering shoots will emerge from these buds. At this stage, the cane is called a floricane. Growth stops after the fruits are formed; the floricane doesn’t produce any over-wintering buds.

A mature blackberry plant produces both primocanes and floricanes at the same time. The floricanes start earlier in the season, since they get a head-start building from last year’s primocanes. By the time the flowers appear in June or early July, the next batch of primocanes are only just beginning to emerge from the underground rhizome.

Unlike other angiosperms, blackberry flowers aren’t generally useful taxonomically¹. The really interesting characters are the primocane leaves, the stem armature (prickles, bristles, hairs and glands) and the mature inflorescences of the floricane. This means the best time to collect specimens is later in the season, when the fruits are forming or ripe and the primocanes are well developed.

First, confirm that you are collecting both a floricane and a primocane, and they both come from the same rootstock. This is important, as we’ve seen a lot of mixed populations!
You make at least two separate sheets for each plant. For the first, select a representative section of the cane from the central part of the primocane, with a few well-formed mature leaves attached. The cane tips are easily damaged, and don’t generally press well. Similarly, the leaves at the base of the cane are often damaged or misshapen.

Pressing the leaves is tricky, particularly on prickly specimens. Keep in mind that on the mounted sheet you will want to see the intact outline of a complete leaf, as well as examples of the upper and lower surface of the leaves.
Make sure the specimen includes a nice segment of the most heavily-armed section of the stem. Sometimes the best leaves and the nastiest prickles are on different sections of the stem. If this is the case, include a section of the prickliest part of the stem, with the leaves removed, in your sample.
Use a second sheet, with a separate newsprint folder, for the floricane. Get a section of the stem with two or three well-formed inflorescence branches. Again, the little shoots at the very top or bottom of the cane are often misshapen, so use the center of the cane. As for the primocane, make sure you’ve got a section of the prickliest cane.

Floricane leaves are incredibly variable. If you can include a few well-formed leaves in the folder, great. They don’t get much attention taxonomically, though, due in large part to their inconstancy.
If you’re collecting tissue for DNA work, the freshly-formed leaves at the end of the primocane are the best source. I had some bad experiences as a grad student, and so I collect enough for at least four extractions, and four flow cytometry analyses. Typically 50-100 cm². Double-bagged, with labels inside and out in case of catastrophe.
Finally, make sure you note the habit of the plant for the label. The major growth forms include:
- highbush blackberries, with upright, ascending, or somewhat arching stems that don’t reach the ground and don’t root at the tip
- doming forms, that start out upright, but arch and may or may not root at the tips, possibly trailing along the ground as well
- trailing forms, which may either grow prostrate on the ground, or trail and scramble over other low vegetation
Do your best to describe what you see. It can be challenging to determine if you’re looking at a trailing form, or a highbush form that is now growing on the ground after having been weighed down by snow or fallen branches. And some highbush species will look strange and stunted when growing in poor conditions.

The shape of the inflorescence should be visible on the sheet. The fruits, however, will either rot or dry down to a pile of seeds. So definitely include a note on their size (length and width). Also flavour, if you get some ripe ones. Better in your belly, with a good description on the label, than pressed into mash untasted!

Note that this protocol reflects the minimum necessary material for a useful herbarium collection. If you have time and space for more than two sheets, and you see interesting bits you’d like to include, you can always collect more. Some of our collections include five or six sheets for a single plant.

What I should say here is that flowers have not been found useful by previous taxonomic research on blackberries. Sadly, previous taxonomic research on blackberries is quite a mess, and I am prepared to accept that anything I think I know about this group is completely wrong. ↩︎