Research is reproducible when others can reproduce the results of a scientific study given only the original data, code, and documentation (Alston and Rick, 2021)
Benefits to the Author:
- Clear and complete documentation of your work makes it easier to share, write up and extend in future work, including responding to reviewers and developing new projects
- Conscientious documentation of your work involves a great deal of error-checking, which is reassuring to you – that you haven’t missed anything, or mis-remembered what you did; and to your readers – that you have conducted your work in a rigourous manner
- Reproducible work gets cited more, and developing a data archive creates a new citable product from your research.
Benefits to the Community:
- Increases the speed and fidelity with which we can learn and apply new approaches.
- Makes it easier to avoid mistakes (through the care and attention required to create the archive), and to detect and correct them if they do happen (by allowing others to critically review your work)
How to make your work reproducible
There are two related goals in producing a reproducible analysis: portability, and reproducibility. Reproducibility is perhaps obvious, but it’s not enough that you can reproduce your analysis on your computer. You are the only person with access to your computer. Your work should be reproducible on anyone’s computer.
To achieve portability, we need to know which files are needed for an analysis, and to organize them in a way that they can be readily moved from one computer to another. In practice, this means they’ll all be in a single directory, and that directory will only contain files for that particular analysis. You may have several related analyses that share a directory. That’s ok, but take time to organize them in a sensible way.
While you may have related analyses together in a directory, you don’t want to mix unrelated files and data in this directory. That will make it harder to keep track of what is needed and what isn’t, and will waste space on the computers where this work is ultimately archived.
There are many ways you can organize your work within this directory. One absolute requirement is there needs to be a clear guide to your files, a “Table of Contents”. This should be a ‘plain text’ file - something that can be opened by any text editor. Your archive might be around for decades, and you don’t know if your readers will be able to find a copy of MSWord97 when they need to read it. We can be reasonably confident that plain text files will be accessible for a long time to come.
By convention, this file is called
README.txt, and some data archiving
services (DataDryad) require that you include a
file with this name. You should probably use this name, unless you have a
good reason not to.
One minor exception: if you use
RMarkdown, you can use these formats for
README. They are both plain text, and even if your audience doesn’t
use them, they can open a
README.Rmd file in any text
editor. There are other, similar simple markup formats used in different
coding communities. As long as they are saved as plain text files they will
meet our requirements.
The contents of your
README should describe as clearly as possible the
contents of your archive. Here is an excerpt from one of mine:
Start with trich.Rmd to read our draft manuscript. See trich-prep.Rmd for the bulk of the code used in generating this manuscript.
trich.Rmd : main manuscript file
plantarum.json : bibliography (Zotero bibliography)
trich-prep.Rmd : The bulk of the code used in the analysis. Loaded from trich.Rmd to regenerate the figures and tables there.
data/ssr_raw.csv : raw microsatellite data. See trich-prep.Rmd (Loading Data) for code to load and translate this to a genind object
data/survey-pops.csv : coordinates of sampled populations
data/eval.opt.2020-06-24.Rda : the output of the Maxent modeling, saved as a binary R Data object. Load into R with the load() function, so you don’t need to repeat the lengthy Maxent analysis.
data/trich-gbif.csv : GBIF records used in the Maxent analysis
data/trich_soil.csv : Soil analysis for each sampled population
data/maps : Maps (shapefiles and rasters) used in the Maxent analysis, and for some of the manuscript plots
I prefer to use
RMarkdown to develop my manuscripts, as this allows me to
keep the code for figures, and the resulting images, together with the text
that describes the methods and interprets the results. If you prefer to
manage your code separately from your writing that’s fine too, but you may
end up structuring your archive a little differently.
If you only have a few files, you may not need to do any further organization of your archive. I find it helpful to use subfolders to keep things organized. Depending on the needs of a project, I use:
- unprocessed data used in the analysis
- intermediate data files, generated from data and not stored permanently
- storage of large external datasets used in the analysis, but not stored permanently in the archive (e.g., WorldClim Climate Data); be sure to include links to the source of any external data you are not archiving yourself!
- images generated by the analysis
- code used in the analysis, if it’s not in the top-level directory
These are not hard rules. You can use whatever structure suits your
project. Just be sure to explain it in your
File organization gets us most of the way to portability. There are a few things we need to do in our coding to complete this arrangement, and of course to ensure we can reproduce the analysis once we move it to a new computer.
Use Relative Paths
absolute path is the location of a file on a particular computer. The
absolute path to the file I’m editing right now is
That location will only ever exist on a Linux computer with a user named
smithty. If I moved it to a Windows computer, it might be located at
If I try to refer to this file by its absolute path on Linux on the Windows
machine, I won’t find it.
On the other hand, from the top directory of my blog,
file will have the same
relative path on both machines:
content/tutorials/2022-10-17-data-management/index.md. That means links
to this file using the relative path will work just fine on both machines.
We should use
relative paths in our analyses too.
For example, I could use an absolute path to load my data:
samples <- read.csv("/home/smithty/nextcloud/trich/2020-06-25/data/survey-pops.csv")
But this won’t load on anyone else’s computer. If we do this instead:
samples <- read.csv("data/survey-pops.csv")
Anyone with our archive can run the code as it is, as long as they have the working directory set properly. For my projects, I do this by running my code from the top directory of the archive. In this case, I have the directory structure:
├── data │ └── survey-pops.csv ├── README.md ├── trich-prep.Rmd └── trich.Rmd
When I run code from
trich.Rmd, I set the working directory to the
location of that file.
RStudio manages this for you with its project
support. If you don’t use that feature, you can tell R to use that location
when it starts. After that, stick to relative file paths, and you don’t
need to worry about your code breaking when you move to a different
Structure Your Code to Run On Its Own
Two common problems that make it hard to run your code are mixing your ‘good’ code with non-working code, and writing code that requires you to update it by hand to finish your analysis.
Mixing good and bad code might look like this:
## Load in our data: myData <- read.data("data/myfile.csv") myDataScaled <- scale(myData) myDataScaled <- scale(myData, center = FALSE) myData <- myData[, -1] myDataScaled <- scale(myData, scale = FALSE)
It’s not unusual to accumulate various versions of code (should I scale this, center it, both? Do I need the first column?). While you’re actively working on your code, you may find you have multiple versions in the same file. They need to be clearly commented, for your own benefit! But more importantly, when you decide which version you’re going to use, remove the rest. Don’t expect yourself (and certainly don’t expect anyone else) to figure out which lines they should run, and which they should skip.
## Load in our data: myData <- read.data("data/myfile.csv") ## drop the name columnn myData <- myData[, colnames(myData) != "name"] ## don't scale, use raw data ## myDataScaled <-scale(myData)
If you have a few lines of code that you might want to revisit, comment them out and leave yourself a note in a comment. If you have large blocks of code you aren’t using, but want to keep a record of, put them in a separate file. The end product should be a file that you can run from start to finish, without deciding which lines to skip and which to run.
A similar problem occurs when you have code that works, but requires you to manually update it in order to complete your analysis. e.g.,
myData <- read.table("data/experiments.csv") myExperiment <- subset(myExperimentData, exp == 1) myExperimentResult <- processingCode(myExperiment) ## repeat for exp 1-20
Code structured like this requires you to edit and re-edit the code many
times to complete your work. That is tedious, and it’s easy to make a
mistake. And when you update
processingCode, you need to rerun everything
by hand. You can avoid this with loops and lists.
Do Not Save Your Workspace
R offers to save your current workspace when you close the terminal. This is sometimes convenient, but can cause hard to detect problems with your analysis. If you happen to alter one of the objects you are working with in an R session, but don’t capture the code in your script file, you won’t have a record of what you’ve done. If you then save your workspace, the next time you work on your code, you will be able to keep using that modified object. At this point, your code and data are out of sync, with nothing to indicate how they differ.
At best, this is inconvenient. At worst, if undetected, you can waste weeks or months analyzing the wrong data! Better to avoid the risk and set R to never save your workspace.
How to get there from here
Now we have an idea of what we want our data archive to look like, how do
we get there? Start by setting up your directory structure, and making a
README file. If you’re starting from scratch, make it a practice to
review your directory regularly, to see what you’ve done, what you’re no
longer using, and updating your
README to capture that. This kind of
regular reflection is useful to track your progress, and helps you keep on
top of your archive work so you don’t have a huge mess to wrangle at the
end of your project.
If you’re already well-along in your project, it may be easier to create an
‘aspirational’ directory and populate it from your existing work. I do this
regularly! I often find I’ve charged into an analysis without thinking
about archiving, and after a few weeks I have an unholy mess of files and
data to deal with. In that case, I create a new directory, a
copy over the main code file I’m using to that directory. When I get to the
first data file in that directory, I copy it over to the
in the new archive, adjust the code to use the relative path, if necessary,
and continue. This can be very helpful in clarifying what code and files
you actually need and use, and what can be left behind.
This is also a good opportunity to ensure that your analysis is structured in a way that it can run start-to-finish without manual editing
You don’t need to delete anything in the old directory, you can keep it in case you later decide you want to revisit some ideas in there.
What to do with it all
One of the benefits of structuring your code in a single directory is there are lots of tools that you can use to manage it. git and GitHub are popular, and very powerful for managing code, especially tracking different versions of the same files. However, they require a certain amount of discipline to get the full benefit, and they are challenging to learn. RStudio does have good support for GitHub repositories.
You can also keep it simple, and sync your directory to Google Drive, Dropbox, or many other options. Once your work is published, you can archive it permanently on Zenodo, DataDryad, or other online services.
All of these options will support housing a single directory and its subdirectories. None of these options will be easy to deal with if you have files spread across multiple directories and mixed with files from other projects!
I’ve been doing some version of this for my own work for years, but have only recently moved to permanent, public archives of my work. Here are three recent examples:
- Hayes et al. (2022): The Genetic Diversity of Triploid Celtis pumila and its Diploid Relatives
- Nowell et al. (2022): Conservation assessment of a range-edge population of Trichophorum planifolium
- Foster et al. (2022): Testing the assumption of environmental equilibrium in an invasive plant species over a 130 year history
I’m still figuring out how best to do this, and my practice will definitely continue to change and evolve. Regardless of the specifics, I have benefited enormously from investing the time needed to make coherent archives of my projects.
Alston, J. M., and J. A. Rick. 2021. A Beginner’s Guide to Conducting Reproducible Research. The Bulletin of the Ecological Society of America 102: e01801.
Foster, S. L., H. M. Kharouba, and T. W. Smith. 2022. Testing the assumption of environmental equilibrium in an invasive plant species over a 130 year history. Ecography.
Hayes, A., S. Wang, A. T. Whittemore, and T. W. Smith. 2022. The Genetic Diversity of Triploid Celtis Pumila and its Diploid Relatives C. Occidentalis and C. Laevigata (Cannabaceae). Systematic Botany 47: 441–451.
Nowell, V. J., S. Wang, and T. W. Smith. 2022. Conservation assessment of a range-edge population of Trichophorum Planifolium (Cyperaceae) reveals range-wide inbreeding and locally divergent environmental conditions. Botany 100: 631–642.