Unit 08.03: Check reproducibility, do not assert it

Reproducible means somebody else can get your numbers. The only way to know is to try.

Check it, do not assert it

The claim 'this is reproducible' is usually made from memory, and memory does not know about the file you fixed by hand in week two or the package you installed and never mentioned.

A checklist makes the claim testable: raw data present and untouched by code, scripts in one place, outputs in a folder that can be deleted, no absolute paths anywhere, and the R version recorded.

The real test is destructive, and it is the only one that settles the question. Delete everything the code produced, restart R, run the scripts from the top, and compare the outputs with what you had. Anything that does not come back was made by hand.

This block builds a small project and runs the checks against it.

# Run the checks. Do not assert reproducibility from memory.
project <- file.path(tempdir(), "final-project")
unlink(project, recursive = TRUE)
for (d in c("data-raw", "data-clean", "R", "outputs")) {
  dir.create(file.path(project, d), recursive = TRUE)
}
writeLines("ward,visits\nNorth,412", file.path(project, "data-raw", "returns.csv"))
writeLines("# reads data-raw, writes outputs", file.path(project, "R", "01-clean.R"))
writeLines("x", file.path(project, "outputs", "by-ward.csv"))

checks <- list(
  "Raw data present and untouched by code" = file.exists(file.path(project, "data-raw", "returns.csv")),
  "All scripts live in one folder"         = length(list.files(file.path(project, "R"))) > 0,
  "Outputs are in a folder that can be deleted" = dir.exists(file.path(project, "outputs")),
  "No absolute paths in the scripts"       = !any(grepl("/Users/", readLines(file.path(project, "R", "01-clean.R")))),
  "R version recorded"                     = nzchar(R.version.string)
)

for (nm in names(checks)) {
  cat(sprintf("[%s] %s\n", ifelse(checks[[nm]], "x", " "), nm))
}

cat("\nPassed:", sum(unlist(checks)), "of", length(checks), "\n")
cat("The real test is destructive: delete outputs/, run everything, compare.\n")
cat("R version for the report:", R.version.string, "\n")

All 5 checks pass here: raw data present, scripts in one folder, outputs in a deletable folder, no absolute paths in the scripts, and the R version recorded as 4.6.1. The version matters because package behaviour changes between releases, and a result that cannot be tied to a version cannot be reproduced later. But passing the list is necessary, not sufficient — only the destructive rebuild is conclusive.

The mistake this prevents

The mistake is testing reproducibility without restarting R. Objects still in memory make scripts appear to work when they depend on a step that is no longer in any file.

Takeaway

Run the checklist, then do the destructive test: delete the outputs, restart R, run everything, compare. Record your R version and the packages you used in the report.

Mark complete