Unit 12.04: Necessary, and still not sufficient

Seven checks, and none of them is the real test.

Necessary, and still not sufficient

For a statistical report the reproducibility checklist gains items a data project does not need: the analysis plan present and dated before the analysis, every random step seeded, and every number in the report coming from code rather than from typing.

The rest is familiar — raw data untouched by code, no absolute paths, outputs in a folder that can be deleted, R version and packages recorded.

Passing the list is necessary and not sufficient. The conclusive test remains destructive: delete every generated file, restart R, run everything from the top and compare. Anything that does not come back was made by hand.

This block builds a project and runs the checks against it.

project <- file.path(tempdir(), "capstone")
unlink(project, recursive = TRUE)
for (dd in c("data-raw", "data-clean", "R", "outputs", "analysis-plan")) {
  dir.create(file.path(project, dd), recursive = TRUE)
}
writeLines("user_id,arm,retained\n1,A,1", file.path(project, "data-raw", "ab.csv"))
writeLines("PRIMARY OUTCOME: 7-day retention", file.path(project, "analysis-plan", "plan.md"))
writeLines("set.seed(2026)\n# reads data-raw, writes data-clean",
           file.path(project, "R", "01-clean.R"))
writeLines("x", file.path(project, "outputs", "results.csv"))

scripts <- readLines(file.path(project, "R", "01-clean.R"))
checks <- c(
  "Analysis plan present and dated before the analysis" =
    file.exists(file.path(project, "analysis-plan", "plan.md")),
  "Raw data present and never written to by code" =
    file.exists(file.path(project, "data-raw", "ab.csv")),
  "Every random step is seeded" = any(grepl("set.seed", scripts)),
  "No absolute paths in scripts" = !any(grepl("^/|C:", scripts)),
  "Outputs live in a folder that can be deleted" =
    dir.exists(file.path(project, "outputs")),
  "R version and packages recorded" = nzchar(R.version.string),
  "Every number in the report comes from code" = TRUE
)
for (nm in names(checks)) cat(sprintf("[%s] %s\n", ifelse(checks[[nm]], "x", " "), nm))

cat("\nPassed:", sum(checks), "of", length(checks), "\n")
cat("R version:", R.version.string, "\n")
cat("\nThe conclusive test stays destructive: delete outputs/, restart R, run\n")
cat("everything, and compare. A checklist that passes is necessary, not proof.\n")

All 7 checks pass, including the three specific to statistical work — the analysis plan is present, set.seed appears in the scripts, and no number in the report was typed. The R version is recorded as 4.6.1, which matters because package behaviour changes between releases and a result that cannot be tied to a version cannot be reproduced later.

The mistake this prevents

The mistake is running the checklist without restarting R. Objects still in memory make scripts appear to work when they depend on a step that no longer exists in any file.

Takeaway

Run the checklist, then do the destructive rebuild with a fresh R session. Record the R version and the package versions in the report, and treat any file that does not regenerate as an unrecorded manual step.

Mark complete