Unit 01.00: The folder that makes a test choice checkable

A statistical project needs one thing a data project does not: a written plan, dated before the analysis ran.

The folder that makes a test choice checkable

The folder structure is the one you already know — raw, clean, code, outputs — with an addition. A statistical analysis makes claims that depend on choices: which outcome, which comparison, which test, one-sided or two, what counts as significant. Made after seeing the data, those choices are unfalsifiable, because there is always a version of them that produces the result you were hoping for.

Writing them down first is what separates a finding from an artefact. The plan does not have to be long — five lines is often enough — but it has to exist before the analysis and be stored alongside it.

None of this requires ceremony. It requires a file with a date on it.

This block builds the structure and writes the plan for an A/B test.

# A statistical project needs one folder a data project does not: the plan.
project <- file.path(tempdir(), "ab-analysis")
unlink(project, recursive = TRUE)

folders <- c("data-raw", "data-clean", "R", "outputs", "analysis-plan")
for (f in folders) dir.create(file.path(project, f), recursive = TRUE)

writeLines(c("Question: does the new checkout reduce abandonment?",
             "Outcome: abandoned (TRUE/FALSE), one row per session",
             "Comparison: variant B vs variant A",
             "Primary test: two-proportion comparison, two-sided, alpha 0.05",
             "Decided BEFORE looking at the data."),
           file.path(project, "analysis-plan", "plan.md"))

cat("Folders:", paste(basename(list.dirs(project, recursive = FALSE)), collapse = "  "), "\n\n")
cat(readLines(file.path(project, "analysis-plan", "plan.md")), sep = "\n")
cat("\n\nThe plan is written before the analysis, and it is version controlled\n")
cat("alongside it. That is what makes a later test choice checkable.\n")

Five folders, with analysis-plan beside the usual four. The plan names the question, the outcome, the comparison and the primary test — including that it is two-sided at alpha 0.05 — and states that it was decided before the data was seen. Every one of those is a choice a reader would otherwise have to take on trust.

The mistake this prevents

The mistake is deciding the test after looking at the distributions. It feels responsible — you are matching the method to the data — and it quietly turns a 5% error rate into something much larger, because you chose among several possible answers.

Takeaway

Add an analysis-plan folder to every statistical project. Name the outcome, the comparison, the primary test and the significance level before you run anything. Any change afterwards is legitimate if it is recorded as a change.

Mark complete