Unit 01.00: The folder that makes a test choice checkable

A statistical project needs one folder a data project does not: the plan, dated before the analysis ran.

The folder that makes a test choice checkable

The layout is the familiar one — raw, clean, code, outputs — plus an analysis-plan folder. A statistical analysis makes claims that depend on choices: which outcome, which comparison, which test, one-sided or two-sided, what counts as significant. Made after seeing the data, those choices cannot be falsified, because there is always a version of them that produces the result you were hoping for.

Writing them down first is what separates a finding from an artefact. Five lines is usually enough, provided the file exists before the analysis and lives beside it.

This block builds the structure and writes the plan for a wellbeing study.

import pathlib, tempfile

project = pathlib.Path(tempfile.mkdtemp()) / "wellbeing-study"
for folder in ["data-raw", "data-clean", "src", "outputs", "analysis-plan"]:
    (project / folder).mkdir(parents=True)

plan = """\
QUESTION      Does the four-day week pilot change reported wellbeing?
OUTCOME       WHO-5 wellbeing score, 0-100, one row per employee
UNIT          employee, measured once at week 12
COMPARISON    pilot sites vs comparison sites
PRIMARY TEST  two-sample mean comparison, two-sided, alpha = 0.05
WRITTEN       before any data was extracted
"""
(project / "analysis-plan" / "plan.md").write_text(plan)

print("Folders:", sorted(p.name for p in project.iterdir()))
print()
print(plan)
print("The plan folder is the one a data project does not need. It exists so")
print("that a reader can check the test was chosen before the data was seen.")

Five folders, with analysis-plan alongside the usual four. The plan names the question, the outcome and its scale, the unit, the comparison and the primary test — including that it is two-sided at alpha 0.05 — and records that it was written before any data was extracted. Every one of those is a choice a reader would otherwise have to take on trust.

The mistake this prevents

The mistake is choosing the test after looking at the distributions. It feels responsible — you are matching the method to the data — and it quietly turns a 5% error rate into something considerably larger, because you selected among several possible answers.

Takeaway

Add an analysis-plan folder to every statistical project. Name the outcome, the comparison, the primary test and the alpha before running anything, and record any later change as a change.

Mark complete