Unit 11.02: Decide before you can be tempted

Every line of a pre-analysis plan exists because somebody, under pressure, did the other thing.

Decide before you can be tempted

A pre-analysis plan fixes the primary outcome, the unit, the comparison, the test, the significance level, the minimum effect worth acting on, the sample size, the stopping rule, the exclusions and the subgroups — all before any data is seen.

None of this is bureaucracy. Each item closes off a specific way of producing a result that will not replicate, and every one of them is a decision that feels perfectly reasonable when made after the fact.

The stopping rule is the one most often missing. Repeatedly checking a test and stopping when it reaches significance drives the false-positive rate far above alpha, because you get many chances at the threshold.

This block lays out a complete plan and names what each line prevents.

plan <- c(
  "PRIMARY OUTCOME    : 7-day retention (binary), one row per user",
  "UNIT OF ANALYSIS   : user, first assignment only",
  "COMPARISON         : variant B vs variant A",
  "PRIMARY TEST       : two-proportion test, two-sided, alpha = 0.05",
  "MINIMUM EFFECT     : 2 percentage points (below this we ship neither)",
  "SAMPLE SIZE        : 3,900 per arm (80% power for 2pp at a 30% base)",
  "STOPPING RULE      : analyse once, at 3,900 per arm. No peeking.",
  "EXCLUSIONS         : sessions under 10s (bot traffic); internal accounts",
  "SECONDARY OUTCOMES : session length, support contacts (reported, not decisive)",
  "SUBGROUPS          : none pre-specified"
)
cat(plan, sep = "\n")

cat("\n\nWhat each line prevents:\n")
prevents <- c(
  "one primary outcome"   = "picking whichever of six metrics moved",
  "stated alpha and sides" = "switching to one-sided when p = 0.07",
  "minimum effect"        = "declaring a 0.2pp win a success",
  "stopping rule"         = "stopping the moment the result looks good",
  "exclusions in advance" = "removing the rows that spoil the result",
  "no subgroups"          = "finding an effect in left-handed users in Belgium"
)
for (nm in names(prevents)) cat(sprintf("  %-22s %s\n", nm, prevents[[nm]]))

Ten lines, and the second half of the output pairs each with the specific abuse it blocks: one primary outcome stops you picking whichever of six metrics moved; a stated alpha and sidedness stops the switch to one-sided at p = 0.07; a 2 percentage point minimum stops a 0.2pp win being declared a success; a stopping rule stops peeking; pre-declared exclusions stop the removal of inconvenient rows; and no pre-specified subgroups stops the discovery of an effect among left-handed users in Belgium.

The mistake this prevents

The mistake is treating the plan as paperwork to be written afterwards. Written afterwards it is a description of what you did, not a constraint on what you could do.

Takeaway

Write the plan before collecting data and store it with a date in the project. Any deviation is allowed and must be reported as a deviation, with its reason.

Mark complete