Unit 11.04: What could this test have seen?

Power is a question you answer before the data exists. Afterwards it is too late to be useful.

What could this test have seen?

Power is the probability of detecting an effect of a given size if it is really there. The minimum detectable effect is the smallest effect the test can reliably find at a given sample size.

Running the calculation before the test tells you how much data you need to answer the question. Skipping it means a non-significant result is uninterpretable: you cannot tell whether there was no effect or whether the test was never capable of seeing one.

Power rises steeply with sample size and with effect size, so the useful framing is not 'how much data do I need' in the abstract but 'how much do I need for the smallest effect worth acting on'.

This block simulates detection rates across three sample sizes and three effect sizes.

# Power is a planning question, answered before the data exists.
power_for <- function(n, p1, p2, reps = 2000) {
  mean(replicate(reps, {
    a <- rbinom(1, n, p1); b <- rbinom(1, n, p2)
    suppressWarnings(prop.test(c(a, b), c(n, n))$p.value) < 0.05
  }))
}

set.seed(805)
cat("Base rate 30%. Chance of detecting each effect at alpha 0.05:\n\n")
cat(sprintf("%10s %10s %10s %10s\n", "n per arm", "+1pp", "+2pp", "+5pp"))
for (n in c(1000, 4000, 16000)) {
  cat(sprintf("%10d %9.0f%% %9.0f%% %9.0f%%\n", n,
              power_for(n, 0.30, 0.31) * 100,
              power_for(n, 0.30, 0.32) * 100,
              power_for(n, 0.30, 0.35) * 100))
}

cat("\nRead it the useful way round: with 4,000 per arm you will usually miss\n")
cat("a 1-point effect and usually catch a 2-point one.\n")
cat("The minimum detectable effect is what the test can see. Deciding it in\n")
cat("advance is what stops an underpowered test being read as 'no effect'.\n")

At 4,000 per arm a 1-point effect is detected 14% of the time, a 2-point effect 48%, and a 5-point effect 100%. Reaching 97% for a 2-point effect needs 16,000 per arm — four times the data. Read the other way: an experiment with 1,000 per arm will miss a 2-point effect 86% of the time, so a null result from it says almost nothing.

The mistake this prevents

The mistake is computing power after a null result to explain it. Post-hoc power is a deterministic function of the p-value and adds no information at all.

Takeaway

Calculate the sample size before running the test, using the smallest effect worth acting on. State the minimum detectable effect alongside any null result so a reader knows what the test could have found.

Mark complete