Unit 05.01: The share of null worlds at least this extreme

A p-value answers exactly one question, and it is not the one most people think it answers.

The share of null worlds at least this extreme

The test statistic puts the observed difference on a standard scale — how many standard errors from the null value. The p-value is the probability of a statistic at least that extreme *if the null were true*.

That conditional is the whole thing. H0 is assumed true in order to compute the p-value, so the p-value cannot possibly be the probability that H0 is true. Nor is it the probability the result was chance, nor one minus the probability the effect is real.

Simulation makes this concrete: build thousands of datasets in a world where H0 holds, and count how many produce a statistic at least as extreme as yours.

This block computes the p-value twice — once by simulation, once by formula.

# A p-value answers ONE question: if H0 were true, how often would data
# this extreme appear? Simulation makes that literal.
set.seed(202)
observed <- rnorm(30, mean = 104, sd = 15)
obs_t <- as.numeric(t.test(observed, mu = 100)$statistic)
cat("Observed mean:", round(mean(observed), 2), "  t =", round(obs_t, 3), "\n\n")

# Build the null distribution by simulating from a world where H0 holds.
null_ts <- replicate(20000, {
  s <- rnorm(30, mean = 100, sd = 15)
  as.numeric(t.test(s, mu = 100)$statistic)
})

simulated_p <- mean(abs(null_ts) >= abs(obs_t))
cat("Share of null simulations at least this extreme:", round(simulated_p, 4), "\n")
cat("p-value from t.test()                          :",
    round(t.test(observed, mu = 100)$p.value, 4), "\n\n")

cat("The two agree because the formula computes exactly this share.\n")
cat("A p-value is NOT the probability that H0 is true; H0 was ASSUMED true\n")
cat("in order to compute it.\n")

The observed mean is 106.85, giving t = 2.418. Of 20,000 simulated datasets drawn from a true null, 2.39% produced a statistic at least that extreme. t.test() reports 0.0221. The two agree because they are the same quantity — the formula is a shortcut for the simulation, not a different idea.

The mistake this prevents

The mistake is 'p = 0.02, so there is a 2% chance this is a fluke'. That sentence reverses the conditional: the 2% is the frequency of data like yours in a null world, not the probability of a null world given your data.

Takeaway

Read a p-value as 'data this extreme would arise this often if the null were true'. When the definition slips, simulate it — the null distribution is usually a dozen lines of R.

Mark complete