Unit 05.01: The share of null worlds at least this extreme

A p-value answers exactly one question, and it is not the one most people think it answers.

The share of null worlds at least this extreme

The test statistic puts the observed difference on a standard scale — how many standard errors from the null value. The p-value is the probability of a statistic at least that extreme *if the null were true*.

That conditional is everything. H0 is assumed true in order to compute the p-value, so the p-value cannot be the probability that H0 is true. Nor is it the probability the result was chance, nor one minus the probability the effect is real.

Simulation makes it concrete: build thousands of datasets in a world where H0 holds, and count how many produce a statistic at least as extreme as yours.

This block computes the p-value twice — once by simulation, once by formula.

import numpy as np
from scipy import stats

rng = np.random.default_rng(202)
observed = rng.normal(104, 15, 30)
obs_t = stats.ttest_1samp(observed, popmean=100).statistic
print(f"Observed mean: {observed.mean():.2f}   t = {obs_t:.3f}\n")

# Build the null distribution by simulating a world where H0 is TRUE.
null_ts = np.array([
    stats.ttest_1samp(rng.normal(100, 15, 30), popmean=100).statistic
    for _ in range(20_000)
])
simulated_p = float(np.mean(np.abs(null_ts) >= abs(obs_t)))
formula_p = stats.ttest_1samp(observed, popmean=100).pvalue

print(f"Share of null simulations at least this extreme: {simulated_p:.4f}")
print(f"p-value from scipy                             : {formula_p:.4f}")
print(f"Difference                                     : {abs(simulated_p - formula_p):.4f}")
print()
print("They agree because the formula computes exactly this share.")
print("A p-value is NOT the probability that H0 is true -- H0 was ASSUMED")
print("true in order to compute it.")

The observed mean is 106.56, giving t = 2.703. Of 20,000 simulated datasets drawn from a true null, 1.12% produced a statistic at least that extreme. scipy reports 0.0114 — a difference of 0.0002. The two agree because they are the same quantity; the formula is a shortcut for the simulation, not a different idea.

The mistake this prevents

The mistake is 'p = 0.01, so there is a 1% chance this is a fluke'. That sentence reverses the conditional: the 1% is the frequency of data like yours in a null world, not the probability of a null world given your data.

Takeaway

Read a p-value as 'data this extreme would arise this often if the null were true'. When the definition slips, simulate it — the null distribution is a dozen lines of numpy.

Mark complete