Unit 05.02: Alpha delivers exactly the rate you set

Alpha is not a discovery threshold. It is the false-positive rate you agree to accept, and it delivers exactly that rate.

Five per cent of true nulls will be rejected

Setting alpha at 0.05 means that in a world where the null is true, you reject it 5% of the time. Not approximately — that is the definition, and the procedure keeps the promise.

The consequence people underestimate is multiplicity. Each test carries its own 5% chance, so the probability that *at least one* of several tests produces a false positive climbs quickly. At twenty tests it is closer to a coin flip than to 5%.

This is why the number of tests run must be reported, and why exploratory findings need either an adjustment or a replication before being treated as results.

This block simulates ten thousand studies in which the null is always true.

import numpy as np
from scipy import stats

rng = np.random.default_rng(203)

# 10,000 studies in a world where the null is TRUE every single time.
p_values = np.array([stats.ttest_1samp(rng.normal(0, 1, 25), popmean=0).pvalue
                     for _ in range(10_000)])

for alpha in (0.10, 0.05, 0.01):
    print(f"alpha = {alpha:.2f}  ->  {np.mean(p_values < alpha) * 100:5.2f}%"
          " of true nulls rejected")

print("\nEvery one of those is a false positive. The rate is not an accident")
print("of this data -- it is exactly what alpha promises.\n")

print("Twenty independent tests, all nulls true, alpha 0.05:")
any_sig = np.array([
    (np.array([stats.ttest_1samp(rng.normal(0, 1, 25), popmean=0).pvalue
               for _ in range(20)]) < 0.05).any()
    for _ in range(3000)
])
print(f"  P(at least one 'significant') = {any_sig.mean():.3f}")
print(f"  1 - 0.95**20 predicts        = {1 - 0.95 ** 20:.3f}")

The rejection rates come out at 10.20%, 4.95% and 0.99% for alpha of 0.10, 0.05 and 0.01 — exactly what was promised, and every one is a false positive. Then twenty independent tests at alpha 0.05: the chance that at least one is 'significant' is 0.645, against the theoretical 1 − 0.95²⁰ of 0.642. Two thirds of such families produce a false positive.

The mistake this prevents

The mistake is running many comparisons and reporting the significant ones. With twenty tests the expected number of false positives is one, so finding one is not a finding.

Takeaway

Fix alpha before analysing, report how many tests you ran, and adjust or replicate when there are many. Treat a single significant result among twenty as a hypothesis, not a conclusion.

Mark complete