Unit 05.06: Small p, small effect

A p-value measures evidence against the null. It says nothing whatever about how big the effect is.

Small p, small effect

Because the p-value depends on both effect size and sample size, a large study can produce a tiny p-value from an effect nobody would act on, and a small study can miss an important effect entirely.

That is why 'highly significant' is not a meaningful phrase. It sounds like a statement about magnitude and is a statement about sample size.

Three other habits to drop: calling p = 0.06 'a trend' — the threshold was fixed in advance, so this is arguing with your own rule; reporting a non-significant result as 'no effect'; and quoting p-values to six decimal places, which implies precision the estimate does not have.

This block runs two studies with very different sample sizes.

import numpy as np
from scipy import stats

rng = np.random.default_rng(208)

studies = {
    "small": (rng.normal(100, 15, 20),    rng.normal(106, 15, 20)),
    "large": (rng.normal(100, 15, 10_000), rng.normal(100.7, 15, 10_000)),
}
for name, (a, b) in studies.items():
    res = stats.ttest_ind(b, a, equal_var=False)
    ci = res.confidence_interval()
    print(f"{name:5s} n={len(a):5d} per group   difference {b.mean() - a.mean():5.2f}"
          f"   p = {res.pvalue:.5f}   CI [{ci.low:5.2f}, {ci.high:5.2f}]")

print("\nThe large study finds a much smaller difference with a much smaller")
print("p-value. A p-value measures evidence against the null, not effect size.\n")
print("Say this : 'a difference of X (95% CI a to b, p = ...)'")
print("Not this : 'highly significant'   -- p is not a magnitude")
print("Not this : 'p = 0.06, a trend'    -- the threshold was set in advance")
print("Not this : 'no effect'            -- absence of evidence is not evidence")
print("Report p to 3 significant figures, or as p < 0.001 when smaller.")

The small study finds a difference of 4.33 with p = 0.43338 and an interval from −6.74 to 15.39 — inconclusive. The large study finds 0.78, less than a fifth the size, with p = 0.00024 and an interval from 0.36 to 1.20. The smaller effect has by far the smaller p-value, purely because n was 500 times larger.

The mistake this prevents

The mistake is ranking findings by p-value. The strongest evidence in a results table is frequently attached to the least important effect.

Takeaway

Report the estimate, its interval and the p-value together, to three significant figures or as p < 0.001. Never use the p-value as a measure of how large or how important an effect is.

Mark complete