Unit 07.05: Two thresholds, one set by you

With enough data, everything is significant. Whether anything is worth doing is a separate question that statistics cannot answer.

Two thresholds, one set by you

Statistical significance says the effect is distinguishable from zero. Practical significance says it is large enough to act on. The first is computed; the second is a judgement about costs and benefits made by someone who knows the domain.

Because the standard error shrinks with sample size, a large enough study finds a statistically significant difference for essentially any non-zero effect. At that point the p-value stops carrying information and the interval carries all of it.

Setting the practical threshold in advance keeps this honest. Chosen afterwards, it will always sit just below whatever was observed.

This block uses fifty thousand observations per group and a tiny true effect.

import numpy as np
from scipy import stats

rng = np.random.default_rng(406)
a = rng.normal(100.0, 15, 50_000)
b = rng.normal(100.3, 15, 50_000)
res = stats.ttest_ind(b, a, equal_var=False)
ci = res.confidence_interval()
diff = b.mean() - a.mean()

MEANINGFUL = 2.0     # decided in advance: below this, nobody would act

print(f"n per group : {a.size:,}")
print(f"Difference  : {diff:.3f} points")
print(f"95% CI      : [{ci.low:.3f}, {ci.high:.3f}]")
print(f"p           : {res.pvalue:.3g}")
print(f"Cohen's d   : {diff / 15:.3f}\n")

print(f"Smallest difference worth acting on : {MEANINGFUL}")
print(f"Statistically distinguishable from 0: {res.pvalue < 0.05}")
print(f"Whole interval below the threshold  : {ci.high < MEANINGFUL}\n")
print("So the result is statistically significant and practically irrelevant.")
print("The data RULES OUT a meaningful effect rather than demonstrating one.")
print("Decide the threshold before the analysis, or it will always turn out")
print("to sit just below whatever you observed.")

The difference is 0.283 points with an interval from 0.095 to 0.470 and p = 0.00307 — unambiguously significant, with Cohen's d of 0.019. Against a pre-set threshold of 2.0 points, the *entire* interval falls below it. The correct conclusion is not 'a significant improvement' but the opposite: the study has ruled out an effect large enough to matter.

The mistake this prevents

The mistake is reporting a significant result from a very large sample as a finding. At n = 50,000 significance is nearly automatic and tells you almost nothing.

Takeaway

Set the smallest worthwhile effect before the analysis and compare the interval to it. Say explicitly when a significant result is too small to act on — that is a finding, not a failure.

Mark complete