Unit 07.05: Two thresholds, one set by you

With enough data, everything is significant. Whether anything is worth doing is a separate question that statistics cannot answer.

Two thresholds, one set by you

Statistical significance says the effect is distinguishable from zero. Practical significance says it is large enough to act on. The first is computed; the second is a judgement about costs and benefits that has to be made by someone who knows the domain.

Because the standard error shrinks with sample size, a large enough study will find a statistically significant difference for essentially any non-zero effect. At that point the p-value stops carrying information and the interval carries all of it.

Setting the practical threshold in advance is what keeps this honest. Chosen afterwards, it will always sit just below whatever was observed.

This block uses fifty thousand observations per group and a tiny true effect.

set.seed(406)

# A trivially small effect, made 'significant' by sample size alone.
a <- rnorm(50000, mean = 100.0, sd = 15)
b <- rnorm(50000, mean = 100.3, sd = 15)
tt <- t.test(b, a)

cat("n per group      :", length(a), "\n")
cat("Difference       :", round(as.numeric(diff(rev(tt$estimate))), 3), "points\n")
cat("95% CI           : [", round(tt$conf.int[1], 3), ",",
    round(tt$conf.int[2], 3), "]\n")
cat("p                :", signif(tt$p.value, 3), "\n")
cat("Cohen's d        :", round(as.numeric(diff(rev(tt$estimate))) / 15, 3), "\n\n")

MEANINGFUL <- 2.0    # decided in advance: below this, nobody would act
cat("Smallest difference worth acting on:", MEANINGFUL, "points\n")
cat("Whole interval below that threshold:",
    tt$conf.int[2] < MEANINGFUL, "\n\n")

cat("So the result is statistically significant and practically irrelevant.\n")
cat("The data rules out a meaningful effect rather than demonstrating one.\n")
cat("Decide the threshold before the analysis, or the effect will always\n")
cat("turn out to be just above whatever you needed.\n")

The difference is 0.348 points with an interval from 0.163 to 0.533 and p = 0.000234 — unambiguously significant, with Cohen's d of 0.023. Against a pre-set threshold of 2 points, the *entire* interval falls below it. The correct conclusion is not 'a significant improvement' but the opposite: the study has ruled out an effect large enough to matter.

The mistake this prevents

The mistake is reporting a significant result from a very large sample as a finding. At n = 50,000 significance is nearly automatic and tells you almost nothing.

Takeaway

Set the smallest worthwhile effect before the analysis and compare the interval to it. Say explicitly when a significant result is too small to act on — that is a finding, not a failure.

Mark complete