Unit 05.07: Small p, small effect

A p-value measures evidence against the null. It says nothing whatever about how big the effect is.

Small p, small effect

Because the p-value depends on both effect size and sample size, a large study can produce an extremely small p-value from an effect nobody would act on, and a small study can miss an important effect entirely.

That is why 'highly significant' is not a meaningful phrase. It sounds like a statement about magnitude and is a statement about sample size.

Three other habits to drop: calling p = 0.06 'a trend' — the threshold was fixed in advance, so this is arguing with your own rule; reporting a non-significant result as 'no effect'; and quoting p-values to six decimal places, which implies precision the estimate does not have.

This block runs two studies with very different sample sizes.

set.seed(208)

# Two studies. Same true effect. Different sample sizes.
small <- list(a = rnorm(20, 100, 15),    b = rnorm(20, 106, 15))
large <- list(a = rnorm(10000, 100, 15), b = rnorm(10000, 100.7, 15))

for (nm in c("small", "large")) {
  s <- get(nm)
  tt <- t.test(s$b, s$a)
  cat(sprintf("%-5s n=%4d per group  difference %5.2f  p = %.4f  CI [%.2f, %.2f]\n",
              nm, length(s$a), as.numeric(diff(rev(tt$estimate))),
              tt$p.value, tt$conf.int[1], tt$conf.int[2]))
}

cat("\nThe large study finds a smaller difference with a smaller p-value.\n")
cat("A p-value measures evidence against the null, not the size of an effect.\n\n")

cat("Say this          : 'a difference of X (95% CI a to b, p = ...)'\n")
cat("Not this          : 'highly significant'  -- p is not a magnitude\n")
cat("Not this          : 'p = 0.06, a trend'   -- the threshold was set in advance\n")
cat("Not this          : 'no effect'           -- absence of evidence is not evidence\n")
cat("Report p to 3 significant figures, or as p < 0.001 when smaller.\n")

The small study finds a difference of 6.21 with p = 0.2639 and an interval from −4.90 to 17.32 — inconclusive. The large study finds a difference of 0.92, less than a seventh the size, with p below 0.0001 and an interval from 0.50 to 1.34. The smaller effect has the far smaller p-value, purely because n was 500 times larger.

The mistake this prevents

The mistake is ranking findings by p-value. The strongest evidence in a results table is frequently attached to the least important effect.

Takeaway

Report the estimate, its interval and the p-value together, to three significant figures or as p < 0.001. Never use the p-value as a measure of how large or how important an effect is.

Mark complete