Unit 07.04: Width is the information content

Two studies can both reject the null and only one of them tell you anything about the size of the effect.

Width is the information content

A confidence interval on an effect answers the question a decision actually needs: what range of effect sizes is the data consistent with? A significant result whose interval runs from trivial to enormous has established that something is there and nothing about how much.

The useful discipline is to name the smallest effect that would matter before the analysis, then compare the interval to it. Four outcomes follow: the interval is entirely above it, entirely below it, contains it and zero, or straddles it.

Only the third of those is genuinely 'inconclusive', and it is a different finding from 'no effect'.

This block runs a small study and a large one on the same underlying effect.

set.seed(405)

# Two studies, same point estimate, very different information.
small <- t.test(rnorm(15, 106, 15), rnorm(15, 100, 15))
large <- t.test(rnorm(600, 103, 15), rnorm(600, 100, 15))

for (nm in c("small", "large")) {
  tt <- get(nm)
  cat(sprintf("%-6s difference %5.2f  CI [%6.2f, %5.2f]  width %5.2f  p = %.4f\n",
              nm, as.numeric(diff(rev(tt$estimate))),
              tt$conf.int[1], tt$conf.int[2], diff(tt$conf.int), tt$p.value))
}

cat("\nBoth studies reject the null. Only one of them tells you the size of\n")
cat("the effect: the small study's interval is", round(diff(small$conf.int) /
    diff(large$conf.int), 1), "times wider and spans everything\n")
cat("from a trivial difference to a large one.\n\n")

cat("Ask of every interval: what is the smallest effect that would matter?\n")
cat("An interval containing both that value and zero means the study is\n")
cat("inconclusive -- a different finding from 'no effect'.\n")

Both reject the null. The small study reports 9.67 with an interval from 1.09 to 18.25 and p = 0.0286; the large one reports 2.62 with an interval from 0.96 to 4.27 and p = 0.0020. The small study's interval is 5.2 times wider and spans everything from a trivial difference to a large one — it has demonstrated existence and nothing else.

The mistake this prevents

The mistake is reading a large point estimate from a small study as evidence of a large effect. Small studies that reach significance systematically overstate the effect, because only the larger estimates clear the threshold.

Takeaway

Name the smallest effect that would matter before analysing, and compare the whole interval to it. Report interval width, and be sceptical of large estimates from small samples.

Mark complete