Unit 03.03: Overlapping error bars are not a test

Two error bars that overlap do not mean there is no difference. This is the most common misreading of a chart in applied statistics.

Read the interval around the difference

Comparing two groups by eye from their individual confidence intervals is not a valid test. The interval that answers 'do these groups differ' is the interval around the difference, and it is narrower than the two individual intervals would suggest, because the difference has a smaller standard error than the sum of the two separate uncertainties implies.

So overlapping intervals are entirely compatible with a clearly significant difference. The reverse — non-overlapping intervals — does imply significance, which is why the error is asymmetric and easy to miss.

There is a second problem with error bars: unlabelled, they could be a standard deviation, a standard error or a 95% interval, and those are three very different lengths from the same data.

This block computes both group intervals and the difference interval.

suppressPackageStartupMessages(library(dplyr))

# Constructed so the two group intervals overlap and the difference is still
# significant -- the case that makes the eyeball test fail.
set.seed(41)
a <- scale(rnorm(30))[, 1] * 8 + 50.0
b <- scale(rnorm(30))[, 1] * 8 + 55.5
d <- data.frame(group = rep(c("A", "B"), each = 30), value = c(a, b))

stats <- d |> group_by(group) |>
  summarise(n = n(), mean = mean(value), sd = sd(value),
            se = sd(value) / sqrt(n()),
            ci_lo = mean(value) - qt(0.975, n() - 1) * se,
            ci_hi = mean(value) + qt(0.975, n() - 1) * se,
            .groups = "drop")

for (i in 1:2) {
  cat(sprintf("%s  mean %.1f  SD %.1f  SE %.2f  95%% CI [%.1f, %.1f]\n",
              stats$group[i], stats$mean[i], stats$sd[i], stats$se[i],
              stats$ci_lo[i], stats$ci_hi[i]))
}

overlap <- stats$ci_hi[1] > stats$ci_lo[2]
cat("\nDo the two 95% intervals overlap?", overlap, "\n")
tt <- t.test(value ~ group, data = d)
cat("Two-sample t-test p-value       :", signif(tt$p.value, 3), "\n")
cat("Difference CI                   : [",
    round(tt$conf.int[1], 2), ",", round(tt$conf.int[2], 2), "]\n\n")

cat("Overlapping intervals do NOT imply no difference. The interval you must\n")
cat("read is the one around the DIFFERENCE, and here it excludes zero.\n")
cat("Always label an error bar: SD, SE and 95% CI are three different lengths.\n")

Group A's interval is [47.0, 53.0] and group B's is [52.5, 58.5] — they overlap. The two-sample test gives p = 0.01, and the interval around the difference is [-9.63, -1.37], which excludes zero. Judging by eye from the overlapping bars would have reached the opposite conclusion.

The mistake this prevents

The mistake is 'the error bars overlap, so there is no difference'. It is stated confidently in meetings and it is simply not what overlapping intervals mean.

Takeaway

Test the difference rather than eyeballing two intervals, and report the interval around the difference. Label every error bar with what it is — SD, SE or 95% CI.

Mark complete