Unit 03.03: Overlapping error bars are not a test

Two error bars that overlap do not mean there is no difference. This is the most common misreading of a chart in applied statistics.

Read the interval around the difference

Comparing two groups by eye from their individual confidence intervals is not a valid test. The interval that answers 'do these groups differ' is the interval around the difference, and it is narrower than the two separate intervals suggest.

So overlapping intervals are entirely compatible with a clearly significant difference. The converse — non-overlapping intervals — does imply significance, which is why the error is asymmetric and easy to miss.

There is a second problem: an unlabelled error bar could be a standard deviation, a standard error or a 95% interval, and those are three very different lengths from the same data.

This block builds two groups whose intervals overlap, then tests the difference.

import numpy as np
from scipy import stats

rng = np.random.default_rng(41)
# Constructed so the two group intervals overlap and the difference is still
# significant -- the case where judging by eye fails.
def shaped(mean, sd, n):
    """Draw n values, then force exactly this mean and sample sd."""
    x = rng.normal(size=n)
    x = (x - x.mean()) / x.std(ddof=1)
    return x * sd + mean

a = shaped(35.0, 8.0, 30)
b = shaped(40.0, 8.0, 30)

for name, x in [("A", a), ("B", b)]:
    se = x.std(ddof=1) / np.sqrt(x.size)
    lo, hi = stats.t.interval(0.95, x.size - 1, loc=x.mean(), scale=se)
    print(f"{name}  mean {x.mean():5.1f}  SD {x.std(ddof=1):4.1f}"
          f"  SE {se:4.2f}  95% CI [{lo:.1f}, {hi:.1f}]")

lo_a, hi_a = stats.t.interval(0.95, 29, loc=a.mean(),
                              scale=a.std(ddof=1) / np.sqrt(30))
lo_b, hi_b = stats.t.interval(0.95, 29, loc=b.mean(),
                              scale=b.std(ddof=1) / np.sqrt(30))
res = stats.ttest_ind(b, a, equal_var=False)
d_ci = res.confidence_interval()

print(f"\nDo the two intervals overlap? {hi_a > lo_b}")
print(f"Two-sample p-value            : {res.pvalue:.4f}")
print(f"CI for the DIFFERENCE         : [{d_ci.low:.2f}, {d_ci.high:.2f}]")
print(f"Difference interval excludes 0: {d_ci.low * d_ci.high > 0}")
print("\nOverlapping intervals do not mean no difference. Read the interval")
print("around the difference. And label every error bar: SD, SE and 95% CI")
print("are three different lengths from the same data.")

Group A's interval is [32.0, 38.0] and group B's is [37.0, 43.0] — they overlap. The two-sample test gives p = 0.0186 and the interval around the difference is [0.87, 9.13], which excludes zero. Judging by eye from the overlapping bars would have reached the opposite conclusion. Note also that the SE is 1.46 while the SD is 8.0 — error bars drawn from those two would look nothing alike.

The mistake this prevents

The mistake is 'the error bars overlap, so there is no difference'. It is stated confidently in meetings and it is not what overlapping intervals mean.

Takeaway

Test the difference rather than eyeballing two intervals, and report the interval around the difference. Label every error bar with what it is.

Mark complete