Unit 07.04: Width is the information content

Two studies can both reject the null and only one of them tell you anything about the size of the effect.

Width is the information content

A confidence interval on an effect answers the question a decision needs: what range of effect sizes is the data consistent with? A significant result whose interval runs from trivial to enormous has established that something is there and nothing about how much.

The useful discipline is to name the smallest effect that would matter before the analysis, then compare the interval to it. Four outcomes follow: the interval is entirely above it, entirely below it, contains it and zero, or straddles it.

Only the third is genuinely 'inconclusive', and it is a different finding from 'no effect'.

This block runs a small study and a large one, both constructed to reach significance.

import numpy as np
from scipy import stats

rng = np.random.default_rng(405)

def shaped(mean, sd, n):
    """Draw n values, then force exactly this mean and sample sd, so the
    comparison below is a property of the design rather than of the draw."""
    x = rng.normal(size=n)
    x = (x - x.mean()) / x.std(ddof=1)
    return x * sd + mean

studies = {
    "small": (shaped(100, 15, 15),  shaped(112, 15, 15)),
    "large": (shaped(100, 15, 600), shaped(103, 15, 600)),
}
widths = {}
for name, (a, b) in studies.items():
    res = stats.ttest_ind(b, a, equal_var=False)
    ci = res.confidence_interval()
    widths[name] = ci.high - ci.low
    print(f"{name:6s} difference {b.mean() - a.mean():5.2f}"
          f"   CI [{ci.low:6.2f}, {ci.high:5.2f}]"
          f"   width {ci.high - ci.low:5.2f}   p = {res.pvalue:.4f}")

print(f"\nThe small study's interval is {widths['small'] / widths['large']:.1f}"
      " times wider and spans everything from a")
print("trivial difference to a large one. Both studies reject the null; only")
print("one of them tells you the size of the effect.\n")
print("Ask of every interval: what is the smallest effect that would matter?")
print("An interval containing both that value and zero means the study is")
print("inconclusive -- a different finding from 'no effect'.")

The small study reports 12.00 with an interval from 0.78 to 23.22 and p = 0.0369; the large one reports 3.00 with an interval from 1.30 to 4.70 and p = 0.0006. Both reject the null. The small study's interval is 6.6 times wider and spans everything from a trivial difference to a large one — it has demonstrated existence and nothing else.

The mistake this prevents

The mistake is reading a large point estimate from a small study as evidence of a large effect. Small studies that reach significance systematically overstate the effect, because only the larger estimates clear the threshold.

Takeaway

Name the smallest effect that would matter before analysing, and compare the whole interval to it. Report interval width, and be sceptical of large estimates from small samples.

Mark complete