Unit 06.04: Small samples and the error bar problem

Four categories with four different sample sizes are not four comparable measurements.

The interval shrinks with the square root of n

The same population sampled at four sizes.

The code reports the interval at each.

import numpy as np

rng = np.random.default_rng(4)
print(f"{'n':>5} {'mean':>8} {'std err':>9} {'95% interval':>20}")
for n in (5, 20, 100, 500):
    sample = rng.normal(100, 15, n)
    se = sample.std(ddof=1) / np.sqrt(n)
    print(f"{n:>5} {sample.mean():>8.1f} {se:>9.2f} "
          f"{f'{sample.mean() - 1.96 * se:.1f} to {sample.mean() + 1.96 * se:.1f}':>20}")

print("""
At n=5 the interval spans 20 points; at n=500 it spans 3. A chart showing four
bars with no error bars presents all four as equally certain.

If you cannot show the interval, show the sample size next to each category.
A reader who knows one bar is five observations will discount it themselves.
""")

At five observations the interval spans about twenty points; at five hundred, about three. A chart with four bars and no intervals presents all four as equally certain.

The category with the fewest observations is usually the most interesting - a new region, a new product - which is exactly where the estimate is least stable.

The mistake this prevents

The mistake is dropping small categories to avoid the problem. That removes the information rather than qualifying it. Show the count, or the interval, and let the reader discount it themselves.

Takeaway

Show the sample size or the interval next to every category. Small categories carry wide uncertainty and are often the ones people act on.

Mark complete