Unit 07.01: A difference divided by the spread

Cohen's d expresses a difference in standard deviations, which makes it comparable across measures and unreadable on its own.

A difference divided by the spread

Cohen's d is the difference between two means divided by their pooled standard deviation. It answers 'how big is this difference relative to how much the values vary', which lets a result on one scale be compared with a result on another.

The price is that d responds to the spread as much as to the difference. Two studies with the same raw gap report very different d values simply because one measured a more homogeneous group.

The conventional labels — 0.2 small, 0.5 medium, 0.8 large — are a last resort. They were offered as rough placeholders, and in any field where you know what the numbers mean the raw difference is better.

This block computes d, then computes it again for the same size of gap on a much less variable population.

import numpy as np

def cohens_d(x, y):
    nx, ny = len(x), len(y)
    pooled = np.sqrt(((nx - 1) * x.var(ddof=1) + (ny - 1) * y.var(ddof=1))
                     / (nx + ny - 2))
    return (x.mean() - y.mean()) / pooled, pooled

rng = np.random.default_rng(402)
a = rng.normal(100, 15, 50)
b = rng.normal(107, 15, 50)
d_wide, pooled_wide = cohens_d(b, a)

print(f"Raw difference : {b.mean() - a.mean():.2f} points")
print(f"Pooled SD      : {pooled_wide:.2f}")
print(f"Cohen's d      : {d_wide:.3f}\n")
print(f"d says the groups differ by {d_wide:.2f} standard deviations.")
print("The conventions -- 0.2 small, 0.5 medium, 0.8 large -- are a last")
print("resort, not a substitute for knowing your own field's scale.\n")

# The same 7-point gap measured on a much less variable population.
a_tight = rng.normal(100, 5, 50)
b_tight = rng.normal(107, 5, 50)
d_tight, pooled_tight = cohens_d(b_tight, a_tight)

print("The same kind of gap, measured on two different populations:")
print(f"  spread ~15: raw difference {b.mean() - a.mean():5.2f}"
      f"   pooled SD {pooled_wide:5.2f}  -> d = {d_wide:.2f}")
print(f"  spread ~5 : raw difference {b_tight.mean() - a_tight.mean():5.2f}"
      f"   pooled SD {pooled_tight:5.2f}  -> d = {d_tight:.2f}")
print(f"  the raw gaps differ by a factor of"
      f" {(b_tight.mean() - a_tight.mean()) / (b.mean() - a.mean()):.2f},"
      f" the d values by {d_tight / d_wide:.1f}")
print("\nd responds to the spread as much as to the gap. That is what makes it")
print("comparable across measures and unreadable on its own -- report the raw")
print("difference too.")

A raw difference of 7.37 points against a pooled SD of 16.90 gives d = 0.436. The same kind of gap — 7.55 points — measured on a population with a pooled SD of 5.21 gives d = 1.45. The raw gaps differ by a factor of 1.02 and the d values by a factor of 3.3. Everything but the spread was held constant.

The mistake this prevents

The mistake is quoting d as though it were a property of the treatment. It is a property of the treatment *and* the population's variability, so it moves when you change who you study.

Takeaway

Report the raw difference first and d alongside it when comparing across scales. Treat the small/medium/large labels as a last resort, and state the pooled SD so a reader can convert back.

Mark complete