Unit 07.00: Report the difference in real units

The best effect size is usually the one that needs no explaining: the difference, in the units the reader already uses.

Report the difference in real units

The raw mean difference is directly interpretable. Milliseconds, pounds, points on a scale the reader knows — no translation, no conventions, no argument about whether 0.3 counts as small.

Its interval carries the uncertainty in the same units, so a decision can be made against a threshold set in those units.

Standardised measures have their place when scales differ across studies, but they should accompany the raw difference rather than replace it.

This block compares page load times between two versions.

import numpy as np
from scipy import stats

rng = np.random.default_rng(401)
control = rng.normal(240, 30, 60)     # page load, milliseconds
treated = rng.normal(228, 30, 60)

res = stats.ttest_ind(treated, control, equal_var=False)
ci = res.confidence_interval()
diff = treated.mean() - control.mean()

print(f"Control mean : {control.mean():.1f} ms")
print(f"Treated mean : {treated.mean():.1f} ms")
print(f"Difference   : {diff:+.1f} ms")
print(f"95% CI       : [{ci.low:.1f}, {ci.high:.1f}] ms")
print(f"p            : {res.pvalue:.4f}\n")

print("The mean difference is the effect size that needs no explaining: it is")
print("in units the reader already understands.")
print(f"As a percentage of the control mean: {diff / control.mean():+.1%}")
print(f"\nThe interval spans {ci.high - ci.low:.1f} ms, so the data is consistent")
print("with both a barely perceptible saving and a clearly worthwhile one.")
print("That range, not the point estimate, is what a decision turns on.")

The treated version loads at 221.9 ms against the control's 239.3 — a difference of 17.3 ms, or 7.2% of the control mean, with p = 0.0013. The interval runs from 6.9 to 27.8 ms and spans 20.9 ms; it is the part a decision turns on, because the data is consistent with a barely perceptible saving and with a clearly worthwhile one.

The mistake this prevents

The mistake is reporting only a standardised effect size. 'd = 0.5' requires the reader to know your scale's standard deviation to recover anything actionable.

Takeaway

Report the mean difference in its natural units with its interval, and give the percentage change when the base is meaningful. Add a standardised measure only when comparing across different scales.

Mark complete