Unit 07.00: Report the difference in real units

The best effect size is usually the one that needs no explaining: the difference, in the units the reader already uses.

Report the difference in real units

The raw mean difference is directly interpretable. Milliseconds, pounds, points on a scale the reader knows — no translation, no conventions, no argument about whether 0.3 counts as small.

Its interval carries the uncertainty in the same units, so a decision can be made against a threshold that was set in those units.

Standardised measures have their place when scales differ across studies, but they should accompany the raw difference rather than replace it.

This block compares page load times between two versions.

set.seed(401)
control <- rnorm(60, mean = 240, sd = 30)   # page load, milliseconds
treated <- rnorm(60, mean = 228, sd = 30)

tt <- t.test(treated, control)
difference <- as.numeric(diff(rev(tt$estimate)))

cat("Control mean:", round(mean(control), 1), "ms\n")
cat("Treated mean:", round(mean(treated), 1), "ms\n")
cat("Difference  :", round(difference, 1), "ms\n")
cat("95% CI      : [", round(tt$conf.int[1], 1), ",",
    round(tt$conf.int[2], 1), "] ms\n")
cat("p           :", signif(tt$p.value, 3), "\n\n")

cat("The mean difference is the effect size that needs no explaining: it is\n")
cat("in the units the reader already understands.\n")
cat("As a percentage of the control mean:",
    round(difference / mean(control) * 100, 1), "%\n\n")
cat("The interval is what the decision turns on. It spans",
    round(diff(tt$conf.int), 1), "ms, so the data is consistent with\n")
cat("both a barely noticeable change and a clearly useful one.\n")

The treated version loads at 224.3 ms against the control's 239.4 — a difference of 15.1 ms, or 6.3% of the control mean, with p = 0.0055. The interval runs from 4.5 to 25.7 ms and is the part a decision turns on: the data is consistent with a barely perceptible saving and with a clearly worthwhile one, and it spans 21.1 ms.

The mistake this prevents

The mistake is reporting only a standardised effect size. 'd = 0.5' requires the reader to know your scale's standard deviation to recover anything actionable.

Takeaway

Report the mean difference in its natural units with its interval, and give the percentage change as well when the base is meaningful. Add a standardised measure only when comparing across different scales.

Mark complete