Unit 07.02: Four honest ways to state a rate difference

For a binary outcome the effect size is a difference in rates, and there are four honest ways to state it.

Percentage points, relative change, and what it means in practice

The absolute difference in percentage points determines how many extra events actually happen. The relative difference is the percentage change and is what gets quoted. Both describe the same result.

Two derived figures are often more useful than either. Extra events per 1000 puts the difference on a scale people can picture. The number needed to treat — how many units you need for one extra event — is the most direct translation into effort.

All four should carry the interval, because the uncertainty in a rate difference is usually larger than people expect.

This block reports one conversion result four ways.

converted <- c(control = 96, treated = 132)
n         <- c(control = 1200, treated = 1200)
rates <- converted / n

cat("Control rate:", rates["control"], "  Treated rate:", rates["treated"], "\n\n")

abs_diff <- as.numeric(rates["treated"] - rates["control"])
rel_diff <- abs_diff / as.numeric(rates["control"])

test <- prop.test(converted, n)
cat("Absolute difference (percentage points):", round(abs_diff * 100, 2), "\n")
cat("95% CI                                 : [",
    round(-test$conf.int[2] * 100, 2), ",",
    round(-test$conf.int[1] * 100, 2), "] pp\n")
cat("Relative difference                    :", round(rel_diff * 100, 1), "%\n\n")

cat("Both describe the same result. They sound very different:\n")
cat("  'conversion rose by 3 percentage points'\n")
cat("  'conversion rose by 37.5 per cent'\n\n")
cat("Extra conversions per 1000 visitors:", round(abs_diff * 1000, 1), "\n")
cat("Number needed to treat (visitors per extra conversion):",
    round(1 / abs_diff, 1), "\n")

Rates of 0.08 and 0.11 give an absolute difference of 3 percentage points with an interval from 0.57 to 5.43 points — and a relative difference of 37.5%. The same result yields 30 extra conversions per 1000 visitors, and a number needed to treat of 33.3: one extra conversion for every 33 visitors sent through the new version.

The mistake this prevents

The mistake is quoting the relative change without the base rate. A 37.5% improvement sounds transformative; three percentage points sounds marginal; they are the same finding.

Takeaway

Report the absolute difference with its interval, the relative change, and at least one practical translation such as events per 1000 or number needed to treat.

Mark complete