Unit 07.02: Four honest ways to state a rate difference

For a binary outcome the effect size is a difference in rates, and there are four honest ways to state it.

Percentage points, relative change, and what it means in practice

The absolute difference in percentage points determines how many extra events actually happen. The relative difference is the percentage change and is what gets quoted. Both describe the same result.

Two derived figures are often more useful than either. Extra events per 1,000 puts the difference on a scale people can picture. The number of units needed for one extra event is the most direct translation into effort.

All four should carry the interval, because the uncertainty in a rate difference is usually larger than people expect.

This block reports one conversion result four ways.

import numpy as np
from statsmodels.stats.proportion import confint_proportions_2indep

converted = np.array([96, 132])          # control, treated
n = np.array([1200, 1200])
rates = converted / n

abs_diff = rates[1] - rates[0]
rel_diff = abs_diff / rates[0]
lo, hi = confint_proportions_2indep(converted[1], n[1], converted[0], n[0],
                                    compare="diff", method="wald")

print(f"Control rate: {rates[0]:.3f}   Treated rate: {rates[1]:.3f}\n")
print(f"Absolute difference : {abs_diff * 100:+.2f} percentage points")
print(f"95% CI              : [{lo * 100:+.2f}, {hi * 100:+.2f}] pp")
print(f"Relative difference : {rel_diff:+.1%}\n")

print("Both describe the same result, and they sound very different:")
print(f"  'conversion rose by {abs_diff * 100:.0f} percentage points'")
print(f"  'conversion rose by {rel_diff:.1%}'\n")
print(f"Extra conversions per 1,000 visitors : {abs_diff * 1000:.0f}")
print(f"Visitors needed for one extra        : {1 / abs_diff:.0f}")

Rates of 0.080 and 0.110 give an absolute difference of +3.00 percentage points with an interval from +0.66 to +5.34, and a relative difference of +37.5%. The same result yields 30 extra conversions per 1,000 visitors, and 33 visitors needed for one extra conversion.

The mistake this prevents

The mistake is quoting the relative change without the base rate. A 37.5% improvement sounds transformative; three percentage points sounds marginal; they are the same finding.

Takeaway

Report the absolute difference with its interval, the relative change, and at least one practical translation such as events per 1,000 or units needed per extra event.

Mark complete