Unit 07.06: Natural frequencies are understood most reliably

The same result stated four ways will produce four different decisions. Choosing the framing is part of the analysis.

Natural frequencies are understood most reliably

Percentage points are accurate and feel small. Relative change is persuasive and routinely misread. Natural frequencies — out of every 1,000, this many before and this many after — are understood most reliably by non-specialists and are hardest to misinterpret. Business units connect the result to a decision.

None of these is the honest one and the others dishonest. The dishonest move is picking whichever framing best supports the conclusion you already wanted, and giving only that one.

The part almost always dropped is the uncertainty, and it is the part that changes what a reader should do.

This block states one conversion result four ways, then attaches the interval.

import numpy as np
from statsmodels.stats.proportion import confint_proportions_2indep

converted, n = np.array([96, 132]), np.array([1200, 1200])
rates = converted / n
abs_diff = rates[1] - rates[0]
lo, hi = confint_proportions_2indep(converted[1], n[1], converted[0], n[0],
                                    compare="diff", method="wald")
MONTHLY_VISITORS = 10_000

print("--- the same result, four ways ---\n")
print(f"1. Percentage points : {abs_diff * 100:.1f} pp increase")
print(f"2. Relative          : {abs_diff / rates[0]:.0%} increase")
print(f"3. Natural frequency : out of every 1,000 visitors,"
      f" {rates[0] * 1000:.0f} converted before and {rates[1] * 1000:.0f} after")
print(f"4. Business units    : at {MONTHLY_VISITORS:,} visitors a month, about"
      f" {abs_diff * MONTHLY_VISITORS:.0f} extra conversions\n")

print("With the uncertainty attached -- the part usually dropped:")
print(f"   between {lo * MONTHLY_VISITORS:.0f} and {hi * MONTHLY_VISITORS:.0f}"
      " extra conversions a month\n")
print("Natural frequencies are understood most reliably. Relative change is")
print("the most persuasive and the most misleading -- pair it with an")
print("absolute figure, and carry the interval into the decision's units.")

The same finding is 3.0 percentage points, a 38% increase, a move from 80 to 110 conversions per 1,000 visitors, and about 300 extra conversions a month at 10,000 visitors. With the interval attached, that last figure becomes between 66 and 534 extra conversions a month — a range wide enough that a plan built on 300 may not survive.

The mistake this prevents

The mistake is presenting only the relative change because it is the most impressive. Pairing it with an absolute figure is the minimum, and carrying the interval into the business units is what makes the number usable.

Takeaway

Give at least two framings, always including an absolute one, and prefer natural frequencies for non-technical readers. Carry the confidence interval into whatever units the decision is made in.

Mark complete