Unit 12.03: Four possible verdicts, not two

There are four possible verdicts, not two, and this result is the one people report wrongly.

Compare the whole interval to the bar you set

With a pre-specified minimum worth acting on, comparing the confidence interval to it gives four outcomes. The interval lies entirely above the bar: act. Entirely below it while excluding zero: a real effect, too small to matter. Containing zero and the bar: inconclusive. Excluding zero but straddling the bar: a real effect of uncertain importance.

Only the first two support a decision. The last two are requests for more data, and both are routinely reported as successes because the p-value was below 0.05.

Naming the verdict explicitly forces the sample-size conversation to happen, which is where it belongs.

This block compares a result against a pre-specified 2 percentage point bar.

import numpy as np
from statsmodels.stats.proportion import (proportions_ztest,
                                          confint_proportions_2indep)

rng = np.random.default_rng(903)
n_arm = 5000
a = rng.binomial(n_arm, 0.300)
b = rng.binomial(n_arm, 0.313)
_, p = proportions_ztest([b, a], [n_arm, n_arm])
lo, hi = confint_proportions_2indep(b, n_arm, a, n_arm,
                                    compare="diff", method="wald")

diff_pp = (b - a) / n_arm * 100
MINIMUM = 2.0     # pre-specified in the analysis plan

print(f"Retention A: {a / n_arm:.4f}   B: {b / n_arm:.4f}")
print(f"Difference : {diff_pp:+.2f} pp")
print(f"95% CI     : [{lo * 100:+.2f}, {hi * 100:+.2f}] pp")
print(f"p          : {p:.4f}\n")

print(f"Pre-specified minimum worth launching : {MINIMUM} pp")
print(f"Distinguishable from zero             : {p < 0.05}")
print(f"Interval entirely above the minimum   : {lo * 100 > MINIMUM}")
print(f"Interval contains the minimum         : {lo * 100 < MINIMUM < hi * 100}\n")

lo_pp, hi_pp = lo * 100, hi * 100
if lo_pp > MINIMUM:
    verdict = "launch: the whole interval clears the bar"
elif hi_pp < MINIMUM and lo_pp > 0:
    verdict = "a real effect, but too small to act on"
elif lo_pp < 0 < MINIMUM < hi_pp:
    verdict = ("inconclusive: the interval contains both zero and the bar, so"
               " the data cannot tell them apart")
else:
    verdict = ("a real effect of uncertain importance: the interval excludes"
               " zero but straddles the bar")
print(f"Verdict: {verdict}\n")
print("Four outcomes are possible, not two. Only the first two support a")
print("decision; the others are a request for more data.")

The difference is +1.86 percentage points with an interval from +0.06 to +3.66 and p = 0.0429. Distinguishable from zero: True. Entirely above the 2pp bar: False. Containing the bar: True. The verdict is therefore *a real effect of uncertain importance* — the interval excludes zero but straddles the threshold, so the data cannot say whether the effect clears the bar.

The mistake this prevents

The mistake is reading p = 0.043 as 'we should launch'. Significance says the effect is not zero; it says nothing about whether it clears the bar you set, and here the interval covers both sides of it.

Takeaway

Set the practical bar in advance and compare the whole interval to it as well as to zero. Name which of the four verdicts applies, and say 'inconclusive' or 'uncertain importance' when that is what it is.

Mark complete