Unit 12.03: Four possible verdicts, not two

There are four possible verdicts, not two, and 'inconclusive' is one of them.

Compare the whole interval to the bar you set

With a pre-specified minimum worth acting on, comparing the confidence interval to it gives four outcomes. The interval lies entirely above the bar: act. Entirely below it, and excluding zero: a real effect, too small to matter. Containing zero and the bar: inconclusive — the study cannot distinguish no effect from a worthwhile one. Straddling the bar but excluding zero: a real effect of uncertain importance.

Only the first two support a decision. The third is a request for more data and is routinely misreported as either a success or a failure.

Reporting the verdict this way forces the sample-size conversation to happen, which is where it belongs.

This block compares a result against a pre-specified 2 percentage point bar.

set.seed(903)
n_arm <- 5000
a <- rbinom(n_arm, 1, 0.300)
b <- rbinom(n_arm, 1, 0.313)
pt <- prop.test(c(sum(a), sum(b)), c(n_arm, n_arm))

diff_pp <- (mean(b) - mean(a)) * 100
ci <- c(-pt$conf.int[2], -pt$conf.int[1]) * 100
MINIMUM <- 2.0     # pre-specified in the analysis plan

cat("Retention A:", round(mean(a), 4), "  B:", round(mean(b), 4), "\n")
cat("Difference :", round(diff_pp, 2), "pp\n")
cat("95% CI     : [", round(ci[1], 2), ",", round(ci[2], 2), "] pp\n")
cat("p          :", signif(pt$p.value, 3), "\n\n")

cat("Pre-specified minimum worth launching:", MINIMUM, "pp\n")
cat("Statistically distinguishable from zero:", pt$p.value < 0.05, "\n")
cat("Interval entirely above the minimum    :", ci[1] > MINIMUM, "\n")
cat("Interval contains the minimum          :", ci[1] < MINIMUM & ci[2] > MINIMUM, "\n\n")

cat("The interval contains BOTH zero and the", MINIMUM,
    "pp bar. So the data cannot\n")
cat("distinguish 'no effect' from 'an effect worth launching'. The honest\n")
cat("sentence is that the test was inconclusive at this sample size -- which\n")
cat("is a request for more data, not a launch and not a rejection.\n")

The difference is 1.24 percentage points with an interval from −0.59 to 3.07 and p = 0.187. Statistically distinguishable from zero: FALSE. Entirely above the 2pp bar: FALSE. Containing the bar: TRUE. The interval covers both zero and the threshold, so the study cannot separate 'no effect' from 'an effect worth launching'. The correct report is neither a launch nor a rejection but a statement that the test was inconclusive at this sample size.

The mistake this prevents

The mistake is collapsing this to 'not significant, so no effect'. The data here is entirely consistent with an effect well above the launch threshold.

Takeaway

Set the practical bar in advance and compare the whole interval to it as well as to zero. Name the verdict explicitly, and say 'inconclusive' when that is what it is.

Mark complete