Unit 11.04: What could this test have seen?

Power is a question you answer before the data exists. Afterwards it is too late to be useful.

What could this test have seen?

Power is the probability of detecting an effect of a given size if it is really there. The minimum detectable effect is the smallest effect the test can reliably find at a given sample size.

statsmodels.stats.power computes both. NormalIndPower().power(...) gives the power at a sample size, and solve_power(power=0.8, ...) gives the sample size for a target power — with the effect expressed by proportion_effectsize for a binary outcome.

Running the calculation before the test tells you how much data you need. Skipping it means a non-significant result is uninterpretable: you cannot tell whether there was no effect or whether the test could never have seen one.

This block computes detection rates across three sample sizes and three effect sizes, then solves for the sample size.

import numpy as np
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

power_calc = NormalIndPower()
BASE = 0.30

print("Base rate 30%. Chance of detecting each effect at alpha 0.05:\n")
print(f"{'n per arm':>10s}{'+1pp':>10s}{'+2pp':>10s}{'+5pp':>10s}")
for n in (1000, 4000, 16000):
    row = [f"{n:10d}"]
    for lift in (0.01, 0.02, 0.05):
        es = proportion_effectsize(BASE + lift, BASE)
        row.append(f"{power_calc.power(es, n, 0.05):9.0%}")
    print("".join(row))

print("\nRead it the useful way round: with 4,000 per arm you will usually")
print("miss a 1-point effect and usually catch a 2-point one.\n")

need = power_calc.solve_power(proportion_effectsize(BASE + 0.02, BASE),
                              power=0.8, alpha=0.05)
print(f"For 80% power on a 2pp lift you need {np.ceil(need):.0f} per arm.")
print("The minimum detectable effect is what the test can see. Deciding it in")
print("advance is what stops an underpowered test being read as 'no effect'.")

At 4,000 per arm a 1-point lift is detected 16% of the time, a 2-point lift 49%, and a 5-point lift 100%. Reaching 97% for a 2-point lift needs 16,000 per arm. Solving directly, 80% power on a 2pp lift from a 30% base requires 8,393 per arm — more than twice what the 4,000-per-arm row would suggest is enough. An experiment with 1,000 per arm will miss a 2-point effect 84% of the time.

The mistake this prevents

The mistake is computing power after a null result to explain it. Post-hoc power is a deterministic function of the p-value and adds no information at all.

Takeaway

Calculate the sample size before running the test, using the smallest effect worth acting on. State the minimum detectable effect alongside any null result so a reader knows what the test could have found.

Mark complete