Module 11 Summary

What this module established

A control group is only a comparison if it is comparable. Adopters outscored non-adopters by 6.68 against a true effect of 2 — because their baseline engagement was 55.8 against 46.0 before the feature existed.

Carry forward

• Randomisation balances the variables you never thought of. The observational estimate was significant at p = 3e-59 for an effect of exactly zero; the randomised one gave +0.006, p = 0.983.
• Every line of a pre-analysis plan blocks a specific abuse, and the stopping rule is the one most often missing.
• One primary metric decides; guardrails can only veto. Retention did not move and support contacts rose by 2.1 points.
• 80% power on a 2pp lift from a 30% base needs 8,393 per arm. Twenty subgroup tests on a zero effect produced exactly one 'winner'.

Before moving on

Move on when your plan is dated, your primary metric is named, and any subgroup finding is reported with how many were tested.

Mark complete