Module 11 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. Comparing users who chose to adopt a feature with those who did not measures…
- A. the treatment effect
- B. the treatment effect plus the pre-existing difference
- C. nothing at all
- D. only the pre-existing difference
2. Randomisation is more powerful than adjustment because it balances…
- A. only the measured covariates
- B. everything, including what you never measured
- C. the sample sizes
- D. the variances
3. The pre-analysis-plan line most often missing is…
- A. the primary outcome
- B. the stopping rule
- C. the alpha level
- D. the unit of analysis
4. For 80% power on a 2pp lift from a 30% base you need roughly…
- A. 1,000 per arm
- B. 4,000 per arm
- C. 8,400 per arm
- D. 16,000 per arm
5. Twenty subgroup tests on a treatment with exactly zero effect produced one 'significant' result. That is…
- A. evidence of an effect in that subgroup
- B. exactly what chance predicts
- C. a sign of a coding error
- D. reason to run more subgroups
---
Answer Key and Explanations
Check these only after attempting every question.
1. B — the treatment effect plus the pre-existing difference
Adopters' baseline engagement was 55.8 against 46.0, and the naive difference was 6.68 for a true effect of 2.
2. B — everything, including what you never measured
The randomised estimate was correct without anyone knowing health mattered.
3. B — the stopping rule
Repeatedly checking and stopping at significance drives the false-positive rate far above alpha.
4. C — 8,400 per arm
solve_power returns 8,393 per arm — more than twice what the 4,000-per-arm row would suggest.
5. B — exactly what chance predicts
At alpha 0.05 the expected number of false positives among twenty tests is one.
Practical Check
Apply this module to your own work: complete the module activity for *Experiments, A/B Tests, and Causal Caution*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.