Module 11 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. Comparing users who chose to adopt a feature with those who did not measures…
- A. the treatment effect
- B. the treatment effect plus the pre-existing difference
- C. nothing at all
- D. only the pre-existing difference
2. Randomisation is more powerful than adjustment because it balances…
- A. only the measured covariates
- B. everything, including what you never measured
- C. the sample sizes
- D. the variances
3. The pre-analysis-plan line most often missing is…
- A. the primary outcome
- B. the stopping rule
- C. the alpha level
- D. the unit of analysis
4. A guardrail metric can…
- A. decide a launch
- B. block a launch but never justify one
- C. replace the primary metric
- D. be chosen after the test
5. Twenty subgroup tests on a treatment with exactly zero effect produced one 'significant' result. That is…
- A. evidence of an effect in that subgroup
- B. exactly what chance predicts
- C. a sign of a coding error
- D. reason to run more subgroups
---
Answer Key and Explanations
Check these only after attempting every question.
1. B — the treatment effect plus the pre-existing difference
Adopters' baseline engagement was 52.8 against 45.2, and the naive difference was 6.45 for a true effect of 2.
2. B — everything, including what you never measured
Adjustment fixes the confounders you thought of. The randomised estimate was correct without anyone knowing health mattered.
3. B — the stopping rule
Repeatedly checking and stopping at significance drives the false-positive rate far above alpha.
4. B — block a launch but never justify one
Only the primary metric decides. Naming guardrails in advance is what makes an adverse move a finding.
5. B — exactly what chance predicts
At alpha 0.05 the expected number of false positives among twenty tests is one.
Practical Check
Apply this module to your own work: complete the module activity for *Experiments, A/B Tests, and Causal Caution*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.