Module 12 Worksheet: Experiments, A/B Tests, and Statistical Reporting
Scenario
A comparison was run and stopped, and you have been asked to write it up.
Data
Dataset: ab-test-summary.csv
| variant | visitors | signups | completion_rate | avg_time_seconds | notes |
|---|---|---|---|---|---|
| A_old_button | 6000 | 480 | 0.0800 | 96 | control variant |
| B_new_button | 5900 | 515 | 0.0873 | 101 | new variant |
| C_highlighted_button | 1200 | 111 | 0.0925 | 108 | small follow-up sample |
Your Task
Write a final evidence review memo using the A/B-test summary.
Questions
- What was the primary measure, and was it named in advance?
- When was the comparison stopped, and on what basis?
- How many measures were tracked in total?
- What was the difference, with its interval?
- What threshold was agreed beforehand?
- What does the comparison establish?
- What does it not establish?
Write Your Conclusion
Write the summary paragraph, naming the primary measure, the stopping basis, the number of measures and what the result does not show.
Self-Check
- Did I check whether the primary measure was pre-specified?
- Did I report the stopping basis?
- Did I say how many measures were tracked?
- Does my summary state what was not established?