Module 12 Lesson: Experiments, A/B Tests, and Statistical Reporting
Classroom Explanation
An experiment earns causal language because assignment was random. Everything else — however large, however significant — supports association, and the difference is what you had to know beforehand rather than how much data you gathered.
In this module, the goal is to bring the course together in a careful evidence report. The important part is not only the calculation. The important part is the thinking before and after the calculation.
Why This Matters
The common mistake is declaring the highest observed rate as the winner without checking sample size, practical size, and test design. This mistake can make a report sound more confident than the evidence deserves.
Key Ideas
1. A/B tests compare variants, usually with a treatment and control.
An A/B test compares versions under a planned design. The result should still be checked for denominator, size, uncertainty, and practical value.
2. Random assignment helps make a causal claim more credible.
An A/B test compares versions under a planned design. The result should still be checked for denominator, size, uncertainty, and practical value.
3. A good report states question, data, method, result, uncertainty, limits, and next step.
An A/B test compares versions under a planned design. The result should still be checked for denominator, size, uncertainty, and practical value.
Worked Example
Variant B has a signup rate of about 8.73% versus 8.00% for A. Variant C is higher, but its sample is much smaller.
Here is the dataset used in this module.
| variant | visitors | signups | completion_rate | avg_time_seconds | notes |
|---|---|---|---|---|---|
| A_old_button | 6000 | 480 | 0.0800 | 96 | control variant |
| B_new_button | 5900 | 515 | 0.0873 | 101 | new variant |
| C_highlighted_button | 1200 | 111 | 0.0925 | 108 | small follow-up sample |
Decide the primary measure, the sample size and the stopping point before collecting anything. Checking repeatedly and stopping when a result crosses the threshold gives many chances at it, which inflates the error rate far above the level you set.
How To Think Through It
- Name the primary measure in advance.
- Fix the sample size and the stopping rule.
- Report the difference with its interval.
- State how many measures were tracked.
- Write what the result does and does not establish.
Common Mistake
The common mistake is declaring the highest observed rate as the winner without checking sample size, practical size, and test design.
How To Write The Result
Variant B has a modestly higher signup rate than A in this synthetic summary; Variant C needs more evidence because its sample is much smaller.
Practice Prompt
Write a final evidence review memo using the A/B-test summary.
Takeaway
A pre-specified primary measure and a stopping rule are what make the reported error rate real.