Module 12 Lesson: Experiments, A/B Tests, and Statistical Reporting
Classroom Explanation
Imagine we are in class and someone puts a small table on the board. The table looks simple, so it is tempting to jump straight to the answer. In statistics, we slow down first. We ask what question the table can answer, what question it cannot answer, and what kind of claim would be safe.
In this module, the goal is to bring the course together in a careful evidence report. The important part is not only the calculation. The important part is the thinking before and after the calculation.
Why This Matters
The common mistake is declaring the highest observed rate as the winner without checking sample size, practical size, and test design. This mistake can make a report sound more confident than the evidence deserves.
Key Ideas
1. A/B tests compare variants, usually with a treatment and control.
An A/B test compares versions under a planned design. The result should still be checked for denominator, size, uncertainty, and practical value.
2. Random assignment helps make a causal claim more credible.
An A/B test compares versions under a planned design. The result should still be checked for denominator, size, uncertainty, and practical value.
3. A good report states question, data, method, result, uncertainty, limits, and next step.
An A/B test compares versions under a planned design. The result should still be checked for denominator, size, uncertainty, and practical value.
Worked Example
Variant B has a signup rate of about 8.73% versus 8.00% for A. Variant C is higher, but its sample is much smaller.
Here is the dataset used in this module.
| variant | visitors | signups | completion_rate | avg_time_seconds | notes |
|---|---|---|---|---|---|
| A_old_button | 6000 | 480 | 0.0800 | 96 | control variant |
| B_new_button | 5900 | 515 | 0.0873 | 101 | new variant |
| C_highlighted_button | 1200 | 111 | 0.0925 | 108 | small follow-up sample |
Read the table slowly. First name the unit. Then name the variables. Then name the comparison or pattern. Only after that should you write the conclusion.
How To Think Through It
- State the question in one sentence.
- Name the unit of analysis.
- Identify the outcome and comparison.
- Check the denominator, sample, uncertainty, or assumption.
- Write only what the data supports.
- Add one limit so the result does not overclaim.
Common Mistake
The common mistake is declaring the highest observed rate as the winner without checking sample size, practical size, and test design.
How To Write The Result
Variant B has a modestly higher signup rate than A in this synthetic summary; Variant C needs more evidence because its sample is much smaller.
Practice Prompt
Write a final evidence review memo using the A/B-test summary.
Takeaway
When a statistical result feels obvious, pause and ask: what exactly was measured, compared, and assumed?