Module 12 Checkpoint: Experiments, A/B Tests, and Statistical Reporting

Scenario

An A/B test was checked daily and stopped on day 9, when variant B first reached p < 0.05 on one of five metrics.

Questions

1. Name two problems with how this test was run.
2. What should have been decided before the test started?
3. Why does checking daily and stopping at significance inflate the error rate?
4. Five metrics were tracked. How does that affect one significant result?
5. Write the honest summary of what this test established.

Pass Standard

You pass when you can name the stopping-rule and multiple-metric problems, and write a summary that does not claim more than the test supports.