Module 11 Activity

Scenario

Design an experiment, or say plainly why your evidence is observational.

What you build

A pre-analysis plan, a balance check and a result reported with the causal language it earns.

Steps

1. Establish whether group membership was assigned or chosen. If chosen, compare the groups on pre-treatment characteristics.
2. Randomise where you can and report a balance table. Do not re-randomise to chase balance.
3. Write the pre-analysis plan: primary outcome, unit, comparison, test, alpha, minimum effect, sample size, stopping rule, exclusions, subgroups.
4. Name one primary metric, the secondaries and the guardrails, and let only the primary decide.
5. Compute the sample size with statsmodels.stats.power from the smallest effect worth acting on, and state the minimum detectable effect alongside any null result.
6. Report the total number of subgroups tested alongside any that reached significance.

Evidence to hand in

• The assignment mechanism and, if observational, the baseline imbalance.
• The balance table.
• The dated pre-analysis plan.
• The primary, secondary and guardrail metrics with their results.
• The power calculation and minimum detectable effect.
• The subgroup count.

Review checklist

• Group membership is identified as assigned or chosen.
• The plan predates the data and includes a stopping rule.
• Only the primary metric can decide; guardrails can only block.
• The minimum detectable effect is reported with any null result.
• Causal language is used only where assignment was randomised.

Mark complete