Activity: Repair an Invalid Experiment

Activity ID: ML-M05-A01 Estimated active time: 45-75 minutes

Goal

Produce the main Module 05 evidence: Repaired experiment with before/after explanation and a clean evaluation protocol.

Inputs

• data/module_05_leakage_repair_v1.csv
• data/data-dictionary.md
• notebooks/module-05-worked.ipynb
• notebooks/module-05-practice.ipynb

Instructions

1. Fit a model using every column, including the post-outcome ones. Record the score.
2. Identify which column is doing the work — a depth-1 tree is enough to find it.
3. Remove the post-outcome columns and refit. Record the new score.
4. Explain the gap between the two scores in one sentence.
5. Plot or tabulate a learning curve on the honest feature set.
6. State whether more data or a different model is the constraint, and how the curve told you.

Quality checklist

• [ ] Both scores are reported: with leakage and without.
• [ ] The leaking column is named.
• [ ] The explanation says why that column could not be available at prediction time.
• [ ] The learning curve is computed on the honest feature set only.
• [ ] The conclusion about more-data-versus-better-model cites the curve.

Submission format

Submit the notebook output or worksheet table plus a 4-6 sentence explanation.

Safety note

Use only supplied fictional or synthetic data. Do not add or upload real learner data.