# Capstone · one defensible experiment

## Deliverables

1. `protocol.md`: one-row meaning, available features, target, IID assumption, alternative group/time split if needed, fixed candidate budget, seeds, primary selection metric and test boundary
2. Completed `exercises/stages.py`, `exercises/capstone.py` and public-check output
3. Validation-only selection output and one learning-curve visualization with text interpretation
4. Final assessment output after selection is frozen; report actual counts, metric units and naive baseline
5. A 600–900-word report explaining shape flow, one gradient calculation, selection decision, validation error analysis, reproducibility and limits

Do not submit virtual environments, downloaded wheels, caches or secret credentials. The source, small data/outputs and report are sufficient. The bundled data and reference test outcomes are visible; disclose prior exposure. For genuine learner assessment, an instructor should generate a new held-out seed after the model is frozen.

## Commands and stopping boundary

```sh
python check_learner.py
python -m exercises.capstone
# Read outputs/learner_capstone/selection.json and freeze your written plan.
python -m exercises.capstone --final
```

`--final` requires an existing selection record whose configuration/rationale hash matches the current plan. It retrains the selected model reproducibly, checks the validation result, then evaluates test once and creates `final.json` exclusively. A second final invocation refuses to overwrite the assessment. Deleting files to keep tuning on the same test is not a valid fresh assessment.

The example reference path is separate:

```sh
python -m solutions.capstone
python -m solutions.capstone --final
```

The source package includes the author's reference results. Those two reference commands will therefore refuse to overwrite the existing final report until you choose a separate output folder. To reproduce unchanged reference mechanics, use `--output-dir outputs/my_reference_reproduction` on both commands. That rerun is a reproducibility check, not additional test evidence.

## Rubric: 100 points

- Question and data contract, 15: explicit row/target, available inputs, IID assumption and deployment limits
- Evaluation integrity, 20: disjoint IDs, train-only fit, validation-only choices, no test-driven revisions
- Implementation, 20: shapes/dtypes/losses correct, gradients cleared, weights update, eval/no_grad correct
- Baselines and selection, 15: train-mean and linear baseline, bounded candidate set, best-checkpoint evidence
- Diagnostics, 10: one validation error pattern, a plausible alternative explanation and discriminating next check
- Reproducibility, 10: versions/seeds/configs, source runnable, checks reported accurately
- Communication, 10: actual measured values/counts, uncertainty and limits, no unsupported mastery or deployment claims

Suggested pass: at least 70 points and all integrity gates. Gate failures require correction regardless of score: split overlap, preprocessing fitted on held-out rows, test-tuned model choice, invented results or a mismatch between code and claimed execution. A high accuracy alone earns no exemption.

Optional CNN extension: compare the supplied flattened linear model and 170-parameter CNN using validation only. Explain every shape and which augmentations preserve the target. Transfer learning and attention are conceptual extensions, not required capstone deliverables.