Module 05 Activity

Scenario

A model "is not learning". Diagnose it with measurements rather than guesses.

Task

1. Train at learning rates 0.0001, 0.01, 0.1 and 1.5. Record which crawl and which diverge.
2. Compare SGD, SGD with momentum, and Adam at matched step counts.
3. Measure gradient magnitude at the first and last layer of a 12-layer network under three initialisations.
4. Show the train/eval difference in BatchNorm output.
5. Train on random targets with and without dropout and weight decay. Record best validation and the step it occurred.
6. Log train loss, validation loss and gradient norm every epoch, then report epochs_after_best.

Deliverable

A diagnostics report as JSON: first and last epoch, best validation and its epoch, epochs trained past best, and a stable/diverged verdict.

Check your work

• 0.0001 barely moves; 1.5 diverges to NaN.
• With poor initialisation the first layer's gradient is orders of magnitude off the last layer's.
• On random targets, best validation arrives long before the final step.

The number that matters

epochs_after_best. If it is large, you trained past the useful point and threw the good model away.

Mark complete