Module 05 Summary
The idea this module was built around
The learning rate is the first thing to check and the most common cause of a model that "does not learn". Change it by factors of ten before changing anything else.
What you can now do
- Recognise the two failure directions: crawling and diverging
- Choose an optimiser knowing what each adds
- Diagnose vanishing and exploding gradients by measuring at both ends of the network
- Read a learning curve and know which of three situations you are in
The trap this module removed
Forgetting model.eval() with BatchNorm present. The layer normalises using the validation batch's own statistics, so the score depends on how the validation set happened to be batched.
Figures worth remembering
Regularisation should raise training loss. If it does not, it is not doing anything.
Before you move on
Plot both curves for a model you already trained. The shape of the pair tells you what to change next.