Module 11 Activity

Scenario

Review a trained model the way a sceptical colleague would, and write what you find.

Task

1. Train at increasing data sizes and record the train/validation gap at each.
2. Bucket predictions by confidence and compare predicted against actual rates.
3. Compute saliency for one input and rank the features.
4. Evaluate on progressively shifted data, recording accuracy and confidence.
5. Split accuracy and positive rate by group.
6. Write a model card including an out-of-scope section.

Deliverable

A review memo: loss checkpoints, accuracy against baseline, confusion matrix, per-slice accuracy, confident-mistake count, and limitations.

Check your work

• A closing gap as data grows means more data helps. A flat validation curve with a wide gap means the

model is the limit.

• Accuracy falls under distribution shift while confidence does not.
• The under-represented group's accuracy differs materially from the overall figure.

The line that matters most

Confident mistakes. Those are where the model was both wrong and sure, and in real data they are usually labelling errors or genuine gaps.

Mark complete