Module 11 Summary
The idea this module was built around
A model is confidently wrong outside its training range. You cannot use its own probability to decide whether to trust it.
What you can now do
- Read a learning curve to decide between more data and a different model
- Check calibration by bucketing predictions against actual rates
- Treat saliency as a lead rather than an explanation
- Report accuracy and positive rate per group
The trap this module removed
Relying on low confidence to flag inputs the model cannot handle. On genuinely novel inputs, confidence is often higher than on hard in-distribution cases.
Figures worth remembering
Under distribution shift, accuracy degrades while confidence holds. That divergence is the danger.
Before you move on
Write the out-of-scope section of a model card for something you have built. If you cannot name a situation where it should not be used, you have not looked hard enough.