Activity: Create a probability-quality report
Estimated active time: 45-75 minutes
Scenario
Use the synthetic learner-support dataset. The fictional platform wants to decide which model families deserve a place in an algorithm portfolio.
Task
Fit a probabilistic classifier, then check whether its probabilities mean what they say by bucketing predictions against outcomes.
Required sections
- Problem and decision.
- Validation accuracy, and a calibration table with at least four confidence buckets.
- For each bucket: predicted probability, actual rate, and the count of rows.
- Whether the model is over- or under-confident, with the buckets as evidence.
- Keep, delay, or reject against the Module 1 baselines.
- Limitation note.
Rules
- A bucket with fewer than about ten rows is noise — report the count beside every rate.
- Do not describe a model as calibrated without showing the table.
- Accuracy and calibration are separate properties. Report both.
Output
Use the relevant learner-resource template from learner-resources/ or write a one-page memo with the same sections.