Module 07 Activity

Scenario

The model produces numbers. This activity works out which of them mean anything.

What you build

An evaluation with a cost-derived threshold, an error analysis, and per-slice results defined in advance.

Steps

1. Report accuracy alongside the majority-class baseline and recall on the class you care about.
2. Build the confusion matrix and price both error types. State the cost ratio.
3. Sweep the threshold and find the minimum-cost operating point. Report how far it is from 0.5.
4. Sort the errors by confidence and inspect the ten most confident. Report what they share.
5. Define your slices from collection metadata BEFORE looking at results, then report each with its row count.
6. Mark any slice under 100 rows as unmeasured rather than as measured.

Evidence to hand in

• Accuracy, baseline and recall together.
• The confusion matrix with both error costs and the ratio.
• The threshold sweep and the chosen operating point.
• The ten most confident errors and what they share.
• Per-slice results with row counts and stability verdicts.

Review checklist

• The majority baseline appears next to every accuracy figure.
• The threshold follows from the cost ratio, not from 0.5.
• The confident errors were inspected for a shared property.
• Slices were defined before results were seen.
• Small slices are marked unmeasured, not reported as fact.

Mark complete