Applied ML · Module 6
Answers and explanations · Read classification metrics as counts and questions
Compare these references after your own attempt. You can read them at any time; they do not record a live model or vendor run.
Lesson 06 · Classification
Exercise 06.1
TP = 8, FN = 2, FP = 6 and TN = 34. The counts sum to 50. Accuracy is (8 + 34)/50 = 84%. Precision is 8/14, approximately 57.14%; recall is 8/10 = 80%. Specificity is 34/40 = 85%, so balanced accuracy is (80% + 85%)/2 = 82.5%. Alert fraction is 14/50 = 28%.
Always predicting on time produces 40/50 = 80% accuracy but zero recall. The model improves accuracy by four percentage points while also finding eight late jobs and requiring fourteen reviews. The workload and error tradeoff should be reported, not compressed into the four-point difference.
Exercise 06.2
A's squared probability errors are 0.04, 0.04 and 0.16; its Brier score is 0.24/3 = 0.08. B's are 0.16, 0.16 and 0.2025; its Brier score is 0.5225/3, approximately 0.17417. A has lower Brier loss on these three examples. Both systems produce hard labels [1, 0, 1] at threshold 0.5, so their thresholded confusion counts are identical. This illustrates information discarded by hard decisions, not population-level proof that A is universally better calibrated.
Exercise 06.3
High AUC supports a ranking claim in the measured population, not trustworthy numerical probabilities. Assess log loss/Brier score and a reliability analysis with bin counts on suitable held-out rows. It also does not specify review burden: choose a declared threshold and report alert count/fraction, confusion counts and capacity behavior on the relevant period. Even a threshold feasible during development may exceed capacity later. All claims need the actual sample and class prevalence.
Exit check 06.A
When on-time jobs dominate, calling every job on time correctly labels most observations. It still finds none of the late jobs, so its positive-class recall is zero.
Exit check 06.B
Ranking concerns ordering; probability quality concerns the numerical estimates relative to outcomes; the decision policy combines probabilities with a threshold, costs and constraints. Improving one does not automatically verify the others.