Read classification metrics as counts and questions

Predicting lateness adds a probability layer

The label is 1 if duration_hours > promised_hours and 0 otherwise. A classifier can produce a score, a probability estimate or a hard label. These outputs serve different purposes. Ranking jobs requires a useful ordering; interpreting a value as a chance requires probability quality; creating a review list requires a decision threshold. A single accuracy number cannot verify all three.

Logistic regression starts with a weighted combination of prepared features, called a logit. The sigmoid converts it into a number between zero and one: probability equals one divided by one plus exp of negative logit. A logit of zero yields 0.5. A logit of approximately 1.386 yields 0.8. The curve changes smoothly, so the same coefficient does not add the same number of probability points everywhere. Despite its name, logistic regression is a classification model here.

The course compares a constant prior-probability baseline, logistic regression and a random forest classifier in training-only temporal validation. Its main probability-policy integration uses logistic regression as a deliberately fixed family. That is a pedagogical choice, not a claim that logistic regression beats every other family. Its regularization and class-weight options are chosen later with a declared training-CV criterion.

Worked miniature: turn alerts into a confusion matrix

Imagine 20 new jobs, four of which are late. A threshold produces six alerts: three late jobs and three on-time jobs. We have three true positives, three false positives, one false negative and thirteen true negatives. Count all four cells: 3 + 3 + 1 + 13 = 20.

Accuracy is correct decisions divided by all jobs: (3 + 13)/20 = 0.80. Precision is late jobs among alerts: 3/6 = 0.50. Recall is alerted jobs among all late jobs: 3/4 = 0.75. Specificity is correctly unflagged on-time jobs divided by all on-time jobs: 13/16 = 0.8125. Balanced accuracy averages recall and specificity, giving 0.78125. F1 is twice precision times recall divided by their sum: 0.60.

The majority baseline also obtains 0.80 accuracy by calling every job on time. It produces no alerts and catches none of the four late jobs. The equal accuracies conceal different workflows. Precision tells the reviewer how often an alert is right on this sample; recall tells operations how many observed delays were caught. Neither measures the number of reviews directly, which is six, or 30% of jobs.

If there are no predicted positives, precision's denominator is zero. A library may return zero by an explicit convention, but the report should still say there were no alerts. Likewise, ROC AUC cannot be meaningfully estimated on a sample containing only one observed class. Record undefined situations rather than hiding them behind a convenient default.

Ranking and probability quality ask different questions

ROC AUC evaluates ranking across positive-negative pairs. Informally, it asks how often a randomly selected late job receives a larger score than a randomly selected on-time job, with ties handled appropriately. It does not specify a review threshold, a probability interpretation or an alert budget. Under strong imbalance, the false-positive rate can look small while the absolute number of false alerts is large.

A precision-recall curve shows how precision and recall change as the threshold changes. Average precision summarizes the precision achieved at recall increases. scikit-learn's average precision is not simply the trapezoidal area under an interpolated curve; the metrics guide gives the precise distinction. The positive-class prevalence is essential context, because a task with many more late jobs has a different baseline precision.

Probability metrics use the probabilities themselves. Binary Brier score averages squared differences between probability and observed 0/1 outcome. For two miniature predictions, p = 0.8 with y = 1 and p = 0.3 with y = 0, the score is [(0.8 − 1) squared + (0.3 − 0) squared]/2 = 0.065. Log loss penalizes confident wrong probabilities especially strongly. For a positive outcome the contribution is negative log(p); for a negative outcome it is negative log(1 − p). Lower values are better for both metrics. Neither proves calibration by itself.

Imbalance is not a command to rebalance everything

A rare positive outcome may be exactly what occurs in deployment. Balancing the evaluation sample would then change the prevalence and some operational interpretations. Keep evaluation representative of its intended period, and report its class counts. Weighted fitting can emphasize the minority class, but its probabilities may need calibration. The class-weight option belongs inside training-only model selection.

Resampling before splitting can copy or derive examples across evaluation boundaries. Synthetic oversampling also has assumptions about meaningful neighborhoods. This course does not require it. First establish a valid split, a baseline, a suitable loss and clear error counts. Complexity should answer an identified problem, not compensate for a misunderstood metric.

Run python examples/06_classification.py. Reconstruct a confusion matrix from the printed probabilities and threshold. Check that counts sum to the evaluation sample, recompute precision and recall by hand for one result, and state the positive class. Compare a probability baseline with a fitted model using the same rows. Record actual results from outputs/reference_results.json; the 20-job miniature is only a teaching trace.

Independent practice

1. 06.1 · Rebuild the report. Among 50 jobs, ten are late. A model alerts on eight late jobs and six on-time jobs. Calculate all four confusion counts, accuracy, precision, recall, balanced accuracy and alert fraction. Compare accuracy with an always-on-time baseline.
2. 06.2 · Score probabilities. For outcomes [1, 0, 1], system A predicts [0.8, 0.2, 0.6] and system B predicts [0.6, 0.4, 0.55]. Calculate Brier scores. Explain why using threshold 0.5 alone would miss the difference between the systems.
3. 06.3 · Choose evidence for a claim. A colleague says, “AUC is high, so probabilities are trustworthy and the review queue will stay small.” Identify the two unsupported conclusions and specify evidence needed for each, including an actual count or fraction.

Exit checks

• 06.A: How can an all-on-time classifier have high accuracy while being useless for finding late jobs?
• 06.B: Which distinction separates ranking quality, probability quality and a thresholded decision policy?