Calibration and threshold tradeoffs

A probability and a decision are different outputs

A predicted late probability of 0.70 says something about uncertainty. An alert says someone should take an action. Connecting them requires a decision rule. Better probability estimates do not automatically produce a better rule, and a useful rule does not establish that every probability is trustworthy.

Our label remains late when duration_hours > promised_hours. Equality is on time. The synthetic example concerns planning attention for depot jobs; it does not authorize consequential scoring of people or real operations. Inputs include the promise known at intake. Never introduce the completed duration, label_available_at, final delay reason or close cost as a predictor to improve the result.

Calibration asks whether groups assigned similar probabilities experience events at similar frequencies. If a large, relevant collection of jobs receives probabilities near 0.20, roughly one fifth being late would support that part of the probability scale. One job is not “20% late”; four observations cannot establish reliable calibration. Probability calibration

Worked trace: check probability quality

Consider four miniature predictions: probabilities 0.20, 0.20, 0.70 and 0.90, with outcomes 0, 1, 1 and 1. These are miniature arithmetic examples.

The Brier score for a binary event is the average of squared differences between probability and outcome. The four squared differences are 0.04, 0.64, 0.09 and 0.01. Their sum is 0.78; dividing by four gives 0.195. Lower is better for this loss. Confidently assigning a low probability to an event that happens produces a large penalty. Brier score reference

For a tiny reliability display, put the first two predictions in a low-probability bin. Their average prediction is 0.20, while the observed late fraction is one out of two, or 0.50. The other bin has mean probability 0.80 and observed fraction 1.00. These differences describe this sample. Two observations per bin make a strong “miscalibrated everywhere” conclusion unjustified.

Choose bins before comparing models and show bin counts. Empty bins contain no empirical event rate. A lower Brier score alone does not prove better calibration, because overall probability performance also depends on separating easy and difficult cases. Read probability loss, the reliability display and discrimination together.

Allocate evidence before fitting calibration

The validation period has 294 jobs across days 150–198. Allocate days 150–173 to calibration: 24 days times six jobs gives 144. Leave day 174 as an embargo. Allocate days 175–198 to threshold choice: another 144 jobs. The arithmetic reconciles as 144 plus 6 plus 144 equals 294.

The logistic classifier tunes C across 0.1, 1, 10 and class weight across none or balanced using training-only negative log loss. Its family is fixed for this probability exercise; comparison with dummy and forest models does not imply logistic wins. Calibration fits a mapping using the base model's predictions and labels from the calibration block. The base classifier stays frozen. In scikit-learn 1.8, FrozenEstimator supports calibration of an already fitted estimator; disjoint fitting data remains the practitioner's responsibility. Sigmoid calibration is a restrained choice for this small teaching block; flexible isotonic calibration can overfit small samples. Calibration implementation guidance

Verify that training labels existed at the base-model fit boundary and calibration labels existed before fitting the mapping for the threshold period. A six-row embargo is not that verification. Reporting fit-block calibration as if it were independent evaluation is another error: the mapping has already used those labels.

Worked trace: threshold changes the workload

Take four new miniature jobs with probabilities 0.15, 0.40, 0.55 and 0.85 and outcomes 0, 1, 0 and 1. Define an alert when probability is greater than or equal to the threshold. At threshold 0.50, alert jobs three and four. There is one true positive, one false positive, one false negative and one true negative.

At threshold 0.30, alert jobs two, three and four. Now there are two true positives, one false positive, no false negatives and one true negative. Recall rises from one half to one, while alerts rise from two to three. Precision changes from one half to two thirds. Lowering a threshold cannot remove existing alerts under this fixed rule, but precision need not change monotonically on a finite dataset.

Suppose this miniature exercise assigns three cost units to a missed late job and one unit to a false alert. The first threshold costs 3 times 1 plus 1 times 1, or four units. The second costs one unit. These are invented instructional costs, not measured depot economics. Decision-threshold guidance

Name the decision assumptions

For perfectly calibrated probabilities, zero cost for correct decisions and constant error costs, alerting has expected cost C_FP times (1 minus p); not alerting has expected cost C_FN times p. Alert when p is at least C_FP divided by (C_FP plus C_FN). The miniature costs above yield 0.25. This arithmetic is a useful starting point, not a universal operational optimum.

Real review capacity, changing costs, uncertain probabilities and differences between jobs can break those assumptions. The course threshold comparison therefore records false positives, false negatives, precision, recall and number of alerts. Define the primary criterion and any capacity constraint in advance. If no candidate meets both, report that conflict rather than quietly relaxing a requirement.

Selecting a threshold uses labels even though it does not change the classifier's weights. Keep the final test closed during that choice. The course searches 0.10–0.90 in 0.05 steps, minimizing FP plus 4FN with at most 40% alerts. Ties prefer fewer alerts, then higher thresholds. If none is feasible, the implementation records an explicit no-alert fallback. Test alert volume may exceed 40%; report it.

Run and explain

Run python examples/08_calibration.py. Read the measured output in outputs/reference_results.json; distinguish calibration-block fit information from threshold-block evaluation. Confirm that the positive class is late, the probability column matches that class and equality uses the declared alert rule. Explain the chosen threshold in workload and error terms.

Independent practice

Exercise 08.1 · Audit probabilities

For probabilities 0.10, 0.40, 0.60, 0.90 and outcomes 0, 1, 0, 1, calculate binary Brier loss. Use the first two and last two observations as bins; calculate each bin's mean prediction and observed rate. State why these bins cannot establish population calibration.

Exercise 08.2 · Compare two decisions

Probabilities are 0.10, 0.35, 0.45, 0.60, 0.80, 0.90; outcomes are 0, 1, 0, 1, 0, 1. Use alerts at probability at least the threshold. Compare thresholds 0.50 and 0.30: confusion counts, precision, recall, alert count and cost with C_FN = 4 and C_FP = 1. Then impose a maximum of three alerts and explain the conflict.

Exercise 08.3 · Repair reused evidence

A teammate calibrates on all 294 validation jobs, chooses a threshold on those same labels and calls that score independent validation. Reallocate the stated day ranges correctly and explain which claimed independence is lost. Identify when the final test may be opened.

Exit checks

• 08.A: Can changing only the threshold improve calibration? Explain what changes and what stays fixed.
• 08.B: Why does a cost-optimal threshold require assumptions beyond a low Brier score?