Answers and explanations · Calibration and threshold tradeoffs

Lesson 08 · Calibration and thresholds

Exercise 08.1

Squared probability errors are 0.01, 0.36, 0.36 and 0.01. Their sum is 0.74; binary Brier loss is 0.74 divided by 4, or 0.185.

The low bin has mean prediction (0.10 plus 0.40) divided by 2 = 0.25 and observed event rate 1/2 = 0.50. The high bin has mean prediction (0.60 plus 0.90) divided by 2 = 0.75 and observed event rate 1/2 = 0.50. Each bin has only two observations. The gaps suggest what to examine with more appropriate evidence, but cannot establish population calibration or stable bin-level miscalibration.

Exercise 08.2

At threshold 0.50, alert jobs four, five and six. The counts are TP = 2, FP = 1, FN = 1, TN = 2. Precision is 2/3, recall is 2/3 and alert count is three. Error cost is 1 plus 4 times 1 = 5 units.

At threshold 0.30, alert jobs two through six. The counts are TP = 3, FP = 2, FN = 0, TN = 1. Precision is 3/5, recall is 3/3 = 1 and alert count is five. Error cost is 2 plus 4 times 0 = 2 units.

The lower threshold minimizes the stated error cost among these two options, but violates a three-alert capacity limit. The 0.50 threshold is the feasible candidate of the pair. State that constrained choice and its additional missed event; do not claim the lower-cost option satisfies capacity. These exercise counts differ from the course's percentage-based policy, which must be assessed on its own threshold block.

Exercise 08.3

Keep the base pipeline fitted on days 0–148 using training-only tuning. Fit its frozen-pipeline sigmoid calibrator on days 150–173, 144 jobs. Leave day 174 unused. Choose the threshold on days 175–198, another 144 jobs. Audit that calibration labels are known by the mapping's fit boundary, rather than relying only on the day-174 embargo.

Using all 294 validation labels to fit calibration means that the reported validation probability performance is partly in-sample. Choosing a threshold on the same labels introduces additional selection reuse. Reallocating the roles for a corrected run makes the procedure coherent, but does not erase the fact that the earlier analysis saw the labels. Document that history. The final 300 jobs may be opened only after all decisions and the evaluation plan are frozen, provided those test labels truly remained unused. They can then evaluate the final selected procedure independently of the development history.

Exit check 08.A

Changing a threshold changes the yes/no decisions, confusion counts and alert workload. It leaves the underlying probability estimates unchanged, so it does not improve their calibration. Calibration concerns probabilities versus outcome frequencies, not which side of an action boundary a job occupies.

Exit check 08.B

A low Brier score summarizes overall probability prediction quality; it does not specify the cost of a missed event, false alert, workload constraints or relevant future conditions. The analytic cost threshold also assumes calibrated probabilities, fixed error costs and the specified treatment of correct decisions. A practical policy must state and test these assumptions rather than infer them from one score.