Unit 10.06: Calibration is the model's job; the threshold is yours

The model gives you a probability. What to do at each probability is a business decision, not a statistical one.

Calibration is the model's job; the threshold is yours

A model is calibrated if, among cases it assigns a probability of 0.3, roughly 30% actually occur. Calibration is checkable directly: bin the predictions and compare the mean predicted probability with the observed rate in each bin.

Choosing the threshold at which you act is entirely separate, and no statistical criterion decides it. It depends on what a missed case costs against what a false alarm costs, and those are facts about the world.

The default of 0.5 is a convention with no special status. On an imbalanced problem it is usually the wrong choice.

This block checks calibration by bin, then varies the threshold.

set.seed(707)
n <- 4000
x <- rnorm(n)
y <- rbinom(n, 1, 1 / (1 + exp(-(-1 + 1.1 * x))))
model <- glm(y ~ x, family = binomial)
p_hat <- predict(model, type = "response")

cat("Calibration: among cases predicted at p, does p actually happen?\n")
bins <- cut(p_hat, breaks = seq(0, 1, 0.2), include.lowest = TRUE)
for (b in levels(bins)) {
  sel <- bins == b
  if (sum(sel) > 0) {
    cat(sprintf("  %-12s n=%4d  mean predicted %.3f  observed %.3f\n",
                b, sum(sel), mean(p_hat[sel]), mean(y[sel])))
  }
}

cat("\nPredicted and observed track each other closely: the model is calibrated.\n\n")
cat("Threshold choice is a separate, non-statistical decision:\n")
for (t in c(0.2, 0.5, 0.8)) {
  pred <- p_hat > t
  cat(sprintf("  threshold %.1f -> flagged %4d  caught %3d of %3d positives  false alarms %4d\n",
              t, sum(pred), sum(pred & y == 1), sum(y == 1), sum(pred & y == 0)))
}
cat("\nThe model gives a probability. What to do at each probability depends on\n")
cat("what a miss costs against what a false alarm costs.\n")

Across five bins the mean predicted and observed rates track closely — 0.116 against 0.119, 0.291 against 0.299, 0.679 against 0.679. The model is calibrated. The threshold table then shows the trade-off: at 0.2 it flags 2433 cases and catches 989 of 1176 positives with 1444 false alarms; at 0.8 it flags only 48 and catches 43, with just 5 false alarms. Same model, same probabilities, three completely different operating points.

The mistake this prevents

The mistake is treating 0.5 as the natural cut-off. It optimises nothing in particular, and on a rare outcome it flags almost nothing.

Takeaway

Check calibration by binning predictions against observed rates. Choose the threshold from the relative cost of misses and false alarms, and report the flagged count, the catch rate and the false alarms at the threshold you picked.

Mark complete