Unit 10.06: Calibration is the model's job; the threshold is yours

The model gives you a probability. What to do at each probability is a business decision, not a statistical one.

Calibration is the model's job; the threshold is yours

A model is calibrated if, among cases it assigns a probability of 0.3, roughly 30% actually occur. Calibration is checkable directly: bin the predictions with pd.cut and compare the mean predicted probability with the observed rate in each bin.

Choosing the threshold at which you act is entirely separate, and no statistical criterion decides it. It depends on what a missed case costs against what a false alarm costs — facts about the world, not about the data.

The default of 0.5 is a convention with no special status. On an imbalanced problem it is usually the wrong choice.

This block checks calibration by bin, then varies the threshold.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(707)
n = 4000
d = pd.DataFrame({"x": rng.normal(size=n)})
d["y"] = rng.binomial(1, 1 / (1 + np.exp(-(-1 + 1.1 * d.x))))
model = smf.logit("y ~ x", data=d).fit(disp=False)
d["p_hat"] = model.predict(d)

print("Calibration: among cases predicted at p, does p actually happen?")
bins = pd.cut(d.p_hat, np.arange(0, 1.01, 0.2), include_lowest=True)
cal = d.groupby(bins, observed=True).agg(
    n=("y", "size"), mean_predicted=("p_hat", "mean"), observed=("y", "mean"))
print(cal.round(3).to_string())
print("\nPredicted and observed track each other closely: it is calibrated.\n")

print("Threshold choice is a separate, non-statistical decision:")
for t in (0.2, 0.5, 0.8):
    flagged = d.p_hat > t
    print(f"  threshold {t:.1f} -> flagged {flagged.sum():4d}"
          f"   caught {int((flagged & (d.y == 1)).sum()):4d}"
          f" of {int(d.y.sum())} positives"
          f"   false alarms {int((flagged & (d.y == 0)).sum()):4d}")
print("\nThe model gives a probability. What to do at each probability depends")
print("on what a miss costs against what a false alarm costs.")

Across five bins the mean predicted and observed rates track closely — 0.118 against 0.123, 0.296 against 0.290, 0.490 against 0.491. The model is calibrated. The threshold table then shows the trade-off: at 0.2 it flags 2642 cases and catches 1108 of 1275 positives with 1534 false alarms; at 0.8 it flags only 58 and catches 53, with just 5 false alarms. Same model, same probabilities, three completely different operating points.

The mistake this prevents

The mistake is treating 0.5 as the natural cut-off. It optimises nothing in particular, and on a rare outcome it flags almost nothing.

Takeaway

Check calibration by binning predictions against observed rates. Choose the threshold from the relative cost of misses and false alarms, and report the flagged count, the catch rate and the false alarms at the threshold you picked.

Mark complete