Unit 11.01: Calibration and confidence caution

A model that outputs 0.9 should be right about 90% of the time. Most are not, and nothing in the output warns you.

Bucketing predictions against outcomes

Group predictions by confidence, then compare the average predicted probability in each bucket against the actual outcome rate. A calibrated model has those two columns matching.

The data here has genuine noise, so perfect confidence is not achievable — which is exactly when calibration matters:

import torch
from torch import nn

torch.manual_seed(0)
X = torch.randn(800, 4)
y = ((X[:, 0] + torch.randn(800) * 1.5) > 0).float().unsqueeze(1)   # genuinely noisy
Xtr, ytr, Xva, yva = X[:600], y[:600], X[600:], y[600:]

model = nn.Sequential(nn.Linear(4, 64), nn.ReLU(), nn.Linear(64, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(800):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(model(Xtr), ytr).backward()
    opt.step()

with torch.no_grad():
    probs = torch.sigmoid(model(Xva)).squeeze(1)

print("bucket        n   predicted   actual")
for low in (0.0, 0.2, 0.4, 0.6, 0.8):
    high = low + 0.2
    m = (probs >= low) & (probs < high)
    if m.sum() < 5:
        continue
    print(f"{low:.1f}-{high:.1f}  {int(m.sum()):>4}    {probs[m].mean():.3f}      "
          f"{yva.squeeze(1)[m].mean():.3f}")

# A calibrated model's predicted column matches its actual column. When they
# diverge, "90% confident" does not mean right 9 times in 10 -- and any
# decision threshold built on that number is on sand.

Where the predicted and actual columns diverge, the probability is not trustworthy as a probability. Over-confidence is the usual direction for modern networks, especially after long training.

This matters most when a threshold drives a decision. "Act when confidence exceeds 0.8" means something quite different on a miscalibrated model.

The mistake this prevents

Using raw network outputs as probabilities in a decision rule. They are scores that happen to lie between 0 and 1, and treating them as calibrated probabilities is an assumption you have not tested.

Takeaway

Check calibration before any threshold-based decision. Bucket predictions and compare against actual rates.

Mark complete