Unit 11.03: Robustness, distribution shift, and out-of-distribution inputs

Accuracy falls when inputs drift away from training data. Confidence does not — and that combination is what makes shift dangerous.

Testing outside the training range

Evaluate the same model on progressively more distant data: same distribution, shifted mean, larger spread, and far outside anything seen in training.

The accuracy column behaves as you would expect. The confidence column is the surprise:

import torch
from torch import nn

torch.manual_seed(0)
# Train on one distribution.
Xtr = torch.randn(500, 4)
ytr = ((Xtr[:, 0] + Xtr[:, 1]) > 0).float().unsqueeze(1)

model = nn.Sequential(nn.Linear(4, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(400):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(model(Xtr), ytr).backward()
    opt.step()


def accuracy(X):
    y = ((X[:, 0] + X[:, 1]) > 0).float().unsqueeze(1)
    with torch.no_grad():
        return round(((torch.sigmoid(model(X)) > 0.5).float() == y).float().mean().item(), 3)


print("same distribution      :", accuracy(torch.randn(200, 4)))
print("shifted (mean +2)      :", accuracy(torch.randn(200, 4) + 2))
print("scaled  (std x3)       :", accuracy(torch.randn(200, 4) * 3))
print("far out of distribution:", accuracy(torch.randn(200, 4) * 10 + 20))

# Confidence does not drop the way accuracy does -- that is the danger.
with torch.no_grad():
    near = torch.sigmoid(model(torch.randn(200, 4))).squeeze(1)
    far = torch.sigmoid(model(torch.randn(200, 4) * 10 + 20)).squeeze(1)
print(f"\nmean confidence in-distribution : {near.max(dim=0).values.item():.3f}")
print(f"mean confidence far out of it   : {far.max(dim=0).values.item():.3f}")
print("A model is confidently wrong outside its training range. Detect and")
print("refuse such inputs rather than trusting the probability.")

Accuracy degrades with distance from the training distribution, which is unsurprising. But confidence stays high — the model is *confidently* wrong far outside its range.

That is why out-of-distribution detection is a separate mechanism. You cannot use the model's own probability to decide whether to trust the model.

The mistake this prevents

Relying on low confidence to flag inputs the model cannot handle. On genuinely novel inputs, confidence is often higher than on hard in-distribution cases.

Takeaway

Test explicitly outside the training range, and detect out-of-distribution inputs with something other than the model's own confidence.

Mark complete