Unit 06.05: Error analysis for image models

A confusion matrix tells you *what* is being confused. Error analysis tells you *which images* to open — and that is what actually leads to a fix.

Sorting mistakes by confidence

Not all errors are equal. A wrong prediction the model was unsure about is ordinary noise. A wrong prediction it was *confident* about usually means something specific: a mislabelled image, a genuine gap in the training data, or a shortcut the model has learned.

Separate the two piles, then rank:

import torch
from torch import nn

torch.manual_seed(0)
model = nn.Sequential(nn.Flatten(), nn.Linear(64, 3))
images = torch.randn(40, 1, 8, 8)
labels = torch.randint(0, 3, (40,))

with torch.no_grad():
    probs = torch.softmax(model(images), dim=1)
confidence, predicted = probs.max(dim=1)
wrong = predicted != labels

print("errors:", int(wrong.sum()), "of", len(labels))

# The two piles worth separating: confident mistakes and unconfident ones.
confident_errors = wrong & (confidence > 0.5)
print("confident mistakes  :", int(confident_errors.sum()), "<- look at these first")
print("uncertain mistakes  :", int((wrong & (confidence <= 0.5)).sum()))

# Rank errors by confidence: the most confident wrong prediction is usually
# either a mislabelled image or a genuine gap in the training data.
if wrong.any():
    idx = torch.nonzero(wrong).flatten()
    ranked = idx[confidence[idx].argsort(descending=True)]
    print("\nworst offenders (index, predicted, true, confidence):")
    for i in ranked[:3]:
        print(f"  {i.item():>3}  {predicted[i].item()}  {labels[i].item()}  {confidence[i]:.3f}")
print("\nA confusion matrix says WHAT is wrong. This says WHICH images to open.")

The ranked list gives you a work queue. Open the most confident mistakes first — in real datasets a surprising share of them turn out to be labelling errors rather than model failures.

The split between confident and uncertain mistakes is also a signal in itself. Mostly uncertain errors suggest a hard problem; mostly confident ones suggest a data problem.

The mistake this prevents

Stopping at the confusion matrix. It says class 2 is confused with class 1, which does not tell you whether the cause is lighting, cropping, mislabelling, or something else entirely.

Takeaway

Rank errors by confidence and look at the images. The fix is usually visible within the first ten.

Mark complete