Unit 06.04: Evaluating image classification

On an imbalanced dataset, accuracy can look respectable while the model completely ignores a class. The number does not warn you.

Where a single metric hides the failure

Here 80% of images are class 0. A model that predicts class 0 for everything scores 0.80 without learning anything at all.

The model below is a little better than that, but only a little — and the per-class breakdown is what exposes it:

import torch
from sklearn.metrics import classification_report, confusion_matrix

torch.manual_seed(0)
# An imbalanced 3-class problem: class 0 dominates.
true = torch.cat([torch.zeros(80), torch.ones(15), torch.full((5,), 2.0)]).long()
pred = true.clone()
pred[80:90] = 0            # the model calls most class-1 images class 0
pred[95:] = 0              # and misses class 2 entirely

accuracy = (pred == true).float().mean().item()
print(f"accuracy: {accuracy:.3f}  <- looks respectable")
print("but predicting class 0 for everything scores:",
      f"{(true == 0).float().mean().item():.3f}")

print("\nconfusion matrix (rows = true, cols = predicted):")
print(confusion_matrix(true.numpy(), pred.numpy()))
print("\n" + classification_report(true.numpy(), pred.numpy(),
                                    target_names=["cat", "dog", "fox"], zero_division=0))
# Class 2 has recall 0.00. Accuracy hid that completely, which is why an
# imbalanced problem needs per-class recall, not a single number.

Overall accuracy looks tolerable. Then read the confusion matrix rows: class 2 has recall 0.00. Every single example of it was misclassified, and accuracy never mentioned this.

Precision, recall and F1 per class are the minimum reporting standard for imbalanced problems.

The mistake this prevents

Reporting accuracy alone on imbalanced data. It is dominated by the majority class, and the classes you most likely care about are the rare ones.

Takeaway

Report per-class recall and the confusion matrix. Accuracy is a summary of the majority class.

Mark complete