Unit 07.00: Accuracy on imbalanced classes tells you nothing

On an imbalanced dataset, accuracy reports the class balance rather than the performance.

A useless predictor beating a real model

Always predicting the majority class, against a model that finds half the damage.

The code scores both on accuracy and on recall.

import numpy as np

truth = np.array([0] * 940 + [1] * 60)
always_zero = np.zeros_like(truth)
model = truth.copy()
model[:20] = 1                              # 20 false positives
model[940:970] = 0                          # 30 missed damages

for name, pred in [("always 'undamaged'", always_zero), ("the model", model)]:
    accuracy = (pred == truth).mean()
    recall = (pred[truth == 1] == 1).mean()
    print(f"{name:20} accuracy {accuracy:.1%}   damage recall {recall:.1%}")

print("\nthe useless predictor beats a model that finds half the damage")

# Accuracy is dominated by the majority class. On a 94%-negative dataset it
# reports the class balance, not the performance -- and it reports it as a
# number that sounds good.

The constant predictor wins on accuracy and finds nothing. The model finds half the damage - the entire purpose of the system - and scores worse on the headline metric.

That is not a subtlety of metric choice. It means an accuracy figure on this dataset carries no information about whether the model is useful.

The mistake this prevents

The mistake is reporting accuracy because it is the metric everyone understands. On a 94%-negative dataset it is understood to mean something it does not, which is worse than reporting nothing.

Takeaway

Accuracy on imbalanced data reports the class balance. Report recall on the class you care about, always alongside the majority-class baseline.

Mark complete