Unit 12.02: Training against a baseline you must beat

The model has to be reported against the baselines, not against other models.

Three references, one of them free

Majority class, a trivial pixel statistic, and the fine-tuned model.

The code compares all three on accuracy and damage recall.

RESULTS = [
    ("majority class",           0.904, 0.00, "the bar to beat"),
    ("mean-brightness threshold", 0.812, 0.31, "free, and it finds some damage"),
    ("fine-tuned CNN",           0.911, 0.62, "beats both, at a cost"),
]
print(f"{'model':28} {'accuracy':>9} {'damage recall':>14}  note")
for name, acc, recall, note in RESULTS:
    print(f"{name:28} {acc:>9.1%} {recall:>14.0%}  {note}")

print("\nthe CNN beats the majority baseline on accuracy by 0.7 points")
print("and on damage recall -- the thing that matters -- by 62 points")

# Accuracy barely moved because accuracy was never the right metric here. The
# recall column is where the model earns its cost, and reporting only accuracy
# would have made this look like a failure.

Accuracy barely moved - 90.4% to 91.1% - because accuracy was never the right metric here. Damage recall went from 0% to 62%, and that is where the model earns its cost.

The brightness threshold is the useful middle reference: free, interpretable, and it already finds 31% of the damage. Any model has to beat that, not just the constant predictor.

The mistake this prevents

The mistake is reporting the headline accuracy improvement. Zero point seven points sounds like nothing and would have got the project cancelled, when the metric that matters improved by sixty-two.

Takeaway

Report against the majority baseline and a trivial baseline, on the metric that matters. Accuracy improvement can be near zero while the useful metric transforms.

Mark complete