Unit 06.04: A baseline you must beat before any of this

Before any of this, there is a number you have to beat.

Three baselines, all cheap

Majority class, a trivial pixel statistic, and what a person does in two seconds.

The code computes the first and lists the others.

from collections import Counter

LABELS = ["undamaged"] * 940 + ["crush"] * 38 + ["tear"] * 17 + ["wet"] * 5
counts = Counter(LABELS)
n = len(LABELS)

majority = counts.most_common(1)[0]
print(f"majority-class baseline: always say {majority[0]!r} "
      f"-> {majority[1] / n:.1%} accuracy")
print(f"                          -> {0:.0%} recall on damage")

BASELINES = [
    ("majority class", f"{majority[1] / n:.1%} accuracy, 0% damage recall"),
    ("mean pixel brightness threshold", "cheap, and often surprisingly hard to beat"),
    ("a human glancing for 2 seconds", "the bar that matters commercially"),
]
print()
for name, note in BASELINES:
    print(f"  {name:34} {note}")

# Report the model against all three. A 94% accuracy that sounds impressive is
# below the majority-class baseline here, and would have been caught in one
# line before any training happened.

The majority-class baseline is 94% accuracy with zero damage recall. A model reporting 94% accuracy has achieved nothing, and it would sound impressive in a summary.

The human baseline is the one that decides whether the project is worth running. If a person glancing for two seconds is as good and the volume is low, the model is a cost with no benefit.

The mistake this prevents

The mistake is comparing model variants against each other without an absolute reference. Three architectures scoring 91%, 92% and 94% look like progress until you notice the constant predictor scores 94%.

Takeaway

Compute the majority-class baseline before training and report every model against it. Include a human baseline - it is what decides whether the project is worth doing.

Mark complete