Unit 05.01: Class imbalance you created by collecting

Class imbalance in a vision dataset is usually something you created by how you collected.

94% of one class

A realistic damage dataset: 940 undamaged, 60 damaged across three types.

The code shows the distribution and the resulting baseline.

from collections import Counter

COLLECTED = (["undamaged"] * 940 + ["crush"] * 38 + ["tear"] * 17 +
             ["wet"] * 5)
counts = Counter(COLLECTED)
total = len(COLLECTED)

print(f"{'class':12} {'count':>6} {'share':>7}")
for name, n in counts.most_common():
    print(f"{name:12} {n:>6} {n / total:>7.1%}")

majority = counts.most_common(1)[0][1] / total
print(f"\nalways predicting 'undamaged' scores {majority:.1%} accuracy")
print(f"and finds {0:.0%} of the damage, which is the entire point of the system")

# The imbalance is not a property of the world -- it is a property of how you
# collected. Photographing every carton gives you this; photographing every
# carton a human flagged gives you something completely different.

Always predicting the majority class scores 94% accuracy and finds none of the damage - which is the entire purpose of the system. Any accuracy figure on this data has to be read against that 94%.

The imbalance is a collection artefact. Photographing every carton gives you this; photographing every carton a human already flagged gives you a completely different distribution and a completely different problem.

The mistake this prevents

The mistake is fixing imbalance by resampling before understanding where it came from. If it reflects the real world, resampling changes what the model is calibrated for; if it reflects your collection process, the fix is to collect differently.

Takeaway

Report the majority-class baseline alongside every accuracy figure. And ask whether the imbalance is the world's or your collection process's - the remedies differ.

Mark complete