Unit 05.04: Auditing a sample of labels by hand

The agreement rate between two labellers is a ceiling on what any model can achieve.

Hand-check a sample before training

Sixty images re-checked by a second labeller.

The code shows a sample and the agreement rate.

SAMPLE = [
    ("img-004", "crush", "crush", True),
    ("img-018", "undamaged", "crush", False),
    ("img-023", "tear", "tear", True),
    ("img-041", "crush", "crush", True),
    ("img-055", "undamaged", "undamaged", True),
    ("img-072", "wet", "undamaged", False),
]
agree = sum(1 for *_, ok in SAMPLE if ok)
print(f"{'image':10} {'label':12} {'re-check':12} agree")
for name, label, recheck, ok in SAMPLE:
    print(f"{name:10} {label:12} {recheck:12} {ok}")

rate = agree / len(SAMPLE)
print(f"\nagreement {agree}/{len(SAMPLE)} = {rate:.0%}")
print(f"a model cannot exceed this ceiling on data labelled this way")

# Hand-check a sample before training anything. A 67% agreement rate means the
# labels disagree with each other a third of the time, and no architecture
# choice fixes that.

Two thirds agreement means the labels disagree with each other a third of the time. A model trained on this cannot exceed that ceiling, and any architecture comparison run on it is measuring noise.

It is also the cheapest possible diagnostic: sixty images, one afternoon, before any training happens.

The mistake this prevents

The mistake is treating a disappointing model score as a modelling problem. Check label agreement first - if it is 67%, the labels are the constraint, and no amount of tuning addresses it.

Takeaway

Measure inter-labeller agreement on a sample before training. It is a hard ceiling on model performance and takes an afternoon to establish.

Mark complete