Unit 05.00: The label schema is the hard part

The label schema is the hard part of a vision dataset, and it is usually decided in a meeting that lasts ten minutes.

Every schema leaves something undecided

Four schemas, each with the question it does not answer.

The code lists them.

SCHEMAS = [
    ("damaged / not damaged", 2, "what counts as damaged? a scuff? a dent?"),
    ("damage type: crush, tear, wet, none", 4, "what about a crushed AND wet box?"),
    ("damage severity 1-5", 5, "two labellers will disagree on 3 vs 4"),
    ("damage present + bounding box", 2, "where does the box end on a gradual dent?"),
]
print(f"{'schema':40} {'classes':>8} the question it does not answer")
for schema, classes, question in SCHEMAS:
    print(f"{schema:40} {classes:>8} {question}")

print("""
Every schema leaves something undecided, and whatever it leaves undecided the
labellers will decide differently from each other.

Write the edge cases into the schema before labelling starts: one worked
example per class, plus three cases that look borderline and the ruling on
each.
""")

Whatever the schema leaves undecided, the labellers will decide differently from each other - and the disagreement appears as label noise that no model can see past.

The fix is to write the edge cases into the schema before labelling: one worked example per class, plus three genuinely borderline cases with the ruling on each.

The mistake this prevents

The mistake is refining the schema during labelling. The images labelled before the refinement now follow different rules from those after, and unless you re-label them the dataset contains two incompatible standards.

Takeaway

Write the edge-case rulings into the schema before any labelling starts. Whatever is undecided becomes label noise, and no model sees past it.

Mark complete