Unit 12.01: Building the dataset with its splits and audit

The dataset section is where most of the honest work in a vision project lives.

Schema rulings, grouped split, duplicate check, label audit

Counts, the edge-case rulings, how the split was made, and what the audit found.

The code prints the record.

import json

dataset = {
    "images": 1_040,
    "label_schema": {"classes": ["undamaged", "crush", "tear", "wet"],
                     "edge_cases_ruled": ["scuff = undamaged",
                                          "crushed AND wet = crush",
                                          "damage under tape = crush"]},
    "class_counts": {"undamaged": 940, "crush": 38, "tear": 17, "wet": 5},
    "split": {"by": "scene id, not by file", "train": 8, "val": 2, "test": 2},
    "near_duplicate_check": "cosine > 0.95 across all pairs, before splitting",
    "label_audit": {"sampled": 60, "agreement": 0.67,
                    "action": "re-labelled with the edge-case rules, re-audited"},
    "excluded": {"faces visible": 12, "EXIF stripped": "all"},
}
print(json.dumps(dataset, indent=2))

n = sum(dataset["class_counts"].values())
print(f"\nmajority baseline: {dataset['class_counts']['undamaged'] / n:.1%}")
print(f"'wet' has 5 examples -- report it, do not pretend to measure it")

The label audit found 67% agreement, and the record says what was done about it: re-labelled with the edge-case rules, then re-audited. That sequence is the work, and it happened before any training.

'wet' has five examples. The record says to report it rather than pretend to measure it - five examples cannot support any performance claim.

The mistake this prevents

The mistake is reporting dataset size and class counts and stopping. The split method, the duplicate check and the label audit are what determine whether the numbers downstream mean anything.

Takeaway

Record the schema rulings, the split method, the duplicate check and the label audit. A class with five examples is reported, not measured.

Mark complete