Unit 05.02: Splitting so the same scene is not on both sides

Three photographs of one carton are three views of one object, and a random split puts them on both sides.

Split by scene, not by file

Grouping by scene keeps every view of an object on one side of the split.

The code compares a naive split with a grouped one.

IMAGES = [
    ("scene-01_a.jpg", "scene-01"), ("scene-01_b.jpg", "scene-01"),
    ("scene-01_c.jpg", "scene-01"), ("scene-02_a.jpg", "scene-02"),
    ("scene-02_b.jpg", "scene-02"), ("scene-03_a.jpg", "scene-03"),
]
naive_train = [f for f, _ in IMAGES[:4]]
naive_test = [f for f, _ in IMAGES[4:]]
overlap = {g for f, g in IMAGES if f in naive_train} & \
          {g for f, g in IMAGES if f in naive_test}
print("naive random split")
print(f"  scenes on both sides: {sorted(overlap)}  <- leakage")

groups = sorted({g for _, g in IMAGES})
train_groups, test_groups = set(groups[:2]), set(groups[2:])
grouped_train = [f for f, g in IMAGES if g in train_groups]
grouped_test = [f for f, g in IMAGES if g in test_groups]
print("\ngrouped split")
print(f"  train scenes: {sorted(train_groups)}  test scenes: {sorted(test_groups)}")
print(f"  overlap: {train_groups & test_groups or 'none'}")

# Three photos of one carton are three views of the same object. Split by
# scene, not by file, or the test set is measuring memorisation.

The naive split has scenes on both sides. The test set then contains objects the model saw during training from a slightly different angle, and the score measures memorisation rather than generalisation.

The inflation can be large - several points on a small dataset - and it survives every other methodological precaution you take.

The mistake this prevents

The mistake is splitting by file because the files look independent. Independence is a property of the underlying object, not the filename, and burst photography, video frames and multi-angle capture all break it.

Takeaway

Split by the underlying object or scene, never by file. Multiple views of one object on both sides of a split measure memorisation.

Mark complete