Unit 07.04: Slices where the model is quietly worse

An overall accuracy figure is mostly a report on your largest slice.

Per-group, with row counts

Four slices with different sizes and different accuracies.

The code reports each with a stability note.

import numpy as np

rng = np.random.default_rng(2)
SLICES = {"bright": 620, "dim": 280, "wet surface": 60, "night": 40}
print(f"{'slice':14} {'n':>5} {'accuracy':>9} {'note':>28}")
for name, n in SLICES.items():
    base = {"bright": 0.94, "dim": 0.88, "wet surface": 0.71, "night": 0.62}[name]
    acc = base + rng.normal(0, 0.01)
    note = "too small to be stable" if n < 100 else ""
    print(f"{name:14} {n:>5} {acc:>9.1%} {note:>28}")

overall = sum(n for n in SLICES.values())
print(f"\noverall n={overall}, dominated by 'bright' ({620 / overall:.0%} of it)")
print("the overall figure is mostly a report on the largest slice")

# Define the slices before you look at results, or you will find whichever
# split makes the number look best. And report row counts: 40 night images
# cannot support a stable accuracy figure however interesting it looks.

The overall figure is dominated by the 'bright' slice, which is 62% of the data. The night slice, at 40 images, has an accuracy figure that cannot be trusted to within several points in either direction.

Reporting the row count next to every slice metric is what makes that visible. A 62% accuracy over 40 images and over 4,000 images are very different claims.

The mistake this prevents

The mistake is defining the slices after seeing the results. You will find whichever split makes the number look best, and the analysis becomes a search for a favourable framing rather than a check.

Takeaway

Define slices before looking at results, report them with row counts, and treat small slices as unmeasured rather than as measured badly.

Mark complete