Unit 12.03: Evaluating per slice, not just overall

The deployment decision is per slice, not overall.

Recall by group, with a usability verdict

Four slices with accuracy, damage recall and whether the slice is large enough to judge.

The code reports each.

SLICES = {"day shift": (620, 0.93, 0.66), "night shift": (140, 0.79, 0.41),
          "phone A": (700, 0.92, 0.64), "phone B": (60, 0.74, 0.29)}
print(f"{'slice':14} {'n':>5} {'accuracy':>9} {'damage recall':>14} {'usable?':>10}")
for name, (n, acc, rec) in SLICES.items():
    usable = "yes" if n >= 100 and rec >= 0.5 else "NO"
    print(f"{name:14} {n:>5} {acc:>9.1%} {rec:>14.0%} {usable:>10}")

print("\nthe model finds 41% of night-shift damage and 66% of day-shift damage")
print("deploying it on the night shift means missing three in five")

# Overall recall of 62% is an average over slices that differ by 25 points. The
# deployment decision is per slice, and for two of these the honest answer is
# not yet.

Overall recall of 62% averages over slices differing by twenty-five points. On the night shift the model finds 41% of damage - it misses three in five - and deploying it there would be a decision nobody consciously made.

Phone model B has 60 images and 29% recall. The honest statement is that it is unmeasured, not that it performs badly.

The mistake this prevents

The mistake is deploying on an overall figure. The overall figure is a weighted average dominated by the largest slice, and the slices where it is weakest are usually the ones with the fewest images.

Takeaway

Make the deployment decision per slice. An overall recall figure averages over groups that differ enough to change the answer.

Mark complete