Unit 11.01: Measuring performance per group

An overall accuracy figure can hide a twenty-point gap between groups.

Per-group, with row counts and a stability verdict

Four groups with different sizes and different accuracies.

The code reports each.

import numpy as np

rng = np.random.default_rng(3)
GROUPS = {"day shift": (620, 0.93), "night shift": (140, 0.79),
          "phone model A": (700, 0.92), "phone model B": (60, 0.74)}

print(f"{'group':16} {'n':>5} {'accuracy':>9} {'stable?':>9}")
for name, (n, acc) in GROUPS.items():
    measured = acc + rng.normal(0, 0.005)
    stable = "yes" if n >= 100 else "NO -- too few"
    print(f"{name:16} {n:>5} {measured:>9.1%} {stable:>13}")

worst, best = 0.74, 0.93
print(f"\ngap between best and worst group: {best - worst:.0%}")
print("the overall figure hides a 19-point gap")

# Define the groups before looking at results. And report row counts -- a
# 74% accuracy over 60 images has a confidence interval wide enough to include
# both 60% and 85%.

The gap between the best and worst group is nineteen points, and the overall figure shows none of it. The worst group is also the smallest, which is the usual pattern and the reason the gap goes unnoticed.

Sixty images cannot support a stable accuracy estimate. Reporting it without the row count invites a comparison the data cannot bear.

The mistake this prevents

The mistake is choosing the groups after seeing the results. Define them from the collection metadata - shift, device, site, lighting - before measuring anything.

Takeaway

Report accuracy per group with row counts, and define the groups before measuring. The worst group is usually the smallest, which is why the gap hides.

Mark complete