Unit 09.04: Error analysis and bias checks for text models

An overall accuracy averages a working slice with a broken one and reports something in between.

Split the metric before you trust it

The failure injected here affects only the minority group. Overall accuracy absorbs it, because the majority group dominates the average.

The fix is not clever — it is simply computing the metric per slice:

import torch
from sklearn.metrics import confusion_matrix

torch.manual_seed(0)
# Predictions carrying a group label, so performance can be split by slice.
n = 200
group = torch.randint(0, 2, (n,))          # 0 = majority, 1 = minority
true = torch.randint(0, 2, (n,))
pred = true.clone()
# Inject a failure that only affects the minority slice.
flip = (group == 1) & (torch.rand(n) < 0.4)
pred[flip] = 1 - pred[flip]

overall = (pred == true).float().mean().item()
print(f"overall accuracy: {overall:.3f}  <- looks acceptable")

print("\nby slice:")
for g, name in [(0, "majority"), (1, "minority")]:
    m = group == g
    acc = (pred[m] == true[m]).float().mean().item()
    print(f"  {name:9} n={int(m.sum()):>3}  accuracy {acc:.3f}")

print("\nminority confusion matrix:")
m = group == 1
print(confusion_matrix(true[m].numpy(), pred[m].numpy()))

# The overall number averaged a working slice with a broken one. Always report
# per-slice performance for any group the model could plausibly treat
# differently -- and decide the slices before you look at the results.

The two slices differ substantially, and the minority confusion matrix shows the specific error pattern.

Decide the slices *before* looking at results. Choosing them afterwards means you are selecting the split that tells the story you want, which is not a measurement.

The mistake this prevents

Reporting a single overall figure for a model that will be applied to identifiable groups. The number is true and it conceals exactly the thing a reviewer needs to know.

Takeaway

Define slices up front, report per-slice metrics, and treat the worst slice as the honest headline.

Mark complete