Unit 11.04: Fairness and dataset limitations

When one group is 10% of the data and follows a different pattern, the loss function has almost no incentive to fit it.

Two rates, not one

Group B here is both under-represented and governed by a genuinely different relationship. Training minimises average loss, and the average is dominated by group A.

Report accuracy *and* positive rate per group — they can diverge, and each reveals a different problem:

import torch
from torch import nn

torch.manual_seed(0)
# Group B is under-represented AND noisier, which is the usual real pattern.
n_a, n_b = 450, 50
Xa, Xb = torch.randn(n_a, 4), torch.randn(n_b, 4)
ya = ((Xa[:, 0] + Xa[:, 1]) > 0).float().unsqueeze(1)
yb = ((Xb[:, 0] - Xb[:, 1]) > 0).float().unsqueeze(1)     # different relationship
X = torch.cat([Xa, Xb]); y = torch.cat([ya, yb])
group = torch.cat([torch.zeros(n_a), torch.ones(n_b)])

model = nn.Sequential(nn.Linear(4, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(500):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(model(X), y).backward()
    opt.step()

with torch.no_grad():
    pred = (torch.sigmoid(model(X)) > 0.5).float()

print(f"overall accuracy: {(pred == y).float().mean().item():.3f}")
for g, name, n in [(0, "group A", n_a), (1, "group B", n_b)]:
    m = group == g
    acc = (pred[m] == y[m]).float().mean().item()
    rate = pred[m].mean().item()
    print(f"  {name}  n={n:>3}  accuracy {acc:.3f}  positive rate {rate:.3f}")

print("""
Group B is 10% of the data and the model largely ignores it: the overall number
is dominated by group A. Report per-group accuracy AND per-group positive rate,
state who is under-represented, and say plainly what the model should not be
used to decide for them.
""")

Overall accuracy looks acceptable and conceals a substantial gap between the groups. The positive rates differ too, which is a separate concern: even at equal accuracy, systematically different flagging rates have consequences.

Neither number is visible in the aggregate.

The mistake this prevents

Treating a fairness check as a post-hoc audit. If a group is 10% of the training data, no metric computed afterwards fixes that — the constraint was set when the data was collected.

Takeaway

Report accuracy and positive rate per group, name who is under-represented, and state what the model must not decide for them.

Mark complete