Unit 11.00: Learning curves, confusion matrices, and error slices

Three diagnostics, each answering a question a single accuracy number cannot.

Would more data help?

Train the same model on increasing amounts of data and watch the train/validation gap. A closing gap means more data will keep helping. A stable gap with flat validation means the model, not the data, is the constraint.

That distinction decides whether your next week goes into collection or architecture:

import torch
from torch import nn
from sklearn.metrics import confusion_matrix

torch.manual_seed(0)
X = torch.randn(600, 6)
y = ((X[:, 0] + X[:, 1]) > 0).float().unsqueeze(1)


def train_with(n):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(6, 32), nn.ReLU(), nn.Linear(32, 1))
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    for _ in range(300):
        opt.zero_grad()
        nn.BCEWithLogitsLoss()(model(X[:n]), y[:n]).backward()
        opt.step()
    with torch.no_grad():
        tr = ((torch.sigmoid(model(X[:n])) > 0.5).float() == y[:n]).float().mean().item()
        va = ((torch.sigmoid(model(X[500:])) > 0.5).float() == y[500:]).float().mean().item()
    return round(tr, 3), round(va, 3), model


print(f"{'n':>5} {'train':>7} {'val':>7} {'gap':>7}")
for n in (25, 50, 100, 200, 400):
    tr, va, model = train_with(n)
    print(f"{n:>5} {tr:>7} {va:>7} {tr - va:>7.3f}")

with torch.no_grad():
    preds = (torch.sigmoid(model(X[500:])) > 0.5).float()
print("\nconfusion matrix:\n", confusion_matrix(y[500:].numpy().ravel(), preds.numpy().ravel()))
print("\nA closing gap as n grows means more data will help. A flat validation")
print("curve with a large gap means the model, not the data, is the limit.")

The gap column tells the story directly. At small n it is wide — the model memorises easily. As n grows it narrows and validation improves.

The confusion matrix then answers a different question: not how much error there is, but which kind.

The mistake this prevents

Reading a learning curve from training loss alone. It falls with more data in almost every case, including when validation is getting worse.

Takeaway

Learning curve for whether more data helps; confusion matrix for which errors you have.

Mark complete