Unit 07.03: Small-data risks and data leakage

Transfer learning is used precisely when data is scarce — which is exactly when leakage is easiest to introduce and hardest to notice.

The repeated unit is not always the row

Forty photographs sounds like forty independent examples. If they are four shots each of ten people, there are ten independent units, not forty.

Split randomly and photographs of the same person land on both sides. The model recognises the person, not the property you asked about, and the test score rewards it:

import torch
from torch import nn

torch.manual_seed(0)

# 40 photographs, but only 10 subjects -- four shots of each.
subject = torch.arange(40) % 10
X = torch.randn(40, 6) + subject.unsqueeze(1).float() * 0.9   # subject identity leaks in
y = (subject % 2 == 0).float().unsqueeze(1)

def score(train_idx, test_idx):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(6, 16), nn.ReLU(), nn.Linear(16, 1))
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(300):
        opt.zero_grad()
        nn.BCEWithLogitsLoss()(model(X[train_idx]), y[train_idx]).backward()
        opt.step()
    with torch.no_grad():
        return ((torch.sigmoid(model(X[test_idx])) > 0.5).float() == y[test_idx]).float().mean().item()

# Wrong: a random split puts photos of the SAME subject on both sides.
perm = torch.randperm(40)
print("random split       :", round(score(perm[:30], perm[30:]), 3))

# Right: split by subject, so no subject appears in both.
train_mask = subject < 7
print("grouped by subject :", round(score(torch.nonzero(train_mask).flatten(),
                                          torch.nonzero(~train_mask).flatten()), 3))

# The random split scores higher and means nothing. With small data, ask what
# the repeated unit is -- patient, subject, session -- and split on that.

The random split scores markedly higher than the grouped split — and it is the *lower* number that is honest. The random split measured the model's ability to recognise faces it had already seen.

Ask what the repeated unit is: patient, subject, session, document, site. Split on that.

The mistake this prevents

Trusting a good score from a random split on small data. The inflated number is convincing, survives review, and collapses in production.

Takeaway

Identify the repeated unit before splitting. On small data, group-aware splitting is usually the difference between a real result and a fictional one.

Mark complete