Unit 12.01: Dataset card and split plan

The dataset card records what the data is. The split plan records how you will avoid fooling yourself with it.

Finding the repeated unit

600 rows sounds like 600 examples. If each learner appears three times, there are 200 independent units.

A random row split puts the same learner on both sides, and the model learns to recognise the learner rather than the pattern:

import json

import torch

torch.manual_seed(0)
n = 600
learner_id = torch.arange(n) % 200          # each learner appears ~3 times
week = torch.arange(n) % 3
X = torch.randn(n, 5)
y = (X[:, 0] > 0).float()

# Splitting by ROW would put the same learner in train and test.
naive_train, naive_test = set(learner_id[:480].tolist()), set(learner_id[480:].tolist())
overlap = len(naive_train & naive_test)

# Split by learner instead.
train_learners = set(range(160))
train_mask = torch.tensor([lid.item() in train_learners for lid in learner_id])

print(json.dumps({
    "rows": n,
    "unique_learners": int(learner_id.unique().numel()),
    "rows_per_learner": round(n / int(learner_id.unique().numel()), 1),
    "naive_row_split_leaks_learners": overlap,
    "grouped_split": {"train_rows": int(train_mask.sum()),
                      "test_rows": int((~train_mask).sum()),
                      "learner_overlap": 0},
    "class_balance": round(y.mean().item(), 3),
    "known_gaps": ["no learners who never started", "one cohort only"],
}, indent=2))
print("\nThe repeated unit here is the learner, so the split must be by learner.")

The naive_row_split_leaks_learners count is the number of learners appearing in both train and test. Grouping by learner reduces it to zero.

The card also records class balance and known gaps — here, no learners who never started, and only one cohort. Both limit what any result can claim.

The mistake this prevents

Documenting the split as "80/20 random" without stating what the row represents. The reader cannot tell whether it leaked, and neither can you six months later.

Takeaway

Identify the repeated unit, split on it, and record the gaps you know the data has.

Mark complete