Unit 02.04: Train, validation, and test splits for deep learning

Three splits, three different jobs. Confusing them is the fastest way to report a score that does not survive contact with new data.

What each split is for

Train fits the parameters. Validation chooses between models — architecture, learning rate, when to stop. Test is looked at once, at the end, to estimate performance on data nobody tuned against.

Split with a seeded generator so the division is reproducible:

import torch
from torch.utils.data import TensorDataset, random_split

torch.manual_seed(0)
data = TensorDataset(torch.randn(500, 4), torch.randint(0, 2, (500, 1)).float())

train, val, test = random_split(data, [350, 75, 75],
                               generator=torch.Generator().manual_seed(0))
print("train/val/test:", len(train), len(val), len(test))
print("no rows lost  :", len(train) + len(val) + len(test) == len(data))

# The splits must not overlap. Checking is cheap; assuming is expensive.
train_idx, val_idx, test_idx = set(train.indices), set(val.indices), set(test.indices)
print("train n val overlap:", len(train_idx & val_idx))    # 0
print("train n test overlap:", len(train_idx & test_idx))  # 0

# Seeded splits are reproducible -- rerun and you get the same rows.
again = random_split(data, [350, 75, 75], generator=torch.Generator().manual_seed(0))
print("split is reproducible:", set(again[0].indices) == train_idx)

# Test data is touched once, at the end. Every time you look at it to make a
# decision, it becomes validation data and stops measuring generalisation.

The overlap checks are the point. Two splits sharing even a few rows inflates the score, and nothing in the output warns you. Checking costs one line.

Re-running with the same seed reproduces the identical split, which means a reviewer can rebuild your exact experiment.

The mistake this prevents

Using the test set to decide anything — a learning rate, an architecture, when to stop. Every look turns it into validation data, and the final number stops measuring generalisation.

Takeaway

Fit on train, decide on validation, report on test once. Verify the splits do not overlap rather than assuming it.

Mark complete