Decide what the model is allowed to learn from

Outcome: create fixed, disjoint splits, fit preprocessing on training observations and inspect batches without losing input–target pairing. Prerequisite: Lessons 01–03 and the train/validation/test distinction from Applied Machine Learning.

Open the diagram at full size

Read the diagram explanation

Six hundred samples split into 360 training, 120 validation and 120 test rows. Preprocessing and weights fit only train, model selection uses validation, final test evaluates frozen choice.

A correct derivative can optimize a misleading experiment. If information from the eventual test observations influences preprocessing or model choice, the reported score answers an easier question than prediction on genuinely unseen data. Data handling belongs inside the learning method, rather than being housekeeping before it.

Fix the evaluation question first

The core course fixtures contain independently generated synthetic observations. We assign 60% of their row IDs to training, 20% to validation and 20% to test, then preserve those memberships. An ID identifies an observation independently of its current position in a batch. Fixed membership lets us compare models against the same evidence instead of benefiting from a newly convenient split.

Training observations supply the optimization loss and preprocessing estimates. Validation observations guide choices such as learning rate, hidden width and stopping epoch. The test set is reserved for one final evaluation of the selected procedure. Repeatedly changing a model after reading its test result makes the test set part of development, even if no test gradient was computed.

Random row splitting fits the independent-sample setup here. It would not automatically fit repeated readings from the same patient, device or customer. If the intended prediction concerns unseen groups, keep groups separate. For prediction into the future, use a time-aware holdout. “Random” does not by itself mean “representative of the deployment question.”

Standardization is a learned transformation

Suppose a training feature has the two values 1 and 3. Its mean is 2. Its population variance is [(1-2)² + (3-2)²] / 2 = 1, so its population standard deviation is 1. Standardization subtracts that training mean and divides by that training standard deviation. The two transformed values are -1 and 1.

A later observation of 5 becomes (5-2)/1 = 3. We do not recompute a new mean using that observation. Otherwise the transformation would change with the evaluation batch, and a future observation would help determine how the model interprets itself.

For each feature, let mu name the training mean and scale its training population standard deviation. The transformed feature is (value - mu) / scale. The population convention divides squared deviations by the number of training observations. This is a scaling choice; we are not claiming to produce an unbiased estimate of an unknown population variance. In PyTorch, unbiased=False expresses this convention; the current documented spelling is correction=0.

If a feature is constant during training, its scale is zero. Dividing by zero creates invalid values. The example clamps scale to at least 1e-8, matching the staged checker and reference implementation, so constant training values become zero. A future departure from that constant can become very large after scaling; flag it for review rather than treating the clamp as a general solution. Record that condition: a constant training feature cannot teach the model how changes in that feature relate to the target.

Make the contract visible in code

This self-contained demonstration generates 100 independent synthetic sensor observations. The target combines a linear effect, a curved effect and fresh noise. The exact formula is for learning the workflow; it is not a physical model of a real sensor.

import torch
from torch.utils.data import TensorDataset, DataLoader

data_rng = torch.Generator().manual_seed(140)
X = torch.randn(100, 1, generator=data_rng)
y = 1.5 * X + 0.4 * X.square()
y = y + 0.1 * torch.randn(100, 1, generator=data_rng)

split_rng = torch.Generator().manual_seed(41)
ids = torch.randperm(100, generator=split_rng)
train_ids, val_ids, test_ids = ids[:60], ids[60:80], ids[80:]
assert len(set(ids.tolist())) == 100
assert not (set(train_ids.tolist()) & set(val_ids.tolist()))
assert not (set(train_ids.tolist()) & set(test_ids.tolist()))
assert not (set(val_ids.tolist()) & set(test_ids.tolist()))

mu = X[train_ids].mean(dim=0)
scale = X[train_ids].std(dim=0, unbiased=False)
scale = scale.clamp_min(1e-8)
train_X = (X[train_ids] - mu) / scale
val_X = (X[val_ids] - mu) / scale
train_data = TensorDataset(train_X, y[train_ids], train_ids)
val_data = TensorDataset(val_X, y[val_ids], val_ids)
train_loader = DataLoader(
    train_data, batch_size=16, shuffle=True, num_workers=0,
    generator=torch.Generator().manual_seed(42),
)
for batch_X, batch_y, batch_ids in train_loader:
    assert batch_X.shape == batch_y.shape
    assert batch_X.shape[0] == batch_ids.shape[0]
    print(batch_X.shape, batch_ids.tolist())

TensorDataset retrieves matching positions from its tensors. Keeping IDs in the example makes pairing inspectable. DataLoader groups retrieved samples into batches and shuffles training order when requested. It does not independently shuffle features and targets. With 60 training samples and batch size 16, the batch sizes are 16, 16, 16 and 12 when drop_last remains false.

An epoch is one traversal of the training loader. Shuffling changes order between epochs, not split membership. We leave validation and test order unshuffled and retain every observation. The CPU examples use num_workers=0 to keep loading easy to inspect. A seed helps reproduce a particular configuration; it does not guarantee identical results across every PyTorch release or device.

Practice

• 04A · Audit membership: write checks that the three split-ID sets are pairwise disjoint, have sizes 60/20/20, and together contain exactly IDs 0 through 99. Deliberately insert a duplicate across splits and confirm a check fails.
• 04B · Catch a leaking scaler: training values are [2, 4, 6] and one validation value is 20. Calculate train-only mean and population standard deviation, then transform 20. Compare with a scaler fitted to all four values and explain the information leak.
• 04C · Count batches: for 35 training observations and batch size eight, predict the batch sizes with drop_last=False and True. State which setting you would use for complete evaluation and why.

Exit check

Print one batch with its IDs and show that every target remains paired with its original observation. Then change validation feature values in a copy of the dataset and confirm that the already-fitted training mean and scale remain unchanged.

Further reading: PyTorch 2.9 data loading, standard deviation, and reproducibility; scikit-learn's data-leakage discussion.