Unit 04.04: Scaling and encoding input features

Gradient descent takes the same-sized step in every direction. When one feature ranges to a million and another to fifty, that is a problem you can measure.

Why unscaled features stall training

Age spans roughly 18–68. Income spans 100,000–1,000,000. The gradient with respect to income is enormous compared with age, so a learning rate small enough to be stable for income barely moves age at all.

Standardising — subtract the mean, divide by the standard deviation — puts both on comparable footing:

import torch
from torch import nn

torch.manual_seed(0)
# Two features on wildly different scales: age in years, income in rupees.
age = torch.rand(300, 1) * 50 + 18
income = torch.rand(300, 1) * 900000 + 100000
X_raw = torch.cat([age, income], dim=1)
y = ((0.02 * age + 0.000002 * income) > 1.6).float()


def train(X, steps=300):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 1))
    opt = torch.optim.SGD(model.parameters(), lr=0.01)
    for _ in range(steps):
        opt.zero_grad()
        nn.BCEWithLogitsLoss()(model(X), y).backward()
        opt.step()
    with torch.no_grad():
        return ((torch.sigmoid(model(X)) > 0.5).float() == y).float().mean().item()


X_scaled = (X_raw - X_raw.mean(0)) / X_raw.std(0)
print("raw features   :", round(train(X_raw), 3))
print("scaled features:", round(train(X_scaled), 3))

# One-hot for unordered categories: encoding them as 0/1/2 invents an order
# the data does not have.
categories = torch.tensor([0, 2, 1, 0])
print("one-hot:\n", torch.nn.functional.one_hot(categories, num_classes=3))

The scaled version reaches a much better accuracy with identical architecture, optimiser and step count. Nothing changed except the input range.

The one-hot block covers the other half: encoding unordered categories as 0, 1, 2 tells the model that category 2 is somehow twice category 1, which is a relationship the data does not contain.

The mistake this prevents

Computing scaling statistics over the whole dataset before splitting. Validation data influences the transformation applied to training data, and the score is inflated by an amount you cannot measure.

Takeaway

Standardise numeric features using training statistics only, and one-hot anything unordered.

Mark complete