Unit 03.06: Project step: train a tiny model end to end

Everything in this module, assembled: a dataset, a loader, a model, a loop, and — the part most write-ups omit — a baseline to compare against.

A problem a linear model cannot solve

Points inside a circle are not linearly separable, so the hidden layer has real work to do. The loop is the one you built in Unit 04.04, now driven by a DataLoader with shuffling and mini-batches.

Note model.train() and model.eval(): they matter as soon as dropout or batch normalisation is present, and getting into the habit now costs nothing.

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(0)

# Binary classification: is the point inside the unit circle?
X = torch.rand(600, 2) * 2 - 1
y = ((X ** 2).sum(dim=1) < 0.6).float().unsqueeze(1)
train_ds = TensorDataset(X[:480], y[:480])
Xva, yva = X[480:], y[480:]
loader = DataLoader(train_ds, batch_size=32, shuffle=True)

# A circle is not linearly separable, so the hidden layer is doing real work.
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.BCEWithLogitsLoss()

for epoch in range(30):
    model.train()
    for xb, yb in loader:
        optimizer.zero_grad()
        loss_fn(model(xb), yb).backward()
        optimizer.step()

model.eval()
with torch.no_grad():
    logits = model(Xva)
    accuracy = ((torch.sigmoid(logits) > 0.5).float() == yva).float().mean()

baseline = max(yva.mean().item(), 1 - yva.mean().item())
print(f"validation accuracy: {accuracy.item():.3f}")
print(f"majority baseline  : {baseline:.3f}")
print("beat the baseline  :", accuracy.item() > baseline)
# Always report the baseline next to the accuracy. A number on its own says
# nothing about whether the model learned anything.

Accuracy lands around 0.94 against a majority baseline of roughly 0.55. Both numbers are printed, and that pairing is the point: an accuracy figure alone says nothing about whether the model learned anything.

The mistake this prevents

Reporting accuracy with no baseline. On an imbalanced problem, a model that always predicts the majority class can score 0.9 and be worthless. The comparison is what makes the number meaningful.

Takeaway

Always print the baseline next to the score. A result without a comparison is not yet a result.

Mark complete