Unit 12.03: Training evidence and diagnostics

The evidence that the training run was sound, kept as you go rather than reconstructed afterwards.

What to log every epoch

Training loss, validation loss, and the gradient norm. Together they distinguish a clean run from one that diverged and partially recovered — which a final-epoch summary cannot.

epochs_trained_past_best is the number that most often surprises people:

import json

import torch
from torch import nn

torch.manual_seed(0)
X = torch.randn(600, 5)
y = ((X[:, 0] + X[:, 1]) > 0).float().unsqueeze(1)
Xtr, ytr, Xva, yva = X[:450], y[:450], X[450:], y[450:]

model = nn.Sequential(nn.Linear(5, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)

history, best = [], {"val_loss": float("inf"), "epoch": -1}
for epoch in range(250):
    opt.zero_grad()
    train_loss = nn.BCEWithLogitsLoss()(model(Xtr), ytr)
    train_loss.backward()
    grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1e9).item()
    opt.step()
    with torch.no_grad():
        val_loss = nn.BCEWithLogitsLoss()(model(Xva), yva).item()
    history.append({"epoch": epoch, "train": round(train_loss.item(), 4),
                    "val": round(val_loss, 4), "grad_norm": round(grad_norm, 3)})
    if val_loss < best["val_loss"]:
        best = {"val_loss": round(val_loss, 4), "epoch": epoch}

print(json.dumps({
    "seed": 0, "epochs": len(history),
    "first": history[0], "last": history[-1], "best_validation": best,
    "epochs_trained_past_best": len(history) - 1 - best["epoch"],
    "diverged": any(h["val"] != h["val"] for h in history),
    "evidence_kept": ["per-epoch train and val loss", "gradient norms", "seed"],
}, indent=2))
print("\nIf best validation is far from the last epoch, you trained too long.")

If best validation occurred at epoch 60 of 250, you spent 190 epochs making the model worse and, without a checkpoint, threw the good one away.

The diverged check tests for NaN explicitly, because NaN compares false against every threshold and slips silently through a naive minimum check.

The mistake this prevents

Reporting only the final epoch's numbers. A run that spiked at epoch 150 and settled by 250 looks identical to one that trained smoothly, and the two deserve very different trust.

Takeaway

Log every epoch. The trajectory is evidence; the final number alone is not.

Mark complete