Evidence Habits for Deep Learning

A training run produces a number. What makes it usable by someone else is everything recorded alongside it.

What to record, and why each line earns its place

The seed makes the run repeatable. The architecture and parameter count make the capacity claim checkable. The row counts make the score interpretable. The limitation tells a reader what the number cannot support.

Here the targets are deliberately random, so the honest finding is that validation loss should *not* improve:

import json

import torch
from torch import nn

torch.manual_seed(0)
X, y = torch.randn(120, 4), torch.randn(120, 1)
Xtr, ytr, Xva, yva = X[:90], y[:90], X[90:], y[90:]

model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)

for _ in range(300):
    opt.zero_grad()
    nn.MSELoss()(model(Xtr), ytr).backward()
    opt.step()

with torch.no_grad():
    train_loss = nn.MSELoss()(model(Xtr), ytr).item()
    val_loss = nn.MSELoss()(model(Xva), yva).item()

# The record you keep, not just the number you liked.
record = {
    "seed": 0,
    "architecture": "4-16-1 ReLU",
    "parameters": sum(p.numel() for p in model.parameters()),
    "train_rows": len(Xtr), "val_rows": len(Xva),
    "train_loss": round(train_loss, 4), "val_loss": round(val_loss, 4),
    "limitation": "targets are random noise, so validation loss should NOT improve",
}
print(json.dumps(record, indent=2))
print("\noverfitting gap:", round(val_loss - train_loss, 4))

The overfitting gap — validation loss minus training loss — is the number to watch. On random targets it grows, because the model memorises the training rows and learns nothing that transfers.

The mistake this prevents

Recording only the metric you liked. A result with no seed, no row counts, and no limitation cannot be checked or reproduced, which means a reviewer has to take it on trust. Most will not.

Takeaway

Report the number with its seed, its population, and its limitation. That is what makes it evidence rather than a claim.

Mark complete