Why Neural Networks Need More Data, Compute, and Care

A network with enough parameters can fit anything, including data with no pattern in it at all. That is not a hypothetical risk. You can demonstrate it in twenty lines.

Capacity without data is memorisation

Here the targets are pure random noise. There is nothing to learn — no function maps these inputs to these outputs. A model that reports a low training loss on this data has memorised 40 random numbers, not discovered anything.

Watch what happens as the hidden layer grows:

import torch
from torch import nn

torch.manual_seed(0)

# The cost of capacity: a bigger model fits training data it should not.
X = torch.randn(40, 4)
y = torch.randn(40, 1)          # pure noise: there is nothing to learn

def fit(width, steps=800):
    model = nn.Sequential(nn.Linear(4, width), nn.ReLU(), nn.Linear(width, 1))
    opt = torch.optim.Adam(model.parameters(), lr=0.02)
    for _ in range(steps):
        opt.zero_grad()
        nn.MSELoss()(model(X), y).backward()
        opt.step()
    return nn.MSELoss()(model(X), y).item(), sum(p.numel() for p in model.parameters())

for width in (2, 64, 512):
    loss, params = fit(width)
    print(f"width {width:>3}: {params:>6} parameters, training loss {loss:.4f}")

# Loss on pure noise falls towards zero as capacity grows. The model is
# memorising 40 random numbers. More parameters need more data, or the
# training score stops meaning anything.

Training loss falls towards zero as width increases, on data that contains no signal whatsoever. The 512-unit model has more parameters than it has training examples, so it can simply store them.

The mistake this prevents

Reporting training loss as evidence that a model works. Training loss measures how well the model memorised what it already saw. Only a held-out score says anything about new data — and on this dataset, no held-out score could ever be good.

Takeaway

More parameters demand more data. When capacity outruns your dataset, the training score stops being evidence of anything.

Mark complete