Unit 05.00: Learning rate and optimizer choice

If a model "is not learning", the learning rate is the first thing to check. It fails in two opposite directions and both are visible immediately.

Too small crawls, too large diverges

The learning rate scales every parameter update. Too small and the model inches towards a solution it will not reach in the time you have. Too large and each step overshoots, the loss grows, and it eventually becomes NaN.

Four rates on identical data and architecture:

import torch
from torch import nn

torch.manual_seed(0)
X = torch.randn(300, 3)
y = (2 * X[:, 0] - X[:, 1]).unsqueeze(1)


def final_loss(lr, steps=200):
    torch.manual_seed(0)
    model = nn.Linear(3, 1)
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        nn.MSELoss()(model(X), y).backward()
        opt.step()
    return nn.MSELoss()(model(X), y).item()


for lr in (0.0001, 0.01, 0.1, 1.5):
    loss = final_loss(lr)
    verdict = "too slow" if lr <= 0.0001 else "diverged" if (loss != loss or loss > 100) else "works"
    print(f"lr={lr:<8} final loss {loss:>12.4f}   {verdict}")

# Learning rate is the first thing to tune and the most common cause of a
# model that "does not learn". Too small crawls; too large diverges to NaN.

At 0.0001 the loss barely moves — the model is learning, just far too slowly to be useful. At 1.5 it diverges outright. Between them is a wide band that works.

That band is usually wider than people expect, which is why order-of-magnitude search (0.1, 0.01, 0.001) beats careful tuning.

The mistake this prevents

Concluding the architecture is wrong when the learning rate is wrong. People rebuild models, add layers, and change losses to fix what one hyperparameter caused.

Takeaway

Change the learning rate by factors of ten before changing anything else.

Mark complete