When fitting better means predicting worse

A model can become increasingly accurate at reproducing its training rows while becoming less useful on new rows. This is not a contradiction. Training asks an optimizer to reduce one finite collection of errors. Generalization asks whether the resulting rule survives another sample from the intended population. Those are related goals, not identical goals.

Suppose a sensor reading contains a stable pattern plus measurement noise. With enough flexibility, a network can fit both. A future measurement contains different noise, so copying the training noise is wasted effort. We call this overfitting. The evidence is not merely “the network has many parameters.” Look for a meaningful training–validation gap under comparable measurement conditions, and for validation performance worsening as training continues. A gap can also arise from distribution shift, different label quality or a bug. Diagnose before prescribing.

Compare comparable curves

Our training loop records two training losses. The online training loss is accumulated while parameters are changing and dropout may be active. The evaluation-mode training loss is measured after the epoch with one fixed parameter state and dropout disabled. Validation is measured the same way. Compare evaluation-mode training loss with validation loss when diagnosing fit. A lower validation loss than online training loss is not automatically leakage: the online calculation may have been deliberately harder.

Open the diagram at full size

Read the diagram explanation

Measured training and validation MSE over 300 epochs. Best validation at epoch 22, loss 0.5413. Final training loss 0.00979 and validation loss 3.1869. Axis clips high initial values at 4.

The chart is generated from actual course runs. Its horizontal axis is epoch, meaning a complete pass through the training rows. Its vertical axis is mean loss. The highlighted epoch is the lowest observed validation loss within the declared training budget. The curve after that epoch still matters: it shows why keeping the final weights can be a different decision from keeping the best validation checkpoint.

Run python -m dlcourse.diagnostics. The small-noisy-training case deliberately keeps only 24 training rows and adds noise to their targets. Validation retains the original process. Compare the wider network with a narrower regularized candidate. This is a diagnostic stress test, not a claim about all regression problems. The script honestly reports whether regularization wins here; do not rewrite the story to guarantee a preferred result.

Three different interventions

A smaller model restricts the set of functions that can be fitted. It may reduce overfitting, but it can also underfit a real nonlinear relationship. A linear model that cannot separate concentric rings needs a better representation, not merely more epochs.

Weight decay shrinks parameters during optimization. In AdamW, the decay is separate from the adaptive gradient calculation. With learning rate 0.01 and decay coefficient 0.1, the decay-only multiplier for a weight is 1 - 0.01*0.1 = 0.999 for that step. A weight of 2 becomes 1.998 before considering the gradient contribution. This is a small cumulative pressure, not a rule forcing all weights to zero. It is not interchangeable with every implementation of adding an L2 penalty to an adaptive optimizer. Our teaching model applies decay to all parameters for simplicity; advanced practice often uses parameter groups.

Dropout randomly suppresses intermediate values during training. With probability 0.25, a value is zeroed; retained values are multiplied by 1/0.75. For an input value 3, the training output is either 0 or 4. Its expectation is still 3. At evaluation, ordinary dropout is the identity operation. This is why model.eval() is part of the inference contract. Dropout is not a guarantee of better results and need not help a tiny clean dataset.

Checkpoint selection limits how long a fitted trajectory is allowed to keep adapting to training details. Our implementation trains the full budget and restores the best validation checkpoint; it does not stop computation early. A patience-based early-stopping loop would save computation, but the selection criterion must still be fixed without consulting test results.

Validation is a resource

Trying fifty combinations and selecting the best noisy validation score can itself overfit validation. The core course declares just two candidates per task and a fixed epoch budget. Report what was tried, including the losing baseline. For serious studies, use the cross-validation and nested-evaluation ideas from Applied ML, or reserve additional evaluation data appropriate to the deployment question. Do not “repair” a disappointing test score by choosing another checkpoint using that test score.

Exercises

• 7A · Dropout expectation. With dropout probability 0.4 and input value 6, list the possible training outputs and their probabilities. Compute their expected value and the evaluation output.
• 7B · Choose a checkpoint. Validation losses at epochs 10, 20, 30, 40 are 0.48, 0.31, 0.35, 0.39. Training losses keep falling. Which checkpoint does the declared minimum-loss rule choose? Why is “epoch 40 has lowest training loss” insufficient?
• 7C · A controlled experiment. Read the noisy-case configuration and final history rows. Write a four-sentence comparison naming sample size, candidate differences, best validation epoch and the limitation created by changing several regularization choices together.

Exit check: explain why no_grad() does not disable dropout, and distinguish best-checkpoint restoration from actual early stopping.