Debug the learning process before enlarging the model

A falling loss is reassuring only after you know what loss was actually computed. A shape bug can optimize the wrong comparison; leakage can make a weak model look excellent; a disconnected graph can leave parameters unchanged. Practical deep learning starts with small checks that separate these causes. A larger network usually makes a broken experiment harder to understand.

Begin with a contract

Print one batch's feature shape, target shape, dtype and finite-value range. For the rings classifier, features are [batch, 2], logits and floating targets are [batch, 1], and targets contain only 0 or 1. For bars, features are [batch, 1, 8, 8], logits are [batch, 2] and integer class indices are [batch]. The same word “classification” does not imply interchangeable shapes.

Now check the first prediction and loss. At binary logit zero, sigmoid is 0.5 and binary cross-entropy is approximately 0.693147 for either class. At two equal class logits, categorical cross-entropy is also approximately 0.693147. This gives a useful scale for an initially uncertain classifier. It is not a required initial value: random weights may produce unequal logits.

The diagnostic script deliberately subtracts predictions [3,1] from targets [3]. The result is a [3,3] matrix of all pairwise differences. Predictions 1, 2, 3 perfectly match their own targets, yet the wrong broadcasted MSE is 4/3 instead of zero. Broadcasting is useful when intended, but a valid tensor operation is not proof of valid statistics. Shape assertions are cheap insurance.

Follow one update

Record a parameter before the training step and after optimizer.step(). If it never changes, inspect whether it belongs to the optimizer, whether the loss depends on it, whether gradients exist, and whether training accidentally runs inside no_grad(). A parameter can legitimately have zero gradient in one batch, especially behind an inactive ReLU, so inspect a few meaningful examples rather than declaring a global bug from one zero.

Next inspect gradient clearing. For L = w² at w = 1, one backward pass adds gradient 2. A second backward pass on a newly computed loss, without clearing gradients, adds another 2, giving 4. This is accumulation, not a recalculation that replaces the old gradient. Accumulation can be deliberate for micro-batches, but then the scaling and update schedule need an explicit design. Our simple loop clears gradients before every batch.

Make the problem intentionally easy

Try fitting a handful of clean training examples. If a sufficiently flexible model cannot reduce their loss, investigate implementation before generalization. Check that labels correspond to the correct rows, the loss matches the task, the output activation is correct, and the optimizer sees the parameters. Passing this test only establishes learning mechanics. It says nothing about held-out performance.

The five-point examples/03_miniloop.py is such a mechanics test. Its target is exactly 2*x + 1. Its fitted weight should be close to 2 and bias close to 1. We deliberately avoid calling its training fit a test result.

Recognize unstable updates

For the toy objective L = 0.5*w², the derivative is w. One SGD step gives w_new = (1 - learning_rate)*w. A learning rate of 0.5 halves the magnitude every step. A learning rate of 2.2 changes its sign and multiplies its magnitude by 1.2, so the squared loss grows by 1.44 each step. Run the large-step diagnostic and verify that pattern. This exact threshold belongs to this particular quadratic; do not apply “learning rate below 2 is safe” to arbitrary neural networks.

When a real loss becomes non-finite, inspect inputs, target scaling, loss usage and learning rate. Shrinking the learning rate may help, but can also hide a shape or preprocessing error. Gradient clipping limits update-driving gradient magnitudes in some settings; it is not included in the core implementation because the first task is to identify the cause.

Metrics tell different stories

The rings report includes true negatives, false positives, false negatives and true positives. At the fixed probability threshold 0.5, predicted class 1 means logit at least zero. Precision asks what fraction of predicted positives are correct; recall asks what fraction of actual positives are found. The report uses zero when a denominator is zero and documents that convention. Accuracy alone can conceal a model that predicts only a majority class. These synthetic classes are approximately balanced within random splits, but that is not an assumption for real work.

Inspect errors against input location or image orientation on validation first. Do not use the test set as an interactive debugging dashboard. A surprisingly perfect synthetic result can simply mean the toy task is easy. It does not demonstrate safety, fairness, calibration, real image recognition or readiness for deployment.

Exercises

• 8A · Broadcast trap. Write the 3-by-3 difference matrix for predictions and targets 1, 2, 3. Calculate its MSE and explain why a diagonal-only comparison is intended.
• 8B · Update or no update? Name three distinct causes of unchanged parameters and a concrete observation that distinguishes each.
• 8C · Confusion counts. For TN=60, FP=10, FN=20, TP=30, calculate accuracy, precision and recall. What changes if the cost of false negatives rises?

Exit check: present a debugging order that starts with data/shape checks and reaches model capacity only after the basic update is verified.