Follow a prediction into its loss

Outcome: trace a forward pass and choose a loss whose inputs match the prediction task. Prerequisite: Lesson 01.

A forward pass is the computation that transforms input into a prediction using the current parameters. Nothing is learned merely because a forward pass happened. Learning requires measuring an error, finding how parameters affect that error and changing them. This lesson establishes the first two parts without hiding their meanings behind a training loop.

A small network with visible arithmetic

Let x = 2 be one input, w = 0.5 its first weight, and b = 0.1 the first bias. The first weighted sum is z = w×x + b = 1.1. The letter z names the value before activation.

Apply the rectified linear unit, or ReLU: h = max(0, z). Here max chooses the larger argument, so h = 1.1. The letter h names the hidden activation. With an output weight v = -1 and output bias c = 0.2, the prediction is yhat = v×h + c = -0.9. We write yhat for a prediction and y for the observed target.

ReLU leaves positive values unchanged and replaces negative values with zero. That change allows a network to behave differently in different input regions. Without an intervening nonlinear operation, a sequence of affine layers can be combined into one affine transformation. Adding layers alone would then add parameters without creating the nonlinear function family we need.

No activation is placed after this regression output: a continuous prediction may need to be negative. Output design follows the task. An attractive-looking diagram with the same activation everywhere is insufficient justification.

Squared error measures distance in target units

Suppose the observed target is y = 0.5. The prediction error is yhat - y = -1.4. For hand differentiation, we use L = 0.5×(yhat - y)² = 0.98, where L names the loss and the superscript two means square. The factor 0.5 simplifies the derivative in Lesson 03.

PyTorch's default MSELoss uses the mean of squared errors without that factor. For this single observation it returns 1.96. For predictions [1, 4] and targets [2, 2], the squared errors are 1 and 4, so their mean is 2.5. The mean is over all output elements. If a sample has several predicted values, those elements also enter the average.

For one-output regression, our contract is floating-point predictions (B, 1) and floating-point targets (B, 1). B is batch size. Matching those shapes prevents the cross-observation broadcasting mistake from Lesson 01. Squaring penalizes a two-unit error four times as much as a one-unit error; large errors therefore have substantial influence.

Binary classification starts with one logit

For two classes, we can produce one unrestricted real number per observation, called a logit. Sigmoid converts a logit a into p = 1 / (1 + exp(-a)), where p is the modeled probability of class 1 and exp is the exponential function. A logit of zero gives probability 0.5.

BCEWithLogitsLoss receives the raw logits and includes the needed probability transformation in a numerically stable computation. Do not apply sigmoid first. Our binary contract uses logits (B, 1) and floating-point labels of the same shape, with values zero or one. The loss also supports targets between zero and one, but this course's binary fixtures use hard labels.

For a class-1 observation, binary cross-entropy is -log(p), where log means natural logarithm. Probability 0.5 gives about 0.6931; probability 0.75 gives about 0.2877. Greater confidence in the correct class lowers this loss. A confidently incorrect prediction receives a larger penalty. Accuracy cannot express that distinction because it only counts the selected labels.

Several exclusive classes need several logits

For one of C mutually exclusive classes, emit C logits per observation. C means the number of classes. Use CrossEntropyLoss with logits (B, C) and integer class indices (B,) of dtype torch.long. In our class-index setup, valid labels run from zero through C - 1. Do not apply softmax before this loss.

Softmax converts scores into probabilities by exponentiating each score and dividing by the sum of those exponentials. For logits [log(2), 0, 0], exponentials are [2, 1, 1], giving probabilities [0.5, 0.25, 0.25]. A target of class zero gives loss -log(0.5), about 0.6931. The correct-class probability determines the unweighted single-example loss.

These classes are exclusive. A photograph that may contain both a bicycle and a dog poses a different, multi-label question. Do not select a loss solely from the number of label names.

import torch
from torch import nn

reg_pred = torch.tensor([[1.0], [4.0]])
reg_y = torch.tensor([[2.0], [2.0]])
assert reg_pred.shape == reg_y.shape
print(nn.MSELoss()(reg_pred, reg_y).item())

binary_logits = torch.tensor([[0.0], [1.0]])
binary_y = torch.tensor([[1.0], [0.0]])
print(nn.BCEWithLogitsLoss()(binary_logits, binary_y).item())

class_logits = torch.tensor([[1.0, 0.0, -1.0], [0.0, 2.0, 0.0]])
class_y = torch.tensor([0, 1], dtype=torch.long)
print(nn.CrossEntropyLoss()(class_logits, class_y).item())

Practice

• 02A · Follow the inactive path: repeat the scalar forward pass with x = -2, leaving all parameters and the target unchanged. Calculate z, h, yhat, half-squared loss and default MSE.
• 02B · Write three contracts: specify output shape, target shape, target dtype and loss for a batch of six house-price predictions, six binary ring labels, and six three-class observations.
• 02C · Confidence changes loss: using logits [log(2), 0, 0], calculate the loss when the target is class two. Explain why the prediction is unchanged but the loss differs from the worked class-zero case.

Exit check

For each code example, identify exactly what one output entry means. Explain why the hand-worked loss and MSELoss differ by a factor of two, and why displaying a binary probability requires a sigmoid while both classification training losses receive raw logits.

Further reading: PyTorch 2.9 ReLU, MSELoss, BCEWithLogitsLoss, and CrossEntropyLoss.