Unit 03.00: Forward pass and loss

Training needs one number to push downwards. The forward pass produces predictions; the loss collapses them into that single number.

From a batch of predictions to one scalar

The forward pass runs inputs through the model and returns one prediction per example. The loss function then compares those predictions against the targets and reduces the whole batch to a single value.

That reduction is essential. Gradients are defined with respect to a scalar, so a loss that returned five numbers could not be backpropagated:

import torch
from torch import nn

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1))

x = torch.randn(5, 4)          # 5 examples, 4 features each
y = torch.randn(5, 1)          # what we wanted the model to say

prediction = model(x)          # the forward pass: inputs -> outputs
loss = nn.MSELoss()(prediction, y)

print("prediction shape:", tuple(prediction.shape))   # (5, 1)
print("loss:", round(loss.item(), 4))                 # a single number
print("loss has a grad_fn:", loss.grad_fn is not None)

# The loss is one number no matter how big the batch is. That is the point:
# training needs a single quantity to push downwards.

Two details worth noticing. The prediction keeps its batch shape (5, 1), while the loss is a single number regardless of batch size. And the loss carries a grad_fn — a record of the operations that produced it, which is what makes the backward pass possible.

The mistake this prevents

Comparing a prediction of shape (n, 1) against a target of shape (n,). MSELoss broadcasts to (n, n), returns a number, and trains on nonsense without ever raising an error.

Takeaway

The loss is one scalar with a grad_fn attached. If either of those is missing, training cannot work.

Mark complete