Unit 02.02: Indexing, reshaping, stacking, and broadcasting

Broadcasting is what lets you add a bias vector to a whole batch in one line. It is also how a shape bug turns into a wrong answer instead of an error.

Alignment happens from the right

When shapes differ, PyTorch lines them up from the trailing dimension and stretches any axis of size 1. That is why a (3,) bias adds cleanly to a (4, 3) batch — one value per feature, applied to every row.

It is also worth separating reshape from transpose: both rearrange, but they produce genuinely different tensors.

import torch

t = torch.arange(12).reshape(3, 4)
print("original:\n", t)
print("row 0      :", t[0].tolist())
print("column 2   :", t[:, 2].tolist())

# reshape vs transpose: same elements, different meaning.
print("reshape(4,3) first row:", t.reshape(4, 3)[0].tolist())   # [0, 1, 2]
print("transpose    first row:", t.T[0].tolist())               # [0, 4, 8]

# Broadcasting: shapes are aligned from the right.
batch = torch.ones(4, 3)
per_feature_bias = torch.tensor([10., 20., 30.])
print("broadcast add:\n", (batch + per_feature_bias)[0].tolist())   # [11, 21, 31]

# The silent bug: a (4,) target against a (4,1) prediction broadcasts to (4,4).
pred = torch.zeros(4, 1)
wrong = torch.zeros(4)
print("wrong shape gives:", tuple((pred - wrong).shape))    # (4, 4) -- not (4, 1)
print("fix with unsqueeze:", tuple((pred - wrong.unsqueeze(1)).shape))

The last two lines are the dangerous case. Subtracting a (4,) tensor from a (4, 1) tensor does not error — it broadcasts to (4, 4), giving you sixteen numbers where you expected four. A loss computed on that shape is meaningless but runs perfectly happily.

The mistake this prevents

Letting a target of shape (n,) meet a prediction of shape (n, 1). The loss trains on a matrix of pairwise differences, the model appears to learn, and the numbers are nonsense throughout.

Takeaway

Print the shape of both operands before any subtraction or loss. Broadcasting fails loudly far less often than it fails quietly.