Unit 03.02: Backpropagation intuition without heavy derivation

Backpropagation has a reputation for being hard. It is the chain rule, applied backwards through a graph you already built. Do it by hand once and it stops being mysterious.

Work it out first, then let autograd confirm

Take four operations: multiply, add, square. Each has a derivative you know. The chain rule says the derivative of the whole is the product of the derivatives along each path — and where a variable feeds two paths, you add the contributions.

The comment block traces it: dloss/da comes out at 150, dloss/db at 90. Now check:

import torch

# Backpropagation is the chain rule applied backwards through the graph.
# Work it out by hand once, then let autograd confirm it.
#
#   a = 3, b = 4
#   c = a * b          dc/da = b = 4
#   d = c + a          dd/dc = 1,  dd/da = dc/da * 1 + 1 = 5
#   loss = d ** 2      dloss/dd = 2d = 2 * 15 = 30
#   dloss/da = 30 * 5 = 150
#   dloss/db = 30 * dd/db = 30 * a = 90

a = torch.tensor([3.0], requires_grad=True)
b = torch.tensor([4.0], requires_grad=True)

c = a * b
d = c + a
loss = d ** 2

loss.backward()

print("d    :", d.item())        # 15.0
print("a.grad:", a.grad.item())  # 150.0  <- matches the hand calculation
print("b.grad:", b.grad.item())  # 90.0

# Nothing about this changes for a real network. There are simply more nodes.

a.grad is 150.0 and b.grad is 90.0, matching the hand calculation exactly. Note that a contributes through two routes — into c and again into d — and autograd sums both, which is why its gradient is larger.

The mistake this prevents

Believing backpropagation does something a network could not be asked to justify. It computes derivatives, nothing more. When a gradient looks wrong, it is almost always the loss or the shapes, not the differentiation.

Takeaway

Backpropagation is the chain rule run backwards. Nothing changes for a real network except the number of nodes.

Mark complete