Unit 03.01: Gradients and `requires_grad`

PyTorch tracks gradients only for tensors you mark. Knowing which tensors those are explains both how training works and why inference is faster.

The flag that decides what is tracked

requires_grad=True tells PyTorch to record every operation involving that tensor, building a graph it can later walk backwards. Model parameters have this set by default. Your data does not — you are not optimising the inputs.

Start with a case simple enough to verify by hand: y = w * x with x = 3, so dy/dw must be 3.

import torch

# requires_grad marks a tensor as something to track gradients for.
w = torch.tensor([2.0], requires_grad=True)
x = torch.tensor([3.0])                       # data: no gradient needed

y = w * x
y.backward()

print("w.grad:", w.grad.item())    # 3.0 -- dy/dw = x
print("x.requires_grad:", x.requires_grad)   # False

# Model parameters have requires_grad=True by default.
from torch import nn
layer = nn.Linear(4, 1)
print("weight tracks gradients:", layer.weight.requires_grad)   # True
print("grad before backward:", layer.weight.grad)               # None

# Inference needs no gradients, and saying so saves memory and time.
with torch.no_grad():
    out = layer(torch.randn(2, 4))
print("out.requires_grad inside no_grad():", out.requires_grad)  # False

w.grad is exactly 3.0, matching the derivative. Before .backward() is called, .grad is None — the attribute exists but holds nothing.

The no_grad() block at the end switches tracking off. The output has requires_grad=False, no graph is built, and memory is not spent recording operations you will never differentiate.

The mistake this prevents

Running validation without torch.no_grad(). It still produces correct numbers, but builds a graph for every batch — slower, and on a large validation set it can exhaust memory for no benefit.

Takeaway

Parameters track gradients; data does not. Wrap every evaluation in torch.no_grad().