Unit 05.03: Batch normalization and layer normalization intuition

Both normalisation layers stabilise training. They differ in what they average over, and that difference has a practical consequence you must not forget.

Across the batch, or across the features

BatchNorm normalises each feature using statistics computed across the batch. LayerNorm normalises each example using statistics across its own features.

That makes BatchNorm dependent on batch composition — and on batch *size*. With very small batches its statistics are noisy.

import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(8, 5) * 10 + 3        # badly scaled activations

bn = nn.BatchNorm1d(5)
ln = nn.LayerNorm(5)

print("input       mean %.2f std %.2f" % (x.mean().item(), x.std().item()))
print("batch norm  mean %.2f std %.2f  (per feature, across the batch)"
      % (bn(x).mean().item(), bn(x).std().item()))
print("layer norm  mean %.2f std %.2f  (per row, across features)"
      % (ln(x).mean().item(), ln(x).std().item()))

# The difference that matters in practice: BatchNorm behaves differently in
# train and eval mode because it uses running statistics at inference.
bn.train(); _ = bn(x)
bn.eval()
with torch.no_grad():
    print("eval-mode output differs from train-mode:",
          not torch.allclose(bn(x), nn.BatchNorm1d(5)(x)))
print("forgetting model.eval() is a real and common bug")

# LayerNorm has no batch dependency, which is why transformers use it.

The critical detail is the train/eval difference. BatchNorm uses batch statistics while training and stored running statistics at inference, so the same input produces different output depending on the mode.

LayerNorm has no batch dependency at all, which is precisely why transformers use it.

The mistake this prevents

Forgetting model.eval() before validation with BatchNorm in the model. The layer normalises using the validation batch's own statistics, so your score depends on how the validation set was batched.

Takeaway

BatchNorm across the batch, LayerNorm across the features — and always switch modes explicitly.

Mark complete