Unit 09.01: Tokenizers, truncation, padding, and attention masks

The mask is not an optimisation. Without it, padding silently contaminates every pooled representation you compute.

Why the mask changes the answer

Text becomes integer ids, padded to a common length. The mask marks which positions are real.

The character-level tokenizer here keeps the mechanics visible — a real tokenizer uses subwords, but the padding and masking behaviour is identical:

import torch
from torch import nn

# A character-level tokenizer, so the mechanics are visible with no library.
texts = ["ok", "this is longer", "fine"]
chars = sorted({c for t in texts for c in t})
stoi = {c: i + 1 for i, c in enumerate(chars)}      # 0 reserved for padding

MAX_LEN = 8
ids = torch.zeros(len(texts), MAX_LEN, dtype=torch.long)
for i, t in enumerate(texts):
    seq = [stoi[c] for c in t][:MAX_LEN]
    ids[i, :len(seq)] = torch.tensor(seq)

mask = ids != 0
print("token ids:\n", ids)
print("\nreal tokens per row:", mask.sum(1).tolist())
print("truncated:", [len(t) > MAX_LEN for t in texts])

# Why the mask matters: mean-pooling without it averages in the padding.
emb = nn.Embedding(len(stoi) + 1, 4, padding_idx=0)
vectors = emb(ids)
naive = vectors.mean(dim=1)
masked = (vectors * mask.unsqueeze(-1)).sum(1) / mask.sum(1, keepdim=True)
print("\nnaive vs masked pooling differ:", not torch.allclose(naive, masked))
print("padding_idx keeps the pad vector at zero:", emb.weight[0].abs().sum().item() == 0.0)

The naive mean and the masked mean differ, and the difference grows with how much padding a sequence needed. A short sentence in a batch of long ones gets its representation diluted towards zero.

padding_idx=0 is the other half: it pins the pad row of the embedding table at zero and excludes it from gradient updates, so the model never learns a meaning for "nothing here".

The mistake this prevents

Mean-pooling over the sequence dimension without applying the mask. It runs, produces plausible numbers, and quietly penalises every short input.

Takeaway

Carry the mask from tokenisation through to pooling, and set padding_idx on the embedding.

Mark complete