Unit 08.01: Text tokens and sequence length

Models need rectangles. Text arrives in ragged lengths. Everything in this unit follows from that mismatch.

Padding, truncation, and the mask that keeps them honest

Short sequences are padded to a fixed length; long ones are truncated. Both are lossy in different ways — truncation discards content outright, and padding adds positions that carry no information.

The mask records which positions are real:

import torch
from torch import nn

sentences = ["the cat sat", "a dog barked loudly at the gate", "hello"]
vocab = {w: i + 1 for i, w in enumerate(sorted({w for s in sentences for w in s.split()}))}
vocab["<pad>"] = 0
print("vocabulary size:", len(vocab))

sequences = [[vocab[w] for w in s.split()] for s in sentences]
for s, seq in zip(sentences, sequences):
    print(f"{len(seq):>2} tokens: {s!r}")

# Batching needs one rectangle, so short rows are padded and long ones truncated.
MAX_LEN = 5
padded = torch.zeros(len(sequences), MAX_LEN, dtype=torch.long)
for i, seq in enumerate(sequences):
    keep = seq[:MAX_LEN]
    padded[i, :len(keep)] = torch.tensor(keep)
print("\npadded batch:\n", padded)

# The mask records which positions are real. Without it the model averages
# padding into the answer and short sentences get diluted.
mask = (padded != 0)
print("\nreal tokens per row:", mask.sum(dim=1).tolist())
print("truncated 7 tokens down to", MAX_LEN, "-- 'at the gate' was discarded")

The 7-token sentence is cut to 5, losing "at the gate" permanently. The 1-token sentence gains four padding positions.

Without the mask, any pooling operation averages those padding positions into the result, so a short sentence gets diluted by however much padding it happened to need.

The mistake this prevents

Choosing a maximum length without checking the length distribution. Set it too low and you silently truncate most of your corpus; too high and you waste computation on padding.

Takeaway

Pad to batch, truncate deliberately, and carry a mask everywhere the padding could be averaged in.

Mark complete