Unit 08.03: Recurrent networks as historical context

Recurrent networks were the answer to word order for years. Understanding what they do — and what it cost — explains why attention replaced them.

Reading left to right, carrying state

An RNN processes one timestep at a time, maintaining a hidden state that summarises everything seen so far. That state is what lets it distinguish orderings.

The data here is built so order is the *only* signal: [1, 2] and [2, 1] contain identical values and opposite labels:

import torch
from torch import nn

torch.manual_seed(0)
# Sequences where only the ORDER differs: [1,2] vs [2,1].
X = torch.tensor([[[1.], [2.]], [[2.], [1.]], [[1.], [2.]], [[2.], [1.]]])
y = torch.tensor([[1.], [0.], [1.], [0.]])

# A bag-of-words style sum cannot tell them apart: both sum to 3.
print("summed inputs:", X.sum(dim=1).flatten().tolist(), "<- identical")

rnn = nn.RNN(input_size=1, hidden_size=8, batch_first=True)
head = nn.Linear(8, 1)
opt = torch.optim.Adam(list(rnn.parameters()) + list(head.parameters()), lr=0.05)

for _ in range(300):
    opt.zero_grad()
    output, hidden = rnn(X)
    nn.BCEWithLogitsLoss()(head(output[:, -1]), y).backward()
    opt.step()

with torch.no_grad():
    output, _ = rnn(X)
    acc = ((torch.sigmoid(head(output[:, -1])) > 0.5).float() == y).float().mean().item()
print("RNN accuracy on order-only data:", acc)
print("output shape:", tuple(output.shape), "= batch, timesteps, hidden")

# RNNs read left to right, carrying a hidden state. That sequential dependency
# is also their weakness: it cannot be parallelised, and long-range signal
# fades. Attention was the answer to both.

The summed inputs are identical for both classes, so any count-based approach is at chance. The RNN separates them, because its hidden state depends on the sequence.

The output shape — batch, timesteps, hidden — shows it produced a state at every position. Taking output[:, -1] uses the final state as a summary of the whole sequence.

The mistake this prevents

Assuming an RNN handles long sequences well. The hidden state is a fixed-size bottleneck, and signal from early positions fades as the sequence grows — the vanishing-gradient problem from Module 5, now along time.

Takeaway

RNNs read sequentially and carry state. That gives them order, and costs them parallelism and long-range memory.

Mark complete