Unit 08.05: Project step: compare simple text representations

Two representations on a task designed so one of them cannot possibly work. The point is the comparison, and the honesty about what it proves.

A task where order is the only signal

Every sequence contains the same tokens. Only their order differs, and the label follows the order. Bag-of-words sees identical count vectors for both classes.

Both approaches are trained on the same split and evaluated the same way:

import json

import torch
from torch import nn

torch.manual_seed(0)
# Label depends on ORDER: "a before b" is 1, "b before a" is 0.
data, labels = [], []
for _ in range(200):
    if torch.rand(1).item() > 0.5:
        data.append([1, 2, 3]); labels.append(1.0)
    else:
        data.append([2, 1, 3]); labels.append(0.0)
X = torch.tensor(data)
y = torch.tensor(labels).unsqueeze(1)
Xtr, ytr, Xva, yva = X[:150], y[:150], X[150:], y[150:]


def bag_of_words():
    torch.manual_seed(0)
    counts = torch.zeros(len(X), 4)
    for i, row in enumerate(X):
        for t in row:
            counts[i, t] += 1
    model = nn.Linear(4, 1)
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(300):
        opt.zero_grad()
        nn.BCEWithLogitsLoss()(model(counts[:150]), ytr).backward()
        opt.step()
    with torch.no_grad():
        return ((torch.sigmoid(model(counts[150:])) > 0.5).float() == yva).float().mean().item()


def sequence_model():
    torch.manual_seed(0)
    emb = nn.Embedding(4, 8)
    rnn = nn.GRU(8, 16, batch_first=True)
    head = nn.Linear(16, 1)
    params = list(emb.parameters()) + list(rnn.parameters()) + list(head.parameters())
    opt = torch.optim.Adam(params, lr=0.02)
    for _ in range(300):
        opt.zero_grad()
        out, _ = rnn(emb(Xtr))
        nn.BCEWithLogitsLoss()(head(out[:, -1]), ytr).backward()
        opt.step()
    with torch.no_grad():
        out, _ = rnn(emb(Xva))
        return ((torch.sigmoid(head(out[:, -1])) > 0.5).float() == yva).float().mean().item()


bow_acc, seq_acc = bag_of_words(), sequence_model()
print(json.dumps({
    "task": "predict which of two tokens came first",
    "majority_baseline": round(max(yva.mean().item(), 1 - yva.mean().item()), 3),
    "bag_of_words": round(bow_acc, 3),
    "sequence_model": round(seq_acc, 3),
    "why": "bag-of-words sees identical counts for both orders, so it cannot beat chance",
    "limitation": "a task built to favour sequence models; on many real tasks BoW is competitive",
}, indent=2))

Bag-of-words sits at chance, as it must. The sequence model separates the classes.

The why field explains the mechanism rather than just reporting the gap, and the limitation is the important part: this task was *built* to favour sequence models. On many real classification problems the gap is small or reversed.

The mistake this prevents

Generalising from a task you designed to prove a point. A benchmark built to favour one method proves that method can do the thing you built for, and nothing more.

Takeaway

Compare representations on your actual task. A demonstration that one *can* win is not evidence that it *will*.

Mark complete