Unit 08.02: Bag-of-words baseline vs learned representation

Bag-of-words counts words and discards their order. That is a real limitation and also, surprisingly often, good enough.

What counting does and does not capture

Each document becomes a vector of word counts. "good service" and "service good" produce identical vectors, because order is thrown away.

That sounds fatal until you notice how much meaning survives in the words themselves — including negation, when "not" is a word in the vocabulary:

import torch
from torch import nn

torch.manual_seed(0)
# "good" and "not good" share every word; only order separates them.
docs = [("good service", 1), ("not good service", 0),
        ("service not good", 0), ("service good", 1)]
vocab = {w: i for i, w in enumerate(sorted({w for d, _ in docs for w in d.split()}))}

bow = torch.zeros(len(docs), len(vocab))
for i, (text, _) in enumerate(docs):
    for w in text.split():
        bow[i, vocab[w]] += 1
labels = torch.tensor([lbl for _, lbl in docs]).float().unsqueeze(1)

print("vocabulary:", list(vocab))
print("bag-of-words rows:\n", bow)
print("\nrows 0 and 3 are identical:", torch.equal(bow[0], bow[3]),
      "-- same label, fine")
print("rows 1 and 2 are identical:", torch.equal(bow[1], bow[2]),
      "-- same label, also fine")
print("but 'not good' vs 'good' differ only by the extra word, not by ORDER")

model = nn.Linear(len(vocab), 1)
opt = torch.optim.Adam(model.parameters(), lr=0.1)
for _ in range(400):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(model(bow), labels).backward()
    opt.step()
with torch.no_grad():
    acc = ((torch.sigmoid(model(bow)) > 0.5).float() == labels).float().mean().item()
print("\nbag-of-words accuracy:", acc)
print("It works here because 'not' is a word. It fails when meaning depends on")
print("word ORDER, which is exactly what sequence models add.")

The model reaches perfect accuracy here, because "not" appears as its own feature and the classifier can weight it negatively.

But look at which rows are identical. Bag-of-words genuinely cannot distinguish orderings — it works on this data because the labels happen to align with word presence, not with sequence.

The mistake this prevents

Dismissing bag-of-words without measuring it. It is fast, interpretable, and on many classification tasks it lands within a few points of a neural model that takes a hundred times longer to train.

Takeaway

Count-based features are a serious baseline. Move to sequence models when you can show order matters.

Mark complete