Unit 09.05: Project step: transformer-assisted classification report

Frozen features against end-to-end fine-tuning, measured the same way, reported together.

The comparison and its caveat

Both paths share the same data and the same split. One caches encoder outputs and fits logistic regression; the other trains the encoder and head jointly with separate learning rates.

The report records both, plus the baseline:

import json

import torch
from torch import nn
from sklearn.linear_model import LogisticRegression

torch.manual_seed(0)
X = torch.randn(300, 6, 16)
y = (X[:, 0, 0] + X[:, 1, 1] > 0).long()
Xtr, ytr, Xte, yte = X[:220], y[:220], X[220:], y[220:]

encoder = nn.TransformerEncoderLayer(16, 4, 32, batch_first=True)
encoder.eval()
with torch.no_grad():
    Ftr, Fte = encoder(Xtr).mean(1), encoder(Xte).mean(1)

frozen = LogisticRegression(max_iter=1000).fit(Ftr.numpy(), ytr.numpy())
frozen_acc = frozen.score(Fte.numpy(), yte.numpy())

torch.manual_seed(0)
enc2 = nn.TransformerEncoderLayer(16, 4, 32, batch_first=True)
head = nn.Linear(16, 1)
opt = torch.optim.Adam([{"params": head.parameters(), "lr": 1e-2},
                        {"params": enc2.parameters(), "lr": 1e-5}])
yt = ytr.float().unsqueeze(1)
for _ in range(150):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(head(enc2(Xtr).mean(1)), yt).backward()
    opt.step()
with torch.no_grad():
    tuned_acc = ((torch.sigmoid(head(enc2(Xte).mean(1))) > 0.5).long().squeeze(1) == yte).float().mean().item()

print(json.dumps({
    "seed": 0,
    "encoder": "1-layer TransformerEncoderLayer, d_model=16, 4 heads",
    "rows": {"train": len(Xtr), "test": len(Xte)},
    "majority_baseline": round(max(yte.float().mean().item(), 1 - yte.float().mean().item()), 3),
    "frozen_features_plus_logreg": round(float(frozen_acc), 3),
    "fine_tuned_end_to_end": round(tuned_acc, 3),
    "limitation": "random vectors, not language; this measures the pipeline, not NLP quality",
}, indent=2))

The limitation line is doing real work here. The inputs are random vectors, not language, so this measures whether the *pipeline* is correct — not whether the approach is good at NLP.

Saying that explicitly is what stops a reader over-reading the numbers.

The mistake this prevents

Presenting a pipeline test as a quality result. The numbers are real; what they measure is that the code runs end to end, which is a much smaller claim.

Takeaway

Report both approaches, the baseline, and what the data does and does not represent.

Mark complete