Unit 07.05: Project step: transfer-learning experiment memo

Two approaches, one comparison, and a verdict that is willing to say the simpler option won.

Running both, then deciding

The same pretrained body is loaded twice: once frozen with only a head trained, once fine-tuned end to end at a much lower backbone rate.

Both are evaluated on the same validation split, and both are reported alongside the majority baseline:

import json

import torch
from torch import nn

torch.manual_seed(0)
X = torch.randn(300, 12)
y = ((X[:, 0] + X[:, 1] * X[:, 2]) > 0).float().unsqueeze(1)
Xtr, ytr, Xva, yva = X[:220], y[:220], X[220:], y[220:]

pretrained = nn.Sequential(nn.Linear(12, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU())


def run(freeze, lr, steps=400):
    torch.manual_seed(0)
    body = nn.Sequential(nn.Linear(12, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU())
    body.load_state_dict(pretrained.state_dict())
    head = nn.Linear(32, 1)
    for p in body.parameters():
        p.requires_grad = not freeze
    params = [{"params": head.parameters(), "lr": 1e-2}]
    if not freeze:
        params.append({"params": body.parameters(), "lr": lr})
    opt = torch.optim.Adam(params)
    for _ in range(steps):
        opt.zero_grad()
        nn.BCEWithLogitsLoss()(head(body(Xtr)), ytr).backward()
        opt.step()
    with torch.no_grad():
        acc = ((torch.sigmoid(head(body(Xva))) > 0.5).float() == yva).float().mean().item()
    trainable = sum(p.numel() for p in body.parameters() if p.requires_grad) + \
        sum(p.numel() for p in head.parameters())
    return round(acc, 3), trainable


frozen_acc, frozen_params = run(freeze=True, lr=0)
tuned_acc, tuned_params = run(freeze=False, lr=1e-4)

print(json.dumps({
    "seed": 0,
    "feature_extraction": {"val_accuracy": frozen_acc, "trainable_params": frozen_params},
    "fine_tuning": {"val_accuracy": tuned_acc, "trainable_params": tuned_params,
                    "backbone_lr": 1e-4, "head_lr": 1e-2},
    "majority_baseline": round(max(yva.mean().item(), 1 - yva.mean().item()), 3),
    "verdict": "fine-tuning helped" if tuned_acc > frozen_acc else
               "freezing was enough -- do not pay for what did not help",
    "limitation": "220 training rows; results on this little data move with the seed",
}, indent=2))

The verdict line is the deliverable. If fine-tuning did not beat freezing, saying so is the correct outcome — you have avoided paying for training time, complexity, and overfitting risk that bought nothing.

The limitation notes 220 training rows, which is small enough that a different seed could reorder the two approaches.

The mistake this prevents

Running only the approach you expected to win. Without the comparison you cannot claim fine-tuning helped, and on small data it frequently does not.

Takeaway

Run both, report both, and be willing to conclude that the cheaper option was sufficient.

Mark complete