Unit 11.06: Project step: deep learning review memo

Every diagnostic from this module in one memo, structured so a reviewer can challenge any part of it.

What a reviewable memo contains

Loss checkpoints show the training trajectory. Test accuracy sits next to the baseline. The confusion matrix separates error types. Per-slice accuracy exposes uneven performance. Confident mistakes point at the specific examples worth opening.

Each of these answers a question a reviewer would otherwise have to ask:

import json

import torch
from torch import nn
from sklearn.metrics import confusion_matrix

torch.manual_seed(0)
X = torch.randn(700, 5)
y = ((X[:, 0] + X[:, 1] * X[:, 2]) > 0).float().unsqueeze(1)
group = (X[:, 4] > 0).long()
Xtr, ytr, Xte, yte = X[:550], y[:550], X[550:], y[550:]

model = nn.Sequential(nn.Linear(5, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
history = []
for epoch in range(300):
    opt.zero_grad()
    loss = nn.BCEWithLogitsLoss()(model(Xtr), ytr)
    loss.backward(); opt.step()
    if epoch % 100 == 0:
        history.append(round(loss.item(), 4))

with torch.no_grad():
    probs = torch.sigmoid(model(Xte)).squeeze(1)
    pred = (probs > 0.5).float()
acc = (pred == yte.squeeze(1)).float().mean().item()
tn, fp, fn_, tp = confusion_matrix(yte.numpy().ravel(), pred.numpy().ravel()).ravel()
slice_acc = {int(g): round((pred[group[550:] == g] == yte.squeeze(1)[group[550:] == g])
                           .float().mean().item(), 3) for g in (0, 1)}
confident_wrong = int(((pred != yte.squeeze(1)) & (probs.clamp(0.5, 1) > 0.8)).sum())

print(json.dumps({
    "seed": 0,
    "loss_checkpoints": history,
    "test_accuracy": round(acc, 3),
    "majority_baseline": round(max(yte.mean().item(), 1 - yte.mean().item()), 3),
    "confusion": {"tn": int(tn), "fp": int(fp), "fn": int(fn_), "tp": int(tp)},
    "accuracy_by_slice": slice_acc,
    "confident_mistakes": confident_wrong,
    "limitations": ["synthetic data with a known interaction",
                    "single seed",
                    "slices defined by a synthetic feature, not a real attribute"],
}, indent=2))

The confident-mistakes count is the most actionable line. Those are the cases where the model was both wrong and sure, which in real data usually means a labelling error or a genuine gap.

The limitations list closes the memo by stating what the numbers cannot support.

The mistake this prevents

Submitting a review memo with a single accuracy figure. Every question a reviewer asks — is that good? for whom? where does it fail? — then requires another round trip.

Takeaway

Anticipate the reviewer's questions and answer them in the memo. That is what makes it a review rather than a claim.

Mark complete