Unit 12.02: Baseline and neural model comparison

Three numbers, always in the same order: majority class, classical model, network. The gaps between them are the finding.

What each comparison rules out

Beating the majority class rules out a model that has learned nothing. Beating logistic regression rules out a problem that did not need a network.

Only after both does the network's score mean anything:

import torch
from torch import nn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

torch.manual_seed(0)
X = torch.randn(700, 6)
y = ((X[:, 0] + X[:, 1] * X[:, 2]) > 0).float().unsqueeze(1)
Xtr, ytr, Xte, yte = X[:550], y[:550], X[550:], y[550:]

results = {}
results["majority"] = max(yte.mean().item(), 1 - yte.mean().item())

logreg = LogisticRegression(max_iter=1000).fit(Xtr.numpy(), ytr.numpy().ravel())
results["logistic"] = accuracy_score(yte.numpy().ravel(), logreg.predict(Xte.numpy()))

torch.manual_seed(0)
net = nn.Sequential(nn.Linear(6, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(net.parameters(), lr=0.01)
for _ in range(500):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(net(Xtr), ytr).backward()
    opt.step()
with torch.no_grad():
    results["neural"] = ((torch.sigmoid(net(Xte)) > 0.5).float() == yte).float().mean().item()

for name, score in results.items():
    print(f"{name:10}: {score:.3f}")
print(f"\nneural over logistic: {results['neural'] - results['logistic']:+.3f}")
print(f"neural over majority: {results['neural'] - results['majority']:+.3f}")
print("\nReport all three. A network that beats only the majority class has not")
print("earned its complexity, its training time, or its harder debugging.")

The two gaps are printed explicitly, because those differences are the result — not the network's accuracy on its own.

If the network beats the majority class but not logistic regression, the honest conclusion is to ship logistic regression: faster, interpretable, and easier to maintain.

The mistake this prevents

Reporting the network alone. Without the comparisons the reader cannot judge the number, and a network that ties a linear model has cost complexity for nothing.

Takeaway

Majority, classical, network. Report all three and let the gaps decide.

Mark complete