Unit 04.05: Comparing a neural net with a classical baseline

A network that beats nothing has proved nothing. Two baselines take about a minute to fit and change how you read every subsequent result.

Two baselines, always

The majority-class baseline is the accuracy you get by always predicting the most common label. Any model below it is worse than a constant.

Logistic regression is the second: fast, interpretable, and often within a point or two of a network on structured data.

The task here has an interaction term, x1 * x2, which logistic regression cannot represent:

import torch
from torch import nn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

torch.manual_seed(0)
X = torch.randn(600, 5)
y = ((X[:, 0] + X[:, 1] * X[:, 2]) > 0).float().unsqueeze(1)
Xtr, ytr, Xte, yte = X[:450], y[:450], X[450:], y[450:]

# Baseline 1: predict the majority class. Never skip this one.
majority = max(yte.mean().item(), 1 - yte.mean().item())

# Baseline 2: logistic regression, seconds to fit.
logreg = LogisticRegression(max_iter=1000).fit(Xtr.numpy(), ytr.numpy().ravel())
logreg_acc = accuracy_score(yte.numpy().ravel(), logreg.predict(Xte.numpy()))

# The neural network.
model = nn.Sequential(nn.Linear(5, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(600):
    opt.zero_grad()
    nn.BCEWithLogitsLoss()(model(Xtr), ytr).backward()
    opt.step()
with torch.no_grad():
    net_acc = ((torch.sigmoid(model(Xte)) > 0.5).float() == yte).float().mean().item()

print(f"majority class     : {majority:.3f}")
print(f"logistic regression: {logreg_acc:.3f}")
print(f"neural network     : {net_acc:.3f}")
print("network earns its complexity:", net_acc > logreg_acc + 0.02)

# The interaction term x1*x2 is why the network wins here. Report all three
# numbers: a network that only matches logistic regression is not a result.

The network wins by a clear margin, and the reason is identifiable — the interaction. That is the kind of claim worth making: not "deep learning is better" but "this relationship is non-linear, and here is the evidence".

The mistake this prevents

Reporting only the network's accuracy. Without the baselines a reader cannot tell whether 0.87 is excellent or embarrassing, and neither can you.

Takeaway

Report three numbers: majority class, a classical model, and the network. The gaps between them are the finding.

Mark complete