Unit 10.05: 94% accuracy can mean catching nothing

On an imbalanced outcome, 94% accuracy can mean the model never predicts the thing you care about.

Accuracy is nearly meaningless when one class is rare

A confusion matrix cross-tabulates predictions against actual outcomes. Accuracy is the share on the diagonal, and when one class is rare a model that always predicts the majority class achieves a high accuracy while being completely useless.

The baseline to compare against is therefore not 50% — it is the majority-class rate.

A separate point: a model can carry a real, strongly significant association and still classify badly. Statistical evidence about a relationship and practical performance as a classifier are different achievements, and a study can succeed at one and fail at the other.

This block fits a model on data where about 6% of cases are positive.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(706)
n = 2000
d = pd.DataFrame({"x": rng.normal(size=n)})
d["y"] = rng.binomial(1, 1 / (1 + np.exp(-(-3.2 + 0.8 * d.x))))   # ~4% positive

model = smf.logit("y ~ x", data=d).fit(disp=False)
p_hat = model.predict(d)
predicted = (p_hat > 0.5).astype(int)

print(f"Positive rate in the data: {d.y.mean():.4f}\n")
print("Confusion matrix at threshold 0.5:")
cm = pd.crosstab(predicted, d.y, rownames=["predicted"], colnames=["actual"],
                 dropna=False).reindex(index=[0, 1], columns=[0, 1], fill_value=0)
print(cm.to_string())

print(f"\nModel accuracy       : {(predicted == d.y).mean():.4f}")
print(f"Always-predict-zero  : {(d.y == 0).mean():.4f}")
print(f"Positives found      : {int(((predicted == 1) & (d.y == 1)).sum())}"
      f" of {int(d.y.sum())}\n")
print("The model is no better than a rule that never predicts a positive,")
print(f"and yet the coefficient on x is real: {model.params['x']:.3f},"
      f" p = {model.pvalues['x']:.3g}\n")
print("Accuracy on an imbalanced outcome is nearly meaningless. A solid")
print("statistical association and a useful classifier are different things.")

The positive rate is 0.0585. At a 0.5 threshold the model predicts a positive for zero cases: it catches 0 of 117 positives. Its accuracy is 0.9415 — identical, to four decimal places, to the accuracy of a rule that always says no. And yet the coefficient on x is real: 0.761 with p = 1.78e-14. The association is solid; the classifier is worthless.

The mistake this prevents

The mistake is reporting accuracy on an imbalanced problem. Always compare it to the majority-class baseline, and look at how many positives were actually found.

Takeaway

Report the confusion matrix, the majority-class baseline, and how many positive cases were caught. Keep the statistical claim about the association separate from the claim about classification performance.

Mark complete