Unit 10.05: 95% accuracy can mean catching nothing

On an imbalanced outcome, 95% accuracy can mean the model never predicts the thing you care about.

Accuracy is nearly meaningless when one class is rare

A confusion matrix cross-tabulates predictions against actual outcomes. Accuracy is the share on the diagonal, and when one class is rare a model that always predicts the majority class achieves a high accuracy while being completely useless.

The baseline to compare against is therefore not 50% — it is the majority class rate.

A separate point: a model can carry a real, strongly significant association and still classify badly. Statistical evidence about a relationship and practical performance as a classifier are different achievements, and a study can succeed at one and fail at the other.

This block fits a model on data where only about 5% of cases are positive.

set.seed(706)
n <- 2000
x <- rnorm(n)
# Only 4% of cases are positive.
y <- rbinom(n, 1, 1 / (1 + exp(-(-3.2 + 0.8 * x))))
model <- glm(y ~ x, family = binomial)

p_hat <- predict(model, type = "response")
predicted <- factor(as.integer(p_hat > 0.5), levels = c(0, 1))

cat("Positive rate in the data:", round(mean(y), 4), "\n\n")
cat("Confusion matrix at threshold 0.5:\n")
print(table(predicted = predicted, actual = y))

cat("\nModel accuracy      :", round(mean(as.integer(as.character(predicted)) == y), 4), "\n")
cat("Always-predict-zero :", round(mean(y == 0), 4), "\n")
cat("Positives found     :", sum(predicted == 1 & y == 1), "of", sum(y == 1),
    "-- the model never predicts a positive at this threshold\n\n")

cat("The model is barely better than a rule that never predicts a positive,\n")
cat("and yet the coefficient on x is real: estimate",
    round(coef(model)[2], 3), ", p",
    signif(summary(model)$coefficients[2, 4], 3), "\n\n")
cat("Accuracy on an imbalanced outcome is nearly meaningless. A statistically\n")
cat("solid association and a useful classifier are different achievements.\n")

The positive rate is 0.051. At a 0.5 threshold the model predicts a positive for zero cases: it catches 0 of 102 positives. Its accuracy is 0.949 — identical, to four decimal places, to the accuracy of a rule that always says no. And yet the coefficient on x is real: 0.854 with p = 2.96e-15. The association is solid; the classifier is worthless.

The mistake this prevents

The mistake is reporting accuracy on an imbalanced problem. Always compare it to the majority-class baseline, and look at how many positives were actually found.

Takeaway

Report the confusion matrix, the majority-class baseline, and how many of the positive cases were caught. Keep the statistical claim about the association separate from the claim about classification performance.

Mark complete