Unit 09.08: R-squared always rises

R-squared always rises when you add a predictor, including a predictor made of pure noise.

Why adjusted R-squared exists

Every additional term gives the model more freedom to fit the particular sample in front of it, so R-squared can only go up. With enough predictors relative to observations, a model can fit noise almost perfectly and predict nothing at all.

Adjusted R-squared penalises the number of terms and can go down, which makes it the honest summary of the two.

The related trap is selecting predictors by their p-values and refitting. At alpha 0.05 one in twenty noise predictors will look significant, so the selected model is built from exactly the variables that got lucky — and the reported p-values in it are no longer valid.

This block regresses an outcome on twenty predictors made entirely of noise.

set.seed(609)
n <- 60
d <- data.frame(y = rnorm(n))
# Twenty predictors, all pure noise, none related to y.
for (i in 1:20) d[[paste0("x", i)]] <- rnorm(n)

full <- lm(y ~ ., data = d)
cat("Twenty pure-noise predictors, n =", n, "\n")
cat("R-squared         :", round(summary(full)$r.squared, 4), "\n")
cat("Adjusted R-squared:", round(summary(full)$adj.r.squared, 4), "\n")
cat("Overall F p-value :",
    signif(pf(summary(full)$fstatistic[1], summary(full)$fstatistic[2],
              summary(full)$fstatistic[3], lower.tail = FALSE), 3), "\n\n")

p_values <- summary(full)$coefficients[-1, 4]
cat("Coefficients with p < 0.05:", sum(p_values < 0.05), "of 20\n")
cat("Expected by chance at alpha 0.05:", 20 * 0.05, "\n\n")

cat("R-squared rose to", round(summary(full)$r.squared, 2),
    "on data with no relationship in it at all.\n")
cat("Adjusted R-squared penalises the extra terms and is the honest one:",
    round(summary(full)$adj.r.squared, 3), "\n")
cat("Selecting the 'significant' predictors here and refitting would produce\n")
cat("a model that looks excellent and predicts nothing on new data.\n")

With 20 noise predictors and n = 60, R-squared reaches 0.4571 on data containing no relationship whatever. Adjusted R-squared is 0.1787 and the overall F-test gives p = 0.0912 — correctly unimpressed. Two of the twenty coefficients have p < 0.05, against exactly the 1 expected by chance. Selecting those two and refitting would produce a model that looks strong and predicts nothing.

The mistake this prevents

The mistake is stepwise selection followed by reporting the surviving p-values as if they were valid. The selection used the same data, so the p-values are optimistic by an amount nobody can quantify.

Takeaway

Compare models on adjusted R-squared or on held-out performance, never on R-squared. Decide the predictors from subject knowledge and the analysis plan, and treat any data-driven selection as exploratory.

Mark complete