Unit 09.08: R-squared always rises

R-squared always rises when you add a predictor, including a predictor made of pure noise.

Why adjusted R-squared exists

Every additional term gives the model more freedom to fit the particular sample in front of it, so R-squared can only go up. With enough predictors relative to observations, a model can fit noise almost perfectly and predict nothing.

Adjusted R-squared penalises the number of terms and can go negative, which makes it the honest summary of the two.

The related trap is selecting predictors by their p-values and refitting. At alpha 0.05 one in twenty noise predictors will look significant, so the selected model is built from exactly the variables that got lucky — and the p-values it reports are no longer valid, because the same data chose them.

This block regresses an outcome on twenty predictors made entirely of noise.

import numpy as np, pandas as pd
import statsmodels.api as sm

rng = np.random.default_rng(6091)
n, p = 60, 20
y = rng.normal(size=n)
X = pd.DataFrame(rng.normal(size=(n, p)),
                 columns=[f"x{i}" for i in range(1, p + 1)])
model = sm.OLS(y, sm.add_constant(X)).fit()

print(f"{p} pure-noise predictors, n = {n}")
print(f"R-squared          : {model.rsquared:.4f}")
print(f"Adjusted R-squared : {model.rsquared_adj:.4f}")
print(f"Overall F p-value  : {model.f_pvalue:.3g}\n")

sig = (model.pvalues.drop("const") < 0.05).sum()
print(f"Coefficients with p < 0.05      : {sig} of {p}")
print(f"Expected by chance at alpha 0.05: {p * 0.05:.0f}\n")

print(f"R-squared reached {model.rsquared:.2f} on data with no relationship in")
print("it at all. Adjusted R-squared penalises the extra terms and is the")
print(f"honest one: {model.rsquared_adj:.3f}")
print("\nSelecting the 'significant' predictors here and refitting would give a")
print("model that looks excellent and predicts nothing on new data -- and its")
print("reported p-values would no longer be valid, because the same data chose")
print("the variables.")

With 20 noise predictors and n = 60, R-squared reaches 0.2625 on data containing no relationship whatever, while adjusted R-squared is −0.1157 — negative, which is the honest verdict — and the overall F test gives p = 0.807. Exactly 1 of the twenty coefficients has p < 0.05, precisely the number expected by chance. Selecting that one and refitting would produce a model that looks strong and predicts nothing.

The mistake this prevents

The mistake is stepwise selection followed by reporting the surviving p-values as if they were valid. The selection used the same data, so they are optimistic by an amount nobody can quantify.

Takeaway

Compare models on adjusted R-squared or on held-out performance, never on R-squared. Decide predictors from subject knowledge and the analysis plan, and treat any data-driven selection as exploratory.

Mark complete