Unit 09.07: Prediction survives; interpretation does not

When two predictors carry the same information, the model cannot tell which is responsible — and says so by making both look insignificant.

Prediction survives; interpretation does not

Multicollinearity is a strong correlation between predictors. The fit is unaffected — predictions and R-squared stay where they were — but the individual coefficients become unstable, their standard errors inflate, and p-values that should be tiny become large.

The signature is distinctive: a model that predicts well overall while none of its correlated predictors is individually significant.

statsmodels.stats.outliers_influence.variance_inflation_factor quantifies it. VIF is 1/(1 − R²) from regressing one predictor on the others; above 5 is a warning, above 10 a problem. Note that it needs the constant column, so pass sm.add_constant(X).

This block puts the same measurement into a model twice, in two units.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf
from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm

rng = np.random.default_rng(608)
n = 200
d = pd.DataFrame({"height_cm": rng.normal(170, 10, n)})
d["height_in"] = d.height_cm / 2.54 + rng.normal(0, 0.35, n)   # nearly the same
d["weight"] = 50 + 0.5 * d.height_cm + rng.normal(0, 6, n)

print(f"Correlation between the two height measures:"
      f" {d.height_cm.corr(d.height_in):.4f}\n")

one = smf.ols("weight ~ height_cm", data=d).fit()
both = smf.ols("weight ~ height_cm + height_in", data=d).fit()
for name, m in [("one predictor ", one), ("both predictors", both)]:
    print(f"{name}: estimate {m.params['height_cm']:7.3f}"
          f"   SE {m.bse['height_cm']:6.3f}"
          f"   p {m.pvalues['height_cm']:.3g}")

X = sm.add_constant(d[["height_cm", "height_in"]])
print()
for i, col in enumerate(X.columns):
    if col != "const":
        print(f"VIF {col:10s}: {variance_inflation_factor(X.values, i):.1f}")
print("Rule of thumb: above 5 is a warning, above 10 a problem.\n")
print(f"R-squared barely moved: {one.rsquared:.4f} -> {both.rsquared:.4f}")
print("Prediction is fine. The individual coefficients are not interpretable.")

The two height measures correlate at 0.9963. Alone, height_cm has an estimate of 0.476 with a standard error of 0.041 and p = 1.09e-23. With both in the model the estimate moves to 0.636 and the standard error explodes to 0.482, giving p = 0.188 — no longer significant. VIF is 134.3 for each. Yet R-squared barely moves, from 0.3993 to 0.3996: the model predicts exactly as well and can no longer say which variable does the work.

The mistake this prevents

The mistake is concluding a predictor does not matter because its p-value is large in a collinear model. The information is there; the model simply cannot attribute it.

Takeaway

Check correlations between predictors before modelling, and compute VIF when several are related. If you only need predictions, collinearity is harmless; if you need to interpret coefficients, drop or combine the redundant variables.

Mark complete