Unit 09.07: Prediction survives; interpretation does not

When two predictors carry the same information, the model cannot tell which of them is responsible — and says so by making both look insignificant.

Prediction survives; interpretation does not

Multicollinearity is a strong correlation between predictors. The fit is unaffected — predictions and R-squared stay where they were — but the individual coefficients become unstable, their standard errors inflate, and p-values that should be tiny become large.

The signature is distinctive: a model that predicts well overall while none of its correlated predictors is individually significant.

The variance inflation factor quantifies it. Regress one predictor on the others; VIF is 1/(1 − R²) from that auxiliary model. Above 5 is a warning, above 10 a problem.

This block puts the same measurement into a model twice, in two units.

set.seed(608)
n <- 200
height_cm <- rnorm(n, 170, 10)
height_in <- height_cm / 2.54 + rnorm(n, 0, 0.35)   # nearly the same variable
weight    <- 50 + 0.5 * height_cm + rnorm(n, 0, 6)

cat("Correlation between the two height measures:",
    round(cor(height_cm, height_in), 4), "\n\n")

one  <- lm(weight ~ height_cm)
both <- lm(weight ~ height_cm + height_in)

cat("One predictor  : estimate", round(coef(one)[2], 3),
    " SE", round(summary(one)$coefficients[2, 2], 3),
    " p", signif(summary(one)$coefficients[2, 4], 3), "\n")
cat("Both predictors: estimate", round(coef(both)[2], 3),
    " SE", round(summary(both)$coefficients[2, 2], 3),
    " p", signif(summary(both)$coefficients[2, 4], 3), "\n\n")

# VIF is 1 / (1 - R^2) from regressing a predictor on the others.
r2 <- summary(lm(height_cm ~ height_in))$r.squared
cat("VIF for height_cm = 1 / (1 -", round(r2, 4), ") =", round(1 / (1 - r2), 1), "\n")
cat("Rule of thumb: above 5 is a warning, above 10 a problem.\n\n")
cat("R-squared barely moved:", round(summary(one)$r.squared, 4), "->",
    round(summary(both)$r.squared, 4), "\n")
cat("Prediction is fine. The individual coefficients are not interpretable.\n")

The two height measures correlate at 0.9964. Alone, height_cm has an estimate of 0.543 with a standard error of 0.043 and p = 4.31e-27. With both in the model the estimate moves to 0.696 and the standard error explodes to 0.511, giving p = 0.174 — no longer significant. VIF is 139.6. Yet R-squared barely moves, from 0.4447 to 0.445: the model predicts exactly as well and can no longer say which variable does the work.

The mistake this prevents

The mistake is concluding a predictor does not matter because its p-value is large in a collinear model. The information is there; the model simply cannot attribute it.

Takeaway

Check correlations between predictors before modelling, and compute VIF when several are related. If you only need predictions, collinearity is harmless; if you need to interpret coefficients, drop or combine the redundant variables.

Mark complete