Unit 08.05: Two different jobs for the same equation

A model that predicts well can be completely wrong about why, and which kind of wrong you can tolerate depends on the question.

Two different jobs for the same equation

For prediction the only thing that matters is whether the forecasts are accurate. A predictor that is merely correlated with the outcome through some third variable is perfectly acceptable.

For explanation you want to know what would happen if you intervened, and a merely correlated predictor is now actively misleading. A confounder — something that causes both variables — produces a strong, significant, entirely spurious relationship.

Adding the confounder to the model usually makes the spurious relationship collapse, which is a useful diagnostic. It only works for confounders you thought of and measured, which is why observational explanation is hard and randomisation is valuable.

This block regresses drownings on ice cream sales, then adds temperature.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(506)
n = 200
d = pd.DataFrame({"temperature": rng.uniform(5, 35, n)})
d["ice_cream"] = 20 + 3 * d.temperature + rng.normal(0, 10, n)
d["drownings"] = 2 + 0.30 * d.temperature + rng.normal(0, 1.5, n)

print(f"Correlation, ice cream and drownings: {d.ice_cream.corr(d.drownings):.3f}\n")

naive = smf.ols("drownings ~ ice_cream", data=d).fit()
print(f"Slope on ice cream alone : {naive.params['ice_cream']:.4f}"
      f"   p = {naive.pvalues['ice_cream']:.3g}")

adjusted = smf.ols("drownings ~ ice_cream + temperature", data=d).fit()
print("\nWith temperature in the model:")
print(f"  ice cream   : {adjusted.params['ice_cream']:+.4f}"
      f"   p = {adjusted.pvalues['ice_cream']:.3g}")
print(f"  temperature : {adjusted.params['temperature']:+.4f}"
      f"   p = {adjusted.pvalues['temperature']:.3g}\n")

print("The ice cream effect vanishes once temperature is accounted for.")
print(f"For PREDICTION the naive model is genuinely useful -- R-squared"
      f" {naive.rsquared:.3f};")
print("ice cream sales do forecast drownings. For EXPLANATION it is worthless,")
print("and banning ice cream would save nobody.")
print("Which question you are answering decides which model is wrong.")

Ice cream sales and drownings correlate at 0.811, and the naive regression gives a slope of 0.0847 with p = 5.4e-48 — about as significant as results get. Adding temperature, the ice cream slope falls to +0.0096 with p = 0.391, while temperature takes a slope of 0.2623. The relationship was entirely temperature. For prediction the naive model is genuinely useful — R-squared 0.658, ice cream sales do forecast drownings — and for explanation it is worthless.

The mistake this prevents

The mistake is reading a significant coefficient from observational data as a cause. The p-value measures evidence against zero association, and a confounded association is a real association.

Takeaway

Decide whether you are predicting or explaining before choosing a model. For explanation, list the plausible confounders and include the ones you have, and say plainly which ones you could not measure.

Mark complete