Unit 09.03: Only as good as the variables you measured

'Adjusted for' is a strong claim. It is only as good as the variables you thought to measure.

Comparing units that match on the other predictors

Adjusting for a variable means comparing observations that share the same value of it. When a confounder drives both the predictor and the outcome, the unadjusted association carries the confounder's effect as well, and adjustment separates them.

The technique works, and only for confounders that are in the model. Anything you did not measure continues to contaminate the estimate, silently and by an unknown amount.

This is the central limitation of observational analysis, and no amount of modelling sophistication removes it. It is why a randomised experiment is worth so much.

This block builds data where experience drives both training hours and salary.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(604)
n = 300
# Experience drives both the training hours and the salary.
experience = rng.uniform(0, 25, n)
d = pd.DataFrame({
    "experience": experience,
    "training": 5 + 1.8 * experience + rng.normal(0, 6, n),
})
d["salary"] = (30_000 + 1800 * d.experience + 50 * d.training
               + rng.normal(0, 4000, n))

unadj = smf.ols("salary ~ training", data=d).fit()
adj = smf.ols("salary ~ training + experience", data=d).fit()

print("Unadjusted: salary ~ training")
print(unadj.params.round(1).to_string())
print("\nAdjusted for experience: salary ~ training + experience")
print(adj.params.round(1).to_string())

u, a = unadj.params["training"], adj.params["training"]
print(f"\nTraining coefficient: {u:.0f} -> {a:.0f}")
print("The value actually used to build the data was 50.")
print(f"The unadjusted estimate was {u / 50:.1f} times too large, because it")
print("was carrying experience's effect as well as training's.")
print("\n'Adjusted for' means comparing units that share the same value of the")
print("other predictor. It is only as good as the variables you measured.")

Unadjusted, the training coefficient is 888. Adjusted for experience it is 82 — and the value actually used to build the data was 50. The unadjusted estimate was 17.8 times too large, because it was carrying experience's effect as well as training's. Adjustment recovered the truth here precisely because the confounder was measured.

The mistake this prevents

The mistake is treating an adjusted estimate as causal. It is adjusted for what you measured; the confounders you never thought of are still in there.

Takeaway

Say 'adjusted for X, Y' rather than 'controlling for confounders', and list the confounders you could not measure in the limitations. Treat a large shift between unadjusted and adjusted estimates as evidence that more may be lurking.

Mark complete