Unit 11.06: Adjustment fixes what you thought of

Adjustment fixes the confounders you thought of. Randomisation fixes the ones you did not.

The difference is what you have to know in advance

In observational data, a confounder that drives both the exposure and the outcome produces a strong, highly significant, entirely spurious association. Adjusting for it removes the spurious part — provided you measured it and thought to include it.

An experiment needs neither. Random assignment severs the link between the exposure and everything that came before it, so no adjustment is required and no knowledge of the confounders is needed.

This is why observational studies support careful, hedged causal language and experiments support plain causal language. The difference is not sample size or sophistication; it is what you had to know beforehand.

This block runs the same question observationally, with adjustment, and as an experiment.

set.seed(807)
n <- 2000
# A confounder drives both the choice and the outcome.
health <- rnorm(n, 50, 10)
takes_supplement <- rbinom(n, 1, 1 / (1 + exp(-(-5 + 0.1 * health))))
recovery <- 30 + 0.5 * health + 0 * takes_supplement + rnorm(n, 0, 4)

naive <- t.test(recovery ~ takes_supplement)
cat("--- observational ---\n")
cat("Difference:", round(as.numeric(diff(rev(naive$estimate))), 2),
    " p =", signif(naive$p.value, 3), "\n")
cat("True effect of the supplement: 0\n\n")

adjusted <- lm(recovery ~ takes_supplement + health)
cat("Adjusted for health:", round(coef(adjusted)[2], 3),
    " p =", signif(summary(adjusted)$coefficients[2, 4], 3), "\n")
cat("Adjustment recovered the truth -- because we measured the confounder.\n\n")

assigned <- sample(rep(0:1, each = n / 2))
recovery_rct <- 30 + 0.5 * health + 0 * assigned + rnorm(n, 0, 4)
rct <- t.test(recovery_rct ~ assigned)
cat("--- randomised, no adjustment ---\n")
cat("Difference:", round(as.numeric(diff(rev(rct$estimate))), 3),
    " p =", signif(rct$p.value, 3), "\n\n")
cat("The experiment got it right without knowing health mattered. That is the\n")
cat("difference: adjustment fixes the confounders you thought of.\n")

Observationally the supplement appears to change recovery by −4.14 with p = 3.82e-49 — as significant as results get, and the true effect is 0. Adjusting for health brings it to −0.078 with p = 0.7, recovering the truth, because health was measured. The randomised version gives −0.002 with p = 0.995 without adjusting for anything at all — and would have worked equally well had nobody ever suspected that health mattered.

The mistake this prevents

The mistake is believing a very small p-value protects against confounding. It does not; a confounded association is a real association, and more data makes the p-value smaller rather than the estimate truer.

Takeaway

Randomise when you can. When you cannot, list the confounders you adjusted for and, explicitly, the ones you could not measure — and keep the language associational.

Mark complete