Unit 11.01: Balance in expectation, on everything

Randomisation balances the variables you never thought of. Nothing else does.

Balance in expectation, on everything

Random assignment makes group membership independent of every characteristic a unit had beforehand — measured, unmeasured, known and unknown. Any remaining difference in the outcome is attributable to the treatment, up to sampling variation.

That is a stronger guarantee than adjustment can ever offer, because adjustment can only handle variables you measured.

A balance check on the covariates you do have is still worth reporting. It cannot verify balance on the unmeasured ones, but a large imbalance on a measured covariate is a signal that the randomisation may have gone wrong mechanically.

This block randomises assignment and checks balance before estimating the effect.

set.seed(802)
n <- 400
engagement <- rnorm(n, 50, 12)         # a confounder we happen to have measured
tenure     <- runif(n, 0, 36)          # and one we do not use in the model

assigned <- sample(rep(c("control", "treated"), each = n / 2))
outcome  <- 20 + 0.6 * engagement + 2 * (assigned == "treated") + rnorm(n, 0, 5)

cat("Balance check on a measured covariate:\n")
cat("  engagement, control", round(mean(engagement[assigned == "control"]), 2),
    " treated", round(mean(engagement[assigned == "treated"]), 2),
    " p =", signif(t.test(engagement ~ assigned)$p.value, 3), "\n")
cat("  tenure,     control", round(mean(tenure[assigned == "control"]), 2),
    " treated", round(mean(tenure[assigned == "treated"]), 2),
    " p =", signif(t.test(tenure ~ assigned)$p.value, 3), "\n\n")

tt <- t.test(outcome ~ assigned)      # estimate is c(control, treated)
effect <- as.numeric(diff(tt$estimate))            # treated - control
ci     <- rev(-tt$conf.int)                        # flip to match
cat("Estimated effect:", round(effect, 2),
    " 95% CI [", round(ci[1], 2), ",", round(ci[2], 2), "]\n")
cat("True effect     : 2\n\n")
cat("Randomisation balances everything -- measured and unmeasured, known and\n")
cat("unknown -- in expectation. That is what no amount of adjustment can buy.\n")

Engagement is balanced — 49.57 against 49.43, p = 0.903 — and tenure is reasonably balanced at 18.13 against 16.4, p = 0.0954, which is the kind of variation randomisation produces. The estimated effect is 1.65 with a 95% interval from −0.06 to 3.36, against a true effect of 2. The estimate is close and the interval, marginally including zero, is an honest statement of what 400 observations can establish.

The mistake this prevents

The mistake is discarding a randomisation because one covariate looks imbalanced. With enough covariates, some will differ by chance — that is what a p-value of 0.05 means — and re-randomising until everything balances breaks the randomisation.

Takeaway

Randomise whenever you can, and report a balance table on the covariates you measured. Do not re-randomise to chase balance, and do not adjust away chance imbalances that were not pre-specified.

Mark complete