Unit 11.01: Balance in expectation, on everything

Randomisation balances the variables you never thought of. Nothing else does.

Balance in expectation, on everything

Random assignment makes group membership independent of every characteristic a unit had beforehand — measured, unmeasured, known and unknown. Any remaining difference in the outcome is attributable to the treatment, up to sampling variation.

That is a stronger guarantee than adjustment can offer, because adjustment can only handle variables you measured.

A balance check on the covariates you do have is still worth reporting. It cannot verify balance on the unmeasured ones, but a large imbalance on a measured covariate signals that the randomisation may have gone wrong mechanically.

This block randomises assignment and checks balance before estimating the effect.

import numpy as np, pandas as pd
from scipy import stats

rng = np.random.default_rng(802)
n = 400
engagement = rng.normal(50, 12, n)      # a confounder we happen to measure
tenure = rng.uniform(0, 36, n)          # and one the model never uses

assigned = rng.permutation(np.repeat(["control", "treated"], n // 2))
treated = (assigned == "treated")
outcome = 20 + 0.6 * engagement + 2 * treated + rng.normal(0, 5, n)

print("Balance check on measured covariates:")
for name, v in [("engagement", engagement), ("tenure", tenure)]:
    p = stats.ttest_ind(v[treated], v[~treated], equal_var=False).pvalue
    print(f"  {name:11s} control {v[~treated].mean():6.2f}"
          f"   treated {v[treated].mean():6.2f}   p = {p:.3f}")

res = stats.ttest_ind(outcome[treated], outcome[~treated], equal_var=False)
ci = res.confidence_interval()
print(f"\nEstimated effect: {outcome[treated].mean() - outcome[~treated].mean():.2f}"
      f"   95% CI [{ci.low:.2f}, {ci.high:.2f}]")
print("True effect     : 2\n")
print("Randomisation balances everything -- measured and unmeasured, known and")
print("unknown -- in expectation. That is what no amount of adjustment buys.")

Both measured covariates are balanced — engagement 50.33 against 49.57 (p = 0.535) and tenure 17.71 against 17.81 (p = 0.920). The estimated effect is 1.08 with a 95% interval from −0.60 to 2.76, against a true effect of 2. The interval contains the truth and also contains zero: an honest statement of what 400 observations can establish about an effect this size.

The mistake this prevents

The mistake is discarding a randomisation because one covariate looks imbalanced. With enough covariates some will differ by chance, and re-randomising until everything balances breaks the randomisation.

Takeaway

Randomise whenever you can, and report a balance table on the covariates you measured. Do not re-randomise to chase balance, and do not adjust away chance imbalances that were not pre-specified.

Mark complete