Unit 11.00: Self-selection puts the answer inside the question

A control group is only a comparison if it is comparable. When people choose their own group, it is not.

Self-selection puts the answer inside the question

Comparing those who adopted a feature with those who did not looks like a treatment-and-control comparison and is not one. The two groups differed before the feature existed, and whatever made them different is also likely to affect the outcome.

The measured difference is then the treatment effect plus the pre-existing gap, with no way to separate them from the data alone.

This is the single most common error in product analytics, and it usually overstates the effect, because the people most likely to adopt are the people most likely to do well anyway.

This block compares adopters with non-adopters where adoption is driven by prior engagement.

set.seed(801)
n <- 400
# Observational: users CHOOSE the new feature, and keen users choose it more.
engagement <- rnorm(n, 50, 12)
adopted    <- rbinom(n, 1, 1 / (1 + exp(-(-4 + 0.08 * engagement))))
outcome    <- 20 + 0.6 * engagement + 2 * adopted + rnorm(n, 0, 5)

cat("--- observational comparison ---\n")
cat("Adopters mean  :", round(mean(outcome[adopted == 1]), 2), "\n")
cat("Non-adopters   :", round(mean(outcome[adopted == 0]), 2), "\n")
cat("Naive difference:", round(mean(outcome[adopted == 1]) -
                               mean(outcome[adopted == 0]), 2), "\n")
cat("True effect built into the data: 2\n\n")

cat("Baseline engagement, adopters vs non-adopters:",
    round(mean(engagement[adopted == 1]), 1), "vs",
    round(mean(engagement[adopted == 0]), 1), "\n")
cat("The groups differed BEFORE the feature existed. That gap is inside the\n")
cat("naive difference and cannot be separated from the effect.\n\n")
cat("A control group is only a comparison if it is comparable. Self-selection\n")
cat("is what randomisation exists to prevent.\n")

Adopters average 53.86 against non-adopters' 47.4 — a naive difference of 6.45, against a true effect of 2 built into the data. The reason is in the next line: baseline engagement was 52.8 for adopters and 45.2 for non-adopters. They differed before the feature existed, and that gap is sitting inside the 6.45.

The mistake this prevents

The mistake is calling the non-adopters a control group. They are a different population, selected by the same thing that predicts the outcome.

Takeaway

Check whether group membership was assigned or chosen. When it was chosen, compare the groups on pre-treatment characteristics and report the imbalance — and treat the difference as an association, not an effect.

Mark complete