Unit 11.00: Self-selection puts the answer inside the question

A control group is only a comparison if it is comparable. When people choose their own group, it is not.

Self-selection puts the answer inside the question

Comparing those who adopted a feature with those who did not looks like a treatment-and-control comparison and is not one. The two groups differed before the feature existed, and whatever made them different is likely to affect the outcome too.

The measured difference is then the treatment effect plus the pre-existing gap, with no way to separate them from the data alone.

This is the most common error in product analytics, and it usually overstates the effect, because the people most likely to adopt are the people most likely to do well anyway.

This block compares adopters with non-adopters where adoption is driven by prior engagement.

import numpy as np, pandas as pd

rng = np.random.default_rng(801)
n = 400
# Observational: users CHOOSE the feature, and keen users choose it more.
engagement = rng.normal(50, 12, n)
adopted = rng.binomial(1, 1 / (1 + np.exp(-(-4 + 0.08 * engagement))))
outcome = 20 + 0.6 * engagement + 2 * adopted + rng.normal(0, 5, n)

print("--- observational comparison ---")
print(f"Adopters mean    : {outcome[adopted == 1].mean():.2f}")
print(f"Non-adopters mean: {outcome[adopted == 0].mean():.2f}")
print(f"Naive difference : {outcome[adopted == 1].mean() - outcome[adopted == 0].mean():.2f}")
print("True effect built into the data: 2\n")

print(f"Baseline engagement, adopters {engagement[adopted == 1].mean():.1f}"
      f" vs non-adopters {engagement[adopted == 0].mean():.1f}")
print("The groups differed BEFORE the feature existed. That gap is inside the")
print("naive difference and cannot be separated from the effect.\n")
print("A control group is only a comparison if it is comparable.")
print("Self-selection is what randomisation exists to prevent.")

Adopters average 54.99 against non-adopters' 48.31 — a naive difference of 6.68, against a true effect of 2 built into the data. The reason is in the next line: baseline engagement was 55.8 for adopters and 46.0 for non-adopters. They differed before the feature existed, and that gap is sitting inside the 6.68.

The mistake this prevents

The mistake is calling the non-adopters a control group. They are a different population, selected by the same thing that predicts the outcome.

Takeaway

Check whether group membership was assigned or chosen. When it was chosen, compare the groups on pre-treatment characteristics and report the imbalance — and treat the difference as an association, not an effect.

Mark complete