Unit 11.05: Slice finely enough and a winner appears

Slice a null result finely enough and a winner will appear. It is guaranteed, not unlucky.

Subgroups are just more tests

When a test shows nothing overall, the temptation is to look for the segment where it worked. Every segment is another hypothesis test at the same alpha, so with twenty slices you should expect one to reach significance even when the effect is exactly zero.

The finding will also come with a story — of course the effect is stronger for email referrals, that segment is more engaged. The story is constructed after the fact and would have been equally available for whichever segment happened to win.

Subgroup analyses are legitimate when pre-specified and reported as exploratory otherwise.

This block runs a treatment with exactly zero effect and then slices it twenty ways.

set.seed(806)
n <- 3000
# The treatment does NOTHING. Every difference below is noise.
d <- data.frame(
  arm      = rep(c("A", "B"), each = n / 2),
  outcome  = rnorm(n),
  country  = sample(c("UK", "US", "DE", "FR", "ES", "IT", "NL", "SE"), n, replace = TRUE),
  device   = sample(c("ios", "android", "web"), n, replace = TRUE),
  segment  = sample(c("new", "returning"), n, replace = TRUE),
  plan     = sample(c("free", "basic", "pro"), n, replace = TRUE),
  referrer = sample(c("search", "social", "direct", "email"), n, replace = TRUE)
)

overall <- t.test(outcome ~ arm, data = d)
cat("Overall effect: p =", round(overall$p.value, 4), " -- nothing, correctly.\n\n")

cat("Now slice by every subgroup and look for a winner:\n")
found <- 0; tested <- 0
for (v in c("country", "device", "segment", "plan", "referrer")) {
  for (lev in unique(d[[v]])) {
    sub <- d[d[[v]] == lev, ]
    p <- t.test(outcome ~ arm, data = sub)$p.value
    tested <- tested + 1
    if (p < 0.05) {
      found <- found + 1
      cat(sprintf("  %s = %-10s p = %.4f  <- 'B wins for these users!'\n", v, lev, p))
    }
  }
}
cat("\nSubgroups tested:", tested, "  'significant':", found, "\n")
cat("Expected by chance:", round(tested * 0.05, 1), "\n")
cat("The treatment effect was exactly zero by construction.\n")

The overall test correctly finds nothing at p = 0.4624. Slicing across five variables produces 20 subgroup tests, and 1 comes back 'significant' — email referrals at p = 0.0498. The expected number by chance is exactly 1. The treatment effect was zero by construction, so that entire finding is noise, and it would have been easy to write a convincing paragraph about it.

The mistake this prevents

The mistake is reporting the winning subgroup without saying how many were tested. One significant result in twenty is precisely what a null effect produces.

Takeaway

Pre-specify any subgroup you intend to test, and report the total number tested alongside any that reach significance. Treat post-hoc subgroup findings as hypotheses for a new experiment, never as results.

Mark complete