Unit 11.03: Primary, secondary, guardrail

One metric decides the launch. The others exist to stop it.

Primary, secondary, guardrail

The primary metric is named in advance and is the only one that can decide the test. Having one is what makes the alpha you set the alpha you actually get.

Secondary metrics are reported for understanding and cannot promote a failing test to a success.

Guardrails are metrics that must not get worse — support load, error rates, latency, unsubscribes. They can block a launch but never justify one. Naming them in advance is what makes an adverse move a finding rather than an argument about whether it counts.

This block reports a primary metric and a guardrail from the same experiment.

set.seed(804)
n_per_arm <- 4000
# One primary metric moves; a guardrail metric moves the wrong way.
retained <- c(rbinom(n_per_arm, 1, 0.300), rbinom(n_per_arm, 1, 0.325))
support  <- c(rbinom(n_per_arm, 1, 0.040), rbinom(n_per_arm, 1, 0.058))
arm <- rep(c("A", "B"), each = n_per_arm)

for (nm in c("retained", "support")) {
  v <- get(nm)
  tab <- c(sum(v[arm == "A"]), sum(v[arm == "B"]))
  pt <- prop.test(tab, c(n_per_arm, n_per_arm))
  cat(sprintf("%-9s A %.3f  B %.3f  diff %+.3f  CI [%+.3f, %+.3f]  p %.4f\n",
              nm, tab[1]/n_per_arm, tab[2]/n_per_arm,
              (tab[2]-tab[1])/n_per_arm, -pt$conf.int[2], -pt$conf.int[1],
              pt$p.value))
}

cat("\nretained is the PRIMARY metric: it decides the launch.\n")
cat("support is a GUARDRAIL: it cannot win the test, only block it.\n\n")
cat("B improves retention and raises support contacts. That is a trade-off\n")
cat("for a human to weigh, and it is only visible because the guardrail was\n")
cat("named in the plan rather than discovered afterwards.\n")

Retention moves from 0.309 to 0.314 — a difference of +0.004 with an interval from −0.016 to +0.025 and p = 0.6993, which is nothing. Support contacts rise from 0.037 to 0.060, an increase of +0.023 with an interval from +0.013 to +0.032 and p below 0.0001. The primary metric did not move and the guardrail moved adversely and unambiguously. That is a clear result, and it is only clear because both were named beforehand.

The mistake this prevents

The mistake is promoting a secondary metric to primary when the primary disappoints. It is the same multiple-comparison problem, dressed as judgement.

Takeaway

Name one primary metric, a short list of secondaries, and the guardrails before the test starts. Report all of them, and let only the primary decide — while any guardrail can veto.

Mark complete