Unit 11.03: Primary, secondary, guardrail

One metric decides the launch. The others exist to stop it.

Primary, secondary, guardrail

The primary metric is named in advance and is the only one that can decide the test. Having exactly one is what makes the alpha you set the alpha you actually get.

Secondary metrics are reported for understanding and cannot promote a failing test to a success.

Guardrails are metrics that must not get worse — support load, error rates, latency, unsubscribes. They can block a launch but never justify one. Naming them in advance is what makes an adverse move a finding rather than an argument about whether it counts.

This block reports a primary metric and a guardrail from the same experiment.

import numpy as np
from statsmodels.stats.proportion import (proportions_ztest,
                                          confint_proportions_2indep)

rng = np.random.default_rng(804)
n_arm = 4000
metrics = {
    "retained": (0.300, 0.325),     # primary
    "support":  (0.040, 0.058),     # guardrail
}
for name, (pa, pb) in metrics.items():
    a = rng.binomial(n_arm, pa)
    b = rng.binomial(n_arm, pb)
    _, p = proportions_ztest([a, b], [n_arm, n_arm])
    lo, hi = confint_proportions_2indep(b, n_arm, a, n_arm,
                                        compare="diff", method="wald")
    print(f"{name:9s} A {a / n_arm:.3f}  B {b / n_arm:.3f}"
          f"  diff {(b - a) / n_arm:+.3f}"
          f"  CI [{lo:+.3f}, {hi:+.3f}]  p {p:.4f}")

print("\nretained is the PRIMARY metric: it decides the launch.")
print("support is a GUARDRAIL: it cannot win the test, only block it.\n")
print("B improves retention and raises support contacts. That trade-off is")
print("for a human to weigh, and it is only visible because the guardrail was")
print("named in the plan rather than discovered afterwards.")

Retention moves from 0.305 to 0.316 — a difference of +0.010 with an interval from −0.010 to +0.031 and p = 0.3218, which is nothing. Support contacts rise from 0.039 to 0.060, an increase of +0.021 with an interval from +0.012 to +0.031 and p below 0.0001. The primary metric did not move and the guardrail moved adversely and unambiguously — a clear result, and clear only because both were named beforehand.

The mistake this prevents

The mistake is promoting a secondary metric to primary when the primary disappoints. It is the same multiple-comparison problem dressed as judgement.

Takeaway

Name one primary metric, a short list of secondaries, and the guardrails before the test starts. Report all of them, let only the primary decide, and allow any guardrail to veto.

Mark complete