Unit 07.02: Metrics worth alerting on

Not everything worth measuring is worth alerting on, and the distinction matters more than the thresholds.

Count, rate, gate

Approval violations are a count with a threshold of zero. Format failures are a rate worth paging on. Correctness is neither - it needs labels, so it belongs in a release gate.

The code sorts seven metrics.

METRICS = [
    ("approval violations",        "count", 0,     "page immediately"),
    ("format failure rate",        "rate",  0.01,  "page"),
    ("refusal rate",               "rate",  0.35,  "alert on a big move either way"),
    ("p95 latency",                "ms",    2000,  "alert"),
    ("cost per request",           "usd",   0.005, "alert"),
    ("grounding rate",             "rate",  0.95,  "daily report, not a page"),
    ("correctness",                "rate",  0.90,  "release gate, not an alert"),
]
print(f"{'metric':24} {'kind':6} {'threshold':>10}  response")
for name, kind, threshold, response in METRICS:
    print(f"{name:24} {kind:6} {threshold:>10}  {response}")

print("""
Two distinctions worth making. Approval violations are a COUNT with a threshold
of zero -- a rate would hide a single breach in a busy day.

And correctness is not an alert. It needs a labelled set to compute, so it
belongs in a release gate. Alerting on things you cannot compute in real time
produces alerts nobody can act on.
""")

Correctness cannot be an alert because it cannot be computed in real time without labelled data. Alerting on things you cannot compute produces alerts nobody can act on, and eventually alerts nobody reads.

The refusal rate is worth watching in both directions. A sharp fall means the system stopped refusing - the threshold drifted or the corpus grew - and a sharp rise means retrieval broke. Neither is visible if you only alert on it going up.

The mistake this prevents

The mistake is putting every quality metric on the same dashboard with the same alerting. Distinguish what pages someone, what gets reviewed daily, and what gates a release - mixing them produces either alert fatigue or a gate nobody enforces.

Takeaway

Separate alerts, daily reviews and release gates. Approval violations are a count that must be zero; correctness needs labels and belongs in a gate, not an alert.

Mark complete