Unit 12.04: The report and the gates it proposes

The report's verdict is the test of whether it was actually run.

Numbers, verdict, and the gates that follow

Quality, operations, verdict, proposed gates, limits, untested areas, next steps.

The code prints the complete report.

import json

report = {
    "application": "support policy assistant",
    "quality": {"correctness": "36/40 (0.90)", "grounding": "35/40 (0.88)",
                "refusal_accuracy": "12/13 (0.92)", "format": "40/40 (1.00)"},
    "operations": {"p95_latency_ms": 1840, "cost_per_request_usd": 0.0031,
                   "retry_rate": 0.18, "refusal_rate": 0.33},
    "verdict": "DOES NOT SHIP -- grounding 0.88 against a 0.95 bar",
    "proposed_gates": [
        {"metric": "approval_violations", "bar": 0, "op": "==", "severity": "blocker"},
        {"metric": "grounding", "bar": 0.95, "op": ">=", "severity": "blocker"},
        {"metric": "newly_broken_cases", "bar": 0, "op": "==", "severity": "blocker"},
        {"metric": "p95_latency_ms", "bar": 2000, "op": "<=", "severity": "blocker"},
        {"metric": "cost_per_request_usd", "bar": 0.005, "op": "<=",
         "severity": "warning"},
    ],
    "known_limits": ["English only", "staleness not auto-detected",
                     "citation check is term overlap"],
    "not_tested": ["concurrent sessions", "index rebuild mid-request"],
    "next_steps": ["fix grounding: add the recency filter and re-run",
                   "reduce the 18% retry rate", "grow the set to 60 cases"],
}
print(json.dumps(report, indent=2))

print("\nA report whose verdict is DOES NOT SHIP is a report that was actually")
print("run. One that always concludes otherwise is a formality.")

The verdict is DOES NOT SHIP, on grounding at 0.88 against a 0.95 bar. A report that reaches that conclusion is a report that was run against bars set beforehand - and a process that never reaches it is a formality.

The proposed gates are the report's real output. They turn this one measurement into a standing check, so the next release is evaluated automatically against the bars this exercise established.

The mistake this prevents

Close with what happens after the release, not only what gated it. Name the metrics monitored in production, the thresholds that page someone, and what would trigger re-running this whole evaluation - a model version change, a corpus update, a shift in the questions users ask. A report that ends at the release gate covers the day you ship and no day after it.

The mistake is writing the report as a summary of work done. Its purpose is a decision and a set of gates; if it produces neither, the measurement happened and nothing followed from it.

Takeaway

End with a verdict and the gates that make it repeatable. A reliability report that never blocks a release is a report nobody is running.

Mark complete