Unit 12.04: Reporting quality with its limits

The report's credibility comes from the two fields nobody is required to include.

Scored separately, with limits and gaps

Retrieval, citation and refusal scored apart, latency percentiles against the budget, known limits, and what has not been tested.

The code prints the full report.

import json

report = {
    "eval_set": {"cases": 24, "kinds": ["direct", "paraphrase", "compound",
                                        "unanswerable", "conflict", "stale"],
                 "refusal_share": 0.33},
    "scored_separately": {"retrieval_recall": "20/24", "citation_supported": "18/24",
                          "refusal_decision": "23/24"},
    "latency": {"p50_ms": 640, "p95_ms": 1180, "budget_ms": 800,
                "over_budget_share": 0.18},
    "known_limits": [
        "single corpus, English only",
        "staleness surfaced but not auto-detected",
        "citation support is term overlap, not entailment",
        "18% of requests exceed the 800ms budget",
    ],
    "not_yet_tested": ["concurrent sessions", "index rebuild mid-request",
                       "a source document being deleted while cited"],
    "next_steps": ["grow the eval set to 40", "add a reranker and re-measure",
                   "investigate p95 latency"],
}
print(json.dumps(report, indent=2))

print("\nThe two fields that make this credible are `known_limits` and")
print("`not_yet_tested`. A report with neither has not been reviewed.")

over_budget_share: 0.18 states that nearly a fifth of requests miss the latency target set in the scope. That is the sort of number a report is tempted to aggregate away, and it is one of the most actionable things in the document.

not_yet_tested is what separates a report from a claim. "A source document being deleted while cited" is a real gap, and naming it lets a reviewer weigh it rather than discover it.

The mistake this prevents

The mistake is reporting a single quality percentage. It cannot be acted on, it hides which of three subsystems is weak, and it invites a comparison against other systems measured differently.

Takeaway

Report the three scores separately, state latency against the budget including the share that misses it, and list both known limits and untested areas. A report with neither has not been reviewed.

Mark complete