Unit 13.04: The evaluation checklist and its stated limits

The deliverable is a checklist someone else can run, ending in a verdict.

Eval set, results, operations, boundary checks

Composition, three scored dimensions, operational numbers, and the boundary flags.

The code prints the report.

import json

report = {
    "eval_set": {"cases": 24, "kinds": ["direct", "paraphrase", "unanswerable",
                                        "out of scope", "compound"],
                 "refusal_share": 0.33},
    "results": {"correct": "20/24", "cited_correctly": "19/24",
                "refusal_decision": "23/24", "format_valid": "24/24"},
    "operations": {"p95_latency_ms": 2840, "usd_per_day": 1.60,
                   "retry_rate": 0.09},
    "boundary_checks": {"auto_send_possible": False,
                        "card_numbers_rejected": True,
                        "key_in_repo": False},
    "known_limits": ["English only", "published policies only, not account data",
                     "citation check is term overlap, not entailment"],
    "not_tested": ["concurrent sessions", "policy document updated mid-request"],
    "verdict": "ready for a supervised pilot with the send button manual",
}
print(json.dumps(report, indent=2))

print("\n`boundary_checks` are the ones that must be exactly true, not high")

boundary_checks are the ones that must be exactly true rather than high. auto_send_possible: False is a structural fact, and a value of True would block release regardless of every other number.

The verdict is a supervised pilot with the send button manual - a real decision that follows from the numbers rather than a summary of them.

The mistake this prevents

The mistake is reporting a single quality percentage. It cannot be acted on, it hides which of three dimensions is weak, and it invites comparison against systems measured differently.

Takeaway

Report the dimensions separately, keep the boundary checks as exact facts, and end with a verdict. A report that never blocks a release is a formality.

Mark complete