Unit 12.00: Choosing the application and its quality bar

The capstone begins with an application, a consequence of error, and a bar that names its own judge.

Four things a bar must name

The set, the judge, at least one zero-tolerance metric, and bounds on latency and cost.

The code defines an application and checks all four.

import json

app = {
    "application": "support policy assistant for agents",
    "users": "support agents, during a live conversation",
    "consequence_of_error": "agent tells a customer the wrong policy",
    "quality_bar": {
        "set": "40 cases, 7 kinds, 33% expecting refusal",
        "correctness": 0.90, "grounding": 0.95, "refusal_accuracy": 0.95,
        "format": 0.99, "approval_violations": 0,
        "p95_latency_ms": 2000, "cost_per_request_usd": 0.005,
    },
    "judge": "rubric-based human labelling, two labellers on a 20% sample",
}
print(json.dumps(app, indent=2))

checks = [
    ("names the set", "set" in app["quality_bar"]),
    ("names the judge", bool(app["judge"])),
    ("has a zero-tolerance metric", app["quality_bar"]["approval_violations"] == 0),
    ("bounds latency and cost", "p95_latency_ms" in app["quality_bar"]),
]
for check, ok in checks:
    print(f"  {'OK  ' if ok else 'FAIL'} {check}")

approval_violations: 0 is the zero-tolerance metric - a count rather than a rate, for the reason Module 6 gives. Every quality bar should have at least one thing that must never happen, because a bar made entirely of rates permits every failure at some frequency.

The judge is specified as rubric-based human labelling with two labellers on a sample. That is a real commitment of effort, and stating it is what makes the correctness number mean anything.

The mistake this prevents

The mistake is writing the bar without naming the judge. "90% correct" with no stated judge gets computed differently by different people at different times, and the number becomes uncomparable across the runs it was supposed to compare.

Takeaway

Name the set, the judge, a zero-tolerance metric and the operational bounds. A bar made entirely of rates permits every failure at some frequency.

Mark complete