Check the judge as well as the answer

The route says escalate. The accompanying response says, “A staff member must review this. Your refund is approved.” The route-only evaluator passes the label even though the response contradicts the staff-only boundary. Adding a score is easy; deciding what the score means is the harder work.

You will build a small human-review rubric, inspect disagreement, and calculate a grader's false acceptances using explicit denominators. The program aggregates supplied synthetic human judgments. It does not understand language, call a model judge, or claim that its ratings came from actual reviewers.

Anchor criteria in observable behavior

Start from the fictional policy in operations.json: public Saturday hours are 10:00–16:00; staff decide individual refunds; the public router cannot disclose attendee contact details. We use three binary criteria:

A rubric should tell reviewers how to handle uncertainty and applicability. Here a clarification that asks what “it” refers to can pass grounded because it asserts no unsupported fact. A genuinely unjudgeable output in real review should be flagged for adjudication rather than coerced into a convenient pass. This small schema has only booleans and therefore cannot represent every useful review state; do not silently interpret a missing rating as true.

Inspect a disagreement before averaging it away

Run from project:

python3 -B opslab.py review

Four outputs have three criteria and two synthetic reviewers. We compare the reviewers on each output/criterion pair, giving twelve comparisons. They agree on eleven. Raw agreement is therefore 11/12 ≈ 91.7%.

For output s2, both reviewers fail within_scope, but they disagree about grounded. One reviewer treated the supported first sentence as sufficient; the other applied the “every factual assertion” anchor. That disagreement suggests rubric clarification. It does not justify accepting the approval claim. The program conservatively lists s2 and s3 as failed outputs when any criterion rating fails and returns REVIEW.

Raw agreement is not accuracy against truth, and it is not adjusted for chance or class imbalance. Two people could agree on every output while both miss the same failure. A high agreement number is evidence about consistency under this comparison, not proof that the rubric protects users.

Output s3 deliberately illustrates a false consensus: both supplied reviewers pass grounded and within_scope even though the contact assertion and claimed handoff are unsupported. Its privacy criterion still fails, but that does not repair the other mistaken ratings. A domain review should challenge that consensus and version corrected judgments. The fixture preserves the mistake so you can see why agreement is not a reference-truth label.

Calibrate a proposed judge against a reference

Imagine a proposed automated judge that says whether an output is acceptable. Before trusting it, compare its decisions with reviewed reference decisions. Our six calibration records are invented to make the calculation visible. “Positive” explicitly means acceptable:

Run:

python3 -B opslab.py calibrate

Acceptance precision asks, “Of the three outputs this judge accepted, how many were reference-acceptable?” It is TP/(TP+FP) = 2/3. Acceptance recall asks, “Of the three reference-acceptable outputs, how many did it accept?” It is TP/(TP+FN) = 2/3. The false-acceptance rate asks, “Of the three reference-unacceptable outputs, how many slipped through?” It is FP/(FP+TN) = 1/3.

These questions have different denominators even though two happen to give the same number. If there were no reference-unacceptable examples, the last denominator would be zero; the program returns null, meaning undefined. Reporting zero would falsely suggest a tested absence of false acceptances.

Choose where automation helps

An exact rule can check IDs, required fields, or an allowed route. A rubric-based human review can inspect meaning, context, and competing interpretations. A model judge may reduce manual effort in a future system, but its errors, prompt sensitivity, and agreement with domain review need their own evaluation. Saving a judge version alongside a system version prevents a changed grader from being mistaken for an improved application.

OpenAI's evaluation guidance discusses calibrating automated scoring with human judgment. Our worked numbers illustrate that principle; they are not a provider performance result. The original reviewer votes and reference labels here are teaching artifacts, not an independent adjudication study.

Practice

1. Apply the three anchors to “Which workshop would you like to move?” Explain why a lack of factual claims is different from a fabricated claim.
2. Compute all three calibration rates without running the command, then compare. Explain the denominator of the false-acceptance rate in words.
3. In memory, call grader_calibration with one record whose reference_ok and judge_ok are both True. Which rate becomes undefined, and what evidence would you collect next?
4. Explain how a route-only pass and a semantic failure can coexist for s2. Write a two-sentence recommendation that preserves the disagreement rather than hiding it in 91.7% agreement.

Your exit artifact is a rubric note and a calibration table with named positive class and denominators. Explained solutions.