Outcome: Score a workplace summary against its source and decide whether to accept, revise, or reject it.
Prerequisites: The complete source, the summary, its intended audience, and permission to use the material. No coding needed.
A summary can read smoothly while turning a suggestion into a decision. Review it with a short evidence-based rubric before using it in meeting notes, a handover, or a project update. Keep the original open throughout the review.
Set the review target first
Write down the summary's purpose and a checklist of essential facts before grading. For a manager deciding next steps, that might include the result, its limitations, decisions, unresolved issues, owners, and deadlines. Otherwise, a polished answer can quietly redefine what “good” means.
Anthropic's evaluation guidance distinguishes checks for source support, key-fact coverage, and source quality. The rubric below adapts that practical separation for human review of summaries. Its scoring rules are a teaching tool, not a published industry standard. Anthropic evaluation guidance
The illustrative source and summary
All details below are synthetic examples created for this lesson, not a live model evaluation.
Source paragraphs:
-
S1: Twelve support staff tested a workflow from 1–14 September 2026.
-
S2: Median ticket resolution time fell from 50 to 40 minutes.
-
S3: Thirty percent of tickets lacked category tags, so comparisons remain preliminary.
-
S4: Lina proposed extending the pilot by two weeks. Omar will validate tags by 18 September.
-
S5: No decision was made on rollout or budget.
For the manager's update, require six items: pilot scope and dates; the measured result; the data limitation; extension proposal status; action owner and deadline; unresolved rollout and budget.
Illustrative summary:
“From 1–14 September 2026, twelve support staff tested a workflow. Median resolution time fell from 50 to 40 minutes. Lina approved a two-week extension. Omar will validate tags by 18 September. [S1, S2, S4]”
Score three dimensions from zero to two
-
Factuality: Does each claim preserve the source's meaning?
-
2: Every claim is supported, including quantities, uncertainty, and decision status.
-
1: A minor ambiguity needs clarification but changes no decision, responsibility, or finding.
-
0: Any unsupported or contradicted material claim, or a misleading change in meaning.
Score the example 0. “Approved” changes Lina's proposal into an authorization. Correct dates and numbers elsewhere do not cancel that error.
-
Coverage: Are the required facts included accurately?
-
2: All six checklist items are present.
-
1: Three to five are present.
-
0: Zero to two are present.
Score the example 1. It correctly covers scope and dates, the measured result, and Omar's action: three items. It omits the data limitation and unresolved decisions, and misstates the proposal. Incorrectly mentioning an item earns no coverage credit. For another task, define the checklist and anchors before scoring.
-
Attribution: Can the reader trace claims to supporting evidence?
-
2: Each claim has an accurate paragraph, page, timestamp, or equivalent locator.
-
1: The correct sources exist, but some locators are missing or too vague.
-
0: A source is fabricated, a locator is wrong, or a citation misleadingly supports a claim it does not establish.
Score the example 0. S4 establishes a proposal, never approval. Listing the right paragraph number does not make the altered claim supported. NIST specifically recommends reviewing and verifying generated sources and citations. NIST Profile, MS-2.5-003
Apply hard stops before accepting a score
Reject the current draft if it invents a decision, quotation, or source; changes a consequential number, owner, or deadline; or drops a caveat that changes interpretation. If the source is incomplete or inaccessible, pause the review as unverified.
Our example scores 1/6 and triggers hard stops for invented approval and the missing data caveat. Request revision. For this routine workplace rubric, require 6/6, no hard stops, and human sign-off before use.
Repair and rescore
A corrected summary reads:
“Twelve support staff tested the workflow from 1–14 September 2026 (S1). Median resolution time fell from 50 to 40 minutes (S2), but missing category tags on 30% of tickets make comparisons preliminary (S3). Lina proposed a two-week extension; Omar will validate tags by 18 September (S4). Rollout and budget remain undecided (S5).”
This earns 2 + 2 + 2 = 6/6 against the supplied source.
Mistakes and limits
A faithful summary can repeat a source's own error. Verify the underlying evidence separately when consequences warrant it. Never award factuality points for elegant wording. If AI helps grade, check its decisions against human examples; OpenAI recommends clear criteria and calibration against human judgment. OpenAI evaluation best practices
Try it
Replace “median” with “mean” in the corrected summary. Is it still acceptable?
Answer: No. The source specifies median. Substituting mean changes the reported metric; restore the original term and recheck.
Sources: The three primary sources linked above were checked on 5 October 2026.