Unit 05.01: Did the citation support the claim?

The citation dimension catches the failure that correctness scoring reports as a success.

Correct, cited, and supported are three things

An answer can be right and cite the wrong chunk, or right and cite nothing.

The code scores four cases on correctness and support separately.

CHUNKS = {"c1": "Refunds are allowed within 7 days.",
          "c2": "Exchanges are allowed within 30 days."}

CASES = [
    {"answer": "Refunds within 7 days.", "cites": ["c1"], "correct": True},
    {"answer": "Refunds within 7 days.", "cites": ["c2"], "correct": True},
    {"answer": "Refunds within 7 days.", "cites": [],     "correct": True},
    {"answer": "Refunds within 30 days.", "cites": ["c2"], "correct": False},
]
print(f"{'correct':>8} {'cites':>8} {'supported':>10}  note")
for c in CASES:
    cited = " ".join(CHUNKS[x] for x in c["cites"]).lower()
    supported = bool(c["cites"]) and "7 days" in cited
    note = ("cited chunk does not contain the claim" if c["cites"] and not supported
            else "no source claimed" if not c["cites"] else "")
    print(f"{c['correct']!s:>8} {str(c['cites']):>8} {supported!s:>10}  {note}")

# All of the first three are factually correct. Only one is properly cited.
# Scoring correctness alone reports 3/3 and gives no signal that two thirds of
# the answers were ungrounded.

The first three are all factually correct. Only one is properly cited. Correctness alone reports three out of three and gives no signal that two thirds of the answers were ungrounded.

The third case - correct, no citation - is the one that indicates a broken system most clearly. The retrieval half contributed nothing and the model's general knowledge happened to match, which will stop being true the moment your policy differs from the industry norm.

The mistake this prevents

The mistake is treating a high correctness score as evidence the retrieval pipeline works. An assistant scoring well on correctness and badly on citation support is a language model with an expensive vector database attached to it.

Takeaway

Score citation support separately from correctness. An answer that is right by luck is indistinguishable from a working system until the question changes.

Mark complete