Unit 11.03: Evaluating retrieval and answers apart

The eval set scores three things separately, and the third is the one most suites leave out.

Retrieval, citation, refusal

Each case declares what should be retrieved and whether it should refuse. Each result is scored on all three axes.

The code evaluates three cases.

CASES = [
    {"id": "q1", "question": "refund window?",  "expected_chunk": "c1",
     "expect_refusal": False},
    {"id": "q2", "question": "money back?",     "expected_chunk": "c1",
     "expect_refusal": False},
    {"id": "q3", "question": "office address?", "expected_chunk": None,
     "expect_refusal": True},
]
RESULTS = [
    {"id": "q1", "retrieved": ["c1"], "cited": ["c1"], "refused": False},
    {"id": "q2", "retrieved": ["c3"], "cited": ["c3"], "refused": False},
    {"id": "q3", "retrieved": [],     "cited": [],     "refused": True},
]

print(f"{'case':5} {'retrieval':10} {'citation':10} refusal")
scores = {"retrieval": 0, "citation": 0, "refusal": 0}
for case, result in zip(CASES, RESULTS):
    r_ok = (case["expected_chunk"] in result["retrieved"]
            if case["expected_chunk"] else not result["retrieved"])
    c_ok = result["cited"] == result["retrieved"][:1]
    f_ok = result["refused"] == case["expect_refusal"]
    for key, ok in [("retrieval", r_ok), ("citation", c_ok), ("refusal", f_ok)]:
        scores[key] += ok
    print(f"{case['id']:5} {'hit' if r_ok else 'MISS':10} "
          f"{'ok' if c_ok else 'BAD':10} {'ok' if f_ok else 'WRONG'}")

print(f"\n{ {k: f'{v}/{len(CASES)}' for k, v in scores.items()} }")
print("q2 retrieved the wrong chunk and cited it faithfully -- a retrieval bug")
print("that a single accuracy number would report as an answer problem.")

q2 retrieved the wrong chunk and cited it faithfully. Citation scoring alone calls that a success - the answer cites what it used - and only the retrieval column reveals the problem.

The refusal column is what a suite built from real questions never has, because real questions are ones someone wanted answered. Without an unanswerable case, the refusal path never runs in any test.

The mistake this prevents

A word on how answer_ok gets filled in at scale. Having a model grade the output is often the only affordable option, and the risk is that a judge sharing the generator's training and failure modes forgives exactly the errors you most need caught - a confident extrapolation reads as correct to a model inclined to make the same one. Grade a sample by hand, compare against the automated grades, and report the disagreement rate. If the two diverge, the judge's scores are measuring the judge.

The mistake is scoring only the final answer because it is what users see. It conflates three independent behaviours, and a change that improves one while breaking another shows no movement at all.

Takeaway

Score retrieval, citation and refusal separately, and include unanswerable cases. A faithfully cited wrong chunk passes any citation-only check.

Mark complete