Unit 10.03: Tracing retrieval separately from generation

A retrieval chain is two systems. One accuracy number for both tells you nothing about which to fix.

Two booleans per case

Did retrieval return the expected chunk, and was the answer correct? The pair localises the failure.

The code evaluates three cases and gives a verdict for each.

CASES = [
    {"id": "q1", "expected": "c1", "retrieved": ["c1", "c3"], "answer_ok": True},
    {"id": "q2", "expected": "c1", "retrieved": ["c3"],       "answer_ok": False},
    {"id": "q3", "expected": "c2", "retrieved": ["c2"],       "answer_ok": False},
]
for c in CASES:
    got = c["expected"] in c["retrieved"]
    if got and c["answer_ok"]:
        verdict = "working"
    elif not got:
        verdict = "RETRIEVAL failed -- no prompt change will fix it"
    else:
        verdict = "GENERATION failed -- retrieval was fine"
    print(f"{c['id']}  retrieved_ok={got!s:5} answer_ok={c['answer_ok']!s:5}  {verdict}")

print("\nq2 and q3 both produce a bad answer, for opposite reasons.")

# One combined accuracy number would show both as a single lost point, and the
# team would spend a week tuning whichever stage they happened to suspect.

q2 and q3 both produce a bad answer for opposite reasons. q2 retrieved the wrong chunk - no prompt change fixes that. q3 had the right chunk and answered badly, so no retrieval tuning touches it.

Collapsed into one number, both are a single lost point, and the team spends a week on whichever stage they happened to suspect. The split costs nothing to compute.

The mistake this prevents

The mistake is reporting a single "RAG accuracy" upward. It moves for reasons nobody can attribute, and it hides the case where retrieval improved while generation regressed and the total stayed flat.

Takeaway

Score retrieval and generation as separate booleans per case. The split is free and turns a mysterious number into a specific stage with a specific fix.

Mark complete