Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 08.01: Judging retrieval separately from the answer

A RAG pipeline is two systems in a trench coat. Scoring them together gives one number that tells you nothing about which one to fix.

Two booleans per case, not one

For every case, ask two independent questions: did retrieval return the expected chunk, and was the final answer correct? The pair localises the failure; the combined score does not.

The example below evaluates three cases and prints a verdict for each.

CASE  EXPECTED CHUNK   RETRIEVED    ANSWER OK?   VERDICT
q1    c1               c1, c3       yes          working
q2    c1               c3           no           RETRIEVAL failed
q3    c2               c2           no           GENERATION failed

q2 and q3 both produce a bad answer, for opposite reasons. q2 never had the
right chunk, so no prompt change helps. q3 had it and answered badly anyway,
so no retrieval tuning helps.

One combined score shows both as a single lost point.

q2 and q3 both produce a bad answer. q2 failed because retrieval returned the wrong chunk - no prompt change will fix it. q3 had the right chunk in hand and answered badly anyway, which is a generation problem and no amount of retrieval tuning will touch it.

Collapsed into a single accuracy figure, both appear as one point of lost score, and the team spends a week tuning whichever stage they happened to suspect.

The mistake this prevents

One caution about how answer_ok gets filled in. It is tempting to have a model grade its own pipeline's answers, and for volume there is often no alternative. The risk is that a judge sharing the generator's training and failure modes forgives exactly the errors you most need to catch - a confident extrapolation reads as correct to a model inclined to make the same one. Grade a sample by hand, compare it against the automated grades, and report the disagreement rate. If the judge and the humans diverge, the judge's scores are measuring the judge.

The mistake is reporting a single "RAG accuracy" to stakeholders. It moves for reasons nobody can attribute, and it hides the case where retrieval improved while generation regressed, netting out to no visible change. Report both columns.

Takeaway

Score retrieval and generation as separate booleans per case. The split costs nothing to compute and turns a mysterious number into a specific stage with a specific fix.