Unit 05.03: Diagnosing which half failed

Two booleans per case produce four combinations, and the fourth is the one people forget exists.

Four combinations, four verdicts

Retrieval correct or not, answer correct or not.

The code assigns a verdict to each combination.

CASES = [
    {"id": "q1", "retrieved_ok": True,  "answer_ok": True},
    {"id": "q2", "retrieved_ok": False, "answer_ok": False},
    {"id": "q3", "retrieved_ok": True,  "answer_ok": False},
    {"id": "q4", "retrieved_ok": False, "answer_ok": True},
]
VERDICTS = {
    (True, True):   "working",
    (False, False): "RETRIEVAL failed -- no prompt change fixes it",
    (True, False):  "GENERATION failed -- retrieval was fine",
    (False, True):  "right answer, wrong evidence -- lucky, not reliable",
}
for c in CASES:
    print(f"{c['id']}  retrieval={c['retrieved_ok']!s:5} answer={c['answer_ok']!s:5}  "
          f"{VERDICTS[(c['retrieved_ok'], c['answer_ok'])]}")

# The fourth combination is the one people forget. A correct answer built on the
# wrong evidence came from the model's weights, and it will be wrong on the next
# question where general knowledge and your documents diverge.

The fourth - wrong retrieval, right answer - is the combination that looks like success and is not. The model produced a correct answer from evidence that did not support it, which means it answered from its weights.

That case will be wrong on the next question where general knowledge and your documents diverge, and nothing in a combined score distinguishes it from a case that worked properly.

The mistake this prevents

The mistake is treating the combined accuracy as the number to improve. It conflates two independent systems, and it counts lucky answers as wins - so a change that makes the system more grounded and slightly less lucky reads as a regression.

Takeaway

Score retrieval and answer as separate booleans and read all four combinations. Right answer on wrong evidence is a failure that a combined score records as a success.

Mark complete