Unit 05.00: Scoring retrieval on its own

Retrieval is half the system and can be scored without the model, without labels for the answer, and without a judge.

Recall and rank, plus the empty case

Did the expected chunk come back, and at what position. Both are booleans or integers computed from recorded ids.

The code scores four cases including one that should retrieve nothing.

CASES = [
    {"id": "q1", "expected": "c1", "retrieved": ["c1", "c3", "c7"]},
    {"id": "q2", "expected": "c1", "retrieved": ["c3", "c7", "c9"]},
    {"id": "q3", "expected": "c2", "retrieved": ["c9", "c2", "c4"]},
    {"id": "q4", "expected": None,  "retrieved": []},
]
hits = 0
print(f"{'case':5} {'rank':>5} {'recall@3':>9}")
for c in CASES:
    if c["expected"] is None:
        ok = not c["retrieved"]
        print(f"{c['id']:5} {'n/a':>5} {'ok' if ok else 'LEAK':>9}  (expected nothing)")
        hits += ok
        continue
    rank = (c["retrieved"].index(c["expected"]) + 1
            if c["expected"] in c["retrieved"] else None)
    hits += rank is not None
    print(f"{c['id']:5} {str(rank):>5} {'hit' if rank else 'MISS':>9}")

print(f"\nrecall@3: {hits}/{len(CASES)}")

# Rank matters as much as recall. `q3` found the right chunk at rank 2, which
# is fine at k=3 and a miss at k=1 -- so the rank column tells you what k needs
# to be, and the recall column alone does not.

The rank column tells you what k needs to be. q3 found the target at rank 2, which is a hit at k=3 and a miss at k=1 - so a recall figure without ranks cannot tell you whether k is right.

q4 is the case most retrieval evaluations omit: a question that should retrieve nothing. Scoring it as a leak when something comes back is the only way to catch a threshold that is set too low.

The mistake this prevents

The mistake is evaluating retrieval only on questions that have answers. The threshold is then tuned entirely on recall, drifts down, and the system stops refusing - which shows up as a grounding failure much later and much more expensively.

Takeaway

Score recall, rank and the empty case. Ranks tell you what k should be, and the empty case is what keeps your threshold honest.

Mark complete