Unit 05.04: Regression testing an index rebuild

An index rebuild changes retrieval for every query at once, which makes it the change most likely to fix one case and break another.

Fixed and broken, not the total

Run the retrieval set before and after, and compare per case.

The code compares two runs with identical totals.

BEFORE = {"q1": True, "q2": True, "q3": False, "q4": True, "q5": True}
AFTER  = {"q1": True, "q2": False, "q3": True, "q4": True, "q5": True}

fixed = sorted(q for q in BEFORE if not BEFORE[q] and AFTER[q])
broke = sorted(q for q in BEFORE if BEFORE[q] and not AFTER[q])

print(f"before: {sum(BEFORE.values())}/{len(BEFORE)}")
print(f"after : {sum(AFTER.values())}/{len(AFTER)}")
print(f"fixed : {fixed}")
print(f"broke : {broke}")

print("""
Identical totals, different system. An index rebuild -- new chunk size, new
embedding model, new documents -- changes retrieval for every query at once,
so it is the change most likely to fix one case and break another.

Run the retrieval set before and after every rebuild and read the broke list,
not the total.
""")

Both runs score four out of five. One case was fixed and one was broken - a different system behind an identical number, and reading only the total you would conclude the rebuild was neutral.

Rebuilds are triggered by things that feel routine: a new chunk size, a new embedding model, a batch of added documents. Each is a full re-evaluation of the retrieval layer whether or not anyone treats it as one.

The mistake this prevents

One production signal is worth wiring up alongside this. A refusal rate that jumps after a corpus update is almost always a retrieval or indexing regression rather than a change in what users are asking - the documents are there and retrieval stopped finding them. It is a cheap alarm for the exact failure a rebuild introduces, and it fires without any labelled data.

The mistake is rebuilding the index without running the retrieval set at all, on the grounds that no code changed. Retrieval behaviour is determined by the index as much as by the code, and the index just changed entirely.

Takeaway

Run the retrieval set before and after every rebuild and read the broken list. An index rebuild is a change to the retrieval layer even when no code moved.

Mark complete