Unit 04.03: Measuring whether the rewrite actually helped
Every rewriting strategy sounds reasonable on a whiteboard. The only way to know which one helps your corpus is to score it, and the thing to score is retrieval - not the answer.
Score the stage you changed
A rewrite changes retrieval. So measure retrieval: for each case, did the expected chunk come back? That is a boolean per case, it needs no model, and it isolates the change from everything downstream.
The example below runs a two-case eval set twice, once with the original questions and once with the rewrites.
question : retrieved the expected chunk in 0/2 cases
rewritten : retrieved the expected chunk in 2/2 casesTwo cases is not an eval set; it is a demonstration of the shape. The shape is what matters: the same cases, the same corpus, one variable changed, retrieval scored directly.
Scoring the final answer instead would conflate two things. An improvement in retrieval and a lucky guess by the model look identical in an answer score, and only one of them holds up on the next question. Module 8 makes this separation a rule for the whole eval set.
The mistake this prevents
The mistake is judging a rewrite by reading a few outputs. Rewrites help some queries and hurt others, and reading three examples tells you about three queries. Run the whole set both ways and compare the totals - a strategy that fixes four cases and breaks five is a regression that feels like progress.
Takeaway
Measure a rewrite on retrieval accuracy across a fixed case set, before and after, with nothing else changed. Answer quality is a downstream signal too noisy to attribute.