Unit 05.02: Answer quality once retrieval is fixed

To measure generation you have to hold retrieval constant, or a change in one masks a change in the other.

A fixed context, three kinds of failure

Give every case the correct chunk by construction. Every remaining failure is then a generation failure.

The code shows four answers against one fixed context.

# Hold retrieval constant: every case gets the correct chunk by construction.
FIXED_CONTEXT = "[c1] Refunds are allowed within 7 days, individual plans only."

ANSWERS = [
    ("Refunds within 7 days for individual plans. [c1]", "complete"),
    ("Refunds within 7 days. [c1]", "drops the scope qualifier"),
    ("Refunds within 7 days and processed instantly. [c1]", "adds an unsupported claim"),
    ("Refunds are always available within 7 days. [c1]", "overreaches: 'always'"),
]
for answer, verdict in ANSWERS:
    print(f"{verdict:32} {answer}")

print("""
With retrieval fixed, every remaining failure is a generation failure -- and
they are of three distinct kinds: dropping a qualifier, adding a claim, and
widening a bounded statement.

Measure generation this way, on a fixed context, or a retrieval improvement and
a generation regression cancel out and look like no change.
""")

Three distinct failure kinds appear: dropping a qualifier, adding an unsupported claim, and widening a bounded statement with "always". Each has a different fix, and lumping them into "answer quality" makes none of them addressable.

The fourth is the subtlest. "Refunds are always available within 7 days" is shorter, cleaner, more quotable, and turns a policy with a scope limit into an unconditional promise.

The mistake this prevents

The mistake is measuring generation on live retrieval. A retrieval improvement and a generation regression then cancel out, the combined score does not move, and you conclude the change did nothing.

Takeaway

Measure generation on a fixed context so retrieval cannot confound it, and distinguish dropped qualifiers, added claims and widened statements - they have different fixes.

Mark complete