Unit 04.01: Asserting on schema, not on phrasing

Every false test failure makes the next real one easier to ignore, and exact-match assertions produce nothing but false failures.

Properties the answer must have

Cites something, states the figure, bounded length, no hedging.

The code compares an exact match against five structural assertions.

ANSWER = "Refunds are allowed within 7 days of purchase. [c1]"

brittle = ANSWER == "Refunds are allowed within 7 days. [c1]"
structural = [
    ("cites a chunk", "[c" in ANSWER),
    ("states the figure", "7" in ANSWER),
    ("one to three sentences", 1 <= ANSWER.count(".") <= 3),
    ("does not hedge", not any(w in ANSWER.lower()
                               for w in ("probably", "i think", "roughly"))),
    ("under 60 words", len(ANSWER.split()) < 60),
]
print(f"exact match: {brittle}   <- fails on 'of purchase'")
for name, ok in structural:
    print(f"  {'PASS' if ok else 'FAIL'} {name}")

# The exact-match assertion fails on a harmless addition. A suite of those gets
# disabled after the third false failure, and it is never the test that would
# have caught the real regression that survives.

The exact match fails because the answer says "of purchase" - a harmless addition. All five structural assertions pass.

The hedging assertion is worth copying into any suite. Words like "probably" in what is supposed to be a factual answer signal that the model did not have what it needed, so the failure points at the inputs rather than at the wording.

The mistake this prevents

The mistake is keeping a brittle assertion because it once caught something. It caught it incidentally, and the running cost is that every model update produces failures someone has to triage - until the third time, when the test gets commented out.

Takeaway

Assert on structure: citations, figures, bounds, absence of hedging. Exact-match assertions fail on harmless rewording and get the suite disabled.

Mark complete