Unit 11.02: Asserting on structure, not on wording

A test asserting on exact model output is a test that will fail for a reason nobody cares about.

Properties, not strings

Does it cite, does it contain the figure, is it the right length, does it hedge. Each is a property the answer must have.

The code compares an exact match against five structural assertions.

ANSWER = "Refunds are allowed within 7 days of purchase. [c1]"

brittle = ANSWER == "Refunds are allowed within 7 days. [c1]"
structural = [
    ("cites at least one chunk", "[c" in ANSWER),
    ("states the number 7", "7" in ANSWER),
    ("mentions days", "days" in ANSWER.lower()),
    ("is one to three sentences", 1 <= ANSWER.count(".") <= 3),
    ("does not hedge", not any(w in ANSWER.lower()
                               for w in ("probably", "i think", "may be"))),
]
print(f"exact-match assertion passes: {brittle}   <- fails on any rewording")
print("\nstructural assertions:")
for name, ok in structural:
    print(f"   {'PASS' if ok else 'FAIL'} {name}")

# The exact match fails because of "of purchase", which is a fine addition. A
# suite full of exact matches is a suite that fails for reasons nobody cares
# about, and gets disabled.

The exact match fails because the answer says "of purchase" - a perfectly good addition that changes nothing about correctness. All five structural assertions pass.

The last one is worth copying into your own suite. Hedging language in what is supposed to be a factual answer signals that the model did not have what it needed, and catching it points you at the inputs rather than at the wording.

The mistake this prevents

The mistake is that a suite full of exact matches gets disabled rather than rewritten. After the third false failure someone comments out the test, and it is never the test that would have caught the real regression that survives.

Takeaway

Assert on structure: citations present, figures correct, length bounded, no hedging. Exact-match assertions fail on harmless rewording and get disabled.

Mark complete