Unit 11.02: Asserting on structure, not on wording

Assert on properties, not on strings.

Five structural checks against one exact match

An answer with a harmless extra clause.

The code compares both approaches.

ANSWER = "Refunds are allowed within 7 days of purchase. [policy#2]"

exact = ANSWER == "Refunds are allowed within 7 days. [policy#2]"
structural = [
    ("cites a source", "[policy#" in ANSWER),
    ("states the figure", "7" in ANSWER),
    ("one to three sentences", 1 <= ANSWER.count(".") <= 3),
    ("under 60 words", len(ANSWER.split()) < 60),
    ("does not hedge", not any(w in ANSWER.lower()
                               for w in ("probably", "i think", "roughly"))),
]
print(f"exact match: {exact}   <- fails on 'of purchase'")
for name, ok in structural:
    print(f"  {'PASS' if ok else 'FAIL'} {name}")

# A suite of exact matches gets disabled after the third false failure, and it
# is never the test that would have caught the real regression that survives
# the cull.

The exact match fails on "of purchase". All five structural assertions pass, and each names something you actually require: a citation, the figure, a length bound, no hedging.

The hedging check is worth copying. "Probably" in what should be a factual answer signals the model did not have what it needed, and points at the inputs rather than the wording.

The mistake this prevents

The mistake is keeping a brittle assertion because it once caught something. It caught it incidentally, and the running cost is a triage every time the model updates - until the third time, when someone comments it out.

Takeaway

Assert on citations, figures, length and absence of hedging. Exact-match assertions fail on harmless rewording and end up disabled.

Mark complete