Unit 11.04: A suite that fails for a real reason

Every false failure makes the next real one easier to ignore. That is the cost of a brittle test, and it compounds.

Good, brittle, flaky

Three categories. Only one belongs in the suite.

The code sorts seven tests.

TESTS = [
    ("exact answer string matches",       "brittle", "fails on any rewording"),
    ("answer contains a citation",        "good",    "structural"),
    ("refusal string is exact",           "good",    "you fixed it in code"),
    ("retrieval returns the right chunk", "good",    "the property that matters"),
    ("answer is under 100 words",         "good",    "a real requirement"),
    ("model returns identical text twice", "brittle", "not guaranteed at all"),
    ("latency under 200ms",               "flaky",   "belongs in monitoring"),
]

print(f"{'test':38} {'verdict':9} why")
for test, verdict, why in TESTS:
    print(f"{test:38} {verdict:9} {why}")

good = sum(1 for _, v, _ in TESTS if v == "good")
print(f"\n{good} of {len(TESTS)} would fail only for a reason worth acting on")

# The three to delete are the ones that will fail on a model update, a
# rewording, or a slow CI runner -- and each false failure makes the next real
# one easier to ignore.

The brittle ones assert on things that were never guaranteed: exact wording, and identical output across two calls. Both will fail on a model update, and neither indicates a problem when it does.

The flaky one - latency under 200 milliseconds - is a real requirement in the wrong place. It belongs in monitoring, where a slow CI runner does not make it fail, and where the number that matters is a percentile over real traffic rather than one sample.

The mistake this prevents

The mistake is keeping a brittle test because it once caught something. It caught it incidentally, and the running cost is that every model update produces failures someone has to triage. Replace it with an assertion on the property it accidentally checked.

Takeaway

Keep tests that fail only for reasons worth acting on. Exact-output and timing assertions belong in monitoring or nowhere - false failures eventually get the whole suite ignored.

Mark complete