Unit 03.04: Growing the set from production failures

The loop that actually improves reliability runs from incident to permanent test.

Every incident leaves a test behind

Symptom, root cause, and the case added so it cannot recur silently.

The code shows three incidents and the tests they produced.

INCIDENTS = [
    {"date": "2026-06-14", "symptom": "quoted a 30-day window",
     "root_cause": "stale document still indexed",
     "test_added": "q41: answer must cite a document updated within 365 days"},
    {"date": "2026-07-02", "symptom": "answered an account question",
     "root_cause": "no refusal for out-of-scope categories",
     "test_added": "q42: account-specific question must refuse"},
    {"date": "2026-07-19", "symptom": "empty answer, no error",
     "root_cause": "parse failure defaulted to empty string",
     "test_added": "q43: malformed response must surface, not default"},
]
for inc in INCIDENTS:
    print(f"{inc['date']}  {inc['symptom']}")
    print(f"            cause: {inc['root_cause']}")
    print(f"            test : {inc['test_added']}\n")

print(f"{len(INCIDENTS)} incidents, {len(INCIDENTS)} permanent tests")

# This is the loop that matters. An incident that produces a fix and no test is
# an incident that will happen again, and the second occurrence is always more
# expensive because the first one used up everyone's patience.

Each test is specific enough to fail if the cause returns. "Answer must cite a document updated within 365 days" would have caught the stale document; a general "answers should be current" would not.

The third is the one worth copying: a parse failure that defaulted to an empty string. The test asserts the failure surfaces rather than defaults, which is a class of bug that produces no error and no output and is invisible in every metric.

The mistake this prevents

The mistake is fixing the incident and not adding the test. The fix is a change someone made once; six months later a refactor undoes it, and the second occurrence is more expensive because everyone's patience was used up by the first.

Takeaway

Turn every incident into a case in the set. A fix without a test is a change that will be undone, and the incident will happen again.

Mark complete