Unit 11.01: Turning an incident into a test case

The test that matters is the one that would have caught it in CI.

Two cases, one of them the real one

A case asserting the correct outcome, and a case asserting the condition that caused the failure.

The code shows an incident and the two cases it produced.

incident = {
    "symptom": "answered '30 days' for a refund question",
    "root_cause": "k lowered to 4; the current policy chunk fell outside top-k, "
                  "leaving a stale 2023 chunk as the best match",
    "fix": "restore k=6 and add a recency filter",
}
new_cases = [
    {"id": "q41", "question": "What is the refund window?",
     "expected_chunk": "policies-v4#refunds:2",
     "assert": "cited document updated within 365 days"},
    {"id": "q42", "question": "refund window",
     "expected_chunk": "policies-v4#refunds:2",
     "assert": "retrieval recall holds at k=4 as well as k=6"},
]
print(f"symptom   : {incident['symptom']}")
print(f"root cause: {incident['root_cause']}")
print(f"fix       : {incident['fix']}\n")
for case in new_cases:
    print(f"{case['id']}: {case['question']!r}")
    print(f"      assert: {case['assert']}")

print("\nq42 is the important one: it fails at k=4, so it would have caught")
print("this change in CI before the deploy.")

# A fix without a test is a change someone made once. The test is what makes
# the incident unrepeatable.

q41 asserts the answer cites a recent document - good, and it would have failed only after the deploy. q42 asserts retrieval recall holds at k=4 as well as k=6, which is the condition that actually broke.

That second one is the test that would have failed in CI before the deploy. Writing it requires understanding the root cause rather than the symptom, which is why the test comes after the analysis rather than instead of it.

The mistake this prevents

The mistake is adding a test that reproduces the symptom and stopping there. It catches this exact failure and nothing adjacent - and the next occurrence usually arrives by a slightly different route that the symptom-level test does not cover.

Takeaway

Write the test for the root cause, not the symptom. The useful test is the one that would have failed in CI before the change shipped.

Mark complete