Unit 09.04: The failure report worth writing

A failure report that stops at what happened is an anecdote. What makes it a report is the last field.

Cause, cost, fix, and the test that follows

What stopped it, where the state first went wrong, whether upstream was correct, the root cause, the cost, the fix, and the regression test.

The code prints a complete report for a blocked run.

import json

report = {
    "run_id": "run-8841",
    "outcome": "blocked",
    "stopped_by": "guardrail: reply states 30 days, policy says 7",
    "first_wrong_step": {"task": "draft", "agent": "Support writer"},
    "upstream_was_correct": True,
    "root_cause": ("the writer was given the decision but not the policy text, "
                   "so it filled the window from its own priors"),
    "cost_of_the_failed_run": {"tokens": 6_030, "usd": 0.018},
    "fix": "pass the policy line into the draft task's context",
    "prevented_by": "guardrail on the reply text",
    "would_a_human_have_caught_it": "yes, but only if reading against the policy",
    "regression_test_added": "draft task must include the quoted policy line",
}
print(json.dumps(report, indent=2))

print("\nThe last field is what makes this a report rather than an anecdote.")

upstream_was_correct: true is the field that redirects the investigation. The decision was right and the draft contradicted it, so the fix is in what the drafting task was given - not in the decision logic, which is where the wrong number would have sent you.

regression_test_added is what stops the same failure recurring. Without it the fix is a change someone made once, and the next person to touch the task context has nothing telling them why the policy line is passed in.

The mistake this prevents

The mistake is writing the report and not the test. Reports accumulate in a folder; tests run. If a failure is worth documenting it is worth one assertion, and the assertion is what actually prevents the recurrence.

Takeaway

Record cause, cost, fix and - most importantly - the regression test. A failure report without a test is a description of something that will happen again.

Mark complete