Unit 10.03: Testing the failure paths deliberately

None of the controls in this course fire on a successful run. That is what makes them the untested part of every system.

Eight controls, eight constructed cases

Each guardrail, ceiling and exception path needs a test that constructs the condition on purpose.

The code lists eight failure paths with the test for each.

FAILURE_TESTS = [
    ("guardrail blocks an over-ceiling payment", "post_payment(90000) refuses"),
    ("tool refuses an unapproved payment",       "approved_by='' refuses"),
    ("exception path fires on a missing PO",     "routed to procurement"),
    ("hop ceiling stops a delegation loop",      "run terminates at 6 hops"),
    ("budget ceiling stops a long run",          "run stops, reason recorded"),
    ("invented tool name is rejected",           "error lists the real tools"),
    ("duplicate approval does not double-post",  "second post is a no-op"),
    ("stale knowledge source is flagged",        "age over budget surfaces"),
]

print(f"{'failure path':44} tested by")
for path, test in FAILURE_TESTS:
    print(f"{path:44} {test}")

print(f"\n{len(FAILURE_TESTS)} failure paths, {len(FAILURE_TESTS)} tests.")
print("None of these fire on a successful run, which is why none of them")
print("gets tested unless you construct the case on purpose.")

"Duplicate approval does not double-post" is the one manual testing structurally cannot produce - a person clicking through does each thing once, so the second post never happens in any hand-run test.

"Invented tool name is rejected" needs a test that the error lists the real tools, not merely that the call fails. Failing without information is a failure the agent will repeat.

The mistake this prevents

The mistake is treating a successful end-to-end run as evidence the controls work. It is evidence that they did not fire. Every control needs a test that makes it fire and asserts on what it did.

Takeaway

Write one test per control that constructs the failing condition deliberately. A successful run exercises none of them, so nothing else will.

Mark complete