Unit 05.04: Guardrails that fail closed

The most dangerous guardrail is one that has silently stopped running.

Blocking on error is the correct default

A guardrail can fail in two directions when it errors: allow the action and carry on, or block and stop the run. The first is the tempting default because it keeps things working.

The code runs a broken checker through both wrappers.

def fail_open(raw, checker):
    try:
        return checker(raw)
    except Exception:
        return True, "check errored -- allowing"      # the dangerous default


def fail_closed(raw, checker):
    try:
        return checker(raw)
    except Exception as exc:
        return False, f"check errored ({type(exc).__name__}) -- blocking"


def broken_checker(raw):
    raise KeyError("policy config not loaded")


for name, wrapper in [("fail open", fail_open), ("fail closed", fail_closed)]:
    allowed, reason = wrapper('{"amount": 9999}', broken_checker)
    print(f"{name:12} allowed={allowed!s:5} {reason}")

print("""
A guardrail that errors is a guardrail that is not running. Fail open and the
first misconfiguration silently disables every check you built; fail closed and
it stops the run loudly.

Blocking on error is correct even though it causes false stops. A false stop is
visible and cheap; an undetected disabled guardrail is neither.
""")

Fail-open lets a 9,999 refund through because the policy config did not load. Nothing alerts, the run completes, and every subsequent run has no guardrail at all - the first misconfiguration silently disables every check you built.

Fail-closed stops the run with a reason. It causes false stops, and false stops are visible and cheap. An undetected disabled guardrail is neither.

The mistake this prevents

The mistake is wrapping guardrails in a broad try/except during development to stop them interrupting your testing, and leaving it there. The exception handler outlives the reason for it, and by then nothing distinguishes a passing check from an erroring one.

Takeaway

Guardrails must fail closed. A false stop is loud and recoverable; a guardrail that errors into silence is a control you believe you have and do not.

Mark complete