Unit 04.02: Tool tests that need no model

Tools are the one part of an LLM application where testing is free, deterministic and exhaustive.

Every refusal branch, directly

A tool is a function. Call it with each argument combination and assert on the result.

The code covers five cases including four refusals.

MAX_REFUND = 500.0


def issue_refund(account_id, amount, approved_by):
    if not approved_by:
        return {"ok": False, "error": "no approver recorded"}
    if not str(account_id).startswith("ACC-"):
        return {"ok": False, "error": "malformed account id"}
    if not isinstance(amount, (int, float)) or amount <= 0:
        return {"ok": False, "error": "amount must be a positive number"}
    if amount > MAX_REFUND:
        return {"ok": False, "error": f"exceeds the {MAX_REFUND} ceiling"}
    return {"ok": True, "receipt": "rcpt-001"}


CASES = [
    ({"account_id": "ACC-1187", "amount": 120.0, "approved_by": "mgr-2"}, True),
    ({"account_id": "ACC-1187", "amount": 120.0, "approved_by": ""}, False),
    ({"account_id": "1187", "amount": 120.0, "approved_by": "mgr-2"}, False),
    ({"account_id": "ACC-1187", "amount": -5, "approved_by": "mgr-2"}, False),
    ({"account_id": "ACC-1187", "amount": 9_000, "approved_by": "mgr-2"}, False),
]
passed = 0
for args, expected_ok in CASES:
    result = issue_refund(**args)
    ok = result["ok"] == expected_ok
    passed += ok
    print(f"{'PASS' if ok else 'FAIL'} {str(args)[:60]:62} {result}")

print(f"\n{passed}/{len(CASES)} -- every refusal branch covered, no model needed")

Four of the five test refusal branches, and those are precisely the branches a successful run never reaches. A well-behaved agent does not produce a negative amount or a malformed account id, so the code handling them runs for the first time when something goes wrong.

The approver check is the one worth being strict about. approved_by being required at the tool boundary means an approval cannot be skipped by any path that reaches the tool, including one added later by someone who did not read this course.

The mistake this prevents

The mistake is testing tools through the agent. It is slow, costs tokens, is non-deterministic, and systematically fails to reach the refusal branches - which is most of the code you wrote.

Takeaway

Test tools directly, covering every refusal branch. It is the only part of the system where exhaustive testing is affordable, so be exhaustive.

Mark complete