Unit 06.01: Did it call the right tool, in the right order?

Set comparison and order comparison catch different failures, and order is where the control violations live.

The same tools, in the wrong sequence

An agent that calls every expected tool can still call them in an order that inverts a safety property.

The code compares expected and actual trajectories.

EXPECTED = ["read_account", "check_policy", "issue_refund"]
ACTUAL = ["read_account", "issue_refund", "check_policy"]

print(f"expected: {EXPECTED}")
print(f"actual  : {ACTUAL}")
print(f"same set  : {sorted(EXPECTED) == sorted(ACTUAL)}")
print(f"same order: {EXPECTED == ACTUAL}")

order_violations = []
for i, tool in enumerate(ACTUAL):
    if tool in EXPECTED:
        expected_pos = EXPECTED.index(tool)
        earlier = [t for t in ACTUAL[:i] if t in EXPECTED
                   and EXPECTED.index(t) > expected_pos]
        if earlier:
            order_violations.append((tool, earlier))
print(f"\nout-of-order: {order_violations}")

# The same tools, in the wrong order: the refund was issued before the policy
# was checked. Set comparison passes and order comparison fails, which is why
# trajectory scoring has to consider both.

The tools match as a set and not as a sequence: the refund was issued before the policy was checked. Set comparison passes, which is why it cannot be the only check.

The out-of-order detection is worth building rather than eyeballing. On a three-step trajectory the violation is obvious; on a ten-step one it is not, and the violations that matter are exactly the ones where a check ran after the action it was supposed to gate.

The mistake this prevents

The mistake is asserting on the exact sequence. Agents legitimately vary - an extra lookup, a different order between two independent reads. Assert on the constraints that matter: which tools may run, and which must precede which.

Takeaway

Check the tool set and the ordering constraints separately. Assert on required precedence rather than an exact sequence, or the test fails on harmless variation.

Mark complete