Unit 06.04: Scoring a trajectory reproducibly

Trajectory scoring is only useful if two people running it get the same numbers.

Five booleans, no judgement

Reached the goal, used only expected tools, correct order, no redundant calls, approvals respected - all computed from recorded data.

The code scores three trajectories.

def score_trajectory(actual, expected, gated, approvals):
    return {
        "reached_goal": actual[-1] == expected[-1],
        "used_only_expected_tools": set(actual) <= set(expected),
        "correct_order": [t for t in actual if t in expected] == expected,
        "no_redundant_calls": len(actual) == len(set(actual)),
        "approvals_respected": all(approvals.get(t) for t in actual if t in gated),
    }


CASES = [
    (["read_account", "check_policy", "respond"], {"issue_refund": False}),
    (["read_account", "read_account", "check_policy", "respond"], {}),
    (["read_account", "issue_refund", "respond"], {"issue_refund": False}),
]
EXPECTED = ["read_account", "check_policy", "respond"]
for actual, approvals in CASES:
    result = score_trajectory(actual, EXPECTED, {"issue_refund"}, approvals)
    failed = [k for k, v in result.items() if not v]
    print(f"{str(actual)[:56]:58} {'all pass' if not failed else failed}")

# Five booleans, all computed from recorded data with no judgement involved.
# That is what makes trajectory scoring reproducible -- two people running it
# get the same numbers, which "was the path reasonable?" does not give you.

Every check is mechanical. "Was the path reasonable?" produces disagreement that cannot be resolved later; "did it call any tool outside the expected set?" produces a boolean.

no_redundant_calls is the one that catches the expensive-but-correct run from the first unit of this module. It is a set-length comparison, and it makes a cost problem into a scored property.

The mistake this prevents

The mistake is scoring trajectories with a model as judge. It reintroduces everything you were trying to remove - non-determinism, cost, and a grader that shares the agent's blind spots - for a job that plain set operations do exactly.

Takeaway

Score trajectories with booleans computed from recorded data. Mechanical checks are reproducible; a model judging whether a path was reasonable is not.

Mark complete