Unit 08.02: Judging the path as well as the result

Three runs, all correct, and only one acceptable.

Redundant steps and unexpected tools

Outcome correctness, step count, repeated calls, and tools outside the expected set.

The table below compares three runs.

RUN   OUTCOME   STEPS   PATH
r1    correct   3       get_invoice, get_contract, answer
r2    correct   9       get_invoice x6, get_contract, answer, answer
r3    correct   3       add_note, get_invoice, answer

r1  expected path, no repeats                        acceptable
r2  6 repeated lookups -- three times the cost       wasteful
r3  wrote a note nobody asked for, before answering  out of scope

  All three answered correctly. Outcome-only scoring rates them identically.

r2 cost three times as much for the same answer. r3 wrote a note nobody asked for before answering - a small over-action that outcome scoring rates as a success.

Both are computed mechanically from the recorded path: a set-length comparison and a set difference. Neither needs judgement, which is what makes them worth automating.

The mistake this prevents

The mistake is evaluating an agent the way you evaluate a script. A script's path is fixed, so its output is the whole behaviour. An agent chooses its path, so the path is a choice that can be wrong independently of the answer.

Takeaway

Score the path alongside the outcome, using mechanical checks: redundant calls and tools outside the expected set. Correct answers can come from expensive or out-of-scope paths.

Mark complete