Unit 01.03: Reading a crew run from the outside

A crew run produces a lot of prose. Reading it as prose tells you whether you like the output; it does not tell you what happened.

Four numbers per task

Which agent ran it, how many tokens it cost, how long it took, and whether the output met its contract. Those four turn a wall of text into something you can reason about.

The code shows a three-task run in that form.

run = {
    "crew": "support-triage",
    "process": "sequential",
    "tasks": [
        {"task": "classify", "agent": "Triage analyst", "tokens": 1_240,
         "seconds": 3.1, "output_ok": True},
        {"task": "draft_reply", "agent": "Support writer", "tokens": 3_980,
         "seconds": 9.4, "output_ok": True},
        {"task": "policy_check", "agent": "Policy reviewer", "tokens": 2_110,
         "seconds": 5.2, "output_ok": False},
    ],
}

print(f"{'task':14} {'agent':18} {'tokens':>7} {'secs':>6}  ok")
for step in run["tasks"]:
    print(f"{step['task']:14} {step['agent']:18} {step['tokens']:>7,} "
          f"{step['seconds']:>6.1f}  {step['output_ok']}")

total = sum(s["tokens"] for s in run["tasks"])
print(f"\ntotal {total:,} tokens, {sum(s['seconds'] for s in run['tasks']):.1f}s")
print(f"failed at: {[s['task'] for s in run['tasks'] if not s['output_ok']]}")

# Four numbers per task -- agent, tokens, time, whether the output met its
# contract. Without them a crew run is a wall of prose that either looks right
# or does not, and you cannot tell which agent caused what.

The failure is at policy_check, and it is the third task - so the two before it spent 5,220 tokens producing work that was then rejected. That is a cost you can see only if cost is attributed per task.

The token column also shows where the money goes. draft_reply is nearly twice the cost of either other task, which is the kind of thing that decides whether a workflow is worth running at volume.

The mistake this prevents

The mistake is judging a crew run by reading its final output. The final output is one agent's text, shaped by everything upstream, and it looks much the same whether the earlier tasks did well or badly. Read the per-task record instead.

Takeaway

Log agent, tokens, duration and contract-compliance for every task. Without them you can tell that a run went wrong and not which agent caused it or what it cost you.

Mark complete