Unit 10.02: Instrumenting the run for review

Instrumentation is a list of fields, decided before the run rather than after the incident.

Five sections, twenty-odd fields

Per task, per tool call, per run, human review, and budget. Every field answers a question from Module 7.

The code lists the full schema.

import json

instrumentation = {
    "per_task": ["task_id", "agent_role", "resolved_inputs", "output_summary",
                 "tokens", "seconds", "guardrail_verdict"],
    "per_tool_call": ["tool", "arguments", "attempt", "result", "external_id"],
    "per_run": ["run_id", "flow_version", "prompt_version", "policy_version",
                "total_tokens", "total_usd", "outcome"],
    "human_review": ["reviewer", "approved_at", "edited_fields", "rejection_reason"],
    "budget": ["max_tokens", "max_usd", "max_seconds", "stopped_by"],
}
for section, fields in instrumentation.items():
    print(f"{section} ({len(fields)})")
    for field in fields:
        print(f"    {field}")

total = sum(len(v) for v in instrumentation.values())
print(f"\n{total} fields. Every one answers a question from Module 7 Unit 00,")
print("and none can be reconstructed after the run.")

resolved_inputs under per-task is the field that repays the most effort, because it is the difference between what you wrote and what the agent saw. external_id under tool calls is next - it is what lets you reconcile against the finance system rather than searching by timestamp.

The budget section exists so a stopped run explains itself. stopped_by turns "the run ended early" into a specific ceiling and a specific value.

The mistake this prevents

The mistake is instrumenting after the first incident, using whatever would have helped with that one. The incident you instrument for is not the incident you get, and the fields are cheap enough to add all at once.

Takeaway

Decide the full instrumentation schema before the first real run. Resolved inputs and external ids are the two fields that repay the effort fastest.

Mark complete