Unit 07.00: The questions you will ask after a failure

Decide what to log by writing down the questions you will ask when something goes wrong. Every one implies a field.

Seven questions, seven fields

Which task, which agent, what did it receive, did a tool run, what did it cost, did a guardrail fire, did a human approve.

The code lists each with the field it requires.

QUESTIONS = [
    ("which task produced the wrong output?", "task id on every output"),
    ("which agent ran it?",                   "agent role per task"),
    ("what did it actually receive?",         "resolved inputs, not the template"),
    ("did any tool run, and what did it do?", "tool name, args, result, attempt"),
    ("what did it cost?",                     "tokens per task"),
    ("did a guardrail fire?",                 "guardrail name and verdict"),
    ("did a human approve?",                  "reviewer, timestamp, edits"),
]

print(f"{'question':42} needs logged")
for question, field in QUESTIONS:
    print(f"{question:42} {field}")

print(f"\n{len(QUESTIONS)} questions, {len(QUESTIONS)} fields. Decide them now,")
print("because none can be reconstructed after the run has finished.")

"What did it actually receive" is the field most often logged wrongly. Logging the task template tells you what you wrote; logging the resolved inputs tells you what the agent saw, and the difference is exactly where an upstream task's output ends up somewhere unexpected.

None of these can be reconstructed after the run. That is the whole argument for deciding them now rather than after the first incident.

The mistake this prevents

The mistake is logging the final output and the errors. Neither answers any of these seven questions, and the run you most need to investigate is usually one that completed successfully and produced something wrong.

Takeaway

Write down the questions you will ask after a failure and log a field for each. Log resolved inputs, not templates - the difference is where most surprises live.

Mark complete