Unit 07.01: Tracing a single request end to end

One request, every stage, with the versions that produced it.

Spans, detail, and the outcome

Each stage records its duration, whether it succeeded, and enough detail to explain why.

The code shows a request blocked at validation.

import json

trace = {
    "request_id": "r-8841",
    "user_segment": "support-agent",
    "deploy_version": "2026-07-28.3",
    "config": {"prompt": "answer-v3", "policy": "v4", "model": "m-2026-06", "k": 6},
    "spans": [
        {"name": "auth",      "ms": 4,   "ok": True},
        {"name": "retrieval", "ms": 42,  "ok": True,
         "detail": {"returned": 6, "above_threshold": 2, "top_score": 0.91}},
        {"name": "model",     "ms": 780, "ok": True,
         "detail": {"tokens_in": 96, "tokens_out": 22}},
        {"name": "validate",  "ms": 1,   "ok": False,
         "detail": {"error": "unsupported term: 'instantly'"}},
    ],
    "outcome": "blocked by validation",
    "total_ms": 827,
}
print(json.dumps(trace, indent=1))

failed = [s["name"] for s in trace["spans"] if not s["ok"]]
print(f"\nfailed at: {failed}, after {trace['total_ms']}ms and a paid model call")

The failure is at validate, after a paid model call - so this request cost money and produced nothing. That is a cost the format-failure metric should be tracking, and it is invisible unless the trace records where in the sequence the failure landed.

The retrieval span's detail is doing real work: six returned, two above threshold. That tells you the threshold is filtering aggressively, which is the context you need to interpret everything downstream.

The mistake this prevents

Join user feedback to the trace while you are at it. A thumbs-down with a request id attached is an investigable case; the same thumbs-down in a separate feedback table is a number on a chart. Feedback is also the only signal you have for the confidently-wrong failures from Module 1, which by definition produce no error and pass every automated check.

The mistake is recording spans without detail. "Retrieval took 42ms and succeeded" tells you nothing about whether it succeeded *well*, and the difference between six results and two above threshold is the difference between a healthy query and one that nearly refused.

Takeaway

Trace every stage with duration, success and enough detail to interpret it - plus deploy and config versions. Record where in the sequence a failure landed, because failures after the model call cost money.

Mark complete