Unit 12.02: Instrumenting every request with a trace

Every request emits one record, and the fields were decided before the first real run.

Versions, retrieval, generation, latency, retention

What was asked, under which versions, what retrieval returned and what cleared the threshold, what generation cost, and how it compared to the budget.

The code prints a complete trace.

import json

trace = {
    "request_id": "r-8841",
    "asked_at": "2026-07-29T10:14:02Z",
    "versions": {"prompt": "answer-v3", "policy": "support-policies-v4",
                 "model": "m-2026-06", "chain": "rag-answer-v3"},
    "retrieval": {"k": 4, "threshold": 0.75,
                  "returned": [{"id": "support-policies-v4.md#refunds:2",
                                "score": 0.91}],
                  "below_threshold": 3},
    "shown_to_model": ["support-policies-v4.md#refunds:2"],
    "generation": {"tokens_in": 96, "tokens_out": 22, "ms": 780},
    "answer": {"refused": False, "cites": ["support-policies-v4.md#refunds:2"],
               "unsupported_terms": []},
    "latency": {"first_chunk_ms": 210, "total_ms": 825, "budget_ms": 800,
                "over_budget": True},
    "content_retention_days": 30,
}
print(json.dumps(trace, indent=2))

print(f"\nover latency budget: {trace['latency']['over_budget']} "
      f"({trace['latency']['total_ms']}ms vs {trace['latency']['budget_ms']}ms)")

below_threshold: 3 is the field worth copying. It records that three documents were retrieved and discarded, which is how you tell a threshold that is filtering appropriately from one that is throwing away answers.

over_budget: True on an 825ms response against an 800ms budget makes the scope's latency decision measurable on every request rather than in an occasional review.

The mistake this prevents

The mistake is instrumenting only failures. The request that most needs investigation is usually one that completed successfully and produced something wrong - no exception, no error, nothing a failure-triggered log would have caught.

Takeaway

Emit one record per request with versions, retrieval detail including what was discarded, generation cost, and the latency budget comparison. Instrument successes, not only failures.

Mark complete