Unit 13.03: Cost and latency as response fields

Cost and latency in the response make them visible to the caller.

A meta block beside the answer

A response with the prediction at the top level and the operational details beside it.

The code prints one.

import json

response = {
    "category": "billing",
    "confidence": 0.87,
    "meta": {"request_id": "r-8841", "model": "fake-model-2026-06",
             "prompt_version": "classify-v3", "input_tokens": 96,
             "output_tokens": 12, "usd": 0.0000468, "latency_ms": 780,
             "cached": False},
}
print(json.dumps(response, indent=2))

print("""
Putting cost and latency in the response makes them visible to the caller,
who can then decide to cache, batch or back off. Putting the model and prompt
version there makes a support conversation possible six weeks later.

Keep it in a `meta` block so the useful fields stay at the top level and the
contract for those stays clean.
""")

A caller who can see the cost and the latency can decide to cache, batch or back off. A caller who cannot will call you in a loop.

The model and prompt version are what make a support conversation possible six weeks later, when the answer they are querying was produced by a version that no longer exists.

The mistake this prevents

The mistake is putting these fields at the top level alongside the prediction. They then become part of the primary contract, and removing one later is a breaking change - a meta block keeps them separable.

Takeaway

Return cost, latency and versions in a meta block. It lets callers manage their own usage and makes old answers explainable.

Mark complete