Unit 12.02: Measuring cost, latency, and failure rate

The operational numbers, measured over enough requests to mean something.

Percentiles, rates and counts

Latency at three percentiles, cost per request, and counts for format failures, retries, refusals and errors.

The code summarises five thousand requests.

SAMPLE = 5_000
MEASURED = {
    "p50_latency_ms": 640, "p95_latency_ms": 1840, "p99_latency_ms": 3200,
    "cost_per_request_usd": 0.0031,
    "format_failures": 12, "parse_retries": 890, "refusals": 1_640,
    "errors_5xx": 3,
}
print(f"over {SAMPLE:,} requests:")
for name, value in MEASURED.items():
    if name.endswith(("_ms", "_usd")):
        print(f"   {name:24} {value}")
    else:
        print(f"   {name:24} {value:>6,}  ({value / SAMPLE:.2%})")

print(f"\ntotal spend: ${MEASURED['cost_per_request_usd'] * SAMPLE:.2f}")
print(f"retry overhead: {MEASURED['parse_retries'] / SAMPLE:.0%} of requests")
print(f"p99 is {MEASURED['p99_latency_ms'] / MEASURED['p50_latency_ms']:.1f}x p50")

# The refusal rate at 33% is by design and worth stating so nobody reads it as
# a failure. The 18% retry rate is not by design, and it is 18% of your model
# spend.

The refusal rate at 33% is by design and worth stating so that nobody reads it as a failure - it matches the eval set's refusal share, which is the check that the system is refusing about as often as it should.

The 18% retry rate is not by design. It is 18% of your model spend producing nothing, and it is a validation failure rate showing up as a cost line - which makes it both a quality finding and a budget one.

The mistake this prevents

The mistake is reporting only the average latency. p99 here is five times p50, and the users in that tail are having a materially different experience from the one the average describes.

Takeaway

Report latency at p50, p95 and p99, and state which rates are by design. A refusal rate matching your eval set is healthy; a high retry rate is spend producing nothing.

Mark complete