Unit 08.00: Measuring cost per request honestly

The cost of a request is not the price of one model call.

Retries, infrastructure, and storage

Embedding, retrieval, the model call, the retry rate, and trace storage.

The code totals five components.

COMPONENTS = [
    ("embedding the query",   0.00002),
    ("retrieval (infra)",     0.00005),
    ("model call",            0.00280),
    ("validation retry (18% of requests)", 0.00280 * 0.18),
    ("trace storage",         0.00004),
]
total = sum(c for _, c in COMPONENTS)
print(f"{'component':40} {'usd/request':>12} {'share':>7}")
for name, cost in COMPONENTS:
    print(f"{name:40} {cost:>12.5f} {cost / total:>7.0%}")

print(f"\ntotal ${total:.5f} per request, ${total * 100_000:,.0f} per 100k requests")

# The retry line is the one people leave out, and at an 18% retry rate it adds
# 15% to the bill. Cost per SUCCESSFUL request is the honest number, and it is
# not the price of one model call.

The retry line is the one people leave out, and at an 18% retry rate it adds 15% to the bill. That is a validation failure rate showing up as a cost figure, which is a useful way to notice it.

Cost per *successful* request is the honest number. A system that costs three tenths of a cent per call and retries a fifth of them costs more than the per-call price suggests, and the gap grows as quality falls.

The mistake this prevents

When cost rises with traffic flat, the cause is almost always growing context rather than a price change: longer conversation histories, more retrieved chunks after a k increase, or a retry loop that has started firing. All three raise tokens per request while the request count stays the same, so tokens-per-request is the metric that diagnoses it - and a cost chart alone will not.

The mistake is budgeting from the provider's per-token price. That number excludes retries, excludes the infrastructure around the call, and excludes the trace storage the call generates - all of which scale with the same traffic.

Takeaway

Measure cost per successful request, including retries and the infrastructure around the model call. A rising retry rate is a quality problem that appears first on the bill.

Mark complete