Unit 10.03: Cost and latency as design constraints

Retrieval settings are usually chosen for quality and paid for in latency and tokens on every single query, including the ones that never needed them.

What thoroughness actually costs

k, query rewriting and reranking all improve results on some queries. All three cost time and tokens on every query. The trade is only worth making if you have measured the quality difference.

The table below compares three configurations on latency and cost per thousand queries.

CONFIG      k    REWRITE   LATENCY   TOKENS   COST PER 1,000 QUERIES
cheap       3    no        180 ms      900    $2.70
balanced    6    yes       520 ms    2,100    $6.30
thorough    20   yes     1,400 ms    6,800    $20.40

cheap -> thorough:  7.5x the tokens, 7.8x the latency

That is only worth paying if the evaluation set shows the extra chunks change
answers. A configuration chosen without that measurement is a cost decision
made blind.

Going from the cheap config to the thorough one costs roughly 7.5 times the tokens and 7.8 times the latency. Whether that is a good trade depends entirely on a number this table does not contain: how many answers actually change.

Run both configurations against the eval set and count the cases that differ. If thorough fixes two cases out of forty, you are paying eight times the cost for a five percent gain, on every query, forever. That may still be right - but it should be a decision, not a default.

The mistake this prevents

The mistake is treating latency as a performance concern to optimise later. At 1.4 seconds a user waits; at 180 milliseconds they do not, and the difference changes how the assistant gets used. Latency is a product decision that happens to be implemented in the retrieval config.

Takeaway

Measure the quality difference between configurations on your eval set before choosing one. A config chosen without that measurement is a cost and latency decision made blind.

Mark complete