Unit 08.01: Where the latency actually goes

Optimising latency without measuring it means optimising the part you happen to be able to see.

One line you can remove

Seven spans, of which the model dominates and one other is optional.

The code breaks down a request and prices removing the reranker.

SPANS = [("auth", 4), ("embed query", 28), ("retrieval", 42),
         ("rerank", 190), ("prompt build", 2), ("model", 780), ("validate", 1)]
total = sum(ms for _, ms in SPANS)
print(f"{'span':16} {'ms':>6} {'share':>7}")
for name, ms in sorted(SPANS, key=lambda s: -s[1]):
    print(f"{name:16} {ms:>6} {ms / total:>7.0%}")
print(f"{'TOTAL':16} {total:>6}")

without_rerank = total - 190
print(f"\ndropping rerank: {total}ms -> {without_rerank}ms "
      f"({(total - without_rerank) / total:.0%} faster)")
print("worth it only if the eval set shows rerank changes few answers")

# The model dominates, as always. Rerank is the second line and the only one
# you can remove -- so the question is what it buys, measured on the eval set,
# not whether it sounds like a good idea.

The model is 74% of the time and is not something you can shorten. Rerank is 18% and is the one line you can actually remove - which makes it the only real decision in the table.

Whether to remove it is a question for the eval set, not for the latency budget. If rerank changes few answers it is 190 milliseconds bought for nothing; if it changes many, it is 190 milliseconds well spent.

The mistake this prevents

Streaming is the other lever, and it changes a different number. It cannot make the model faster; it moves the moment the user first sees something from the end of the request to the beginning. Time to first token is what users experience as speed, so a streamed 1,800ms response feels faster than a buffered 1,200ms one - measure and budget both, and never trade the user-facing one for the total.

The mistake is optimising the cheap spans because they are yours. Retrieval at 42 milliseconds and prompt building at 2 could both be halved for no perceptible gain, and the effort is time not spent on the two lines that account for 92% of the total.

Takeaway

Measure per span, find the lines you can actually remove, and decide against the eval set rather than the latency budget. The model dominates and is rarely the thing you can change.

Mark complete