Unit 09.02: Measuring latency where it is felt

There are two latency numbers and users experience only one of them.

Time to first chunk versus time to completion

The first is when something appears. The second is when everything has arrived.

The code measures both on a streamed chain.

import time
from langchain_core.language_models.fake_chat_models import FakeListChatModel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

chain = (ChatPromptTemplate.from_messages([("human", "{q}")])
         | FakeListChatModel(responses=["Refunds are allowed within 7 days."])
         | StrOutputParser())

start = time.perf_counter()
first_chunk_at = None
for chunk in chain.stream({"q": "refund window?"}):
    if first_chunk_at is None:
        first_chunk_at = time.perf_counter() - start
total = time.perf_counter() - start

print(f"time to first chunk : {first_chunk_at * 1000:.2f} ms  <- what a user feels")
print(f"time to completion  : {total * 1000:.2f} ms  <- what a dashboard shows")

print("""
Two different numbers, and the first is the one a user experiences. A change
that halves total time while doubling time-to-first-chunk feels slower, and a
dashboard reporting only completion time will report it as an improvement.
""")

A change that halves total time while doubling time-to-first-chunk feels slower, and a dashboard reporting only completion time reports it as an improvement. That is a real and common outcome of adding a reranking step or a larger k.

Report both, and set the budget on the one users feel. A p95 of 1.2 seconds to completion is fine if the first words appear in 200 milliseconds and unacceptable if nothing appears until the end.

The mistake this prevents

The mistake is measuring latency only in aggregate. An average hides the tail entirely, and the tail is where users abandon. Track p50 and p95 for both numbers.

Takeaway

Measure time to first chunk and time to completion separately, report p50 and p95 for both, and set the user-facing budget on the first.

Mark complete