Unit 03.03: Streaming, and what it changes for the user

Streaming changes when the user sees something, not how long the whole thing takes.

Time to first chunk versus time to complete

Chunks accumulated, with both timings measured.

The code reports them.

import time

CHUNKS = ["Refunds ", "are ", "allowed ", "within ", "7 days."]

start = time.perf_counter()
first_at = None
buffer = []
for chunk in CHUNKS:
    if first_at is None:
        first_at = time.perf_counter() - start
    buffer.append(chunk)
total = time.perf_counter() - start

print(f"chunks received : {len(CHUNKS)}")
print(f"reassembled     : {''.join(buffer)!r}")
print(f"time to first   : {first_at * 1000:.3f} ms   <- what the user feels")
print(f"time to complete: {total * 1000:.3f} ms   <- what a dashboard shows")

print("""
Two rules. Accumulate the chunks rather than making a second call to get the
whole value. And never treat partial content as a final answer -- if
validation fails after streaming, you have to replace what the user is
already reading.
""")

The two numbers are different and users experience only the first. A change that halves total time while doubling time-to-first-chunk feels slower, and a dashboard reporting only completion time will call it an improvement.

Accumulating the chunks gives you the complete value for free - making a second non-streamed call to get it is paying twice.

The mistake this prevents

The mistake is treating streamed content as final. If validation fails after the stream, you have to visibly replace text the user is already reading, so anything that must pass a check should not be streamed as an answer.

Takeaway

Stream for perceived speed, accumulate the chunks rather than calling twice, and never present streamed content as final before it has been validated.

Mark complete