Unit 09.03: Handling a stream that breaks midway

A stream that fails halfway leaves the user with a half-sentence and your system with a decision to make.

Partial content is not an answer

The pieces received before the failure are real content and an incomplete response.

The code simulates a stream dropping partway and inspects what arrived.

def unreliable_stream(pieces, fail_at):
    for i, piece in enumerate(pieces):
        if i == fail_at:
            raise ConnectionError("stream dropped")
        yield piece


received = []
try:
    for piece in unreliable_stream(["Refunds ", "are ", "allowed ", "within 7 days."], 2):
        received.append(piece)
except ConnectionError as exc:
    partial = "".join(received)
    print(f"stream failed: {exc}")
    print(f"partial content: {partial!r}")
    print(f"complete: {partial.rstrip().endswith('.')}")

print("""
The user has a half-sentence. Three rules for this path.

Never present partial content as an answer -- mark it incomplete. Never persist
it as if the turn finished. And if you retry, retry the whole request rather
than resuming, because you cannot know how much the model had produced beyond
what reached you.
""")

The partial content ends mid-word and is not a sentence. Three rules follow. Never present it as an answer - mark it incomplete. Never persist it as though the turn finished, or it enters the history as a fact.

And if you retry, retry the whole request rather than resuming. You cannot know how much the model produced beyond what reached you, so resuming from the last received chunk can duplicate or skip content.

The mistake this prevents

The mistake is treating a dropped stream as a rendering problem and leaving the partial text on screen. Users read it as the answer, particularly when it happens to end at a plausible point - which is exactly the case where the truncation is least visible.

Takeaway

Mark partial content as incomplete, do not persist it, and retry the whole request rather than resuming. You have no way to know what was produced beyond what arrived.

Mark complete