Unit 09.00: Streaming a response to a user

Streaming changes nothing about the chain and a great deal about how it feels to use.

invoke waits, stream yields

The same composed chain supports both. Only how you consume it differs.

The code streams a response and reassembles it.

from langchain_core.language_models.fake_chat_models import FakeListChatModel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([("human", "{question}")])
model = FakeListChatModel(responses=["Refunds are allowed within 7 days."])
chain = prompt | model | StrOutputParser()

print("streamed:")
pieces = []
for chunk in chain.stream({"question": "What is the refund window?"}):
    pieces.append(chunk)
    print(f"   {chunk!r}")

print(f"\n{len(pieces)} chunks, reassembled: {''.join(pieces)!r}")

# `invoke` waits for the whole response; `stream` yields as it arrives. The
# chain definition is identical -- only how you consume it changes.

The chunks concatenate back to exactly what invoke would have returned, which is the property that makes streaming safe to add late - nothing downstream needs to change if you buffer.

With a real model the chunks arrive over seconds rather than instantly, and that is the entire point: a user watching text appear waits considerably longer than a user watching a spinner, because each chunk is evidence the system is working.

The mistake this prevents

The mistake is streaming to the user and separately calling invoke to get the value for logging or storage. That is two model calls for one answer. Accumulate the chunks - you already have every piece.

Takeaway

stream yields the same content invoke returns, chunk by chunk. Accumulate the chunks rather than calling twice.

Mark complete