Unit 09.01: Callbacks as observation points

Callbacks are where you observe a chain without changing it, which makes them the right place for timing, counting and assertions.

Handlers that watch, not modify

A handler subclasses BaseCallbackHandler and implements the events it cares about. The chain runs identically whether or not one is attached.

The code records model and chain events during a run.

from langchain_core.callbacks import BaseCallbackHandler
from langchain_core.language_models.fake_chat_models import FakeListChatModel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate


class Recorder(BaseCallbackHandler):
    def __init__(self):
        self.events = []

    def on_chat_model_start(self, serialized, messages, **kwargs):
        self.events.append(("model_start", len(messages[0])))

    def on_llm_end(self, response, **kwargs):
        self.events.append(("model_end", None))

    def on_chain_start(self, serialized, inputs, **kwargs):
        self.events.append(("chain_start", sorted(inputs) if isinstance(inputs, dict) else None))


recorder = Recorder()
chain = (ChatPromptTemplate.from_messages([("human", "{question}")])
         | FakeListChatModel(responses=["ok"]) | StrOutputParser())
chain.invoke({"question": "hi"}, config={"callbacks": [recorder]})

for name, detail in recorder.events:
    print(f"{name:14} {detail}")

# Callbacks observe without changing the chain. That makes them the right place
# for timing, token counting and assertions -- and the wrong place for anything
# that alters the result.

The events give you a timeline for free: when the chain started, what it was given, when the model was called and when it finished. That is enough for latency attribution without touching the chain definition.

The important constraint is in the name. Handlers observe; anything that alters the result belongs in the chain where it is visible. A callback that quietly rewrites an output is a behaviour nobody reading the chain will find.

The mistake this prevents

Streaming gets harder the moment an agent is involved. An agent's intermediate output - a tool call it is considering, a partial reasoning step - is not an answer, and streamed raw it reads as one. Users act on text that appears in the answer position. Stream step-level progress for agent runs and reserve token streaming for the final response, which is the same split the LangGraph course arrives at.

The mistake is putting business logic in a callback because it has convenient access to everything. Six months later someone reads the chain definition, reasons about it correctly, and is wrong - because a handler registered elsewhere is changing the result.

Takeaway

Use callbacks to observe: timing, token counts, assertions. Anything that changes the result belongs in the chain where a reader will see it.

Mark complete