Unit 10.00: Turning a black box into a trace

A trace is one span per step with its inputs, outputs and duration. The shape is the same whether a service collects it or you log it yourself.

Spans with durations

Each step of the chain becomes a span. Reading them together shows where the time goes and what each step produced.

The code prints a five-span trace and computes the model's share.

import json

trace = {
    "run_id": "r-8841",
    "chain": "rag-answer-v3",
    "spans": [
        {"name": "retriever", "ms": 42,
         "output": {"ids": ["c1", "c7"], "scores": [0.91, 0.44]}},
        {"name": "format_docs", "ms": 1, "output": {"chars": 180}},
        {"name": "prompt", "ms": 2, "output": {"messages": 2, "chars": 340}},
        {"name": "model", "ms": 780,
         "output": {"tokens_in": 96, "tokens_out": 22}},
        {"name": "parser", "ms": 1, "output": {"type": "str"}},
    ],
}
print(json.dumps(trace, indent=1))

total = sum(s["ms"] for s in trace["spans"])
model_ms = next(s["ms"] for s in trace["spans"] if s["name"] == "model")
print(f"\ntotal {total} ms, model is {model_ms / total:.0%} of it")

# A trace is one span per step with its inputs, outputs and duration. LangSmith
# collects these automatically when enabled; the shape is the same whether you
# use it or log the spans yourself.

The model is the overwhelming majority of the time, which is typical and worth confirming rather than assuming - if a retriever is taking seconds, that changes what to optimise entirely.

The retriever's span carries ids *and scores*. Those scores are what the next unit uses to find where a wrong answer entered, and they are gone unless the span records them at the time.

The mistake this prevents

Tracing belongs in production, not only in development. The runs worth investigating are production runs, and a system traced only locally can never explain one. What production adds is two controls: a sampling rate, so volume stays affordable, and the data handling from the last unit of this module. Sample successful runs and trace failures in full.

The mistake is enabling tracing and never looking at a trace until something breaks. Read one from a successful run first, so you know what normal looks like - otherwise the first trace you study is an unfamiliar format during an incident.

Takeaway

One span per step with inputs, outputs and duration, and scores on the retrieval span. Read a healthy trace before you need to read a broken one.

Mark complete