Unit 06.03: Stopping rules and runs that never end

Three ceilings, because a run can be cheap and endless or fast and expensive.

Which ceiling fired is itself a metric

Steps, tokens and time, each recorded as the reason a run stopped.

The code shows four runs stopped by three different ceilings.

RUNS = [
    {"id": "r1", "steps": 3,  "tokens": 6_200,  "seconds": 11, "stopped_by": None},
    {"id": "r2", "steps": 12, "tokens": 41_000, "seconds": 96, "stopped_by": "step ceiling"},
    {"id": "r3", "steps": 4,  "tokens": 38_000, "seconds": 22, "stopped_by": "token ceiling"},
    {"id": "r4", "steps": 2,  "tokens": 4_000,  "seconds": 61, "stopped_by": "time ceiling"},
]
CEILINGS = {"steps": 12, "tokens": 40_000, "seconds": 60}
print(f"{'run':4} {'steps':>6} {'tokens':>8} {'secs':>6}  stopped by")
for r in RUNS:
    print(f"{r['id']:4} {r['steps']:>6} {r['tokens']:>8,} {r['seconds']:>6} "
          f" {r['stopped_by'] or 'completed normally'}")

stopped = sum(1 for r in RUNS if r["stopped_by"])
print(f"\n{stopped}/{len(RUNS)} runs hit a ceiling; ceilings: {CEILINGS}")
print("three different ceilings fired -- one alone would have missed two runs")

# Track the stop reason as a metric. A rising rate of ceiling stops is a
# regression even when the answers that do complete are still correct.

Three different ceilings fired across three runs. Any one alone would have missed two of them - r3 used only four steps and forty thousand tokens, r4 used two steps and sixty-one seconds.

Recording the stop reason turns this into a trackable metric. A rising rate of ceiling stops is a regression even when the runs that do complete are still correct, and it is usually the first visible symptom of a prompt or model change.

The mistake this prevents

The mistake is treating ceiling stops as infrastructure noise rather than quality signal. A run that hit a ceiling did not answer the user's question, so it belongs in your failure rate - not in a separate bucket labelled timeouts.

Takeaway

Bound steps, tokens and time independently, and record which ceiling fired. Ceiling stops are failures, and a rising rate is an early warning.

Mark complete