Unit 03.02: Token limits, truncation and the silent cut

Truncation does not raise. It produces a valid string that stops in the middle.

The silent cut

The same request at three output limits.

The code simulates each and reports the finish reason.

def simulate(prompt_tokens, max_output, needed_output):
    produced = min(max_output, needed_output)
    return {"produced": produced,
            "finish_reason": "stop" if produced >= needed_output else "length",
            "text_complete": produced >= needed_output}


for max_output in (500, 200, 60):
    result = simulate(900, max_output, 220)
    print(f"max_output={max_output:>4} produced={result['produced']:>4} "
          f"finish_reason={result['finish_reason']:>7} "
          f"complete={result['text_complete']}")

print("\na truncated answer is well-formed prose that stops mid-thought")

# Truncation does not raise. The response is a valid string that ends in the
# middle of a sentence -- and if you are parsing JSON, it ends in the middle of
# an object and the parse fails somewhere unrelated. Check `finish_reason`.

At a low limit the response is a grammatical fragment. A user reads it as the answer; a JSON parser fails on an incomplete object and reports a syntax error that points nowhere useful.

This is also the most common cause of intermittent parse failures, and it is usually misdiagnosed as a model quality problem.

The mistake this prevents

The mistake is setting max_tokens from the typical response length. Set it from the longest legitimate response, and check finish_reason on every call rather than only when something looks wrong.

Takeaway

A truncated response is valid text that stops mid-thought and raises nothing. Size the limit from the longest legitimate answer and check finish_reason every time.

Mark complete