Unit 05.01: Parsing, and the three ways it fails

Three distinct parse failures, and only one of them is a model problem.

Prose wrapping, truncation, and plain English

Four candidate responses through a JSON parser.

The code shows which succeed.

import json

CANDIDATES = [
    '{"category": "billing"}',
    'Sure! Here it is: {"category": "billing"}',
    '{"category": "billing"',
    'The category is billing.',
]
for raw in CANDIDATES:
    try:
        print(f"OK    {raw[:44]:46} -> {json.loads(raw)}")
    except json.JSONDecodeError as exc:
        print(f"FAIL  {raw[:44]:46} -> {exc.msg}")

print("""
Three distinct failures: prose wrapping valid JSON, truncated JSON, and plain
English. Only the second is a bug in your configuration -- it is what a too-low
max_tokens produces -- and it is the one most often misdiagnosed as a model
quality problem.
""")

Truncated JSON is what a too-low max_tokens produces, and it presents as an intermittent parse failure that looks like model unreliability. It is a configuration bug with a completely different fix.

Prose wrapping valid JSON is common and sometimes handled by the parser. Relying on that is fragile - it depends on the implementation, and a model that starts wrapping differently breaks a chain that assumed it.

The mistake this prevents

The mistake is catching the parse error and returning a default. The metrics then show a healthy success rate while a fraction of responses are silently substituted. Count parse failures separately and alert on the rate.

Takeaway

Count parse failures rather than defaulting past them, and check finish_reason - a rising rate is usually truncation, not the model.

Mark complete