Unit 03.01: Parsers, and what happens when parsing fails

Parsers succeed more often than you would expect and fail in ways worth knowing before you depend on them.

Four candidates, two surprises

Clean JSON, JSON wrapped in prose, plain English, and truncated JSON.

The code runs all four through the parser.

from langchain_core.output_parsers import JsonOutputParser
from langchain_core.exceptions import OutputParserException

parser = JsonOutputParser()

CANDIDATES = [
    '{"category": "billing"}',
    'Sure! Here is the JSON: {"category": "billing"}',
    'The category is billing.',
    '{"category": "billing"',
]
for raw in CANDIDATES:
    try:
        print(f"OK    {raw[:46]:48} -> {parser.parse(raw)}")
    except (OutputParserException, ValueError) as exc:
        print(f"FAIL  {raw[:46]:48} -> {type(exc).__name__}")

# The second case is worth noting: many parsers extract JSON embedded in prose,
# so it succeeds. Do not rely on that -- it depends on the parser, and a model
# that starts wrapping its output differently will break a chain that assumed it.

The second case succeeds. Many parsers extract JSON embedded in surrounding text, which is genuinely helpful and a bad thing to rely on - it depends on the parser implementation, and a model that starts wrapping its output differently will break a chain that assumed the extraction.

The fourth is the one that matters operationally. Truncated JSON is what you get when max_tokens is too low, and the failure looks like a model quality problem rather than a configuration one.

The mistake this prevents

The mistake is catching the parse failure and returning a default. A default silently substitutes for a real answer, and the metrics show a healthy success rate. Count parse failures separately and alert on the rate - a jump usually means a config or model change.

Takeaway

Parsers handle more than clean JSON, and relying on that is fragile. Count parse failures rather than defaulting past them; a rising rate is usually max_tokens or a model change.

Mark complete