Unit 02.03: Format failures and why they are cheapest to catch

Format is the one dimension you can afford to check on every single response in production.

No model, no judge, microseconds

Parsing and type checking need no labelled data and no judgement. They are string operations.

The code checks four responses.

import json

RESPONSES = [
    '{"decision": "not_eligible", "days": 9}',
    '{"decision": "not_eligible", "days": "nine"}',
    'The refund is not eligible.',
    '{"decision": "not_eligible"',
]
for raw in RESPONSES:
    try:
        parsed = json.loads(raw)
        ok = isinstance(parsed.get("days"), int)
        print(f"{'PASS' if ok else 'FAIL':5} {raw[:46]:48} "
              f"{'' if ok else 'days is not an integer'}")
    except json.JSONDecodeError as exc:
        print(f"FAIL  {raw[:46]:48} {exc.msg}")

print("""
Format is the cheapest dimension to check: no model, no judge, no labelling,
microseconds per response. It is also the one that breaks a page rather than
merely being wrong.

Check it on every response in production, not only in the eval set -- it is
the one dimension where that is affordable.
""")

The second case parses cleanly and has "nine" where an integer was required. That passes a JSON check and fails a type check, and it is the kind of drift that breaks downstream arithmetic rather than a page.

Because it costs nothing, format is the dimension to check on every response rather than on a sampled eval set. Every other dimension needs labels or a judge; this one needs a schema.

The mistake this prevents

The mistake is checking format only in the eval set. Format is the dimension most likely to break suddenly - from a max_tokens change, a prompt edit, or a model update - and a weekly eval run means up to a week of broken responses.

Takeaway

Check format on every production response. It needs no labels or judge, costs microseconds, and is the dimension most likely to break between eval runs.

Mark complete