Unit 03.02: Making an LLM step's output checkable

A model's output is untrusted text until something has validated it. Designing the step so validation is possible is a design decision, not a post-processing detail.

Parse, shape, allowed values

Three checks in order: does it parse, does it have the field, and is the value one you accept. Each catches a different failure and the third is the one that catches drift.

The code runs three model outputs through the validator.

import json

ALLOWED = {"billing", "technical", "account", "other"}


def validate(raw):
    """Everything a model returns is untrusted text until this passes."""
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return None, "not valid JSON"
    if not isinstance(parsed, dict) or "category" not in parsed:
        return None, "missing 'category'"
    if parsed["category"] not in ALLOWED:
        return None, f"category {parsed['category']!r} not in the allowed set"
    return parsed, None


for raw in ['{"category": "billing"}',
            '{"category": "Billing Department"}',
            'Sure! Here is the JSON: {"category": "billing"}']:
    parsed, error = validate(raw)
    print(f"{'OK  ' if parsed else 'FAIL'} {raw[:44]:46} {error or parsed}")

# The third case is the one that bites. It is valid English, contains valid
# JSON, and is not valid JSON -- so the parse fails and the node needs a path
# for that, which is what the next unit builds.

The third case is the one that bites: Sure! Here is the JSON: {"category": "billing"}. It is helpful, it is valid English, it contains valid JSON, and it is not valid JSON - so the parse fails.

The second case is subtler. "Billing Department" parses fine and has the right field. It fails only because the value is not in the allowed set, which is why the allowed set has to exist as data rather than as an instruction in the prompt.

The mistake this prevents

A note on temperature=0, which is often proposed as the fix here. It makes a model step more consistent and does not make it deterministic: the same prompt can still produce different text across model versions, across providers, and sometimes across calls, because the sampling is only one source of variation. Treat it as noise reduction, not as a guarantee. The validator is still required, and a step whose correctness depends on identical output across runs is a step that belongs in code.

The mistake is asking for structured output in the prompt and treating that as the guarantee. A prompt is a request. The allowed-value check is what turns model drift - a new phrasing, a slightly different label - into a caught failure rather than a category nothing downstream recognises.

Takeaway

Validate parse, shape and allowed values separately, with the allowed set as data. An LLM step without a validator is a step whose failures reach the next node unchanged.

Mark complete