Unit 03.03: Retrying a malformed response

A retry with the same prompt gets the same class of output. A retry with the error attached gives the model something it did not have.

Feedback, a cap, and a give-up path

Each attempt produces a specific error string, and that string goes into the next attempt.

The code walks three attempts, succeeding on the third.

import json

ATTEMPTS = [
    'The category is billing.',
    '{"category": "billing"',
    '{"category": "billing", "urgency": "low"}',
]
REQUIRED = {"category", "urgency"}


def validate(raw):
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError as exc:
        return None, f"not valid JSON: {exc.msg}"
    missing = REQUIRED - set(parsed)
    if missing:
        return None, f"missing keys: {sorted(missing)}"
    return parsed, None


for attempt, raw in enumerate(ATTEMPTS, 1):
    parsed, error = validate(raw)
    if parsed:
        print(f"attempt {attempt}: OK {parsed}")
        break
    print(f"attempt {attempt}: retry with feedback -> {error!r}")
else:
    print("giving up after 3 attempts")

# The feedback string is the whole point. Retrying with the same prompt gets the
# same class of output; retrying with "not valid JSON: Expecting ',' delimiter"
# gives the model something it did not have. Cap the attempts and have a
# give-up path, or a persistently malformed response loops forever.

"not valid JSON: Expecting ',' delimiter" is information. "please try again" is not, and a retry loop built on the second costs a full call per attempt to learn nothing.

The for/else is doing real work here: the else branch runs when the loop finishes without breaking, which is the give-up path. Without a cap and a give-up path, a persistently malformed response retries until something else stops it.

The mistake this prevents

The mistake is retrying validation failures the same way as parse failures. A response that parses but has a disallowed value will often produce the same value again - the model believes it. Feed back the allowed set explicitly, or fall through to a deterministic default.

Takeaway

Retry with the specific error attached, cap the attempts, and have an explicit give-up path. Feedback is what makes the second attempt different from the first.

Mark complete