Unit 05.03: Retrying with the error attached

A retry with the same prompt gets the same class of output. A retry with the error attached gives the model something new.

Feedback, a ceiling, and a give-up path

Three attempts with a specific error fed back each time.

The code walks them.

import json

ATTEMPTS = ['The category is billing.',
            '{"category": "billing"',
            '{"category": "billing", "urgency": "high"}']
REQUIRED = {"category", "urgency"}
MAX_ATTEMPTS = 3


def check(raw):
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError as exc:
        return None, f"not valid JSON: {exc.msg}"
    missing = REQUIRED - set(parsed)
    return (None, f"missing keys: {sorted(missing)}") if missing else (parsed, None)


for attempt, raw in enumerate(ATTEMPTS[:MAX_ATTEMPTS], 1):
    parsed, error = check(raw)
    if parsed:
        print(f"attempt {attempt}: OK {parsed}")
        break
    print(f"attempt {attempt}: retry with -> {error!r}")
else:
    print(f"gave up after {MAX_ATTEMPTS} attempts")

# The error string is the point. Retrying with the same prompt gets the same
# class of output; retrying with "not valid JSON: Expecting ',' delimiter"
# gives the model something it did not have. Cap the attempts.

"not valid JSON: Expecting ',' delimiter" is information the model did not have. "please try again" is not, and a loop built on it pays for a call per attempt to learn nothing.

The for/else is the give-up path. Without a ceiling, a persistently malformed response retries until something else stops it.

The mistake this prevents

The mistake is retrying validation failures the same way as parse failures. A response that parses but carries a disallowed value will often produce the same value again - feed back the allowed set explicitly, or fall through to a deterministic default.

Takeaway

Retry with the specific error attached, cap the attempts, and have an explicit give-up path. Feedback is what makes attempt two different.

Mark complete