Unit 11.01: Fake responses that drive the hard paths

A scripted fake drives the paths a real model will not produce on demand.

Malformed first, valid second

A model that returns a scripted sequence, driving a retry.

The code counts the calls.

class ScriptedModel:
    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = 0

    def __call__(self, prompt):
        self.calls += 1
        return self.replies.pop(0)


model = ScriptedModel(["not json at all", '{"category": "billing"}'])
import json

for attempt in (1, 2):
    raw = model(prompt="classify")
    try:
        print(f"attempt {attempt}: parsed {json.loads(raw)}")
        break
    except json.JSONDecodeError:
        print(f"attempt {attempt}: unparseable, retrying")

print(f"\nmodel called {model.calls} times -- the retry path was exercised")

# You cannot make a real model return invalid JSON on demand, so the retry
# logic goes untested until it fails in production. A scripted fake drives it
# in a millisecond.

You cannot reliably make a real model return invalid JSON, so the retry logic goes untested until it fails in production. A scripted fake drives it in a millisecond and asserts on the call count.

The same technique reaches the refusal path, the give-up path, and every branch that exists specifically for output you hope not to get.

The mistake this prevents

The mistake is testing against a real model at temperature zero and calling it deterministic. It is neither deterministic nor free, the tests are slow enough that people stop running them, and the failure paths still cannot be triggered on demand.

Takeaway

Script the fake model's responses to drive retry, refusal and give-up paths. Those are precisely the branches a real model will not produce for you.

Mark complete