Unit 04.00: Testing a prompt like a function

A prompt is a function from inputs to a string. It can be tested like one, for free, before any model is involved.

Hostile inputs, not just typical ones

Empty values, missing values, and content that interacts with the templating itself.

The code renders three cases including one that breaks formatting.

TEMPLATE = ("Answer only from CONTEXT. Cite the chunk id.\n\n"
            "CONTEXT:\n{context}\n\nQUESTION: {question}")


def render(context, question):
    return TEMPLATE.format(context=context, question=question)


CASES = [
    ("normal", {"context": "[c1] Refunds within 7 days.", "question": "How long?"}),
    ("empty context", {"context": "", "question": "How long?"}),
    ("braces in input", {"context": "[c1] Use {placeholder} syntax.",
                         "question": "How?"}),
]
for name, args in CASES:
    try:
        out = render(**args)
        checks = {"contains context marker": "CONTEXT:" in out,
                  "question present": args["question"] in out,
                  "no unfilled placeholder": "{" not in out.split("CONTEXT:")[0]}
        print(f"{name:16} {checks}")
    except (KeyError, IndexError) as exc:
        print(f"{name:16} RAISED {type(exc).__name__}: {exc}")

# The third case is the one that bites in production: user content containing
# braces breaks `str.format`. A prompt is a function -- test it with hostile
# inputs like any other.

The third case is the one that bites in production. User content containing braces breaks str.format, and the input that triggers it is ordinary - a code snippet, a template, a set of curly quotes in the wrong encoding.

The empty-context case is the other one worth having. It renders successfully and produces a prompt asking the model to answer from nothing, which is exactly the situation where a refusal path is supposed to have fired earlier.

The mistake this prevents

The mistake is testing prompts only through the model. That is slow, costly and non-deterministic, and it does not test the rendering at all - a prompt that renders wrongly produces a plausible answer to the wrong question, and the model call tells you nothing about which.

Takeaway

Test rendering separately from the model, with hostile inputs. Braces in user content and empty context are the two cases that reach production untested.

Mark complete