Unit 04.04: Testing a template without calling the API

Template rendering is a function and can be tested with hostile inputs, for free.

Empty values, and characters that break formatting

Three cases including a $ in user content.

The code checks the rendered output each time.

import string

template = string.Template("POLICY:\n$policy\n\nQUESTION:\n$question")

CASES = [
    ("normal", {"policy": "Refunds within 7 days.", "question": "How long?"}),
    ("empty policy", {"policy": "", "question": "How long?"}),
    ("dollar sign in input", {"policy": "Cost is $5.", "question": "How much?"}),
]
for name, values in CASES:
    rendered = template.safe_substitute(values)
    checks = {
        "no unfilled placeholder": "$policy" not in rendered
                                   and "$question" not in rendered,
        "question present": values["question"] in rendered,
        "policy section exists": "POLICY:" in rendered,
    }
    print(f"{name:22} {checks}")

# The third case is why `Template` beats `str.format` for user content: a `$`
# or a `{` in the input breaks `format` and does not break this. Test rendering
# with hostile characters, and do it without an API key.

The dollar-sign case is why Template beats str.format for user content. A $ or a { in an input breaks format with an exception that points at your template rather than at the input.

The empty-policy case renders successfully and produces a prompt asking the model to answer from nothing - which is a situation the refusal path should have caught earlier.

The mistake this prevents

The mistake is testing prompts only through the model. That is slow, costly and non-deterministic, and it does not test the rendering at all - a prompt that renders wrongly produces a plausible answer to the wrong question.

Takeaway

Test rendering separately with hostile inputs - empty values and formatting characters. It needs no key and catches the failures a model call would obscure.

Mark complete