Unit 13.03: The offline test suite

The suite runs with no key, which is what makes it run at all.

Forcing the refusal with a parameter

Four assertions covering the refusal path and a normal answer.

The code runs them.

REFUSAL = "The published policies do not cover that."


def draft(question, model, force_refusal=False):
    if force_refusal:
        return {"text": REFUSAL, "cites": [], "refused": True}
    return {"text": model(question), "cites": ["policy#2"], "refused": False}


fake = lambda q: "Refunds are allowed within 7 days. [policy#2]"

TESTS = [
    ("refuses when nothing is retrieved",
     lambda: draft("x", fake, force_refusal=True)["refused"] is True),
    ("refusal carries no citations",
     lambda: draft("x", fake, force_refusal=True)["cites"] == []),
    ("refusal text is exact",
     lambda: draft("x", fake, force_refusal=True)["text"] == REFUSAL),
    ("normal answer cites a policy",
     lambda: "[policy#" in draft("refund", fake)["text"]),
]
passed = sum(bool(t()) for _, t in TESTS)
for name, test in TESTS:
    print(f"{'PASS' if test() else 'FAIL'} {name}")
print(f"\n{passed}/{len(TESTS)} -- no key, no network, no cost")

# Forcing the refusal with a parameter rather than a carefully chosen question
# is what keeps the test stable: it does not depend on the corpus or on a
# threshold that may be retuned.

The refusal is forced with a parameter rather than a carefully chosen question. That keeps the test stable: it does not depend on the corpus, on a threshold that may be retuned, or on the model behaving a particular way.

Three of the four assertions test the refusal from different angles - the flag, the empty citations, the exact string - and each catches a different regression.

The mistake this prevents

The mistake is testing refusal by asking something absurd and hoping nothing matches. Whether it matches depends on the corpus, so the test passes today and fails after the next content update for reasons unrelated to the refusal logic.

Takeaway

Force the refusal branch with a parameter and assert on the flag, the empty citations and the exact string. A carefully chosen question is not a stable test.

Mark complete