Unit 12.03: Writing tests that run offline

The suite runs with no key, no network and no cost, which is what makes it run at all.

The refusal path, tested three ways

Fake model, deterministic embeddings, and a threshold set high enough to force the refusal branch.

The code runs three assertions about refusal behaviour.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.language_models.fake_chat_models import FakeListChatModel
from langchain_core.vectorstores import InMemoryVectorStore

REFUSAL = "The available documents do not cover that."
store = InMemoryVectorStore.from_documents(
    [Document("Refunds are allowed within 7 days.", metadata={"id": "c1"})],
    DeterministicFakeEmbedding(size=64))


def answer(question, model, threshold=0.75):
    scored = store.similarity_search_with_score(question, k=2)
    keep = [(d, s) for d, s in scored if s >= threshold]
    if not keep:
        return {"text": REFUSAL, "cites": [], "refused": True}
    return {"text": model.invoke(question).content,
            "cites": [d.metadata["id"] for d, _ in keep], "refused": False}


TESTS = [
    ("refuses when nothing clears the threshold",
     lambda: answer("office address", FakeListChatModel(responses=["x"]),
                    threshold=0.99)["refused"] is True),
    ("refusal carries no citations",
     lambda: answer("office address", FakeListChatModel(responses=["x"]),
                    threshold=0.99)["cites"] == []),
    ("refusal string is exact",
     lambda: answer("office address", FakeListChatModel(responses=["x"]),
                    threshold=0.99)["text"] == REFUSAL),
]
passed = 0
for name, test in TESTS:
    ok = test()
    passed += ok
    print(f"{'PASS' if ok else 'FAIL'} {name}")

print(f"\n{passed}/{len(TESTS)} -- no key, no network, no cost, no flakiness")

All three test the same path from different angles: the flag is set, no citations are attached, and the string is exactly right. Each would catch a different regression.

Forcing the branch by raising the threshold is the technique worth remembering. You do not need a document that fails to match - you need a bar nothing clears, which is one parameter and works on any corpus.

The mistake this prevents

The mistake is testing refusal by querying for something absurd and hoping nothing matches. Whether it matches depends on the embedding model and the corpus, so the test passes today and fails after re-indexing - for reasons unrelated to the refusal logic.

Takeaway

Force the refusal branch with a threshold parameter rather than a carefully chosen query, and assert on the flag, the empty citations and the exact string.

Mark complete