Unit 11.01: Fake models and deterministic fixtures

Two fixtures make almost every path in a retrieval application testable offline.

Scripted responses and stable vectors

FakeListChatModel returns your responses in order, so a test can drive any path on demand. DeterministicFakeEmbedding gives the same vector for the same text every run.

The code demonstrates both.

from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.language_models.fake_chat_models import FakeListChatModel

# A fake model returns your scripted responses in order -- so a test can drive
# the retry path, the refusal path, or a malformed response on demand.
model = FakeListChatModel(responses=['not json at all',
                                     '{"category": "billing"}'])
print("call 1:", model.invoke("classify").content)
print("call 2:", model.invoke("classify").content)

# Deterministic embeddings give the same vector for the same text, every run.
e = DeterministicFakeEmbedding(size=8)
a, b = e.embed_query("refund policy"), e.embed_query("refund policy")
print("\nembeddings stable across calls:", a == b)

print("""
These two together let you test the parts that matter -- retry logic, refusal
paths, filter behaviour, citation formatting -- with no key, no cost, no
network, and no flakiness.
""")

The scripted responses are what make the hard paths reachable. A test wanting the retry path scripts a malformed response followed by a good one; a test wanting the refusal path scripts whatever it likes and sets the threshold high.

Those paths are exactly the ones a real model will not produce on demand. You cannot reliably make a real model return invalid JSON, so the retry logic goes untested until it fails in production.

The mistake this prevents

The mistake is testing against a real model with a low temperature and calling it deterministic. It is neither deterministic nor free, the tests are slow enough that people stop running them, and the failure paths still cannot be triggered on demand.

Takeaway

Fake models and deterministic embeddings make the retry, refusal and filter paths reachable in tests. Those are the paths a real model will not produce for you.

Mark complete