Unit 05.03: Comparing two retrievers honestly

Comparing retrievers by reading a few outputs tells you about those outputs. A fixed case set tells you about the retrievers.

Same cases, one variable, per-case results

Every configuration runs the same queries against the same corpus, and the result is recorded per case rather than as a total.

The code compares two k settings across three cases.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore

DOCS = [
    Document("Refunds are allowed within 7 days.", metadata={"id": "c1"}),
    Document("Exchanges are allowed within 30 days.", metadata={"id": "c2"}),
    Document("Support replies within one business day.", metadata={"id": "c3"}),
]
store = InMemoryVectorStore.from_documents(DOCS, DeterministicFakeEmbedding(size=64))

CASES = [("refund window", "c1"), ("exchange period", "c2"), ("reply time", "c3")]
CONFIGS = {"k=1": 1, "k=3": 3}

print(f"{'config':8} " + " ".join(f"{q:16}" for q, _ in CASES) + "recall")
for name, k in CONFIGS.items():
    results, hits = [], 0
    for query, expected in CASES:
        ids = [d.metadata["id"] for d in store.similarity_search(query, k=k)]
        found = expected in ids
        hits += found
        results.append("hit" if found else "MISS")
    cells = " ".join(f"{r:16}" for r in results)
    print(f"{name:8} {cells}{hits}/{len(CASES)}")

# Same cases, same corpus, one variable changed. Comparing two retrievers by
# reading a few outputs tells you about those outputs; this tells you about the
# retrievers -- and per-case results show which cases each one breaks.

The per-case row is what matters. A configuration that fixes one case and breaks another has an unchanged total, and reading only the total you would conclude the change did nothing.

Three cases is a demonstration of the shape, not an eval set. The shape generalises: fixed cases, one variable changed, results stored per case so the fixed and broken lists can be computed.

The mistake this prevents

The mistake is comparing retrievers on different corpora or different questions because one was set up more recently. Any difference is then unattributable, and the comparison is worse than not doing it - it produces a confident conclusion with no basis.

Takeaway

Compare retrievers on identical cases with one variable changed, and store results per case. A total hides a change that fixed one case and broke another.

Mark complete