Unit 05.02: Tuning how many documents come back

k is the number of documents in the context window on every single query, including the ones that needed one.

Recall against noise, measured

For each k, whether the answer-bearing chunk was retrieved, at what rank, and how much noise came with it.

The code sweeps k over a corpus with one relevant document among thirty.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore

DOCS = [Document(f"policy statement {i} about various support topics")
        for i in range(30)]
DOCS.append(Document("Refunds are allowed within 7 days of purchase."))
store = InMemoryVectorStore.from_documents(DOCS, DeterministicFakeEmbedding(size=64))

TARGET = "Refunds are allowed within 7 days of purchase."
for k in (1, 3, 10, 20):
    hits = store.similarity_search("refund window", k=k)
    found = any(h.page_content == TARGET for h in hits)
    rank = next((i for i, h in enumerate(hits, 1) if h.page_content == TARGET), None)
    print(f"k={k:<3} target retrieved: {found!s:5} rank: {rank}  "
          f"noise chunks: {k - (1 if found else 0)}")

# Every chunk past the one that answers the question is noise in the context
# window, and it is paid for on every query. Measure recall against your own
# eval set and stop where the curve flattens.

The noise column is the cost. At k=20 the relevant chunk is present alongside nineteen irrelevant ones, each of which is a citable id the model might attach to a claim.

The rank column tells you what k needs to be. If the target sits at rank 2 across your eval set, k=3 is enough and k=10 is buying nothing but tokens - and the only way to know is to measure on your own corpus.

The mistake this prevents

The mistake is raising k when answers are wrong. If the answer chunk was already retrieved, more chunks do not help - the failure is in generation or in citation checking. Check the rank first, then tune the stage that actually failed.

Takeaway

Measure recall and rank across your eval set and stop where the curve flattens. Everything past the answer-bearing chunk is noise paid for on every query.

Mark complete