Unit 03.02: Indexes, neighbours, and how many to fetch

k - the number of chunks you retrieve - looks like a tuning knob and behaves like a design decision. It trades recall against noise, cost and latency all at once.

What each additional chunk buys and costs

Small k risks missing the chunk that holds the answer. Large k guarantees you include it and surrounds it with irrelevant material, which dilutes the context and gives the model more chances to cite the wrong thing.

The example below builds a 200-vector corpus, perturbs one of them into a query, and reports what k=1, 3 and 10 return.

A corpus of 200 chunks. The chunk that answers the question is number 7.

k     CHUNKS RETURNED                 CONTAINS THE ANSWER?   NOISE
1     7                               yes                    0
3     7, 69, 33                       yes                    2
10    7, 69, 33, 134, 199, 53, ...    yes                    9

score gap, rank 1 to rank 2:   0.163   <- a decisive separation
score gap, rank 9 to rank 10:  0.012   <- effectively a tie

The answer is at rank 1, so every setting of k finds it and k=10 adds nine
chunks of noise -- each one a citable id the model might attach to a claim.

The true match is at rank 1, so every setting of k finds it and k=10 adds nine chunks of pure noise. The score gaps say why: 0.163 between ranks 1 and 2, and 0.012 between ranks 9 and 10. The first gap is a real separation; the second is arbitrary ordering of near-identical scores.

That gap between consecutive scores is the number to watch. A large rank-1-to-rank-2 gap means the ranking is decisive. A tiny one means two chunks are effectively tied, and which one lands in slot 1 is close to chance - worth escalating rather than answering from, as Module 10 does.

The mistake this prevents

There is a second reason to keep k modest, and it is what makes the standard two-stage pattern work. Retrieve generously with the cheap vector search - recall first - then re-rank those candidates with a costlier, more accurate relevance model and pass only the top few forward. Re-ranking is expensive per candidate, so it is affordable only over a shortlist, and the shortlist is exactly what k defines. Cheap recall, expensive precision, in that order.

The mistake is raising k when answers are wrong. If the answer chunk was already at rank 1, retrieving ten chunks does not help - the failure is in generation or citation checking, not recall. Diagnose which stage failed before tuning the one that did not.

Takeaway

Choose k by measuring recall on your own eval set and stopping where the curve flattens. Then watch the rank-1-to-rank-2 gap per query: it tells you whether the ranking was decisive or a coin flip.

Mark complete