Module 05 Activity

Scenario

The chunks are indexed. Now retrieval has to be measured rather than assumed.

What you build

A retriever with metadata filters applied before ranking, a k chosen from measured recall, and citation ids that survive a rebuild.

Steps

1. Query for something absent from your corpus and confirm you get results anyway. Record the top score.
2. Add metadata filters for at least one exact condition, applied before ranking. Report which documents they removed from the top-k.
3. Sweep k and record, per case, whether the target was retrieved and at what rank. Choose k where the curve flattens.
4. Compare two retriever configurations on identical cases, storing results per case, and report the fixed and broken lists.
5. Choose a chunk id scheme and prove it survives a rebuild that adds a document at the top of the corpus.
6. Record the score distribution for answerable and unanswerable questions and say whether they separate.

Evidence to hand in

• The absent-query result with its top score.
• Before-and-after filtering, with displaced documents named.
• The recall-and-rank table and your chosen k.
• The two-config comparison with fixed and broken lists.
• The id scheme with rebuild evidence.
• The two score distributions.

Review checklist

• Filters run before ranking.
• k is chosen from measured recall on your own corpus.
• The comparison is per case, not a total.
• Ids survive a rebuild that reorders the corpus.
• The score distributions are plotted or tabulated, and whether they separate is stated plainly.

Mark complete