Module 05 Activity

Scenario

Your RAG system produces bad answers and nobody can say which half is at fault.

What you build

Retrieval, citation and answer quality scored separately, with an index rebuild regression-tested.

Steps

1. Score retrieval alone: recall, rank, and the empty case for questions that should retrieve nothing.
2. Score citation support separately from correctness, and report how many answers are correct but uncited.
3. Measure generation on a fixed context so retrieval cannot confound it. Classify failures as dropped qualifier, added claim, or widened statement.
4. Produce the four-combination diagnosis table and identify any right-answer-wrong-evidence cases.
5. Rebuild the index with a different chunk size and run the retrieval set before and after. Report the fixed and broken lists.
6. State what your retrieval threshold is and what score distribution it came from.

Evidence to hand in

• Recall, rank and empty-case results.
• Citation support scored apart from correctness.
• Generation failures on a fixed context, classified into three kinds.
• The four-combination table.
• Before-and-after rebuild results with the broken list.
• The threshold and its provenance.

Review checklist

• Retrieval is scored on questions that should return nothing, too.
• Correct-but-uncited answers are counted.
• Generation was measured on a fixed context.
• The rebuild comparison is per case, not a total.
• The threshold comes from a measured distribution.

Mark complete