Module 08 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. A RAG evaluation set must test retrieval and generation separately because…
- A. It is traditional
- B. A bad answer may come from a missing passage or from misuse of a correct passage - the fixes differ entirely
- C. It is faster
- D. Models vary
2. Which metric addresses 'did the right passage get retrieved at all?'
- A. Fluency
- B. Recall@k
- C. Latency
- D. Token count
3. A good eval set includes…
- A. Only common questions
- B. Answerable, unanswerable, ambiguous, and conflicting-source questions
- C. Only long questions
- D. Only questions from one document
4. Using the same model to generate and to grade answers risks…
- A. Nothing
- B. Shared blind spots that systematically inflate scores
- C. Slower evaluation
- D. Licensing issues
5. Evaluations should be re-run when…
- A. Only at launch
- B. Documents, chunking, retrieval settings, prompts, or the model change
- C. Once a year
- D. Never
---
Answer Key and Explanations
Check these only after attempting every question.
1. B - A bad answer may come from a missing passage or from misuse of a correct passage - the fixes differ entirely
Without separating the stages you cannot tell whether to fix chunking or fix answering.
2. B - Recall@k
Recall@k measures whether the answer-bearing chunk appears in the top k results.
3. B - Answerable, unanswerable, ambiguous, and conflicting-source questions
The hard behaviours - refusal, conflict handling, ambiguity - only surface if you test for them deliberately.
4. B - Shared blind spots that systematically inflate scores
A judge that shares the generator's failure modes will forgive exactly the errors you most need to catch.
5. B - Documents, chunking, retrieval settings, prompts, or the model change
Every one of those inputs can silently change behaviour; the eval set is the regression test.
Practical Check
Apply this module to your own work: complete the module activity for *Evaluation Sets for RAG*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.