Module 03 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. The best source of eval cases is…
- A. Synthetic questions only
- B. Real user queries, including the ones that failed
- C. The documentation
- D. Model-generated examples
2. An eval set of only answerable questions cannot measure…
- A. Latency
- B. Whether the system invents answers when it should refuse
- C. Cost
- D. Format
3. Expected outputs for open-ended tasks are best expressed as…
- A. Exact strings
- B. Criteria or rubrics describing what a good answer must contain
- C. Nothing
- D. Regexes
4. Eval sets should be versioned because…
- A. Storage is cheap
- B. Comparing results across time is meaningless if the test set silently changed
- C. It is required
- D. Models change
5. A small, carefully curated eval set is often better than a huge one because…
- A. It runs faster only
- B. It can be reviewed by humans and kept genuinely representative and correct
- C. It costs less
- D. It is easier to write
---
Answer Key and Explanations
Check these only after attempting every question.
1. B - Real user queries, including the ones that failed
Real traffic contains the ambiguity and messiness that synthetic sets systematically miss.
2. B - Whether the system invents answers when it should refuse
Unanswerable cases are how you detect confident invention - the highest-risk behaviour.
3. B - Criteria or rubrics describing what a good answer must contain
Exact-match grading fails valid paraphrases and pushes teams to over-constrain outputs.
4. B - Comparing results across time is meaningless if the test set silently changed
A moving benchmark cannot show whether you improved.
5. B - It can be reviewed by humans and kept genuinely representative and correct
An unreviewed large set encodes its own errors as ground truth.
Practical Check
Apply this module to your own work: complete the module activity for *Eval Datasets and Test Case Design*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.