Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 08.03: The cases you would rather leave out

There is a strong, quiet pressure to build an eval set the system passes. Resisting it is most of what makes an eval set useful.

The cases that lower the score on purpose

Unanswerable questions, contradictory documents, informal phrasing, compound questions, recently-changed answers. Every one makes the headline number worse and every one corresponds to a real thing users do.

The table below lists them against the capability each one tests.

- a question the corpus genuinely cannot answer        tests refusal
- two documents that contradict each other             tests conflict handling
- a question phrased in the user's words, not the document's tests retrieval robustness
- a compound question with two parts                   tests partial answering
- a question whose answer changed last month           tests staleness handling

Every one of these makes the eval score look worse, which is exactly why they
belong in the set. An eval built only from questions the system already handles
measures nothing and reports a number that will not survive production.

Each awkward case maps to a module: refusal to Module 6, conflict to Module 7, phrasing to Module 4, partial answering to Module 6, staleness to Module 7. That mapping is the argument for including them - they are not adversarial edge cases, they are the behaviours the system was designed to have.

An eval built only from questions the system already handles reports a number that will not survive contact with production, and will drop sharply the week after launch for reasons nobody can explain.

The mistake this prevents

The mistake is removing a case because it fails. Once, that is reasonable - the case may be badly specified. As a habit it converts the eval set into a record of what the system does rather than a specification of what it should do, and the score becomes a number that only ever goes up.

Takeaway

Deliberately include the cases that lower the score. A falling eval number after adding hard cases is information; a set that always passes is not a measurement.