Unit 11.02: Writing the evaluation set

The eval set is a specification of behaviour, not a test suite bolted on afterwards. Its composition says what the system is supposed to do.

The kinds are the specification

Direct, paraphrase, compound, unanswerable, conflict, stale, restricted. Seven kinds across eight cases - because each kind tests a different behaviour, and eight questions of one kind measure one behaviour eight times.

The table below counts the kinds and reports the refusal share.

{
 "direct": 1,
 "paraphrase": 1,
 "compound": 1,
 "unanswerable": 2,
 "conflict": 1,
 "stale": 1,
 "restricted": 1
}

8 cases, 3 expecting refusal (38%)

The kinds are the specification. A set of eight direct questions measures one
behaviour eight times; this set measures seven. Score retrieval and citation
separately for each, and store per-case results so a change that fixes one and
breaks another is visible.

Three of eight cases expect refusal, roughly 38%. That share is deliberate and it is the number to check first when reviewing anyone's eval set - including your own. A set with no refusal cases cannot detect the failure the entire course has been building towards.

Eight cases is small, and that is fine for the capstone. The composition generalises; the count does not. Growing it means adding cases of the under-represented kinds, not adding more direct questions because they are easy to write.

The mistake this prevents

The mistake is scoring the set with a single number. Each case needs retrieval and citation scored separately, and per-case results stored, so the newly-broken list from Module 8 can be computed. A capstone eval producing one percentage has thrown away most of its own value.

Takeaway

Build the eval set from kinds rather than from questions, keep a substantial refusal share, and store per-case retrieval and citation results so changes can be compared case by case.

Mark complete