Unit 08.00: Building a question set from real usage

An eval set is the only thing standing between "we changed the chunk size" and "we have no idea whether that helped." It is also the artefact teams most often write last, from imagination.

Where the questions come from

Questions invented by the people who built the system are systematically easy. They use the document's vocabulary, they are single-part, and they have answers - because the builder knows what the corpus contains.

Real usage supplies none of those conveniences. The example below shows a small set drawn from a support log, with the provenance of each case recorded.

[
 {
  "id": "q1",
  "question": "What is the refund window?",
  "expected_chunk": "c1",
  "expect_refusal": false,
  "source": "support log"
 },
 {
  "id": "q2",
  "question": "i want my money back",
  "expected_chunk": "c1",
  "expect_refusal": false,
  "source": "support log - informal wording"
 },
 {
  "id": "q3",
  "question": "What is the office address?",
  "expected_chunk": null,
  "expect_refusal": true,
  "source": "support log - not documented"
 },
 {
  "id": "q4",
  "question": "Refund window and how to request one?",
  "expected_chunk": "c1",
  "expect_refusal": false,
  "source": "support log - compound"
 }
]

4 cases, 1 expecting refusal (25%)
If that share is zero, the set cannot detect an over-eager assistant.

Look at the source field on each case. q2 is informal wording, q3 is a question the documents do not cover, q4 is compound. Each one is a specific failure mode from an earlier module, captured as a case.

The refusal share is printed for a reason: if it is zero, the set cannot detect an over-eager assistant, no matter how many cases it contains. That single number is the fastest check on whether an eval set is worth anything.

The mistake this prevents

The mistake is building the set once, at launch, and treating it as fixed. The queries that return nothing (Module 6) and the cases that get escalated (Module 7) are both sources of new eval cases, arriving free from production. A set that does not grow stops representing what users ask within a few months.

Takeaway

Build the eval set from real questions, record where each one came from, and check the refusal share before trusting any score it produces.

Mark complete