Unit 03.00: Building a set from real usage

An eval set written by the people who built the system is systematically easy, because they know what the corpus contains.

Real questions, with their provenance recorded

Each case records where it came from. That field is what lets you tell an invented case from an observed one later.

The code shows a five-case set with kinds and sources.

import json
from collections import Counter

CASES = [
    {"id": "q1", "question": "What is the refund window?", "kind": "direct",
     "source": "support log", "expect_refusal": False},
    {"id": "q2", "question": "i want my money back", "kind": "paraphrase",
     "source": "support log", "expect_refusal": False},
    {"id": "q3", "question": "What is the office address?", "kind": "unanswerable",
     "source": "support log", "expect_refusal": True},
    {"id": "q4", "question": "Refund window and how to request one?",
     "kind": "compound", "source": "support log", "expect_refusal": False},
    {"id": "q5", "question": "Is 30 days right?", "kind": "conflict",
     "source": "incident 2026-06-14", "expect_refusal": False},
]
print(json.dumps(dict(Counter(c["kind"] for c in CASES)), indent=1))
refusals = sum(c["expect_refusal"] for c in CASES)
print(f"\n{len(CASES)} cases, {refusals} expecting refusal "
      f"({refusals / len(CASES):.0%})")
print("sources:", sorted({c['source'] for c in CASES}))

# Every case records where it came from. `q5` came from an incident, which is
# the healthiest source there is -- it means a real failure became a permanent
# test rather than a fix someone made once.

q5 came from an incident, which is the healthiest source there is - it means a real failure became a permanent test rather than a fix someone made once and moved on from.

The refusal share is printed for a reason. If it is zero the set cannot detect an over-eager assistant, however many cases it contains, and that single number is the fastest check on whether an eval set is worth anything.

The mistake this prevents

The mistake is building the set once at launch and treating it as fixed. The queries that return nothing, the escalations, and every incident are all sources of new cases arriving free from production. A set that has not grown has stopped representing what users ask.

Takeaway

Build from real questions, record the source of each, and check the refusal share before trusting any score. Incidents are the best source of new cases.

Mark complete