Unit 03.01: The awkward cases people actually send

There is a quiet pressure to build a set the system passes. Resisting it is most of what makes a set useful.

Seven cases that lower the score on purpose

Unanswerable, contradictory, informally worded, compound, recently changed, typo-laden, and deliberately probing.

The code lists each with the behaviour it tests.

AWKWARD = [
    ("a question the corpus cannot answer",     "refusal path"),
    ("two documents that disagree",             "conflict handling"),
    ("the user's words, not the document's",    "retrieval robustness"),
    ("two questions in one message",            "partial answering"),
    ("an answer that changed last month",       "staleness handling"),
    ("a typo-laden, half-finished sentence",    "real input, not clean input"),
    ("someone testing the assistant on purpose", "misuse handling"),
]
print(f"{'case':46} tests")
for case, tests in AWKWARD:
    print(f"{case:46} {tests}")

print(f"""
{len(AWKWARD)} cases, every one of which lowers the headline score. That is why
they belong in the set.

An eval built only from clean, answerable questions reports a number that will
not survive launch, and will drop sharply in week one for reasons nobody can
explain.
""")

The sixth is the one nobody includes. Real input is not clean: it has typos, half-finished sentences, and pasted fragments, and a set built from carefully typed questions never tests how the system handles any of it.

Every one of these lowers the headline number, and that is the argument for including them. A set built from clean answerable questions reports a figure that will not survive launch and drops sharply in week one for reasons nobody can explain.

The mistake this prevents

None of this argues for a large set. Thirty cases someone has actually read, labelled against a rubric and confirmed correct beats three hundred nobody has checked - an eval set with wrong labels in it produces confident measurements of the wrong thing, and the errors are invisible because nobody re-reads a set that size. Curate for representativeness and correctness; grow it from incidents rather than from bulk.

The mistake is removing a case because it fails. Once, that is reasonable - the case may be badly specified. As a habit it converts the set from a specification of what the system should do into a description of what it does, and the score only ever goes up.

Takeaway

Deliberately include the cases that lower the score, including messy real input. A falling number after adding hard cases is information; a set that always passes is not a measurement.

Mark complete