Unit 08.03: The cases that make the score look worse

There is a quiet pressure to build a set the agent passes.

Six cases that lower the number on purpose

Refusal, missing data, rephrasing, compound requests, staleness, and messy input.

The table below lists each with what it tests.

case                                         tests
a request the agent must refuse              refusal path
data that does not exist                     escalation on missing input
a request phrased three different ways       robustness
two requests in one message                  partial handling
a fact that changed since last week          staleness
a half-finished sentence with a typo         real input, not clean input

6 cases, each of which lowers the headline number. That is the
argument for including them.

The last is the one nobody writes. Real requests arrive truncated, misspelled
and half-pasted, and a set of carefully typed sentences never tests any of it.

The last one is the case nobody writes. Real requests arrive truncated, misspelled and half-pasted, and a set of carefully typed sentences never tests any of it - so the first week in production produces a drop nobody can explain.

Every case here lowers the headline number, which is the argument for including them rather than against it. A number that only goes up is not measuring anything.

The mistake this prevents

The mistake is removing a case because it fails. Once, that is reasonable - the case may be badly specified. As a habit it turns the set into a description of what the agent does rather than a specification of what it should.

Takeaway

Include the cases that lower the score, especially messy real input. A falling number after adding hard cases is information; a set that always passes is not a measurement.

Mark complete