Unit 06.00: Designing the answer that says no

Refusal is a feature with a specification, not the absence of an answer. It is worth designing as carefully as anything the system does produce.

Fixed string, no citation, no hedge

A refusal has three properties worth insisting on. It is a fixed string, so tests can assert on it. It carries no citation, because there is nothing to cite. And it does not apologise its way into a guess.

The example below sets out a response with both branches, for an empty retrieval result and a populated one.

WHAT RETRIEVAL RETURNED       THE RESPONSE
nothing                       "The available documents do not cover that."
                              cites: none      refused: yes

one chunk (c1)                "Based on c1..."
                              cites: c1        refused: no

A refusal has three properties worth insisting on. It is a fixed wording, so
a check can match it exactly. It carries no citation, because there is
nothing to cite. And it does not apologise its way into a guess -- "I am not
sure, but probably 30 days" is not a refusal.

The refused response carries a refused flag alongside the text. That flag is the part that scales: an eval set can count refusals without parsing English, and a dashboard can track the refusal rate over time.

The fixed string matters for the same reason. "I'm not sure, but it may be around 30 days" is not a refusal - it is a guess with a disclaimer, and users act on it exactly as they would on an answer.

The mistake this prevents

Fixed does not have to mean bare. A refusal is more useful when it also says what was searched and where to go next - "the support policy documents do not cover that; an agent can check the account system" - and that is still a fixed string, still assertable, still carrying no citation. What must not vary is the part a test matches on and the absence of a guess. Compose the message from a constant core plus a constant next-step clause, not from whatever the model produces that time.

The mistake is letting the model phrase its own refusals. You then get twenty variants, none of which a test can match, several of which contain a hedged guess. Fix the wording in the design and let the model choose only *whether* to refuse, not how.

Takeaway

Specify the refusal: exact wording, empty citation list, an explicit flag in the response object. All three exist so the behaviour can be tested and counted rather than hoped for.

Mark complete