Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 06.04: Why a fluent guess is the expensive failure

Not all wrong outcomes cost the same, and threshold decisions should follow the costs rather than the accuracy number.

Four outcomes, wildly asymmetric costs

A system can refuse correctly, refuse unnecessarily, answer correctly, or answer fluently and wrongly. Three of those are cheap. One is not.

The table below lays out all four with their consequences.

outcome                                consequence                            cost
refuses correctly                      user asks a colleague                  low
refuses when it could have answered    user is mildly annoyed                 low
answers correctly                      user proceeds                          none
answers fluently and wrongly           user acts on it and finds out later    high

The costs are not symmetric. An unnecessary refusal is visible and recoverable
-- the user asks someone else. A fluent wrong answer is invisible: it is acted
on, and the error surfaces later, somewhere else, attributed to something else.

Tune the threshold towards refusal, and record that you did.

An unnecessary refusal is visible and recoverable: the user is mildly annoyed and asks a colleague. The loss is one interaction. A fluent wrong answer is invisible - it is acted on, and the error surfaces later, somewhere else, usually attributed to something other than the assistant.

That asymmetry has a direct consequence for the retrieval threshold. Raising it produces more refusals and fewer confident errors, which trades a cheap failure for an expensive one. Make that trade knowingly and record that you made it.

The mistake this prevents

The mistake is optimising for answer rate. "The assistant answers 94% of questions" is a metric that rewards guessing, and it will be reported to you as progress. Pair it with the fluent-error rate from your eval set, or it means nothing.

Takeaway

Tune the threshold towards refusal, deliberately, and write down that you did. The costs of the four outcomes are not symmetric, so a threshold chosen to maximise accuracy is a threshold chosen for the wrong objective.