Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 07.04: What to escalate to a human

Escalation criteria should be computable and decided in advance. A system that escalates when a user complains has no escalation policy.

Four signals, all already available

Everything needed to make the decision has been built in this module and the last: do the chunks disagree, are the top two effectively tied, is the best evidence stale, does the answer contain unsupported terms.

The table below combines all four into one triage rule, applied to a clean case and a risky one.

SIGNAL                              CLEAN CASE   RISKY CASE
retrieved chunks disagree           no           YES
top two scores within 0.02          no           YES  (0.91 / 0.90)
best evidence older than a year     no           YES  (800 days)
answer contains unsupported terms   no           YES  ("instantly")

escalate?                           no           YES -- four reasons

The reasons are what a reviewer receives with the case. "This needs review"
wastes their time; four named signals tell them where to look.

The clean case escalates on nothing. The risky one returns four reasons, each traceable to a specific check rather than to a general sense that something is off.

Those reasons are what a human receives along with the query. "This needs review" wastes the reviewer's time; "the top two chunks are tied and state different refund windows, and the older one is from 2024" tells them where to look. The value of the triage rule is as much in what it reports as in what it routes.

The mistake this prevents

The mistake is treating escalation as an error path. It is a designed outcome for cases the evidence cannot settle, and a system with only "answer" and "refuse" forces every ambiguous case into one of them. Ambiguous cases are exactly the ones a person should see.

Takeaway

Decide escalation criteria in advance, compute them from signals you already have, and pass the reasons through to whoever reviews the case. Module 10 makes this one of three routing outcomes in the full workflow.