Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 08.02: Scoring citations, not just correctness

An answer can be factually right and still indicate a broken system. Correctness alone cannot tell the two apart.

Right for the wrong reason

Three answers, all stating the correct refund window. One cites the chunk that contains it, one cites a chunk about exchanges, and one cites nothing at all.

The table below scores each on correctness and on citation support, and the two columns disagree.

CHUNKS
  c1   Refunds are allowed within 7 days.
  c2   Exchanges within 30 days.

ANSWER                        CITES   FACTUALLY CORRECT?   SUPPORTED BY ITS CITATION?
"Refunds within 7 days."      c1      yes                  yes
"Refunds within 7 days."      c2      yes                  NO
"Refunds within 7 days."      none    yes                  NO -- nothing claimed

All three answers are correct. Only the first is properly cited. The third
came from the model's own knowledge and happened to match, which is
indistinguishable from a working system until the question changes.

Case three is the one worth sitting with. The answer is correct and cites nothing, which means it came from the model's weights and happened to match. On this question that is harmless. On the next question - where the model's general knowledge and your policy diverge - the same behaviour produces a confident error.

Correctness scored alone rewards that behaviour. It will show 3 out of 3 and give you no signal that two thirds of the answers were ungrounded.

The mistake this prevents

The mistake is treating citation support as a presentation detail to measure later. It is the measurement that tells you whether the retrieval half of your system is contributing at all. An assistant scoring well on correctness and badly on citation support is a language model with an expensive vector database attached.

Takeaway

Score citation support alongside correctness on every case. An answer that is right by luck is indistinguishable from a working system until the question changes.