Unit 01.03: When RAG is the wrong tool entirely

RAG has become the default answer to "how do we make the model know about our stuff?" For a good number of problems it is the wrong shape, and building it anyway costs months.

Four questions, three of them not RAG

RAG fits one specific case: the answer exists in text you hold, and finding the right passage is the hard part. Change either half of that sentence and something else fits better.

The table below sorts four questions by kind. Read the kind column before the RAG? column - the kind is what decides.

question                                     kind                     RAG?
What is our refund window?                   documented fact          yes
How many refunds did we issue last quarter?  aggregate over records   no
Should we change the refund policy?          judgement call           no
What did the CEO say in the all-hands?       not recorded anywhere    no

RAG answers questions whose answer is written down somewhere retrievable.
  - A count over records is a database query. Retrieval will find documents
    that mention refunds, not compute the number.
  - A judgement call has no retrievable answer at all.
  - Content that was never recorded cannot be retrieved, and an assistant asked
    for it will produce something anyway.

Row 2 is an aggregate over records. It is a SELECT COUNT(*), and no amount of chunking makes a vector search compute it reliably; retrieval will find documents that *mention* refunds, which is a different thing entirely.

Row 3 has no retrievable answer at all - "should we change the policy" is a judgement, and a model handed your policy documents will produce a confident recommendation that appears in none of them. Row 4 is the quietest failure: the all-hands was never written down, so there is nothing to retrieve, and the assistant will answer anyway.

The mistake this prevents

The mistake is scoping by data source rather than by question type. "We have 40,000 documents, so we need RAG" skips the question of what people will actually ask. Collect fifty real questions first and sort them into these four kinds. If most land outside row 1, you are building the wrong system.

Takeaway

RAG answers *what do the documents say about X*. It does not compute, count, or decide, and it cannot retrieve what was never recorded. Scoping the assistant means writing down which kinds of question you accept.

Mark complete