Unit 06.01: Keeping the model inside the retrieved text

A grounding prompt is worth writing as a short list where every line traces to a failure you can name.

Five rules, five failures

Answer only from context, cite every claim, exact refusal wording, ignore outside knowledge, report disagreement.

The code prints the prompt and maps each line to what it prevents.

SYSTEM = """Answer only from the CONTEXT below.
- Cite the chunk id for every claim.
- If the CONTEXT does not contain the answer, reply exactly:
  "The available documents do not cover that."
- Do not use knowledge from outside the CONTEXT, even if you are confident.
- If two chunks disagree, say so and cite both."""

print(SYSTEM)
print()
for rule, failure in [
    ("answer only from context", "answering from model weights"),
    ("cite every claim", "unverifiable assertions"),
    ("exact refusal wording", "a fluent guess when nothing was retrieved"),
    ("ignore outside knowledge", "general knowledge overriding your policy"),
    ("report disagreement", "silently picking one side of a contradiction"),
]:
    print(f"  {rule:26} -> prevents {failure}")

# Every line traces to a failure. The fourth is the one people leave out, and it
# matters because the model often has genuine general knowledge that contradicts
# your documents -- your documents are the authority on your policies.

The fourth rule is the one people leave out, and it addresses a specific situation: the model has genuine general knowledge that contradicts your documents. Refund windows vary by company, the model has read thousands of them, and your documents are the authority on yours.

The exact refusal wording matters for a mechanical reason. Fixed in the prompt and fixed in your tests, it becomes something an eval set can assert on rather than something a human has to read and judge.

The mistake this prevents

The mistake is treating the prompt as the grounding mechanism. It reduces the rate of these failures and bounds nothing. The unsupported-terms check in the last unit of this module is what tells you the rate.

Takeaway

Write the grounding prompt as a list where every line names a failure, and fix the refusal string exactly so tests can assert on it. The prompt moves the rate; measurement tells you by how much.

Mark complete