Unit 08.04: Saying which part of the file an answer came from

An answer that names which part of the file it came from can be checked in seconds.

Citations plus a term check

An answer with two citations, verified against the cited chunks.

The code lists any terms not present in them.

import json

CHUNKS = {
    "notes.txt#2": "Refunds are allowed within 7 days of purchase.",
    "notes.txt#3": "This applies to individual plans only.",
}
answer = {
    "text": "Refunds are allowed within 7 days, for individual plans.",
    "cites": ["notes.txt#2", "notes.txt#3"],
}
cited = " ".join(CHUNKS[c] for c in answer["cites"]).lower()
terms = [w.strip(".,") for w in answer["text"].lower().split()
         if w.strip(".,").isalpha() and len(w) > 4]
missing = sorted({t for t in terms if t not in cited})

print(json.dumps(answer, indent=1))
print(f"\nterms not in the cited chunks: {missing or 'none'}")
for c in answer["cites"]:
    print(f"  {c}: {CHUNKS[c]}")

# The citation names a file and a chunk, so a reader can open it. And the term
# check runs in microseconds on every answer, which is what makes it affordable
# to run on all of them rather than a sample.

The citation names a file and a chunk, so a reader opens it directly. The term check is crude - overlap, not entailment - and it runs in microseconds, which is what makes it affordable on every answer rather than on a sample.

It catches the common case where a figure or a noun came from somewhere other than the cited text.

The mistake this prevents

The mistake is measuring how many answers have citations. That is trivially satisfied by attaching an id to everything. What matters is whether the cited chunk contains the claim.

Takeaway

Cite the file and chunk, and check the answer's substantive terms against the cited text on every response. Citation presence is not grounding.

Mark complete