Unit 04.04: Checking a chunk still makes sense alone

A chunk is retrieved on its own and cited on its own. It has to make sense on its own, and that is checkable without a model.

Three string checks, run at ingestion

Unresolved pronoun, cross-reference, and too short to name its subject. None needs a model, and together they catch most uncitable chunks.

The code runs them over four candidates.

CANDIDATES = [
    "Refunds are allowed within 7 days of purchase.",
    "This applies to individual plans only.",
    "Within 7 days.",
    "See the table above for the full list.",
]


def problems(chunk):
    found = []
    lowered = chunk.lower()
    if lowered.startswith(("this ", "it ", "they ", "these ", "that ")):
        found.append("opens with an unresolved pronoun")
    if any(w in lowered for w in ("above", "below", "the table", "section")):
        found.append("points at text that will not be retrieved with it")
    if len(chunk.split()) < 5:
        found.append("too short to name its own subject")
    return found


failed = 0
for chunk in CANDIDATES:
    issues = problems(chunk)
    failed += bool(issues)
    print(f"{'OK  ' if not issues else 'FAIL'} {chunk!r}")
    for issue in issues:
        print(f"       - {issue}")

print(f"\nfailure rate: {failed}/{len(CANDIDATES)} -- run this at ingestion, on"
      " every chunk, and alert on the rate rather than the individual failures")

Three of four fail. "This applies to individual plans only" is grammatical, retrievable and meaningless alone - it will be cited as evidence for whatever the model assumes "this" refers to. "See the table above" points somewhere the retriever will never follow.

Report the rate rather than the individual failures. A rate of forty percent is a verdict on your splitting strategy; forty individual failures look like forty things to fix by hand.

The mistake this prevents

The mistake is running these checks once before the first ingestion. Document sets change - a new export format, a new source, an edited template - and the rate moves with them. Make it part of the ingestion job and alert on the rate.

Takeaway

Three string checks at ingestion catch the chunks that produce unfollowable citations. Alert on the failure rate, which tells you about your splitting strategy rather than about individual chunks.

Mark complete