Unit 02.04: Checking a chunk can stand on its own

Before an index is built, every chunk in it should pass a check you can run in a loop. Doing this at ingestion is cheap; discovering it from user complaints is not.

Three mechanical tests, no model required

You cannot read 40,000 chunks. You can run three string checks over all of them and read only the failures: does it open with an unresolved pronoun, does it point at text that will not be retrieved with it, and is it too short to name its own subject.

The example below runs those checks against four candidate chunks. One passes.

OK   'Refunds are allowed within 7 days of purchase.'
FAIL 'This applies to individual plans only.'
       - opens with an unresolved pronoun
FAIL 'Within 7 days.'
       - too short to name its own subject
FAIL 'See section 4 above for details.'
       - refers to surrounding text

The dangling pronoun is the subtle one. "This applies to individual plans only." is grammatical, retrievable, and meaningless alone - it will be cited as evidence for whatever the model assumes "this" refers to. "See section 4 above" is worse: it points somewhere the retriever will never follow.

Run the checks over a real corpus, look at the aggregate failure rate, and treat a high rate as a verdict on your splitting strategy rather than as thousands of individual problems to fix.

The mistake this prevents

The mistake is running these checks once, before the first ingestion, and never again. Document sets change - a new export format, a new source system, someone editing a template - and the failure rate moves with them. Make it part of the ingestion job and alert on the rate.

Takeaway

Chunk quality is measurable without a model and without human review. Three string checks at ingestion catch the passages that will produce unfollowable citations, and the failure rate tells you whether your splitting strategy is working.

Mark complete