Module 04 Activity

Scenario

A corpus has to become chunks. Every decision here determines what your citations will look like and what you can filter on later.

What you build

A structure-aware ingestion producing chunks with metadata, a justified chunk size, and a stand-alone check running on every chunk.

Steps

1. Load at least three real documents, turning their structure - headings, sections, dates - into metadata rather than text.
2. State your semantic unit in one sentence, then measure boundary completeness at three chunk sizes.
3. Measure what overlap costs you in index size, and decide whether to use it based on a measurement.
4. Embed once and cache by content. Report what a re-embed of the whole corpus would cost at real provider rates.
5. Run the three stand-alone checks over every chunk and report the failure rate.
6. Re-split the worst offenders and report the rate before and after.

Evidence to hand in

• The metadata schema with where each field came from.
• The semantic unit and boundary completeness at three sizes.
• The overlap measurement and your decision.
• The re-embed cost estimate.
• Stand-alone failure rate before and after re-splitting.

Review checklist

• Structure became metadata, not text.
• Chunk size follows from the stated semantic unit.
• The overlap decision cites a measurement.
• The stand-alone check is reported as a rate.
• The check runs as part of ingestion, not as a one-off.

Mark complete