All courses · Course overview · Lesson sequence

Build a source collection you can inspect

The retrieval team wants to index a longer handbook. A quick script slices every six words into a passage. One result says, “A renewal lasts seven days. A”. Another says, “permitted only when no hold exists.” Both are pieces of the source. Neither is a comfortable unit for explaining the complete renewal policy.

The problem begins before ranking. We need to decide what a retrievable unit means and how someone can trace it back to its source. A chunk is a bounded piece of a source document. Choosing chunk boundaries is a design decision with consequences for evidence completeness, duplication and context budgets.

Preserve the parent before making pieces

Start from a source record whose text and identity you can inspect. Our synthetic record renewal-long-v1 contains two paragraphs:

A renewal lasts seven days. A renewal is permitted only when no hold exists.

Ask before the due date.

The first paragraph connects a duration to its permission condition. The second gives a timing instruction. Our parent record also carries a source identifier and version. A chunk inherits them and adds a parent ID plus character start/end offsets. Its text must equal parent.text[start:end] exactly. If that invariant fails, a displayed quotation could no longer be reconstructed reliably.

A chunk ID such as renewal-long-v1:chars-0-76 identifies a span in a particular parent version. An offset is meaningful only relative to that exact text. Changing the source can move every later offset. Keep the old version identity with its chunks, rebuild the new version deliberately and prevent a mixture of stale chunks and new metadata.

The W3C provenance overview provides a broader vocabulary for entities and their derivation. This lab uses a small explicit parent/span relationship; it does not implement the PROV standard or a complete data lineage system.

Compare two boundary policies

Run:

python3 -B methods.py chunking

The word_windows section uses windows of six whitespace-separated words with two words of overlap. After each window, the start advances by 6−2=4 words. The first text is A renewal lasts seven days. A; the next is days. A renewal is permitted only. A later window contains the hold condition. Each span is an exact slice, but the policy meaning is fragmented.

Overlap means adjacent windows repeat some words. It can help a phrase near one boundary appear intact in another window. It also creates repeated evidence that consumes ranking slots and context space. It cannot guarantee that a long dependency fits into any window. If an exception is thirty words away from the rule and the window holds six words, repeating two words will not connect them.

The paragraph_chunks section packs complete paragraphs within an 80-character target. It returns the first paragraph as characters [0,76) and the second as [78,102). The notation is half-open: include character 0, stop before character 76. The two blank-line characters between paragraphs remain in the parent but need not be independently retrieved. The paragraph about the hold keeps its condition with the duration.

This improvement is specific to the source's sensible paragraphing. A badly formatted handbook could put a title in one paragraph, a rule in another and an exception far away. Paragraph boundaries are clues, not semantic guarantees. Table rows can likewise lose column headings if extracted as unrelated strings. Before indexing a table, retain labels such as “member type” or “maximum duration” with the values they explain. We do not claim this text-only chunker parses PDFs, tables or images.

A condition can fall across a chunk boundary — The source connects a seven-day renewal to the condition that no hold exists. A six-word window cuts the rule into fragments. A paragraph chunk retains the full condition and carries its parent, source version and exact character span.

An oversized paragraph needs a visible decision

Now imagine reducing the paragraph target to 20 characters. The first paragraph is longer than that. Our implementation keeps it whole and marks over_budget: true; it does not silently break the condition to meet the target. The flag means “review the boundary or budget,” not “this chunk will fit a model.”

You can reproduce the policy in a short Python session from project/:

from retrieval import Document
from methods import paragraph_chunks

source = Document("demo-v1", "Renewal", "Renew only if no hold exists.",
                  "synthetic://demo/renewal", "1", "current", "public")
for chunk in paragraph_chunks(source, max_chars=20):
    print(chunk.text, chunk.over_budget)

The printed text remains Renew only if no hold exists. and the flag is True. A later context selector may skip it if its quote budget is too small. Separating boundary creation from context selection lets us diagnose that loss rather than conceal it through arbitrary truncation.

Decide what enters the collection

The current lab indexes only records labeled current and public. Those fields are authored synthetic metadata, not an access-control system. The course does not ingest private records or teach a shortcut around production permissions. In a real project, eligibility would need to come from an approved authoritative source and remain enforced when retrieving and displaying material. A copy of a label in an untrusted file is not enough.

For your own practice, use original synthetic text or material whose use and distribution are already permitted. Record its origin, the date/version you captured and transformations you performed. A checksum can detect a changed byte sequence against a known baseline. It cannot establish that the source is truthful, current, licensed for a new use or approved by its owner. Do not turn an integrity check into a claim about those separate questions.

Our loader rejects duplicate document IDs and unknown status/audience values. This catches accidental ambiguity early. It cannot detect every factual conflict. Two different current records might disagree about opening hours, and the file could still be well formed. Module 6 will make that conflict a context-selection decision.

Inspect one failure before adding scale

Take the first six-word window and try to answer, “When is renewal allowed?” You cannot recover the condition from that window alone. Next inspect the complete paragraph: now the permission condition is present. That comparison is evidence about this source and boundary choice. It is not a universal recommendation for an 80-character chunk size.

Before trying a larger collection, preserve these three checks: each chunk maps back to an exact parent span; version identity survives every transformation; and at least one condition-bearing question is reviewed against the proposed boundaries. Then complete the source/chunk practice and separate solutions. In Module 4, we will rank the pieces with a baseline whose arithmetic we can explain.

Read separate answers and explanations