Unit 08.01: Chunking a document that will not fit

A document that does not fit has to be split, and splitting on characters cuts sentences in half.

Sentence boundaries, measured

The same passage chunked at three sizes, with boundary completeness reported.

The code prints the chunks at the smallest size.

TEXT = ("Refunds are allowed within 7 days of purchase. "
        "This applies to individual plans only. "
        "Enterprise contracts are negotiated per account. "
        "Exchanges are allowed within 30 days.")


def chunk_by_sentence(text, max_chars):
    chunks, current = [], ""
    for sentence in text.split(". "):
        sentence = sentence.strip().rstrip(".") + "."
        if current and len(current) + len(sentence) + 1 > max_chars:
            chunks.append(current.strip())
            current = sentence
        else:
            current = f"{current} {sentence}".strip()
    if current:
        chunks.append(current)
    return chunks


for size in (60, 120, 400):
    chunks = chunk_by_sentence(TEXT, size)
    complete = sum(c.endswith(".") for c in chunks)
    print(f"max {size:>3}: {len(chunks)} chunks, {complete}/{len(chunks)} end at a sentence")

print()
for c in chunk_by_sentence(TEXT, 60):
    print(f"  {c!r}")

# Splitting on sentences rather than characters keeps each chunk a complete
# statement. The second chunk here is "This applies to individual plans only."
# -- true, and meaningless without the one before it.

Every chunk ends at a sentence, so each one is a complete statement. The second chunk is "This applies to individual plans only." - true, grammatical, and meaningless without its predecessor.

That is the failure to design against: a chunk that is retrieved alone and read alone has to make sense alone, and sentence boundaries are necessary but not sufficient for that.

The mistake this prevents

The mistake is choosing the chunk size from the model's context window. The window constrains how many chunks you can send, not what belongs in one. If the semantic unit does not fit alongside the others, send fewer.

Takeaway

Split on sentence boundaries and measure how many chunks end at one. A chunk retrieved alone must make sense alone, which is a stronger requirement than fitting.

Mark complete