Unit 04.01: Choosing a chunk size from the content

Chunk size is usually the first number set and the last one justified. It should follow from the content's own units.

Boundary completeness, measured

For each size, how many chunks end at a sentence boundary. That is a measurement rather than an opinion.

The code splits the same text at three sizes.

from langchain_text_splitters import RecursiveCharacterTextSplitter

TEXT = ("Refunds are allowed within 7 days of purchase. This applies to "
        "individual plans only. Enterprise contracts are negotiated per "
        "account. Exchanges are allowed within 30 days of purchase.")

for size in (40, 100, 400):
    splitter = RecursiveCharacterTextSplitter(chunk_size=size, chunk_overlap=0)
    chunks = splitter.split_text(TEXT)
    complete = sum(c.rstrip().endswith(".") for c in chunks)
    print(f"chunk_size={size:<4} {len(chunks)} chunks, "
          f"{complete}/{len(chunks)} end at a sentence boundary")
    if size == 40:
        for c in chunks[:3]:
            print(f"    {c!r}")

# At 40 characters the splitter cuts mid-sentence: the chunk asserts something
# it does not finish. Size follows from the semantic unit -- here, one policy
# statement -- rather than from a number you picked first.

At 40 characters the splitter cuts mid-sentence, and the printed chunks show what that means: a passage that asserts something it does not finish. Retrieved on its own it is true, unreadable and uncitable.

The right size follows from the semantic unit - here, one policy statement - rather than from a number chosen first and validated afterwards. Pick the unit, measure what size holds it, and check the boundary completeness.

The mistake this prevents

The mistake is choosing chunk size from the model's context window. The window constrains how many chunks you can pass, not what belongs in one. If your semantic unit does not fit alongside k others, reduce k rather than splitting meaning.

Takeaway

Choose the semantic unit first, then the size that holds it, and measure how many chunks end at a real boundary. Context window constrains k, not chunk content.

Mark complete