Unit 05.04: Rebuilding an index without breaking citations

Every citation your system has ever issued resolves through a chunk id. What that id is derived from decides whether a rebuild breaks them.

Three schemes, two of which break

Document-and-position, insertion order, and content hash. Each survives a different set of changes.

The code shows all three for the same chunk.

import hashlib

CHUNKS = [
    {"text": "Refunds are allowed within 7 days.",
     "source": "support-policies-v4.md", "section": "refunds", "position": 2},
]


def stable_id(chunk):
    """Derived from the document and position -- survives a rebuild."""
    return f"{chunk['source']}#{chunk['section']}:{chunk['position']}"


def unstable_id(chunk, index):
    """Derived from insertion order -- changes on every rebuild."""
    return f"doc-{index}"


def content_id(chunk):
    """Derived from the text -- changes when the text is corrected."""
    return hashlib.sha256(chunk["text"].encode()).hexdigest()[:12]


chunk = CHUNKS[0]
print(f"{'scheme':12} {'id':44} survives...")
print(f"{'stable':12} {stable_id(chunk):44} rebuild: yes, edit: yes")
print(f"{'positional':12} {unstable_id(chunk, 0):44} rebuild: NO")
print(f"{'content hash':12} {content_id(chunk):44} rebuild: yes, edit: NO")

# Every citation issued before a rebuild resolves through the id. Positional ids
# break when anything upstream is reordered; content hashes break when a typo is
# fixed. The document-and-position scheme survives both.

Positional ids break on any rebuild that reorders anything - adding one document at the top renumbers everything below it. Content hashes break when the text is corrected, so fixing a typo silently invalidates every citation to that chunk.

The document-and-section-and-position scheme survives both, because it is derived from where the chunk sits in a named document rather than from the index or the bytes.

The mistake this prevents

The mistake is letting the vector store generate ids. The default is usually a UUID assigned at insertion, which is stable within one index and completely different after a rebuild - and rebuilds happen whenever the embedding model or chunking changes.

Takeaway

Derive chunk ids from the document and position, not from insertion order or content. Every archived citation resolves through them, and rebuilds are routine.

Mark complete