Unit 09.04: Retention rules the index has to honour

Retention policies are usually written for document stores and then quietly violated by every derived copy - including yours.

Three places a purge has to reach

A retention rule says content older than some period must be removed. A RAG pipeline holds that content in three forms: the original document, the chunks split from it, and the vectors embedded from those chunks.

The table below computes the age of three chunks against their individual retention rules.

c1  age   28d  retain, 337 days remaining
c2  age  936d  PURGE - retention exceeded
c3  age  570d  retain indefinitely

Retention applies to three places, and a purge that misses any of them has not
happened: the document store, the chunk index, and the vector store. Embeddings
derived from deleted text are still derived from it.

c2 is past its retention window and flagged for purge. c3 has no limit and is retained. The per-chunk rule matters because retention is rarely uniform - a policy document and a support transcript have different obligations even when they arrive in the same ingestion run.

The purge must then reach all three stores. Embeddings are the one that gets forgotten, because they do not look like content. They are derived from it, they support nearest-neighbour search over it, and leaving them in place means the deleted text remains findable in every practical sense.

The mistake this prevents

The mistake is implementing retention as a scheduled job over the document store alone. It runs, it reports success, and the index keeps serving chunks whose sources were deleted months ago - which is worse than not purging, because now the record and the index disagree.

Takeaway

Attach retention to the chunk, run the purge across document store, chunk index and vector store together, and verify by querying for something that should no longer be retrievable.

Mark complete