Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 09.00: Deciding what must never be indexed

The strongest privacy control in a retrieval system is refusing to index something. Everything after ingestion is a weaker guarantee.

The decision belongs at ingestion

Once a chunk is in the index it can be surfaced by a query nobody anticipated, ranked by a similarity nobody inspected, and read by whoever clears the access filter. "We will filter it later" is not a control, it is a hope.

The example below applies an indexing policy to three documents and prints the decision with its reason.

d1  INDEX    policy content, no personal data
d2  EXCLUDE  contains an identifiable individual
d3  INDEX    policy content, no personal data

d2 names an identifiable individual and is excluded. d1 and d3 are policy content and go in - d3 is internal, which is a visibility question rather than an indexing one, and gets handled by the filter in the next unit.

That distinction is worth being precise about. Excluding a document means nobody can ever retrieve it. Restricting one means the right people can. Conflating the two either leaks data or makes the assistant useless to the staff who need it.

The mistake this prevents

The mistake is indexing everything available because it is easier than deciding. A corpus assembled by pointing at a shared drive will contain personal data, drafts, and material from systems with their own access rules - and the retrieval index flattens all of it into one searchable space.

Takeaway

Decide what must never be indexed before the first ingestion run, and record the reason per document. Exclusion and restriction are different controls for different problems.