Unit 11.01: Specifying chunking, metadata, and permissions

Three specifications that have to agree with each other, and one detail that quietly determines whether your citations survive a rebuild.

The three that must be consistent

Chunking decides what a citation points at. Metadata decides what can be filtered, dated and restricted. Permissions decide who sees it. Specify one without the others and you get an index that cannot support the behaviour you designed.

Below is all three together, plus an explicit id format.

{
  "chunk_rule": "one policy statement per chunk, split on sentence boundaries",
  "max_chunk_words": 60,
  "must_stand_alone": true,
  "id_format": "{document}#{section}:{index}",
  "required_metadata": {
    "id": "citation target",
    "source_document": "where a reader checks it",
    "updated": "staleness detection",
    "visibility": "access control",
    "section": "pre-ranking filter"
  },
  "acl_model": "set intersection of chunk visibility and user groups, before ranking",
  "excluded_from_index": [
    "anything naming an individual customer"
  ]
}

The id format matters more than it looks: it must stay stable across
re-indexing, or every citation issued before the rebuild breaks.

id_format - {document}#{section}:{index} - is the detail worth dwelling on. It has to stay stable across re-indexing, because every citation issued before a rebuild resolves through it. An id derived from position in the corpus, or from an embedding hash, changes when anything upstream changes, and the audit trail from Module 9 breaks silently.

must_stand_alone and max_chunk_words together encode the Module 2 test. They are worth stating as specification rather than convention, because conventions do not survive a change of implementer.

The mistake this prevents

The mistake is specifying chunk size without specifying the semantic unit. "60 words maximum" tells an implementer nothing about where to cut; "one policy statement, split on sentence boundaries, maximum 60 words" tells them the rule and the bound.

Takeaway

Specify chunking, metadata and permissions together, and fix an id format that survives re-indexing. Those choices constrain each other, and a specification that treats them separately produces an index that cannot do what the design promised.

Mark complete