Unit 02.02: The metadata you cannot add later

Some metadata can be backfilled automatically later. Some cannot, because the information only existed at ingestion and is gone once the text is in the index. Knowing which is which saves a re-ingestion.

Five fields, five questions a reader will ask

A chunk is not just text. It is text plus everything the system needs to rank it, restrict it, date it and cite it. Below is a fully formed chunk and then maps each field to the question it answers.

Read the mapping rather than the JSON. Every field is there because something downstream is impossible without it.

{
  "id": "policy-1#0",
  "text": "Refunds are allowed within 7 days of purchase.",
  "source_document": "support-policies-v4.md",
  "updated": "2026-06-14",
  "visibility": "public",
  "section": "refunds"
}
id               -> which chunk is this?
source_document  -> where can I check it?
updated          -> is it still true?
visibility       -> am I allowed to see it?
section          -> can we filter before ranking?

visibility is Module 9's access control. updated is Module 7's staleness detection. section is the pre-ranking filter from the next module. source_document is what makes a citation followable, and id is what makes it precise. Drop any one and you lose the corresponding capability, not just a column.

The ones that cannot be recovered later are visibility and any notion of version. They come from the system the document lived in, and that context does not survive the copy into your pipeline. Source and section can be rebuilt from the originals; who was allowed to see it usually cannot.

The mistake this prevents

The mistake is treating metadata as decoration on the text and planning to "add fields later." Later means re-embedding the whole corpus, and for visibility it may mean going back to a source system that has since changed. Capture more than you think you need at ingestion; pruning is cheap.

Takeaway

Decide the metadata schema before the first ingestion run. Every field corresponds to a capability elsewhere in the pipeline, and the two most commonly regretted omissions - visibility and version - are the two that cannot be reconstructed afterwards.

Mark complete