Unit 04.00: Loading a document without losing its structure

Structure discarded at load time cannot be recovered later, and the structure is what the next two modules need.

Headings become metadata, not more text

A structure-aware loader reads the document's own organisation and turns it into fields on each chunk.

The code loads a two-section policy document, attaching the section to every chunk.

from langchain_core.documents import Document

RAW = """# Refunds
Refunds are allowed within 7 days of purchase.
This applies to individual plans only.

# Exchanges
Exchanges are allowed within 30 days."""

# Structure-aware: the heading becomes metadata, not just more text.
docs, section = [], None
for line in RAW.splitlines():
    if line.startswith("# "):
        section = line[2:].strip().lower()
    elif line.strip():
        docs.append(Document(page_content=line.strip(),
                             metadata={"section": section,
                                       "source": "support-policies-v4.md"}))

for d in docs:
    print(f"[{d.metadata['section']:9}] {d.page_content}")

print(f"\n{len(docs)} documents, each carrying the section it came from")

# Loading as one blob loses the section, and the section is what Module 5's
# metadata filter needs. Structure discarded at load time cannot be recovered.

Each document carries the section it came from. That field is what Module 5's metadata filter needs, and it exists only because loading noticed the headings.

Load the same file as one blob and the section is gone. It is still *present*, as the word "Refunds" somewhere above the text, but it is no longer a field - so it cannot be filtered on, and similarity has to recover it from wording.

The mistake this prevents

Real documents resist this more than the example suggests. PDF extraction is the usual culprit: multi-column layouts get read across the columns rather than down them, tables arrive as a stream of cells with no row structure, and scanned pages contain no text at all until something OCRs them. None of these raises an error - you get text, it is simply wrong or empty. Read the extracted output of a few real documents before trusting a loader on a corpus.

The mistake is loading everything with the most general loader available and planning to enrich later. Enrichment after the fact means re-parsing the originals, which you may not have kept, and re-embedding everything, which you will pay for.

Takeaway

Extract the document's structure into metadata at load time. Headings, sections and dates cannot be recovered from a blob, and every later filter depends on them.

Mark complete