Unit 04.03: Embedding once, reusing forever

With a real provider, embedding is a paid call per chunk. The shape of the job matters more than the model choice.

Deterministic means cacheable

The same text always produces the same vector, so a cache keyed on text is always valid.

The code embeds two hundred chunks and confirms the cache holds.

import time
from langchain_core.embeddings import DeterministicFakeEmbedding

embeddings = DeterministicFakeEmbedding(size=64)
CHUNKS = [f"policy statement number {i}" for i in range(200)]

start = time.perf_counter()
vectors = embeddings.embed_documents(CHUNKS)
elapsed = time.perf_counter() - start
print(f"embedded {len(vectors)} chunks in {elapsed * 1000:.1f} ms (fake model)")

# Deterministic: the same text always gives the same vector, so a cache is safe.
cache = {text: vec for text, vec in zip(CHUNKS, vectors)}
again = embeddings.embed_documents(CHUNKS[:3])
print("cache is valid:", all(cache[t] == v for t, v in zip(CHUNKS[:3], again)))

print("""
With a real provider this is a paid call per chunk, so the shape of the job
matters: embed the corpus once, store the vectors, and re-embed only what
changed. Re-embedding everything on each deploy is a common and entirely
avoidable bill.
""")

The fake model makes this instant. With a provider it is a network call per batch and a line on the bill, which changes what the right job shape is: embed the corpus once, store the vectors, and re-embed only what changed.

Re-embedding everything on each deploy is common, entirely avoidable, and usually discovered from the invoice. A content hash per chunk is enough to tell you what actually changed.

The mistake this prevents

One more thing the cache cannot save you from. Vectors from different embedding models are not comparable - they occupy different spaces, and a query embedded with a new model against a corpus embedded with the old one produces nonsense rather than an error. Changing the embedding model means re-embedding the entire corpus, which makes it a budgeted migration rather than a config edit, and a good reason to record which model produced your index.

The mistake is embedding at query time for documents that never change. The query has to be embedded on every request; the corpus does not, and conflating the two turns a one-off cost into a per-request one.

Takeaway

Embed the corpus once and cache by content. Only the query needs embedding per request - re-embedding unchanged documents is a recurring bill for no benefit.

Mark complete