Unit 10.00: Embeddings as reusable features

Once encoded, a vector can serve many tasks. That reuse is the operational argument for representation learning.

One encoding, several downstream jobs

The encoder runs once over the raw data. The resulting vectors are then used to fit two entirely different classifiers, with no re-encoding between them.

That pattern is what makes embeddings practical at scale — the expensive step happens once:

import time

import torch
from torch import nn
from sklearn.linear_model import LogisticRegression

torch.manual_seed(0)
encoder = nn.Sequential(nn.Linear(32, 16), nn.ReLU(), nn.Linear(16, 8))
encoder.eval()
raw = torch.randn(400, 32)
labels = (raw[:, 0] > 0).long()

# Embed once, reuse for every downstream task.
start = time.perf_counter()
with torch.no_grad():
    vectors = encoder(raw)
embed_time = time.perf_counter() - start

print("raw dimensions      :", raw.shape[1])
print("embedded dimensions :", vectors.shape[1])
print(f"embedding took      : {embed_time * 1000:.1f} ms (done once)")

# Two different tasks, same cached vectors, no re-encoding.
task_a = LogisticRegression(max_iter=1000).fit(vectors[:300].numpy(), labels[:300].numpy())
other = (raw[:, 1] > 0).long()
task_b = LogisticRegression(max_iter=1000).fit(vectors[:300].numpy(), other[:300].numpy())

print("\ntask A accuracy:", round(task_a.score(vectors[300:].numpy(), labels[300:].numpy()), 3))
print("task B accuracy:", round(task_b.score(vectors[300:].numpy(), other[300:].numpy()), 3))
print("\nCache embeddings with the model version that produced them. Vectors from")
print("two different checkpoints are not comparable, even for the same input.")

Both tasks train on the same cached vectors. In production this is the difference between an affordable system and an unaffordable one.

The caution at the end matters: vectors are only comparable if they came from the same model version. Cache the version alongside the vectors, or you will eventually mix two incompatible encodings.

The mistake this prevents

Regenerating embeddings after a model update without re-encoding the whole store. Old and new vectors coexist, similarity comparisons cross between them, and retrieval quality degrades in a way that is very hard to trace.

Takeaway

Embed once, cache with the model version, reuse across tasks.

Mark complete