Unit 10.04: When embeddings fail or encode unwanted patterns

If an attribute can be recovered from your vectors, it is influencing every ranking and every retrieval built on them.

The probe: ask whether it leaked

Train a simple classifier to predict a sensitive attribute *from the embeddings alone*. If it succeeds well above the base rate, that attribute is encoded — whether or not you intended it.

Compare against the same probe on the content features, to separate what the encoder added from what was already there:

import torch
import torch.nn.functional as F
from sklearn.linear_model import LogisticRegression

torch.manual_seed(0)
# An embedding that accidentally encodes a protected attribute.
group = torch.randint(0, 2, (400,))
content = torch.randn(400, 6)
embeddings = torch.cat([content, group.float().unsqueeze(1) * 3.0], dim=1)

# The probe: can a simple classifier recover the attribute from the vectors?
probe = LogisticRegression(max_iter=1000).fit(embeddings[:300].numpy(), group[:300].numpy())
leak = probe.score(embeddings[300:].numpy(), group[300:].numpy())
print(f"group recoverable from embeddings: {leak:.3f}")
print("baseline (guessing the majority) :",
      round(max(group.float().mean().item(), 1 - group.float().mean().item()), 3))

# Same probe on the content-only vectors, for comparison.
clean = LogisticRegression(max_iter=1000).fit(content[:300].numpy(), group[:300].numpy())
print("from content alone               :", round(clean.score(content[300:].numpy(), group[300:].numpy()), 3))

# Nearest neighbours cluster by the attribute rather than by meaning.
unit = F.normalize(embeddings, dim=1)
sims = unit @ unit.T
sims.fill_diagonal_(-2.0)
same_group = (group[sims.argmax(1)] == group).float().mean().item()
print(f"\nnearest neighbour shares the group: {same_group:.3f}")
print("Run this probe before shipping any embedding used for retrieval or")
print("ranking. If an attribute is recoverable, it is influencing results.")

The attribute is recoverable at high accuracy from the embeddings, and much less so from content alone. That gap is the encoder's contribution.

The neighbour check shows the practical consequence: nearest neighbours cluster by the attribute rather than by meaning, so any retrieval system built on these vectors will too.

The mistake this prevents

Assuming that not including an attribute as a feature means the model cannot use it. Correlated features reconstruct it, and the probe is the only way to find out.

Takeaway

Probe for every sensitive attribute before shipping an embedding used for retrieval or ranking.

Mark complete