Unit 08.00: What an embedding represents

An embedding turns a discrete thing — a word, a product, a user — into a vector. The vector means nothing until it has been trained.

A lookup table with learnable rows

nn.Embedding is a table with one row per item. Passing an index returns that row. The rows start random and are updated by gradient descent like any other parameter.

The geometry people describe — similar things near each other — is a *result* of training on a task, not a property the table has from the start:

import torch
from torch import nn

torch.manual_seed(0)
vocab = ["cat", "dog", "kitten", "car", "truck"]
embedding = nn.Embedding(len(vocab), 4)

ids = torch.arange(len(vocab))
vectors = embedding(ids)
print("embedding table:", tuple(embedding.weight.shape), "= vocab x dimensions")
print("one word ->", tuple(vectors[0].shape), "numbers")

# Untrained embeddings are random, so "similarity" means nothing yet.
similarity = torch.nn.functional.cosine_similarity(vectors[0], vectors[2], dim=0)
print("\ncat vs kitten (untrained):", round(similarity.item(), 3))

# Hand-place them to show what a TRAINED table would look like.
with torch.no_grad():
    embedding.weight[:] = torch.tensor([
        [1.0, 0.0, 0.0, 0.0],   # cat
        [0.9, 0.1, 0.0, 0.0],   # dog
        [0.95, 0.05, 0.0, 0.0], # kitten
        [0.0, 0.0, 1.0, 0.0],   # car
        [0.0, 0.0, 0.9, 0.1],   # truck
    ])
v = embedding(ids)
pairs = [("cat", "kitten", 0, 2), ("cat", "car", 0, 3), ("car", "truck", 3, 4)]
for a, b, i, j in pairs:
    sim = torch.nn.functional.cosine_similarity(v[i], v[j], dim=0).item()
    print(f"{a:6} vs {b:7}: {sim:.3f}")
# An embedding is only meaningful after training on a task. The geometry is
# learned, not given.

The untrained similarity between "cat" and "kitten" is arbitrary, because the rows are random numbers.

The second half places the vectors by hand to show what a trained table looks like: cat and kitten nearly identical, car and truck close to each other, and the two groups far apart. That structure has to be learned from data.

The mistake this prevents

Reading meaning into an untrained or randomly initialised embedding. Cosine similarity always returns a number, and that number is meaningless until the table has been trained on something.

Takeaway

An embedding is a learnable lookup table. Its geometry comes from the task it was trained on, and nowhere else.

Mark complete