Unit 10.02: Autoencoder intuition and reconstruction limits

An autoencoder compresses input through a bottleneck and rebuilds it. The bottleneck width is the experiment.

Finding the intrinsic dimensionality

The data here is constructed to live on 2 genuine dimensions, embedded in 8 columns — the extra columns are scaled copies carrying no new information.

Training at several bottleneck widths shows where reconstruction becomes possible:

import torch
from torch import nn

torch.manual_seed(0)
# Data that genuinely lives on 2 dimensions, embedded in 8.
latent = torch.randn(500, 2)
X = torch.cat([latent, latent * 0.5, latent * -1.0, latent * 0.2], dim=1)


def train(bottleneck, steps=400):
    torch.manual_seed(0)
    encoder = nn.Sequential(nn.Linear(8, 16), nn.ReLU(), nn.Linear(16, bottleneck))
    decoder = nn.Sequential(nn.Linear(bottleneck, 16), nn.ReLU(), nn.Linear(16, 8))
    opt = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()), lr=0.01)
    for _ in range(steps):
        opt.zero_grad()
        nn.MSELoss()(decoder(encoder(X)), X).backward()
        opt.step()
    with torch.no_grad():
        return nn.MSELoss()(decoder(encoder(X)), X).item()


for bottleneck in (1, 2, 4):
    print(f"bottleneck {bottleneck}: reconstruction MSE {train(bottleneck):.5f}")

# Error collapses at 2, because the data really is 2-dimensional. Squeezing
# below the true dimensionality loses information that cannot be recovered.
print("\nLow reconstruction error means the code retained enough to rebuild the")
print("input. It does NOT mean the code is useful for your downstream task --")
print("test that separately rather than assuming it.")

Error is high at width 1, collapses at width 2, and barely improves at 4. That elbow identifies the true dimensionality of the data.

Below it, information is destroyed and cannot be recovered by any decoder. Above it, the extra capacity has nothing left to encode.

The mistake this prevents

Treating low reconstruction error as proof that the codes are useful. A model can reconstruct perfectly while encoding nothing your downstream task cares about. Test the codes on that task directly.

Takeaway

The bottleneck reveals intrinsic dimensionality. Reconstruction quality and downstream usefulness are separate questions.

Mark complete