Unit 09.03: Near-duplicates you did not know you had

Near-duplicates are how a dataset leaks its test set into its training set without anyone noticing.

Cosine above 0.95, before splitting

All pairs compared, with a threshold that catches near-identical images.

The code finds the duplicate pair in a small set.

import numpy as np

rng = np.random.default_rng(2)
base = rng.normal(size=(6, 16))
base /= np.linalg.norm(base, axis=1, keepdims=True)
vectors = np.vstack([base, base[1] + rng.normal(0, 0.02, 16)])
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
ids = ["img-001", "img-002", "img-003", "img-004", "img-005", "img-006",
       "img-007"]

sim = vectors @ vectors.T
np.fill_diagonal(sim, -1)
pairs = [(ids[i], ids[j], sim[i, j])
         for i in range(len(ids)) for j in range(i + 1, len(ids))
         if sim[i, j] > 0.95]

print("near-duplicate pairs (cosine > 0.95):")
for a, b, s in pairs:
    print(f"  {a} <-> {b}  {s:.3f}")

print(f"\n{len(pairs)} pair(s) found in {len(ids)} images")
print("if these land on both sides of a train/test split, the score is inflated")

# Run this before splitting, not after training. Near-duplicates are how a
# dataset assembled from burst photography or scraped pages leaks the test set
# into the training set without anyone noticing.

Run this before splitting, not after training. A near-duplicate pair straddling a train/test split inflates the test score, and the inflation survives cross-validation and every other precaution.

Burst photography, video frames and scraped pages all produce them in quantity, and none of them is visible by looking at filenames.

The mistake this prevents

The mistake is checking for exact duplicates by file hash. Two frames a tenth of a second apart have different bytes, different hashes, and are the same image for every purpose that matters.

Takeaway

Detect near-duplicates by embedding similarity before splitting. File hashes miss them entirely, and a straddling pair inflates every score you report.

Mark complete