Unit 07.04: Overplotting, and three ways to fix it

At high density a scatter becomes a solid blob, and its darkest region is invisible.

Three fixes, three costs

Five thousand points, with the proportion that land on top of another counted.

The code measures the overplotting and lists the remedies.

import numpy as np

rng = np.random.default_rng(9)
n = 5000
x = rng.normal(50, 12, n)
y = x * 0.8 + rng.normal(0, 10, n)

# how many points land on the same rounded coordinate
cells = {}
for xi, yi in zip(np.round(x), np.round(y)):
    cells[(xi, yi)] = cells.get((xi, yi), 0) + 1
overplotted = sum(v - 1 for v in cells.values() if v > 1)

print(f"{n:,} points, {len(cells):,} distinct rounded positions")
print(f"{overplotted:,} points ({overplotted / n:.0%}) are drawn on top of another")

print("""
FIX                what it shows                       what it costs
transparency       density through shading              exact values at the edges
hexagonal binning  density as a grid of counts          individual points
sampling           the shape, from a readable subset    rare points may vanish

At this density a plain scatter is a solid blob whose darkest region is
invisible. Pick a fix and say which -- sampling in particular changes what the
reader is looking at.
""")

A large share of the points are drawn over another point, so the densest region - usually the most important part of the distribution - is exactly the part that cannot be read.

Each fix costs something. Transparency loses exact values at the edges; hexagonal binning loses the individual points; sampling may drop the rare points entirely, which matters if outliers are the finding.

The mistake this prevents

The mistake is sampling silently. It is often the right fix and it changes what the reader is looking at - a scatter of a thousand points drawn from fifty thousand should say so on the chart.

Takeaway

Measure the overplotting before choosing a fix, and say which you used. Sampling in particular changes what the reader is seeing.

Mark complete