Unit 08.01: Values or ranks

Pearson asks whether the relationship is a straight line. Spearman asks only whether it consistently goes one way.

Values or ranks

Pearson correlates the values, so it measures straight-line association and is sensitive to extreme observations. Spearman correlates the ranks, so it measures any consistently increasing or decreasing relationship and is much less affected by a single unusual point.

When a relationship is monotonic but curved, Spearman is higher — and the gap between the two is itself informative, since it says the relationship is real but not linear.

Either way, report the confidence interval. Recent scipy versions give it directly from pearsonr(...).confidence_interval().

This block correlates a curved monotonic relationship both ways, then adds a contradicting point.

import numpy as np
from scipy import stats

rng = np.random.default_rng(502)
x = rng.uniform(1, 10, 60)
y = np.exp(x / 3) + rng.normal(0, 1, 60)      # monotonic but strongly curved

pear = stats.pearsonr(x, y)
spear = stats.spearmanr(x, y)
print(f"Pearson  (straight-line)  : {pear.statistic:.3f}")
print(f"Spearman (rank, monotonic): {spear.statistic:.3f}\n")
print("Spearman is higher because the relationship always increases even")
print("though it is not a straight line. Pearson only sees the straight part.\n")

xo, yo = np.append(x, 10.5), np.append(y, 2.0)
print("After adding one contradicting point:")
print(f"  Pearson : {pear.statistic:.3f} -> {stats.pearsonr(xo, yo)[0]:.3f}")
print(f"  Spearman: {spear.statistic:.3f} -> {stats.spearmanr(xo, yo)[0]:.3f}\n")

ci = pear.confidence_interval()
print(f"Pearson with its interval: r = {pear.statistic:.3f}"
      f"   95% CI [{ci.low:.3f}, {ci.high:.3f}]   p = {pear.pvalue:.3g}")
print("Report the interval. Even at r = 0.94 with n = 60, the third decimal")
print("is not determined.")

Pearson gives 0.943 and Spearman 0.975 — Spearman higher, because the relationship always increases even though it is not a straight line. Adding one contradicting observation moves Pearson to 0.880 and Spearman to 0.899. The interval on Pearson runs from 0.906 to 0.966: even at r = 0.943 with n = 60, the third decimal is not determined.

The mistake this prevents

The mistake is quoting a correlation to three decimals from a sample of thirty. The interval is usually wide enough to make the third decimal meaningless.

Takeaway

Use Pearson for linear relationships and Spearman when the relationship is monotonic but curved or the data has extreme values. Report the interval, and treat a large Pearson–Spearman gap as evidence of curvature.

Mark complete