Unit 03.00: Robust summaries, and the ddof trap

numpy and pandas disagree about what 'standard deviation' means by default, and on skewed data the mean describes a case that does not exist.

Robust summaries, and the ddof trap

Durations, money and counts are usually skewed: a long right tail and a bulk of small values. The mean sits in the sparse middle and the standard deviation — which squares distances from it — is dominated by the largest observation.

The median and interquartile range use ranks rather than magnitudes, so a single extreme value barely moves them.

There is also a Python-specific trap. np.std() defaults to ddof=0, the population formula. pandas.Series.std() defaults to ddof=1, the sample formula. The same data gives two different numbers depending on which library you called.

This block summarises skewed delivery times with one long delivery present, then removes it.

import numpy as np, pandas as pd

rng = np.random.default_rng(19)
minutes = np.append(np.round(rng.lognormal(3.4, 0.5, 60), 1), 210.0)
s = pd.Series(minutes)

print(f"n      : {s.size}")
print(f"mean   : {s.mean():.2f}")
print(f"median : {s.median():.2f}")
print(f"std    : {s.std():.2f}    (pandas default ddof=1)")
print(f"np.std : {np.std(minutes):.2f}    (numpy default ddof=0)")
print(f"IQR    : {s.quantile(.75) - s.quantile(.25):.2f}")
print(f"quartiles: {s.quantile([.25, .5, .75]).round(2).tolist()}")
print()

trimmed = s[s < 150]
print("Excluding the single 210-minute delivery:")
for name, fn in [("mean", "mean"), ("median", "median"), ("std", "std")]:
    print(f"  {name:7s}{getattr(s, fn)():8.2f} -> {getattr(trimmed, fn)():.2f}")
print(f"  IQR    {s.quantile(.75) - s.quantile(.25):8.2f}"
      f" -> {trimmed.quantile(.75) - trimmed.quantile(.25):.2f}")
print()
print("Mean and std move; median and IQR barely do. Note also that numpy and")
print("pandas disagree on the standard deviation by default -- ddof=0 against")
print("ddof=1 -- so state which you used.")

With the 210-minute delivery included the mean is 34.73 and the median 29.50. The two standard deviations differ: 26.97 from pandas (ddof=1) against 26.74 from numpy (ddof=0). Removing one observation in 61 takes the mean to 31.81 and the standard deviation to 14.50 — almost halved — while the median moves from 29.50 to 29.15 and the IQR from 18.80 to 18.45.

The mistake this prevents

The mistake is reporting np.std() as a sample standard deviation. At n = 61 the difference is small; at n = 10 it is about 5%, and it is a silent, systematic understatement.

Takeaway

Look at the distribution before choosing a summary, and prefer median and IQR for skewed measures while saying they are skewed. Always pass ddof=1 to np.std for a sample, or state which convention you used.

Mark complete