Unit 03.00: Robust summaries for skewed measures

On skewed data the mean and the standard deviation describe a distribution that does not exist.

Robust summaries for skewed measures

Money, durations and counts are usually skewed: a long right tail of large values and a bulk of small ones. The mean sits somewhere in the sparse middle, and the standard deviation — which squares distances from the mean — is dominated by the largest observation.

The median and the interquartile range answer the same two questions using ranks rather than magnitudes, so a single extreme value moves them very little. On skewed data they describe a typical case; the mean and SD describe an average that few observations resemble.

This is not a rule to always prefer the median. It is a rule to look at the distribution first and say which summary you chose and why.

This block summarises skewed revenue with one large value present, then removes it.

set.seed(19)
revenue <- c(round(rlnorm(60, meanlog = 3.2, sdlog = 0.6), 2), 890.00)

cat("n            :", length(revenue), "\n")
cat("Mean         :", round(mean(revenue), 2), "\n")
cat("Median       :", round(median(revenue), 2), "\n")
cat("SD           :", round(sd(revenue), 2), "\n")
cat("IQR          :", round(IQR(revenue), 2), "\n")
q <- quantile(revenue, c(0.25, 0.5, 0.75, 0.95))
cat("Quartiles    :", paste(round(q[1:3], 2), collapse = "  "), "\n")
cat("95th pct     :", round(q[4], 2), "\n\n")

trimmed <- revenue[revenue < 500]
cat("Excluding the one 890 value:\n")
cat("  Mean  :", round(mean(revenue), 2), "->", round(mean(trimmed), 2), "\n")
cat("  Median:", round(median(revenue), 2), "->", round(median(trimmed), 2), "\n")
cat("  SD    :", round(sd(revenue), 2), "->", round(sd(trimmed), 2), "\n")
cat("  IQR   :", round(IQR(revenue), 2), "->", round(IQR(trimmed), 2), "\n\n")

cat("Mean and SD move; median and IQR barely do. On skewed money data,\n")
cat("report the median and IQR and say the distribution is skewed.\n")

With the large value included, the mean is 45.94 and the median 24.55 — nearly double. The standard deviation is 112.18 against an IQR of 22.52. Removing one observation out of 61 moves the mean to 31.87 and the SD to 22.82, a fivefold change, while the median moves from 24.55 to 23.82 and the IQR from 22.52 to 22.12. One row in sixty-one controlled the standard deviation entirely.

The mistake this prevents

The mistake is reporting mean and SD for revenue by habit. It gives a typical value that almost no customer matches and an SD that describes the largest customer rather than the spread.

Takeaway

Look at the distribution before choosing a summary. For skewed measures report the median and IQR, say the data is skewed, and give the mean too if the total matters — but never the mean alone.

Mark complete