Unit 06.05: Compare ranks when means are the wrong summary

Rank-based tests trade a little power for robustness, and give you no effect size at all.

Compare ranks when means are the wrong summary

The Wilcoxon rank-sum test (also called Mann-Whitney) replaces the values with their ranks, so a single extreme observation counts once rather than dominating. Kruskal-Wallis is the same idea for three or more groups. Fisher's exact test replaces chi-square when expected counts are too small for the approximation.

These tests compare whole distributions, not means, so 'the medians differ' is a loose reading rather than a precise one. More importantly they return no effect size, so a median difference has to be reported separately.

They are not automatically the safe choice. On well-behaved data they are slightly less powerful, and on strongly skewed data a bootstrap of the mean is often more informative than either.

This block compares skewed groups both ways, then shows two small-count situations.

set.seed(305)
# Heavily skewed data with an extreme value -- where the t-test struggles.
a <- c(rlnorm(18, 2.5, 0.5), 400)
b <- rlnorm(19, 2.9, 0.5)

cat("Group A: median", round(median(a), 2), " mean", round(mean(a), 2), "\n")
cat("Group B: median", round(median(b), 2), " mean", round(mean(b), 2), "\n\n")

cat("t-test           p =", signif(t.test(a, b)$p.value, 3),
    "  (one value of 400 dominates the mean of A)\n")
cat("Wilcoxon rank-sum p =",
    signif(suppressWarnings(wilcox.test(a, b))$p.value, 3),
    "  (ranks, so the extreme value counts once)\n\n")

# Fisher's exact test: small counts, where chi-square is unreliable.
small <- matrix(c(9, 1, 4, 6), nrow = 2,
                dimnames = list(group = c("A", "B"), outcome = c("yes", "no")))
print(small)
cat("\nchisq p =", signif(suppressWarnings(chisq.test(small))$p.value, 3),
    " (with a warning: expected counts too small)\n")
cat("Fisher p =", signif(fisher.test(small)$p.value, 3), " (exact, no warning)\n\n")
# Three or more groups: Kruskal-Wallis is the rank-based ANOVA.
cc <- rlnorm(19, 3.3, 0.5)
long <- data.frame(g = rep(c("A", "B", "C"), c(length(a), length(b), length(cc))),
                   v = c(a, b, cc))
cat("\nThree groups, Kruskal-Wallis p =",
    signif(kruskal.test(v ~ g, data = long)$p.value, 3),
    " (aov p =", signif(summary(aov(v ~ g, data = long))[[1]][1, 5], 3), ")\n\n")

cat("Rank tests compare distributions, not means. Report a median difference\n")
cat("alongside, because the test itself gives you no effect size.\n")

Group A has a median of 14.81 and a mean of 33.71 — one value of 400 controls the mean entirely, and A's median is actually *lower* than B's 15.6. The t-test gives p = 0.48, the Wilcoxon 0.181. On the small table, chi-square gives 0.0608 with a warning about expected counts while Fisher's exact test gives 0.0573 with none. Across three groups, Kruskal-Wallis finds p = 0.000277 where ANOVA on the same data reports 0.605.

The mistake this prevents

The mistake is treating a rank test as a test of medians and reporting 'the medians differ significantly'. It tests distributions, and it hands you no estimate of how far apart they are.

Takeaway

Reach for rank tests when the outcome is ordinal or heavily skewed, and for Fisher's exact test when expected counts are small. Always report a median difference alongside, because the test provides no effect size.

Mark complete