Unit 06.05: Compare ranks when means are the wrong summary

Rank-based tests trade a little power for robustness, and give you no effect size at all.

Compare ranks when means are the wrong summary

Mann-Whitney U (mannwhitneyu) replaces the values with their ranks, so a single extreme observation counts once rather than dominating. Kruskal-Wallis (kruskal) is the same idea for three or more groups. Fisher's exact test (fisher_exact) replaces chi-square when expected counts are too small for the approximation.

These tests compare whole distributions, not means, so 'the medians differ' is a loose reading. More importantly they return no effect size, so a median difference has to be reported separately.

They are not automatically the safe choice. On well-behaved data they are slightly less powerful, and on strongly skewed data a bootstrap of the mean is often more informative than either.

This block compares skewed groups both ways, then shows two small-count situations.

import numpy as np
from scipy import stats

rng = np.random.default_rng(305)
a = np.append(rng.lognormal(2.5, 0.5, 18), 400.0)   # one extreme value
b = rng.lognormal(2.9, 0.5, 19)

print(f"Group A: median {np.median(a):.2f}   mean {a.mean():.2f}")
print(f"Group B: median {np.median(b):.2f}   mean {b.mean():.2f}\n")
print(f"t-test              p = {stats.ttest_ind(a, b, equal_var=False).pvalue:.3f}"
      "   (the 400 dominates A's mean)")
print(f"Mann-Whitney U      p = {stats.mannwhitneyu(a, b).pvalue:.3f}"
      "   (ranks, so it counts once)\n")

c = rng.lognormal(3.3, 0.5, 19)
print(f"Three groups, Kruskal-Wallis p = {stats.kruskal(a, b, c).pvalue:.3g}")
print(f"                    f_oneway p = {stats.f_oneway(a, b, c).pvalue:.3g}\n")

small = np.array([[9, 4], [1, 6]])
print("Small counts:", small.tolist())
chi2_p = stats.chi2_contingency(small)[1]
fisher_p = stats.fisher_exact(small)[1]
print(f"  chi2   p = {chi2_p:.4f}   (expected counts are too small for it)")
print(f"  Fisher p = {fisher_p:.4f}   (exact, no approximation)\n")
print("Rank tests compare distributions, not means, and give no effect size.")
print("Report a median difference alongside.")

Group A has a median of 12.74 and a mean of 33.70 — one value of 400 controls the mean entirely, and A's median is *lower* than B's 19.32 while its mean is higher. The t-test gives p = 0.546, Mann-Whitney 0.075. Across three groups, Kruskal-Wallis finds p = 0.000193 where f_oneway on the same data reports 0.74. On the small table, chi-square gives 0.0608 while Fisher's exact test gives 0.0573 with no approximation.

The mistake this prevents

The mistake is treating a rank test as a test of medians and reporting 'the medians differ significantly'. It tests distributions, and it hands you no estimate of how far apart they are.

Takeaway

Reach for rank tests when the outcome is ordinal or heavily skewed, and for fisher_exact when expected counts are small. Always report a median difference alongside, because the test provides no effect size.

Mark complete