Unit 06.06: Check by looking; judge power before believing a null

Normality tests are least informative exactly where you need them most, and a small non-significant study tells you almost nothing.

Check by looking; judge power before believing a null

The independence assumption cannot be checked from the data at all — it follows from how the data was collected. Shape can be checked, and the usual way of doing it is worse than looking at a histogram.

A normality test's power depends on n. At small samples it misses real skew; at large samples it flags departures too small to affect anything. So it rejects when you do not need it to and fails to reject when you would want the warning.

The second half of this is power. A study too small to detect the effect you care about produces a non-significant result almost regardless of the truth, and that result is not evidence of absence.

This block measures how often Shapiro-Wilk rejects, then how often a genuine difference is detected.

set.seed(306)

# Assumption 1: independence -- not checkable from the data, only from design.
# Assumption 2: shape. Check it by looking, not by a normality test.
# One Shapiro-Wilk result is a coin toss. Run each situation 500 times.
detect_rate <- function(n, generator, reps = 500) {
  mean(replicate(reps, shapiro.test(generator(n))$p.value) < 0.05)
}
skewed <- function(n) rlnorm(n, 3, 0.9)
normal <- function(n) rnorm(n)

cat("How often Shapiro-Wilk rejects normality:\n")
for (n in c(8, 30, 200)) {
  cat(sprintf("  n = %3d  skewed data %3.0f%%   normal data %3.0f%%\n",
              n, detect_rate(n, skewed) * 100, detect_rate(n, normal) * 100))
}

cat("\nAt n = 8 the test misses genuinely skewed data most of the time.\n")
cat("At n = 200 it catches skew reliably -- by which point the mean's own\n")
cat("sampling distribution is close to normal anyway and the t-test is fine.\n")
cat("The test is least informative exactly where the decision is hardest.\n\n")

# Sample size: what a small study can and cannot detect.
detect <- function(n, effect, sd = 15, reps = 2000) {
  mean(replicate(reps, t.test(rnorm(n, effect, sd), rnorm(n, 0, sd))$p.value) < 0.05)
}
cat("Chance of detecting a 5-point difference (sd 15):\n")
for (n in c(10, 30, 100, 300)) {
  cat(sprintf("  n = %3d per group  -> %.0f%%\n", n, detect(n, 5) * 100))
}
cat("\nA non-significant result from n = 10 tells you almost nothing.\n")

On genuinely skewed data Shapiro-Wilk rejects only 38% of the time at n = 8, but 97% at n = 30 and 100% at n = 200 — and by n = 200 the mean's own sampling distribution is close to normal anyway, so the warning arrives when it no longer matters. On the power side, a real 5-point difference with an SD of 15 is detected 10% of the time at n = 10, 26% at n = 30, 66% at 100 and 98% at 300. A non-significant result from ten per group is close to uninformative.

The mistake this prevents

The mistake is treating a non-significant result from a small study as evidence that there is no effect. At n = 10 the study would have missed a real 5-point difference nine times out of ten.

Takeaway

Check shape by plotting, not by a normality test. Before interpreting any non-significant result, work out what effect size the study could have detected, and report that alongside.

Mark complete