Unit 06.06: Check by looking; judge power before believing a null

Normality tests are least informative exactly where you need them, and a small non-significant study tells you almost nothing.

Check by looking; judge power before believing a null

Independence cannot be checked from the data at all — it follows from how the data was collected. Shape can be checked, and the usual way of doing it is worse than looking at a histogram.

A normality test's power depends on n. At small samples it misses real skew; at large samples it flags departures too small to affect anything. So it rejects when you do not need it to and fails to reject when you would want the warning.

The second half is power. A study too small to detect the effect you care about produces a non-significant result almost regardless of the truth, and that result is not evidence of absence.

This block measures how often Shapiro-Wilk rejects, then how often a genuine difference is detected.

import numpy as np
from scipy import stats

rng = np.random.default_rng(306)

# One Shapiro-Wilk result is a coin toss. Measure the RATE over 500 runs.
def reject_rate(n, draw, reps=500):
    return np.mean([stats.shapiro(draw(n)).pvalue < 0.05 for _ in range(reps)])

skewed = lambda n: rng.lognormal(3, 0.9, n)
normal = lambda n: rng.normal(size=n)

print("How often Shapiro-Wilk rejects normality:")
for n in (8, 30, 200):
    print(f"  n = {n:3d}   skewed data {reject_rate(n, skewed):5.0%}"
          f"   normal data {reject_rate(n, normal):5.0%}")

print("\nAt n = 8 it misses genuinely skewed data most of the time. At n = 200")
print("it catches skew reliably -- by which point the mean's own sampling")
print("distribution is close to normal anyway and the t-test is fine.")
print("The test is least informative exactly where the decision is hardest.\n")

def power(n, effect, sd=15, reps=2000):
    return np.mean([
        stats.ttest_ind(rng.normal(effect, sd, n), rng.normal(0, sd, n),
                        equal_var=False).pvalue < 0.05
        for _ in range(reps)])

print("Chance of detecting a 5-point difference (sd 15):")
for n in (10, 30, 100, 300):
    print(f"  n = {n:3d} per group  -> {power(n, 5):.0%}")
print("\nA non-significant result from n = 10 tells you almost nothing.")

On genuinely skewed data Shapiro-Wilk rejects only 42% of the time at n = 8, but 98% at n = 30 and 100% at n = 200 — and by n = 200 the mean's own sampling distribution is close to normal anyway, so the warning arrives when it no longer matters. On the power side, a real 5-point difference with an SD of 15 is detected 12% of the time at n = 10, 24% at n = 30, 64% at 100 and 99% at 300.

The mistake this prevents

The mistake is treating a non-significant result from a small study as evidence that there is no effect. At n = 10 the study would have missed a real 5-point difference nearly nine times out of ten.

Takeaway

Check shape by plotting, not by a normality test. Before interpreting any non-significant result, work out what effect size the study could have detected, and report it alongside.

Mark complete