Unit 06.04: One F-test, then adjusted comparisons

ANOVA answers one question — are all these means equal — and then leaves you needing a second analysis.

One F-test, then adjusted pairwise comparisons

With three or more groups, running every pairwise t-test inflates the false-positive rate across the family of comparisons. The F-test in an ANOVA asks the single question 'are all the means equal?' at a controlled error rate.

A significant F says no. It does not say which pair differs, and the follow-up is where the multiplicity problem returns. Tukey's HSD makes all pairwise comparisons while adjusting for how many there are, so the family-wise error rate stays at alpha.

Unadjusted pairwise p-values after a significant F are the commonest way of smuggling multiplicity back into an analysis that was designed to avoid it.

This block runs a three-arm ANOVA and then the adjusted comparisons.

suppressPackageStartupMessages(library(broom))
set.seed(304)
d <- data.frame(
  arm = factor(rep(c("placebo", "low", "high"), each = 30),
               levels = c("placebo", "low", "high")),
  y   = c(rnorm(30, 50, 8), rnorm(30, 53, 8), rnorm(30, 58, 8))
)

model <- aov(y ~ arm, data = d)
print(tidy(model))

cat("\nThe F-test asks one question: are all three means equal?\n")
cat("p =", signif(tidy(model)$p.value[1], 4), "-- so no.\n")
cat("It does NOT say which pair differs. That needs a follow-up.\n\n")

cat("Tukey's HSD adjusts for the three comparisons it makes:\n")
print(round(TukeyHSD(model)$arm, 4))

cat("\nUnadjusted pairwise p-values would be smaller and would inflate the\n")
cat("false-positive rate across the family of comparisons.\n")

The F-test gives a statistic of 3.74 and p = 0.0278 — the three arm means are not all equal. Tukey's HSD then finds only one pair separated: high versus placebo at 6.67 with an adjusted p of 0.0260 and an interval from 0.66 to 12.68. Low versus placebo (p = 0.7484) and high versus low (p = 0.1395) are not distinguishable. The significant F did not mean every pair differs.

The mistake this prevents

The mistake is following a significant ANOVA with plain t-tests on each pair. Their unadjusted p-values are smaller and the family-wise error rate is no longer what you set.

Takeaway

Use ANOVA for the overall question and Tukey's HSD for the pairwise follow-up. Report the F-test and the adjusted comparisons together, and never report unadjusted pairwise p-values after it.

Mark complete