Unit 05.04: Four questions choose the test

Four questions choose the test. Everything else is detail.

Outcome type, group count, design, assumptions

What type is the outcome — numeric, binary, categorical? How many groups are being compared — one against a target, two, or more? Is the design paired or independent? And do the method's assumptions hold?

Answering those four picks the test from a short table. Memorising test names without the questions is what produces t-tests on yes/no outcomes.

The paired/independent distinction is not a technicality. Pairing removes between-unit variation from the comparison, and when that variation is large relative to the effect, the paired test can find something the unpaired test has no chance of seeing.

This block lays out the table and then demonstrates why pairing is its own row.

# Four questions choose the test. Nothing else needs to be memorised.
decision <- data.frame(
  outcome  = c("numeric", "numeric", "numeric", "binary", "binary", "numeric"),
  groups   = c("1", "2", "3+", "2", "2+", "2"),
  design   = c("-", "independent", "independent", "independent", "independent", "paired"),
  test     = c("t.test(x, mu=)", "t.test(y ~ g)", "aov(y ~ g)",
               "prop.test()", "chisq.test()", "t.test(paired=TRUE)")
)
for (i in seq_len(nrow(decision))) {
  cat(sprintf("%-9s %-4s %-12s -> %s\n", decision$outcome[i], decision$groups[i],
              decision$design[i], decision$test[i]))
}

cat("\nWhy 'paired' is a separate row, not a detail:\n")
set.seed(205)
before <- rnorm(25, 50, 10)
after  <- before + rnorm(25, 3, 2)     # each person improves by about 3

cat("  unpaired t-test p =", signif(t.test(after, before)$p.value, 3), "\n")
cat("  paired   t-test p =", signif(t.test(after, before, paired = TRUE)$p.value, 3), "\n")
cat("\nSame numbers. The paired test removes the between-person variation and\n")
cat("finds an effect the unpaired test cannot see at all.\n")

Six rows cover most beginner work. Then the same 25 before-and-after measurements are tested both ways: unpaired gives p = 0.316 — nothing — while paired gives p = 1.36e-08. Identical numbers, and the choice of design decides whether a real, consistent three-point improvement is visible at all.

The mistake this prevents

The mistake is running an independent-samples test on before-and-after data. It throws away the pairing, so the between-person spread swamps the within-person change and the effect disappears.

Takeaway

Answer the four questions in writing before choosing a test. Check explicitly whether the two measurements come from the same units, because that single fact can decide the result.

Mark complete