Unit 03.04: Estimate, interval, p-value, sample size

'Significant' is a word that replaces four pieces of information with none of them.

Estimate, interval, p-value, sample size

A statistical sentence has to carry the estimate — how big the effect was; the interval — what range the data is consistent with; the p-value; and the sample size. Compressing all four into 'significant' discards precisely what a reader needs to judge whether the result matters.

The word also carries an everyday meaning — important, substantial — which is not what it means here. A tiny difference is significant with a large enough sample.

The habit worth building is describing the *range* the data supports rather than the point estimate alone, because that is what an interval is for.

This block writes the same result twice.

suppressPackageStartupMessages(library(dplyr))

set.seed(53)
d <- data.frame(group = rep(c("control", "treated"), each = 45),
                value = c(rnorm(45, 62, 11), rnorm(45, 67, 11)))

tt <- t.test(value ~ group, data = d)       # estimate is c(control, treated)
diff <- as.numeric(diff(tt$estimate))       # treated - control
ci   <- rev(-tt$conf.int)                   # flip: CI was for control - treated

cat("Overclaiming:\n")
cat("  \"The treatment works: scores rose significantly.\"\n\n")

cat("What the data supports:\n")
cat(sprintf("  \"Treated participants scored %.1f points higher on average\n", diff))
cat(sprintf("   (95%% CI %.1f to %.1f, p = %.3f, n = %d per group).\n",
            ci[1], ci[2], tt$p.value, 45))
cat("   The interval is consistent with anything from a small to a\n")
cat("   moderate improvement.\"\n\n")

cat("The second names the estimate, the interval, the p-value and the sample\n")
cat("size, and it describes the range the data is consistent with rather than\n")
cat("collapsing all of that into the word 'significant'.\n")

The overclaiming version asserts that the treatment works. The supported version reports 6.7 points higher on average, a 95% interval from 2.1 to 11.4, p = 0.005, with 45 per group — and then says the data is consistent with anything from a small to a moderate improvement. That last sentence is the honest reading of an interval spanning a fivefold range of effect sizes.

The mistake this prevents

The mistake is writing 'significantly higher' and stopping. A reader cannot tell whether the effect was 0.5 points or 50, and the interval usually admits both a trivial and an important effect.

Takeaway

Report the estimate, the interval, the p-value and n in every finding. Describe the range the interval covers, and avoid 'significant' as a substitute for saying how big the effect was.

Mark complete