Unit 04.05: The 95% belongs to the method

The 95% is a property of the method, not of the interval in front of you.

What the confidence level actually counts

A 95% confidence interval comes from a procedure that, applied repeatedly to fresh samples, produces intervals containing the true value 95% of the time. That is a statement about the long run of the method.

Your particular interval either contains the truth or it does not. There is no probability left in it once it is computed — the sampling already happened. Saying 'there is a 95% chance the true mean is in this interval' attributes randomness to a fixed, unknown quantity.

The other frequent error is reading it as a range containing 95% of the data. It is a range for the *mean*, and it is far narrower than the data's spread — narrower by a factor of roughly the square root of n.

This block builds a thousand intervals from a population whose mean is known.

set.seed(106)
TRUE_MEAN <- 100

# What "95% confidence" actually counts: intervals, not this interval.
covered <- 0
for (i in 1:1000) {
  s <- rnorm(25, mean = TRUE_MEAN, sd = 15)
  ci <- t.test(s)$conf.int
  if (ci[1] <= TRUE_MEAN && TRUE_MEAN <= ci[2]) covered <- covered + 1
}
cat("Of 1000 intervals built this way,", covered, "contained the true mean.\n")
cat("That is", covered / 10, "% -- the 95% is a property of the PROCEDURE.\n\n")

s <- rnorm(25, mean = TRUE_MEAN, sd = 15)
ci <- t.test(s)$conf.int
cat("One particular interval:", round(ci[1], 2), "to", round(ci[2], 2), "\n")
cat("It either contains 100 or it does not. Here:",
    ci[1] <= TRUE_MEAN && TRUE_MEAN <= ci[2], "\n\n")

cat("Wrong: 'there is a 95% probability the true mean is in THIS interval'\n")
cat("Wrong: '95% of the data lies in this interval'\n")
cat("Right: 'values in this range are consistent with the data; the method\n")
cat("        that produced it captures the truth 95% of the time'\n")

Of 1000 intervals, 947 contained the true mean of 100 — 94.7%, which is the 95% coverage the method promises, arriving through repetition rather than through any one interval. The single interval shown afterwards, 91.44 to 103.82, either contains 100 or does not; here it does. There is nothing probabilistic about that fact once the sample is drawn.

The mistake this prevents

The mistake is 'we are 95% sure the true value is between these numbers'. It is the natural reading and it is not what the procedure guarantees; the guarantee is about the procedure's long-run behaviour.

Takeaway

Read a confidence interval as the range of values consistent with the data, produced by a method that succeeds 95% of the time. Never describe it as containing 95% of the data or as carrying a 95% probability.

Mark complete