Unit 04.05: The 95% belongs to the method

The 95% is a property of the method, not of the interval in front of you.

What the confidence level actually counts

A 95% confidence interval comes from a procedure that, applied repeatedly to fresh samples, produces intervals containing the true value 95% of the time. That is a statement about the long run of the method.

Your particular interval either contains the truth or it does not. There is no probability left in it once it is computed — the sampling already happened. Saying 'there is a 95% chance the true mean is in this interval' attributes randomness to a fixed unknown.

The other frequent error is reading it as a range containing 95% of the data. It is a range for the *mean*, and it is narrower than the data's spread by roughly the square root of n.

This block builds a thousand intervals from a population whose mean is known.

import numpy as np
from scipy import stats

TRUE_MEAN = 100
rng = np.random.default_rng(106)

covered = 0
for _ in range(1000):
    s = rng.normal(TRUE_MEAN, 15, 25)
    ci = stats.ttest_1samp(s, popmean=0).confidence_interval()
    if ci.low <= TRUE_MEAN <= ci.high:
        covered += 1

print(f"Of 1000 intervals built this way, {covered} contained the true mean.")
print(f"That is {covered / 10:.1f}% -- the 95% is a property of the PROCEDURE.\n")

s = rng.normal(TRUE_MEAN, 15, 25)
ci = stats.ttest_1samp(s, popmean=0).confidence_interval()
print(f"One particular interval: {ci.low:.2f} to {ci.high:.2f}")
print(f"Does it contain 100? {ci.low <= TRUE_MEAN <= ci.high}")
print("Once the sample is drawn there is no probability left in it.\n")

print(f"An interval for the MEAN is far narrower than the data's spread:")
print(f"  interval width : {ci.high - ci.low:.2f}")
print(f"  data range     : {s.max() - s.min():.2f}")
print()
print("Wrong: 'there is a 95% probability the true mean is in THIS interval'")
print("Wrong: '95% of the data lies in this interval'")
print("Right: 'values in this range are consistent with the data; the method")
print("        that produced it captures the truth 95% of the time'")

Of 1000 intervals, 946 contained the true mean of 100 — 94.6%, the coverage the method promises, arriving through repetition rather than through any one interval. The single interval shown afterwards, 91.68 to 103.23, either contains 100 or does not; here it does. And the scale point is stark: that interval is 11.55 wide while the data itself spans 57.66.

The mistake this prevents

The mistake is 'we are 95% sure the true value is between these numbers'. It is the natural reading and it is not what the procedure guarantees.

Takeaway

Read a confidence interval as the range of values consistent with the data, produced by a method that succeeds 95% of the time. Never describe it as containing 95% of the data or as carrying a 95% probability.

Mark complete