Separate variation in people from uncertainty in an average

Imagine repeating a study tomorrow with new participants from the same stable process. The observed mean will usually change. That is sampling variability. It is possible for individual outcomes to remain highly variable while the average across many independent outcomes becomes fairly stable.

This distinction is central to inference. A standard deviation describes how individual outcomes vary. A standard error describes how an estimate would vary across repeated samples under a sampling model. Neither word means “the spreadsheet contains a mistake.”

A complete miniature sampling experiment

Take a tiny fictional population with values 2, 4 and 6 minutes. Its mean is 4. Draw two values independently, with replacement, so drawing a 2 first does not remove it before the second draw. There are nine equally likely ordered samples:

The sampling distribution of the mean is therefore 2 once, 3 twice, 4 three times, 5 twice and 6 once. Its average is also 4. The possible sample means cluster more closely around 4 than individual values do. We have not improved any person's duration; we have made an average less variable by combining independent information.

This example lists all possible samples, so no simulation is necessary. In a real process we rarely know the population well enough to enumerate them. We estimate the variability from the sample and use a model to describe what repeated samples might do.

Work a standard error

For an independent sample from a process with finite variance, an estimated standard error of the mean is the sample SD divided by the square root of the sample size. Suppose the observed SD is 4 minutes and n = 16. The square root of 16 is 4, so SE = 4/4 = 1 minute. An individual participant can still differ from the mean by several minutes even though the average is estimated more precisely.

If n increases to 64 and SD stays around 4, the SE becomes 4/8 = 0.5 minutes. Four times as many independent observations halve the standard error in this simplified comparison. Doubling the sample size does not halve it. This square-root relationship also does not remove selection bias, missingness bias or a poor measurement definition.

In the simulated trial, the control group has n = 20, mean 23.9895 minutes and SD about 4.1254. Its standard error is about 4.1254/√20 = 0.9225 minutes. The SD describes the spread of simulated participant durations; the SE describes uncertainty in the estimated process mean under the study model. Displaying an SE error bar as if it represented most individual participants would conceal much of the participant-to-participant variation.

What the central limit idea does and does not promise

Under suitable conditions, averages of many independent observations have an approximately normal sampling distribution, even when the individual observations are not normal. This is a statement about the distribution of averages across repeated samples. It does not turn the original histogram into a bell curve.

There is no universal threshold at which “n ≥ 30” makes every problem safe. Heavy tails, extreme outliers, dependence and strong skew can make approximations poor. A small sample from a reasonably normal process can be suitable for a t-method; a large collection of strongly dependent records can still be unsuitable for an ordinary independent-observation method.

The trial generator supplies independent normal outcome noise by design, with one observation per simulated participant. The shipment generator deliberately supplies a more complicated process with shared weekly conditions. We use the trial for introductory inference and the shipments for descriptive comparisons. This is method selection based on how the data arose, rather than choosing a method because it produces an interesting answer.

A simulation is a demonstration of assumptions

The optional companion produces repeated 20-observation samples from a stated normal model and computes their means. The changing sample means illustrate variability; they do not validate a model for a real operation. A random seed makes a demonstration repeatable. It does not make one particular simulated result more truthful, and changing the seed until a desired result appears would hide the selection process.

When reporting precision, name the estimator, model, unit and number of independent units. “SE = 0.92” without “minutes, for the control mean, n = 20” is incomplete. When two datasets contain the same number of rows, ask whether they contain the same amount of independent information.

Attempt Practice 6A–6C. Explain why reducing SE is not the same as reducing individual delivery variability, then use the miniature sampling table to distinguish a population distribution from a sampling distribution.

Further reading: NIST's confidence-interval setup and normal distribution reference.