Ask who and what are missing

A table can have perfectly valid numbers and still give a misleading answer. Imagine measuring delivery satisfaction only among people who voluntarily complete a survey. Even if every recorded answer is accurate, respondents may differ from nonrespondents. The problem is selection, not arithmetic.

The shipment file is a complete set of 240 generated records in its fictional eight-week window. Its observed late fraction exactly describes those records after the documented cleaning. It is not automatically an estimate of every future shipment, another season, another city or a real company. To generalize, you need a relationship between the observed data and the target population, plus assumptions about how the process changes.

Random sampling and random assignment solve different problems

Random sampling selects units from a population. If the selection design gives the population suitable coverage, it can support generalization to that population. Random assignment allocates an intervention among enrolled units. It helps separate the intervention's effect from pre-existing differences, in expectation, within the experimental setting. A randomized trial made up of convenient volunteers is still a convenience sample of people.

The separate checklist fixture randomizes 40 simulated participants into 20 control and 20 checklist assignments. Each simulated participant contributes one independent outcome. It illustrates an experiment; it does not demonstrate that an actual checklist works for real workers. Conversely, the shipment file contains no randomized hub assignment. A large difference between hubs could reflect routes and service mix rather than hub management.

Open the diagram at full size

Read the diagram explanation

Random sampling selects study units from a target population. Random assignment divides enrolled units into intervention groups. These support different kinds of conclusions under their assumptions.

Larger is not a cure for biased selection

Suppose an operation handles 900 short jobs and 100 long jobs. Short jobs average 10 minutes and long jobs average 40. The population mean is (900 × 10 + 100 × 40) / 1,000 = 13 minutes. A convenience sample containing 50 of each type has mean (50 × 10 + 50 × 40) / 100 = 25 minutes. Taking 500 of each type would still target an equal mixture and would still have mean 25 under these values. More observations sharpen the wrong comparison if the selection pattern remains wrong.

Weighting could recover 13 here because the true group shares and group means were supplied: 0.9 × 10 + 0.1 × 40. In actual work, weights require a justified sampling or adjustment model. They cannot fix missing groups whose outcomes you never observe without additional assumptions. Do not assign weights merely until the result matches what you expected.

Missing does not mean zero

In the shipment generator, a rating is more likely to be missing when the shipment is late. You can see that mechanism in the documented source because this is a teaching simulation. In a real dataset, the missingness mechanism is usually not fully observable. You can measure different response rates by known groups, but you cannot learn every missing person's opinion from those rates.

Report the response count and denominator alongside a rating summary. Check response rates separately for late and on-time shipments. If late shipments respond less often and tend to receive lower ratings among respondents, the overall respondent average may overstate satisfaction across all shipments. The observed data alone cannot identify the exact size of that bias.

A small bounds example shows the issue. Suppose six of ten people respond with ratings 4, 4, 5, 3, 4, 4. The observed sum is 24 and the respondent mean is 4.0. If the four missing ratings were all 1, the all-person mean would be (24 + 4) / 10 = 2.8. If they were all 5, it would be (24 + 20) / 10 = 4.4. These are worst-case bounds given the 1–5 scale, not a confidence interval and not predictions of what missing people would say. They make the information gap visible.

Independence is about how observations arise

Forty independent participants are not interchangeable with forty observations from four participants. Repeated readings from the same person may move together. Likewise, shipments in the same week may share conditions. Counting correlated records as independent can make a standard error too small and an interval too narrow.

Our shipment generator includes shared weekly conditions. We therefore use its rates and comparisons descriptively in the required work. We do not attach a naive independent-shipment t-test to every comparison. Analyses of clustered, time-series or repeated-measures data need additional methods beyond this course. “The software returned a p-value” does not establish that the independence assumption held.

What a limitations statement should do

A useful limitation identifies how the conclusion could change: “Ratings were missing more often for late shipments, so the respondent average may not represent all shipments.” “The sample is small” is less useful unless you explain that the interval is wide or a group is poorly represented. State selection, measurement, missingness and dependence concerns before the recommendation depends on ignoring them.

Attempt Practice 4A–4C and then Checkpoint A. You should be able to name the target population, describe how units reached the file, and distinguish an unobserved value from an observed zero.

Further reading: NIST on randomized designs and the assumptions behind common analyses.