Decide whether a difference matters and whether the comparison is fair

A result can be statistically detectable and too small to matter. A result can also be practically important but estimated too imprecisely to establish with the current study. These are different problems, so report both effect size and uncertainty before making a recommendation.

The trial's estimated reduction is 3.20 minutes. Suppose a fictional manager says, before seeing results, that a reduction of at least 2 minutes would justify further operational work. That is a practical threshold supplied for this exercise, not a fact about an actual business. The confidence interval permits reductions from roughly 0.55 to 5.85 minutes. The estimate exceeds 2, but the interval includes benefits smaller than 2. Rejecting the zero-effect null does not establish that the effect clears the manager's threshold.

Use units that the decision needs

The absolute effect is −3.2015 minutes for checklist minus control. The relative effect is about −13.35% of the control mean. Both are legitimate if you state the reference. “A 13% improvement” without a denominator could refer to time, late rate, cost or something else.

For rates, distinguish percentage points from percent change. If a late rate falls from 20% to 15%, the absolute change is −5 percentage points. The relative change is (15 − 20)/20 = −25%. Saying “down 5%” would be ambiguous. A fall from 2% to 1% is only 1 percentage point but a 50% relative reduction. Include counts and baseline rates so the reader can judge the scale.

Standardized effect sizes express a difference relative to an outcome's variability. They can help compare studies using different scales, but they discard the original units that a manager often needs. We keep minutes as the primary effect size here. A universal label such as “large effect” should not substitute for actual costs, risks, implementation effort and outcome relevance.

A fair comparison needs a common question

In the recorded shipment file, East's mean actual time is 79.63 minutes and West's is 45.96, a raw difference of about 33.67 minutes. East also has longer routes on average: 36.85 km versus 18.67 km. Priority shipments account for 40/120 East records and 80/120 West records. A raw time comparison mixes these characteristics with hub labels.

Late rates tell a different descriptive story: East has 25/120 = 20.83%, while West has 26/120 = 21.67%. East is about 0.83 percentage points lower, although its mean absolute time is much longer. The two metrics answer different questions. Promise durations and service mix influence the late classification, so near-equal late rates do not establish that the hubs are equivalent either.

The documented generator has no direct hub term in actual delivery time; hub affects the mix of routes and service types. As authors, we know that because the data is invented. A real analyst would not have access to the true generating mechanism. The appropriate lesson is to inspect plausible confounders and avoid unsupported causal attribution, not to assume every observed hub difference must be caused only by the variables present in a CSV.

When aggregation reverses a comparison

Here is a separate, deliberately small example of on-time performance. It is not the 240-row dataset.

B has a higher on-time rate within both route types, yet A has a higher overall rate because A handled mostly easy short routes. The overall rate is a weighted average of group-specific rates. Each hub uses different weights. This pattern is often called Simpson's paradox; the arithmetic is ordinary weighted averaging.

If the decision concerns a population with equal short and long shares, a descriptive standardization gives A: 0.5 × 90% + 0.5 × 10% = 50%; B: 0.5 × 95% + 0.5 × 20% = 57.5%. The answer changes because the target mixture changes. Equal weighting is justified here only because the question explicitly specifies an equal mixture.

Stratifying or standardizing does not automatically create a causal estimate. Important confounders may remain unmeasured; some variables might be consequences of the intervention rather than appropriate adjustment variables; sparse subgroups may be unstable. Causal adjustment requires a substantive model and stronger assumptions than this beginner course provides.

Recommend an action proportional to the evidence

For the fictional checklist decision, a reasonable recommendation is to design a real, controlled field pilot that measures time, implementation burden and relevant quality outcomes. A full rollout justified solely by a p-value would overstate both practical certainty and transfer from simulation. For the hub question, investigate route/service mix and promise design before assigning blame.

Attempt Practice 9A–9C. Write the absolute effect, its reference, its interval and the practical threshold on four separate lines. If those four lines tell a more nuanced story than your headline, revise the headline.

Further reading: ASA guidance on effect size and significance. All hub examples and decision thresholds in this lesson are original fictional teaching scenarios.