Compare groups without turning a p-value into a verdict

The checklist question concerns a difference: does the average task duration change when the checklist is used? Define the contrast before calculating. We use checklist minus control, so a negative difference means the checklist group was faster. Keeping that sign convention throughout prevents an otherwise correct calculation from being narrated backward.

The fixture contains 40 simulated participants, randomly assigned to two groups of 20. Each contributes one task duration. The generator specifies independent normal noise and a constant 3-minute checklist reduction; random assignment and sample noise mean the observed difference need not equal exactly −3. No observations were rerolled to get a preferred p-value. Because the whole study is simulated, it teaches analysis rather than validating a real checklist.

Estimate the difference first

The control mean is 23.9895 minutes with SD 4.1254. The checklist mean is 20.7880 with SD 4.1477. Subtract: 20.7880 − 23.9895 = −3.2015 minutes. The observed checklist mean is about 3.20 minutes lower. Relative to the control mean, −3.2015/23.9895 × 100 ≈ −13.35%. The absolute difference is often easier to connect to a decision than the percentage alone.

We use Welch's independent-samples t method. It does not require equal population variances. Its standard error for the difference combines the two groups' squared standard errors: square each SD, divide by its group size, add, and take the square root. Here that gives about 1.3081 minutes. Software calculates a Welch degrees-of-freedom value of about 38 and uses the corresponding t multiplier.

The compatible two-sided 95% confidence interval for checklist minus control is −5.85 to −0.55 minutes. In reduction language, the interval corresponds to approximately 0.55 to 5.85 minutes faster under the stated model. It excludes zero but spans a wide range of practical benefits. The interval is about a mean effect in the model, not a guarantee that every participant benefits.

Open the diagram at full size

Read the diagram explanation

Checklist minus control is −3.2015 minutes, with a 95% Welch interval of −5.8496 to −0.5534. It excludes zero but crosses the −2-minute hypothetical worthwhile threshold. Simulated trial, 20 per group.

What the test asks

The null hypothesis is that the two model population means are equal, or that checklist minus control equals zero. The two-sided alternative allows a difference in either direction. “Two-sided” is chosen before inspecting the outcome; it is not selected afterward because one direction looks more favorable.

The test statistic is the estimated difference divided by its estimated standard error: approximately −3.2015/1.3081 = −2.4475. Under the null model and its assumptions, the two-sided p-value is the probability of obtaining a test statistic at least as extreme in magnitude as the one observed. Here p ≈ 0.0191. The p-value is not the probability that the null hypothesis is true, that the checklist works, or that the finding occurred “by chance.”

If a 0.05 threshold was specified in advance, the test rejects the zero-difference null at that level. That phrase is a procedural statement. It does not establish importance, eliminate model concerns, or make the experiment representative of a real setting. Report the estimate, interval, design and limitations with the p-value.

Errors and power in everyday terms

A Type I error means rejecting a true null under the decision rule. A test's nominal 5% level controls that error rate under its model across repeated uses, not the probability that this particular rejection is wrong. A Type II error means failing to reject a false null. Power is the probability of rejecting for a specified true effect, sample size, variation and test design.

“Not statistically significant” does not prove no difference or equivalence. A wide interval may include meaningful benefit and meaningful harm. Demonstrating equivalence requires a pre-specified acceptable range and a suitable equivalence procedure, beyond this course. Likewise, do not compute many alternative outcomes, subgroups and exclusion rules and then report only the smallest p-value. Multiple searching and repeated peeking can substantially increase false-positive risk.

In practice, write down the primary outcome, contrast, stopping rule, exclusion policy and minimum worthwhile effect before collecting data. If you explore additional questions, label them exploratory and consider a confirmatory follow-up. This separates useful discovery from claims that require stricter testing.

Match the method to the design

If the same participants performed before and after tasks, their two durations would be paired; you would analyze within-person differences using a suitable paired method. Our fixture uses independent participants, so pairing rows by CSV position would manufacture a relationship. If outcomes were binary, a mean-duration t-test would not be the right introductory method for comparing rates. If teams, rather than individuals, were randomized, team clustering would matter.

For optional Python, scipy.stats.ttest_ind(checklist, control, equal_var=False, alternative="two-sided") requests the method used here. The order of arrays determines the sign. The library's default assumes equal variances, so explicitly setting equal_var=False is important. The tests verify the reference using both formula calculations and SciPy.

Attempt Practice 8A–8C and Checkpoint B. You are ready when you can explain the p-value without treating it as a probability that a hypothesis is true, and when your recommendation survives without the word “significant.”

Further reading: NIST's two-sample methods, the pinned SciPy method, and the ASA's p-value guidance.