Unit 03.04: Estimate, interval, p-value, sample size

'Significant' replaces four pieces of information with none of them.

Estimate, interval, p-value, sample size

A statistical sentence carries the estimate — how big the effect was; the interval — what range the data supports; the p-value; and the sample size. Compressing all four into 'significant' discards exactly what a reader needs to judge whether the result matters.

The word also has an everyday meaning — important, substantial — which is not what it means here. A tiny difference is significant with a large enough sample.

The habit worth building is describing the *range* the data supports rather than the point estimate alone. That is what an interval is for.

This block writes the same result twice.

import numpy as np
from scipy import stats

rng = np.random.default_rng(53)
control = rng.normal(41.0, 7.0, 45)
treated = rng.normal(36.0, 7.0, 45)

res = stats.ttest_ind(treated, control, equal_var=False)
ci = res.confidence_interval()
diff = treated.mean() - control.mean()          # negative: a saving
saving_lo, saving_hi = -ci.high, -ci.low        # state the saving as positive

print("Overclaiming:")
print('  "The new route plan works: delivery times fell significantly."')
print()
print("What the data supports:")
print(f'  "Deliveries on the new plan took {abs(diff):.1f} minutes less on'
      f' average')
print(f'   (95% CI {saving_lo:.1f} to {saving_hi:.1f} minutes saved,'
      f' p = {res.pvalue:.3f}, n = {control.size} per group).')
print("   The interval is consistent with anything from a small saving to a")
print('   substantial one."')
print()
print("The second names the estimate, the interval, the p-value and the")
print("sample size, and describes the range the data supports rather than")
print("collapsing all four into the word 'significant'.")

The overclaiming version asserts that the route plan works. The supported version reports 3.8 minutes saved on average, a 95% interval from 0.6 to 7.1 minutes, p = 0.022, with 45 per group — and then says the data is consistent with anything from a small saving to a substantial one. That last sentence is the honest reading of an interval spanning more than a tenfold range of effects.

The mistake this prevents

The mistake is writing 'significantly faster' and stopping. A reader cannot tell whether the saving was half a minute or ten, and the interval here admits both a negligible and a useful effect.

Takeaway

Report the estimate, the interval, the p-value and n in every finding. Describe the range the interval covers, and avoid 'significant' as a substitute for saying how big the effect was.

Mark complete