Unit 06.04: One F-test, then an adjusted follow-up

f_oneway answers one question and then leaves you stranded — scipy has no post-hoc test at all.

One F-test, then adjusted pairwise comparisons

With three or more groups, running every pairwise t-test inflates the false-positive rate across the family of comparisons. The F-test asks the single question 'are all the means equal?' at a controlled error rate.

A significant F says no. It does not say which pair differs, and scipy stops there. The follow-up lives in statsmodels: pairwise_tukeyhsd makes all pairwise comparisons while adjusting for how many there are, keeping the family-wise error rate at alpha.

Unadjusted pairwise p-values after a significant F are the commonest way of smuggling multiplicity back into an analysis designed to avoid it.

This block runs a three-arm ANOVA and then the adjusted comparisons.

import numpy as np, pandas as pd
from scipy import stats
from statsmodels.stats.multicomp import pairwise_tukeyhsd

rng = np.random.default_rng(304)
d = pd.DataFrame({
    "arm": np.repeat(["placebo", "low", "high"], 30),
    "y": np.concatenate([rng.normal(50, 8, 30),
                         rng.normal(53, 8, 30),
                         rng.normal(58, 8, 30)]),
})

groups = [g["y"].values for _, g in d.groupby("arm", observed=True)]
f_stat, p = stats.f_oneway(*groups)
print(f"One-way ANOVA:  F = {f_stat:.3f}   p = {p:.5f}")
print("The F-test asks one question: are all three means equal? Here, no.")
print("It does NOT say which pair differs, and scipy offers no post-hoc.\n")

tukey = pairwise_tukeyhsd(d["y"], d["arm"], alpha=0.05)
print(tukey.summary())
print("\nTukey's HSD adjusts for the three comparisons it makes. Running plain")
print("t-tests on each pair would give smaller p-values and inflate the")
print("family-wise error rate.")

The F-test gives F = 9.697 with p = 0.00016 — the three arm means are not all equal. Tukey's HSD then separates two pairs: high versus placebo at −9.95 (adjusted p = 0.0001) and high versus low at −6.98 (p = 0.0095), while low versus placebo at −2.97 (p = 0.4092) is not distinguishable. The significant F did not mean every pair differs.

The mistake this prevents

The mistake is following a significant f_oneway with plain ttest_ind on each pair. Their unadjusted p-values are smaller and the family-wise error rate is no longer what you set.

Takeaway

Use f_oneway for the overall question and pairwise_tukeyhsd for the follow-up. Report the F-test and the adjusted comparisons together, and never report unadjusted pairwise p-values after it.

Mark complete