Unit 01.05: Report all of it, then interpret

The results table reports everything you tested. The interpretation cell argues. Neither may do the other's job.

Report all of it, then interpret

A results table containing only the comparisons that worked is not a results table; it is a selection, and a reader who cannot see how many tests were run cannot judge any of them.

So the table carries every planned comparison with its estimate, its interval and its p-value, whether or not the interval excludes zero. The interpretation cell then argues, pointing at rows — which differences are large enough to matter, which intervals are too wide to conclude from.

The separation puts the selection in the open. Emphasising one metric is fine; doing it in prose with the others still visible above is the difference between an argument and a filter.

This block tests three metrics and tabulates all three.

import numpy as np, pandas as pd
from scipy import stats

rng = np.random.default_rng(7)
metrics = {
    "wellbeing":     (58, 64, 9),
    "hours_worked":  (41.0, 40.4, 3.5),
    "output_index":  (100, 101, 12),
}

rows = []
for name, (mu_a, mu_b, sd) in metrics.items():
    a = rng.normal(mu_a, sd, 60)
    b = rng.normal(mu_b, sd, 60)
    res = stats.ttest_ind(b, a, equal_var=False)
    ci = res.confidence_interval()
    rows.append({"metric": name,
                 "pilot_minus_comparison": round(b.mean() - a.mean(), 2),
                 "ci_low": round(ci.low, 2),
                 "ci_high": round(ci.high, 2),
                 "p": float(f"{res.pvalue:.3g}")})

results = pd.DataFrame(rows)
print(results.to_string(index=False))

crosses_zero = ((results.ci_low < 0) & (results.ci_high > 0)).sum()
print(f"\nMetrics tested: {len(results)}   intervals containing zero: {crosses_zero}")
print("Every planned comparison is in the table, including the ones that")
print("concluded nothing. The interpretation cell argues; it does not filter.")

Wellbeing rises by 7.13 with an interval from 4.31 to 9.95 and p = 0.000002. Hours worked moves by 0.01 and the output index by 2.28, both with intervals straddling zero — 2 of 3 metrics are inconclusive. Reporting wellbeing alone would be defensible only because it was the pre-specified primary outcome; presented after the fact it would be one significant result in three.

The mistake this prevents

The mistake is a results section with one table and one row. A reader has no way to know whether it was the only test or the best of twelve.

Takeaway

Tabulate every planned comparison with its estimate, interval and p-value. Argue in the interpretation cell rather than by choosing rows, and say how many tests were run.

Mark complete