Unit 01.05: Report all of it, then interpret

The results table reports everything you tested. The interpretation section says what it means. Neither is allowed to do the other's job.

Report all of it, then interpret

A results table that contains only the comparisons that worked is not a results table. It is a selection, and a reader who cannot see how many tests were run cannot judge any of them.

So the table reports every planned comparison, with its estimate, its interval and its p-value, whether or not the interval excludes zero. The interpretation section then argues — pointing at rows, explaining which differences are large enough to matter and which intervals are too wide to conclude anything from.

The separation matters because it puts the selection in the open. If you want to emphasise one metric, argue for it in prose with the others still visible above.

This block tests three metrics and tabulates all three.

suppressPackageStartupMessages({library(dplyr); library(broom); library(knitr)})

set.seed(7)
outcomes <- data.frame(
  metric = rep(c("time_on_page", "items_viewed", "cart_value"), each = 120),
  group  = rep(rep(c("A", "B"), each = 60), 3),
  value  = c(rnorm(60, 42, 9),   rnorm(60, 45, 9),
             rnorm(60, 6.1, 1.8), rnorm(60, 6.3, 1.8),
             rnorm(60, 31, 11),  rnorm(60, 38, 11))
)

results <- outcomes |>
  group_split(metric) |>
  lapply(function(d) {
    tt <- t.test(value ~ group, data = d)          # estimate is c(mean A, mean B)
    data.frame(metric = d$metric[1],
               b_minus_a = round(as.numeric(diff(tt$estimate)), 2),
               ci_low  = round(-tt$conf.int[2], 2),   # CI is for A - B, so flip
               ci_high = round(-tt$conf.int[1], 2),
               p = signif(tt$p.value, 3),
               row.names = NULL)
  }) |>
  bind_rows()

cat(kable(results, format = "simple"), sep = "\n")

cat("\nThe table reports every metric, including the ones with wide intervals.\n")
cat("Metrics whose interval contains zero:",
    sum(results$ci_low < 0 & results$ci_high > 0), "of", nrow(results), "\n")
cat("An interpretation section says what that means; it does not re-select rows.\n")

Cart value differs by 6.33 with an interval from 1.92 to 10.74 and p = 0.005. Items viewed and time on page have intervals that straddle zero — 2 of 3 metrics are inconclusive. Reporting only cart value would be defensible only if cart value had been named as the primary outcome in advance; presented alone after the fact, it is one significant result out of three tests, which is close to what chance produces.

The mistake this prevents

The mistake is a results section containing one table with one row. The reader has no way to know whether it was the only test run or the best of twelve.

Takeaway

Tabulate every planned comparison with its estimate, interval and p-value. Argue in the interpretation section, not by choosing rows, and state how many tests were run.

Mark complete