Unit 02.00: Outcome, predictor, grouping, unit, window

Five decisions define what an analysis is, and all five are made before any code runs.

Outcome, predictor, grouping, unit, window

The outcome is what you are measuring. The predictor is what you think relates to it. Any grouping says the analysis is run within strata. The unit of analysis says what one row is. The time window says which observations are eligible.

The unit is the one that catches people. The same table can be analysed per session or per user, and the two give different sample sizes from identical data. Since every standard error divides by the square root of n, the choice changes every interval and every p-value in the report.

None of these are technical decisions. They are statements about what question you are answering.

This block states the five and then shows what the unit choice does.

suppressPackageStartupMessages(library(dplyr))

# Five decisions, made before any code, that define what the analysis is.
spec <- data.frame(
  decision = c("Outcome", "Predictor", "Grouping", "Unit of analysis", "Time window"),
  value = c("abandoned (TRUE/FALSE)",
            "variant (A or B)",
            "none -- single stratum",
            "one session",
            "2026-01-01 to 2026-01-31 inclusive")
)
for (i in seq_len(nrow(spec))) {
  cat(sprintf("%-18s %s\n", spec$decision[i], spec$value[i]))
}

sessions <- data.frame(
  session_id = 1:6,
  user_id    = c(1, 1, 2, 3, 3, 3),
  variant    = c("A", "A", "B", "B", "B", "A"),
  abandoned  = c(TRUE, FALSE, TRUE, FALSE, FALSE, TRUE)
)

cat("\nRows:", nrow(sessions), " Distinct users:", n_distinct(sessions$user_id), "\n")
cat("If the unit is the session, n =", nrow(sessions), "\n")
cat("If the unit is the user, n =", n_distinct(sessions$user_id), "\n")
cat("\nThe same data supports two different n. Every p-value depends on which.\n")

The specification names an outcome, a predictor, no grouping, the session as the unit, and a one-month window. The data then has 6 rows drawn from 3 distinct users. If the unit is the session, n = 6. If it is the user, n = 3. The same rows, two different sample sizes, and every inference downstream depends on which one you meant.

The mistake this prevents

The mistake is never stating the unit, and letting it default to whatever the table's row grain happens to be. That is a decision made by the export format rather than by the analyst.

Takeaway

Write all five decisions down before analysing. Be explicit about the unit, and check that the analysis table's grain actually matches it.

Mark complete