Unit 01.03: Build the analysis table on purpose

The analysis table is a deliberate object: one row per unit of analysis, one column per variable the plan named, and nothing else.

Build it on purpose, not by accident

Most statistical mistakes are visible in the table the test was run on. The wrong row grain, an extra variable that invited a comparison nobody planned, a group column left as text so the reference level is alphabetical.

So the analysis table gets built explicitly rather than being whatever the import produced. Select the columns the plan named. Set the grain. Make grouping variables factors with the levels you intend.

Leaving extra columns in is not neutral. Every additional variable is another comparison available to be run when the planned one disappoints, and that is the mechanism behind most irreproducible findings.

This block reduces a raw extract to its analysis table.

suppressPackageStartupMessages(library(dplyr))

# The analysis table is a deliberate object: one row per unit of analysis,
# one column per variable the plan named, and nothing else.
raw <- data.frame(
  session_id = 1:8,
  variant    = c("A", "A", "B", "B", "A", "B", "A", "B"),
  abandoned  = c(TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE),
  browser    = c("chrome", "safari", "chrome", "firefox", "chrome",
                 "safari", "chrome", "chrome"),
  raw_ts     = Sys.time()
)

analysis <- raw |>
  select(session_id, variant, abandoned) |>
  mutate(variant = factor(variant, levels = c("A", "B")))

cat("Unit of analysis: one session\n")
cat("Rows:", nrow(analysis), " Variables:", ncol(analysis), "\n")
cat("Dropped from raw:", setdiff(names(raw), names(analysis)), "\n\n")

print(analysis |> count(variant, abandoned))
cat("\nEvery column here was named in the plan. `browser` was not, so it is\n")
cat("out -- available for a stated secondary analysis, not for quiet reuse.\n")

Eight rows and 3 variables survive; browser and raw_ts are dropped. The counts show four sessions per variant with abandonment split 2–2 in A and 1–3 in B. Dropping browser is not a claim that browser is irrelevant — it is a statement that browser was not in the plan, so any analysis of it is a secondary analysis and must be labelled as one.

The mistake this prevents

The mistake is analysing whatever data frame the import produced. It usually has the wrong grain, and it always has more columns than the plan named.

Takeaway

Construct the analysis table as its own step: select the planned columns, set the grain, convert grouping variables to factors with deliberate levels. Anything not in the plan is a secondary analysis and says so.

Mark complete