Unit 02.05: Six checks, before any test

Run the checks before the test, not after the result surprises you.

Six checks, every time

The analysis table can be wrong in ways the test will never complain about. Duplicated units, a grouping variable left as text, a group too small to support the method, more missingness than you realised, an impossible value that will drag a mean.

Encoding these as a list of TRUE/FALSE checks makes them cheap enough to run every time, and it produces a record that the checks were run.

A failed check is not a line to comment out. It is a question for whoever produced the data, and answering it usually changes the analysis.

This block runs six checks against a table with several problems.

suppressPackageStartupMessages(library(dplyr))

analysis <- data.frame(
  id      = c(1:9, 9),
  group   = factor(c("A","A","A","A","B","B","B","B","B","B")),
  outcome = c(41, 38, 44, 40, 52, 49, 51, NA, 47, 47)
)

checks <- list(
  "One row per unit"            = !any(duplicated(analysis$id)),
  "Outcome is numeric"          = is.numeric(analysis$outcome),
  "Group is a factor"           = is.factor(analysis$group),
  "Both groups have >= 5 rows"  = all(table(analysis$group) >= 5),
  "Missingness under 10%"       = mean(is.na(analysis$outcome)) < 0.10,
  "No impossible outcome"       = all(analysis$outcome >= 0 | is.na(analysis$outcome))
)

for (nm in names(checks)) cat(sprintf("[%s] %s\n", ifelse(checks[[nm]], "x", " "), nm))

cat("\nPassed:", sum(unlist(checks)), "of", length(checks), "\n")
failed <- names(checks)[!unlist(checks)]
cat("Failed:", paste(failed, collapse = "; "), "\n")
cat("Duplicated id:", analysis$id[duplicated(analysis$id)], "\n")
cat("\nRun this before any test. A failed check is a question for the data\n")
cat("owner, not a line to comment out.\n")

3 of 6 pass. Three fail: the id column has a duplicate — id 9 appears twice — one group has fewer than 5 rows, and missingness exceeds 10%. None of these would have stopped t.test() from returning a confident-looking answer. The duplicated id in particular would have counted one unit twice.

The mistake this prevents

The mistake is treating a failed check as an obstacle to the analysis. It is information about the data, and the right response is to ask about it rather than to relax the check.

Takeaway

Run the check list before every analysis and record which checks passed. Treat each failure as a question for the data owner, and re-run the list after any change to the table.

Mark complete