Unit 01.06: Rules, with their reason and their cost

Any exclusion that lives in your head or in a spreadsheet is a hidden manual step, and it will change your headline number without appearing anywhere.

Rules, with their reason and their cost

Statistical analyses accumulate small decisions: drop the test accounts, exclude the bot traffic, ignore the first week while the feature was rolling out. Each is defensible. Together they can move a result substantially, and if they were made by hand nobody can see them.

Written as code, each becomes a named constant with a comment giving its justification, and a printed count of what it removed. That is three lines, and it converts an invisible judgement into a reviewable one.

It also lets a reviewer test the sensitivity of your conclusion by changing the constant.

This block applies one exclusion rule and reports what it cost.

suppressPackageStartupMessages(library(dplyr))

sessions <- data.frame(
  variant   = rep(c("A", "B"), each = 6),
  abandoned = c(TRUE, TRUE, FALSE, FALSE, TRUE, FALSE,
                TRUE, FALSE, FALSE, FALSE, TRUE, FALSE),
  duration  = c(12, 8, 240, 190, 5, 310, 9, 260, 205, 180, 7, 290)
)

# A hidden manual step is any exclusion that lives in your head or a spreadsheet.
# Write it as a rule, with its reason and its cost.
MIN_DURATION <- 10   # sessions under 10s are bot traffic per the plan

kept <- sessions |> filter(duration >= MIN_DURATION)
excluded <- sessions |> filter(duration < MIN_DURATION)

cat("Rule: exclude sessions under", MIN_DURATION, "seconds (bot traffic)\n")
cat("Rows in:", nrow(sessions), " kept:", nrow(kept),
    " excluded:", nrow(excluded), "\n")
print(excluded |> count(variant, name = "excluded"))

cat("\nAbandonment rate, all rows :",
    round(mean(sessions$abandoned) * 100, 1), "%\n")
cat("Abandonment rate, after rule:",
    round(mean(kept$abandoned) * 100, 1), "%\n")
cat("\nThe exclusion moved the headline figure. Written as a rule it is\n")
cat("reviewable; done by hand in a spreadsheet it is invisible.\n")

The rule removes sessions under 10 seconds as bot traffic: 4 of 12 rows, evenly split between the variants. The effect on the headline is large — abandonment falls from 41.7% to 12.5%. The rule may well be right; the point is that a reader can see it, see its cost, and try a different threshold.

The mistake this prevents

The mistake is filtering in a spreadsheet before the data reaches R. The script then starts from an already-edited file, and nothing in the project records what was removed or why.

Takeaway

Encode every exclusion as a named constant with its reason in a comment, print how many rows it removed and from which group, and report the headline figure before and after.

Mark complete