Unit 01.04: Look at the object between every step

Beginners write ten lines and then run them. Experienced analysts write one line, look at what came back, and then write the next one.

Look at the object between every step

A pipeline is a sequence of transformations, and each one either did what you expected or did not. Running the whole chain and inspecting only the final table means that when the answer is wrong you have no idea which step broke it.

The habit is to run a step, then look: how many rows now, how many columns, what are they called, do the values look plausible. Three seconds per step, and it localises every bug to the line that caused it.

This matters more in R than in a spreadsheet because R gives you no view of the intermediate state unless you ask for one.

This block keeps each stage as a named object and reports its shape.

# Run a line, look at the object, then write the next line.
suppressPackageStartupMessages(library(dplyr))

visits <- data.frame(ward = c("North", "South", "East"), count = c(412, 388, 502))

step1 <- visits                              # what came in
step2 <- step1 |> mutate(share = count / sum(count))
step3 <- step2 |> arrange(desc(share))

for (nm in c("step1", "step2", "step3")) {
  obj <- get(nm)
  cat(nm, ": ", nrow(obj), " rows x ", ncol(obj), " cols -- ",
      paste(names(obj), collapse = ", "), "\n", sep = "")
}

cat("\nShares sum to:", sum(step3$share), "\n")
# If that had printed 0.98 you would have found the bug at step 2, not in the report.

step1 is 3 rows by 2 columns, step2 is 3 by 3 — the mutate() added share and touched nothing else — and step3 is the same 3 by 3, because arrange() reorders rows without adding or removing any. The shares sum to 1, which is the check worth doing: had it printed 0.98, the fault would be in step 2, and you would know that before writing a word of the report.

The mistake this prevents

The mistake is assuming a step did what its name suggests. filter() quietly removes rows that are NA on the tested column, and a join can multiply rows. Both are invisible unless you look at the row count either side.

Takeaway

After every step, check the row count, the column count and the column names. Add one arithmetic check — a total, a proportion that should sum to one — at the point where the numbers first mean something.

Mark complete