Unit 08.01: The pipeline, end to end, reporting its cost

Six stages, in the same order every time. The discipline is reporting what each one cost.

The pipeline, end to end

Import with the missing markers declared. Inspect, and count what is wrong before touching it. Clean, with a rule per issue. Summarise at a stated grain, carrying the counts. Visualise with the axis and labels chosen deliberately. Report, saying what the analysis covers and what it does not.

Every stage in this course has been one of these. What makes it an analysis rather than a sequence of commands is that each stage reports its effect: how many rows arrived, how many were dropped and why, how many observations each summary rests on.

Those counts are what let a reader judge the result, and they are the first thing lost when the pipeline is written as one unbroken chain.

This block runs all six stages on a small file with one missing value.

suppressPackageStartupMessages({library(dplyr); library(readr); library(ggplot2)})

# The whole pipeline, end to end, in the order the plan named.
path <- file.path(tempdir(), "returns.csv")
writeLines(c("ward,month,visits,residents",
             "North,Jan,412,12400", "North,Feb,455,12400",
             "South,Jan,388,9800",  "South,Feb,n/a,9800",
             "East,Jan,502,15100",  "East,Feb,498,15100"), path)

raw <- read_csv(path, na = c("", "n/a"), show_col_types = FALSE)   # 1 import
cat("Imported:", nrow(raw), "rows |  missing visits:", sum(is.na(raw$visits)), "\n")  # 2 inspect

clean <- raw |> filter(!is.na(visits))                              # 3 clean
cat("Cleaned :", nrow(clean), "rows |  dropped:", nrow(raw) - nrow(clean), "\n")

by_ward <- clean |>                                                 # 4 summarise
  group_by(ward) |>
  summarise(months = n(),
            per_1000 = round(mean(visits / residents * 1000), 1),
            .groups = "drop") |>
  arrange(desc(per_1000))
print(by_ward)

chart <- ggplot(by_ward, aes(reorder(ward, per_1000), per_1000)) +  # 5 visualise
  geom_col() + scale_y_continuous(limits = c(0, 40)) +
  labs(x = NULL, y = "Visits per 1,000 residents")
out <- file.path(tempdir(), "by-ward.png")
ggsave(out, chart, width = 5, height = 3, dpi = 150)

cat("\nChart written:", file.size(out), "bytes\n")                  # 6 report
cat("Highest rate:", by_ward$ward[1], "at", by_ward$per_1000[1], "per 1,000\n")
cat("South covers", by_ward$months[by_ward$ward == "South"], "month, so its rate is not comparable.\n")

6 rows import with 1 missing visits value; cleaning drops it, leaving 5. The summary puts South top at 39.6 visits per 1,000 — and the very next line says South covers 1 month, so the ranking is not comparable. That sentence is the output of the pipeline as much as the table is. North's 35 and East's 33.1 each rest on two months.

The mistake this prevents

The mistake is a pipeline that prints only the final table. Every count along the way — imported, dropped, grouped — is evidence a reader needs, and none of it is recoverable from the result.

Takeaway

Run the six stages in order and print a count at each one. Make the limitation a line of output, not something you remember to mention.

Mark complete