Unit 05.00: Count first, ask why second, decide third

The first question about a missing value is never how to fill it. It is why it is missing.

Count first, ask why second, decide third

Missing data arrives for reasons, and the reasons are not interchangeable. A form that was never submitted means the visits happened and were not counted. A ward that was closed means there were no visits to count. Filling both with zero asserts the second explanation for both, and one of those assertions is false.

So the order is fixed. Count the missingness, so you know its size. Find out why, which usually means asking whoever produced the file. Only then decide, and record the decision.

Dropping the rows is a legitimate decision, and it is still a decision: it changes what population your result describes.

This block counts the gaps and then looks at what the source says about them.

suppressPackageStartupMessages(library(dplyr))

returns <- data.frame(
  ward   = c("North", "South", "East", "West", "Central"),
  visits = c(412, NA, 502, NA, 388),
  reason = c("", "form never submitted", "", "ward closed", "")
)

# Step one is always counting, never fixing.
cat("Rows:", nrow(returns), "  Missing visits:", sum(is.na(returns$visits)),
    "  Complete rows:", sum(complete.cases(returns)), "\n\n")

# Step two is asking why. These two NAs do not mean the same thing.
print(returns |> filter(is.na(visits)) |> select(ward, reason))

cat("\nMean over what is there :", mean(returns$visits, na.rm = TRUE), "\n")
cat("Rows that mean covers   :", sum(!is.na(returns$visits)), "of", nrow(returns), "\n")

# Filling with zero would assert that West had no visits. It was closed.
cat("Mean if NA became 0     :", mean(ifelse(is.na(returns$visits), 0, returns$visits)), "\n")

Two of five rows are missing visits, leaving 3 complete rows. The reasons differ: South's form was never submitted, West's ward was closed. The mean over what is present is 434, and it covers 3 of 5 rows — a fact worth printing beside it. Filling the gaps with zero drops the mean to 260.4, and that number claims South recorded no visits at all, which nobody believes.

The mistake this prevents

The mistake is na.rm = TRUE used reflexively. It computes over the surviving rows and says nothing about how many that was, so the report quotes an average without saying what it averages.

Takeaway

Count missing values before anything else, find out why they are missing, and state in the output how many rows your summary actually covers. Never let zero stand in for unknown.

Mark complete