Unit 05.04: Normalise before you count

'North', 'north' and 'North ' are three different wards as far as R is concerned, and your totals will reflect that.

Normalise before you count

Text columns used as categories carry invisible variation: leading and trailing spaces, inconsistent capitalisation, double spaces where somebody hit the bar twice. None of it shows in a printed table, and all of it splits one category into several.

The fix is two functions applied before any grouping. str_squish() removes leading, trailing and repeated internal whitespace. str_to_title() puts capitalisation on a consistent footing.

count() on the raw column is the diagnostic. If a category you expect to see once appears three times in slightly different clothes, you have found the problem before it reaches a total.

This block counts before and after normalising.

suppressPackageStartupMessages({library(dplyr); library(stringr)})

entries <- data.frame(
  ward = c("North", "north", "North ", " NORTH", "South", "South"),
  visits = c(412, 88, 55, 30, 388, 12)
)

cat("Before cleaning, count() sees:\n")
print(count(entries, ward))

tidied <- entries |>
  mutate(ward = str_to_title(str_squish(ward)))

cat("\nAfter trimming whitespace and normalising case:\n")
print(count(tidied, ward))

cat("\nWard totals before:", nrow(count(entries, ward)), "categories\n")
cat("Ward totals after :", nrow(count(tidied, ward)), "categories\n")
cat("North's true total:", sum(tidied$visits[tidied$ward == "North"]), "\n")
cat("The uncleaned version would have reported North as:",
    entries$visits[entries$ward == "North"], "\n")

Before cleaning, count() reports 5 categories for what should be two wards: NORTH, North, North and north are all separate, alongside South. After squishing whitespace and normalising case there are 2. The consequence is in the last two lines: North's true total is 585, while the uncleaned data would have reported 412 — a third of the ward's visits filed under variant spellings and simply absent from the answer.

The mistake this prevents

The mistake is trusting a category column because the printed values look fine. A trailing space is invisible in every printout you will ever look at, and it splits a group in two.

Takeaway

Run count() on every category column before grouping. Squish whitespace and normalise case as a standard step, and check that the number of categories afterwards is the number you expected.

Mark complete