Unit 05.02: Impossible, implausible, and who handles which

Some values are impossible. Others are merely unlikely. The difference decides whether a rule or a person handles it.

Write the rules as code, not as intentions

A validation rule is a statement about what the data can be: counts are not negative, percentages do not exceed 100, dates do not fall in the future. These are facts about the world, and code can check every row against them in one pass.

Keeping the check as a column rather than as a filter matters. A filter removes the bad rows and the evidence together; a TRUE/FALSE column lets you count them, look at them, and decide what to do while they are still in front of you.

Implausible is a different category from impossible. A value can be legal and still be wrong, and no rule can settle that — only somebody who knows the process can.

This block applies two rules and separates the impossible from the merely suspicious.

suppressPackageStartupMessages(library(dplyr))

readings <- data.frame(
  ward     = c("North", "South", "East", "West", "Central"),
  visits   = c(412, -8, 502, 9999, 388),
  pct_seen = c(87.5, 91.2, 103.4, 88.0, 79.9)
)

# Write the rules down as code. A rule in your head is not a rule.
checked <- readings |>
  mutate(
    visits_ok = visits >= 0 & visits < 5000,
    pct_ok    = pct_seen >= 0 & pct_seen <= 100
  )

print(checked)

failures <- checked |> filter(!visits_ok | !pct_ok)
cat("\nRows failing a rule:", nrow(failures), "of", nrow(readings), "\n")
cat("Wards affected:", paste(failures$ward, collapse = ", "), "\n\n")

cat("A negative count is impossible.\n")
cat("A percentage of 103.4 is impossible.\n")
cat("9999 is possible but implausible -- that one needs a human, not a rule.\n")

Three of five rows fail a rule: South with -8 visits, East with 103.4% seen, and West with 9999. The first two are impossible and a rule can act on them. The third passes no plausibility test but breaks no law of arithmetic — 9999 is a number a ward could in principle record, and recognising it as a not-recorded sentinel needs somebody who knows the source system.

The mistake this prevents

The mistake is silently filtering out impossible values. The report then covers fewer rows than it claims, and nobody upstream ever hears that their system is emitting negative counts.

Takeaway

Encode every rule you can state as a check column, count the failures, and report them. Escalate implausible-but-legal values to a human rather than inventing a threshold.

Mark complete