Unit 07.04: Failures that produce no error at all

The failure that costs most is the one your monitoring cannot see.

Four runs, one invisible failure

Whether the run reported an error, whether it produced output, and whether that output was correct.

The table below classifies four runs.

RUN   RAISED AN ERROR   PRODUCED OUTPUT   CORRECT   VISIBLE TO MONITORING?
r1    no                yes               yes       n/a -- it worked
r2    yes               no                no        yes
r3    no                yes               NO        NO
r4    no                no                no        yes

  r3 produced a confident, well-formed, wrong answer. No error, no empty
  output, nothing an error rate would count. It is the most expensive
  failure here and the only one monitoring cannot see.

r3 produced a confident, well-formed, wrong answer. No reported error, no empty output, nothing an error rate counts. It is the most expensive failure in the list and the only one that is completely invisible operationally.

That is why Module 8 exists. A case set with known-correct answers is the only instrument that detects this class, and no amount of logging, alerting or error tracking substitutes for it.

The mistake this prevents

The mistake is treating the error rate as the reliability measure because it is the one you already have. It measures the failures that announce themselves, which are the cheap ones - the user retries, or asks a person.

Takeaway

Silent wrong answers produce no error and no empty output, so operational monitoring cannot see them. Detecting them needs a case set with known answers, which is the next module.

Mark complete