Unit 08.04: Re-running the set after every change

The useful output of a case set is not the score. It is the list of cases that used to work and no longer do.

Same total, different behaviour

Per-case comparison between two runs.

The table below computes the fixed and broken lists.

CASE   BEFORE   AFTER
c1     pass     pass
c2     pass     FAIL     <- newly broken
c3     FAIL     pass     <- newly fixed
c4     pass     pass
c5     pass     pass
c6     pass     pass

before: 5/6      after: 5/6      headline change: none

  Identical totals, different system. c2 is the vague request -- the kind
  that is hard to get right and easy to lose. The broken list is the
  decision; the total hides it.

Both runs score five out of six. One case was fixed and one was broken - and the broken one is c2, the vague request, which is precisely the kind of case that is hard to get right and easy to lose.

Reading only the total, this change looks neutral and ships. Reading the broken list, it is a regression in the behaviour that was hardest to build.

The mistake this prevents

The mistake is storing only the aggregate. It is one boolean per case per run, it cannot be reconstructed once the old code is gone, and without it the broken list does not exist.

Takeaway

Store per-case results and compare run to run. The broken list is the decision; an unchanged total can hide a regression in exactly the behaviour that was hardest to build.

Mark complete