Unit 11.04: Closing the loop back into the eval set

The health of an eval set is measured by how much of it came from things that went wrong.

Nine cases added, twenty-four written up front

Growth of the set from launch through three incidents and a red-team exercise.

The code tracks the count at each stage.

GROWTH = [
    ("launch",        24, "built from support logs"),
    ("+ incident 1",  26, "stale document quoted as current"),
    ("+ incident 2",  27, "answered an out-of-scope account question"),
    ("+ red team",    31, "injection, disclosure, and two probes"),
    ("+ incident 3",  33, "parse failure defaulted to an empty answer"),
]
print(f"{'stage':16} {'cases':>6}  source")
for stage, count, source in GROWTH:
    print(f"{stage:16} {count:>6}  {source}")

print(f"""
{GROWTH[-1][1] - GROWTH[0][1]} cases added from things that actually went wrong, against {GROWTH[0][1]}
written up front.

That ratio is the health signal. An eval set that has not grown since launch is
either running a system nothing has gone wrong with, or -- far more likely --
one where the failures never made it back into the set.
""")

Nine of thirty-three cases came from real failures rather than from imagination. That ratio is the signal: it means failures are getting back into the set rather than being fixed and forgotten.

An eval set that has not grown since launch is either running a system nothing has gone wrong with, or - far more likely - one where every incident produced a fix and no test.

The mistake this prevents

The mistake is measuring the eval set by its size. Forty invented cases are worth less than twenty-five that include nine drawn from things that actually broke, because the invented ones share the blind spots of the people who built the system.

Takeaway

Track how many cases came from incidents and red-teaming. A set that has not grown is a set where failures are not making it back in.

Mark complete