Unit 01.03: Setting a bar you could test tomorrow

A quality bar is only useful if someone could run it. Most proposed bars could not.

Set, dimension, threshold, judge

Four components. Drop any one and the bar becomes an aspiration, and aspirations cannot gate a release.

The code sorts five proposed bars by whether they are runnable.

BARS = [
    ("the assistant should be helpful",              False, "not measurable"),
    ("95% of answers are correct",                   False, "no set, no judge"),
    ("on the 40-case set, >=90% correct, >=95% "
     "grounded, p95 under 2s",                       True,  "runnable tomorrow"),
    ("no hallucinations",                            False, "unachievable and untestable"),
    ("refuses >=95% of the 12 unanswerable cases",   True,  "specific and countable"),
]

print(f"{'proposed bar':56} testable?")
for bar, testable, why in BARS:
    print(f"{bar:56} {'yes' if testable else 'NO':4} {why}")

testable = sum(1 for _, t, _ in BARS if t)
print(f"\n{testable} of {len(BARS)} could be run tomorrow")

# A testable bar names the set, the dimension, the threshold and the judge.
# Drop any one and it becomes an aspiration -- and aspirations do not gate
# releases, which is what Module 10 needs them for.

"95% of answers are correct" fails because it names a threshold and no set and no judge - correct according to whom, on which questions? "No hallucinations" fails harder: it is both unachievable and untestable.

The two that pass name all four parts. That is the whole test, and it is worth applying to any quality claim someone hands you.

The mistake this prevents

The mistake is setting the bar before the set exists, on the grounds that the target should come first. A threshold without a set is a number somebody made up, and it gets renegotiated the first time it blocks a release. Build the set, measure where you are, then set the bar.

Takeaway

A testable bar names the set, the dimension, the threshold and the judge. Build the set and measure before choosing the number.

Mark complete