Unit 12.01: Running the evaluation set

The run produces four numbers against four bars, and one of them decides the release.

Per dimension, against its own bar

Passed, total, rate and verdict for each dimension.

The code evaluates a forty-case run.

RESULTS = {
    "correctness":       {"passed": 36, "total": 40},
    "grounding":         {"passed": 35, "total": 40},
    "refusal_accuracy":  {"passed": 12, "total": 13},
    "format":            {"passed": 40, "total": 40},
}
BARS = {"correctness": 0.90, "grounding": 0.95,
        "refusal_accuracy": 0.95, "format": 0.99}

print(f"{'dimension':20} {'result':>10} {'rate':>7} {'bar':>7}  verdict")
failures = []
for name, r in RESULTS.items():
    rate = r["passed"] / r["total"]
    ok = rate >= BARS[name]
    if not ok:
        failures.append(name)
    print(f"{name:20} {r['passed']:>4}/{r['total']:<5} {rate:>7.2f} "
          f"{BARS[name]:>7.2f}  {'PASS' if ok else 'FAIL'}")

print(f"\nfailed: {failures or 'none'}")
print("grounding is 0.875 against a 0.95 bar -- this does not ship")

Grounding is 0.875 against a 0.95 bar and the system does not ship. Correctness passed at exactly 0.90, format was perfect, refusal accuracy was fine - so three of four dimensions say yes.

That is the entire argument for measuring dimensions separately. An average of the four is 0.94 and looks like a comfortable pass, and it would ship a system whose grounding is a fifth below target.

The mistake this prevents

The mistake is renegotiating the bar when the run fails. The bar was set before the measurement precisely so that this conversation would not happen - and a bar that moves when it blocks something is not a bar.

Takeaway

Evaluate each dimension against its own bar and let the weakest decide. An average of four dimensions hides the one that fails.

Mark complete