Unit 08.04: The optimisation that changed the output

Every optimisation is a change to the system, and the metric you were optimising is never the only one that moved.

Latency halved, grounding fell 13 points

The same configuration change measured on four dimensions rather than one.

The code compares before and after.

BEFORE = {"k": 6, "rerank": True,  "correctness": 0.92, "grounding": 0.96,
          "p95_ms": 1840, "usd": 0.0031}
AFTER  = {"k": 3, "rerank": False, "correctness": 0.89, "grounding": 0.83,
          "p95_ms": 940,  "usd": 0.0014}

print(f"{'metric':14} {'before':>9} {'after':>9} {'change':>9}")
for key in ("correctness", "grounding", "p95_ms", "usd"):
    before, after = BEFORE[key], AFTER[key]
    change = (after - before) / before
    print(f"{key:14} {before:>9} {after:>9} {change:>8.0%}")

print("""
Latency halved and cost more than halved -- an excellent optimisation by any
performance measure.

Grounding fell 13 points, which is the dimension nobody was watching. Every
optimisation must be re-run against the full eval set, because the metric you
were optimising is never the only one that moved.
""")

By any performance measure this is an excellent optimisation: latency halved, cost more than halved. Grounding fell from 0.96 to 0.83, which is the dimension nobody was watching because the work was framed as a performance task.

Correctness only fell three points, which is why a single quality number would have made this look acceptable. The dimension that moved most is the one a composite would have absorbed.

The mistake this prevents

The mistake is treating performance work as exempt from the eval set because no prompt changed. Lowering k and dropping the reranker are changes to retrieval, and retrieval determines grounding - the eval set is the only thing that would have caught it.

Takeaway

Re-run the full eval set after every optimisation, including ones framed as performance work. The dimension you were not watching is the one that moved.

Mark complete