Unit 10.01: Catching the quality drop a model update caused

Your system can get worse without anyone changing anything in it.

Nothing in your code changed

Weekly eval results across a provider-side model version change.

The code tracks four weeks.

WEEKLY = [
    ("2026-07-01", "m-2026-05", 0.92, 0.96),
    ("2026-07-08", "m-2026-05", 0.91, 0.96),
    ("2026-07-15", "m-2026-06", 0.88, 0.89),
    ("2026-07-22", "m-2026-06", 0.89, 0.90),
]
print(f"{'date':12} {'model':12} {'correct':>8} {'grounded':>9}")
for date, model, correct, grounded in WEEKLY:
    print(f"{date:12} {model:12} {correct:>8.2f} {grounded:>9.2f}")

drop = WEEKLY[1][3] - WEEKLY[2][3]
print(f"\ngrounding fell {drop:.2f} on the week the model version changed")
print("correctness fell 0.03 -- within noise. grounding fell 0.07 -- not.")

# Nothing in your code changed. Providers update models, sometimes without a
# version bump you notice, so the eval set has to run on a schedule and not
# only on your own deploys.

Grounding fell seven points the week the model version changed. Correctness fell three, which is within the noise of a forty-case set - so a system watching only correctness would have seen nothing worth investigating.

Nothing in the application changed. Providers update models, sometimes without a version bump anyone notices, which means the eval set has to run on a schedule rather than only on your own deploys.

The mistake this prevents

The mistake is running the eval set only in CI, triggered by your commits. That covers changes you make and misses every change made underneath you - the model, the provider's infrastructure, and the corpus if anyone else can add to it.

Takeaway

Run the eval set on a schedule as well as on deploys. Model updates change your system without touching your code, and the dimension that moves is not always the one you watch.

Mark complete