Skip to course content
Free RAG fundamentals course

Introduction to RAG and Knowledge Assistants

Unit 08.04: Re-running the set after every change

The most useful property of an eval set is not the score. It is the per-case comparison between two runs.

Same total, different system

A change that fixes one case and breaks another leaves the headline pass rate untouched. If the total is all you store, that change looks like it did nothing.

The table below compares a baseline run against a run after a change, per case.

CASE   BASELINE   AFTER THE CHANGE
q1     pass       pass
q2     pass       FAIL   <- newly broken
q3     pass       pass
q4     FAIL       pass   <- newly fixed

baseline pass rate:   3 of 4
after the change:     3 of 4

The headline rate is identical. A change that fixed one case and broke
another looks like no change at all unless you compare case by case.

Both runs score 3 out of 4. The comparison shows q4 newly fixed and q2 newly broken - a completely different system behind an identical number.

The newly-broken list is the one that matters operationally. It is a regression list, produced automatically, before the change ships. Without per-case storage it does not exist, and the regression is discovered later by a user who cannot tell you which change caused it.

The mistake this prevents

The mistake is storing only the summary. It is a small amount of data - one boolean per case per run - and reconstructing it after the fact is impossible, because you no longer have the old code. Store per-case results with every run, alongside the change that produced them.

Takeaway

Compare eval runs case by case, never on the summary. The list of newly broken cases is what turns an eval set from a report card into a regression test.