Unit 03.03: Keeping a holdout you do not tune against

Tuning against your eval set produces a number that improves and a system that does not.

Dev improves, holdout does not

Split the cases. Tune against one, and look at the other rarely and on purpose.

The code tracks four changes against both.

CASES = [f"q{i}" for i in range(1, 41)]
DEV, HOLDOUT = CASES[:30], CASES[30:]

history = [
    ("baseline",        0.80, 0.79),
    ("prompt tweak 1",  0.84, 0.79),
    ("prompt tweak 2",  0.88, 0.80),
    ("prompt tweak 3",  0.93, 0.79),
]
print(f"{'change':16} {'dev':>6} {'holdout':>8}  gap")
for name, dev, hold in history:
    print(f"{name:16} {dev:>6.2f} {hold:>8.2f}  {dev - hold:+.2f}")

print(f"\ndev set: {len(DEV)} cases, holdout: {len(HOLDOUT)} cases")
print("dev improved 13 points; holdout moved 0. The tuning fitted the dev set.")

# Without a holdout, "prompt tweak 3" ships as a 13-point improvement. The
# holdout is the only thing that says otherwise, and it only works if you never
# tune against it -- which means looking at it rarely and on purpose.

The dev set gained thirteen points across three prompt tweaks. The holdout moved by one, in the noise. Without the holdout, "prompt tweak 3" ships as a substantial improvement.

What happened is ordinary overfitting: each tweak was chosen because it helped the thirty cases you could see, and the thing being fitted was those thirty cases rather than the task.

The mistake this prevents

The mistake is checking the holdout after every change. That turns it into a second dev set - you are now selecting changes on it too, just more slowly. Look at it when you think you are done, and treat a large dev-holdout gap as evidence the tuning was fitting noise.

Takeaway

Keep a holdout and do not tune against it. A widening gap between dev and holdout is the signal that your improvements are fitting the cases rather than the task.

Mark complete