By Anand Pai V
A better-looking answer does not establish a better prompt. Keep the cases fixed, protect an untouched test set, and inspect which individual cases improve or regress.
You revise a prompt and the next answer looks better. That gives you a candidate worth testing. To discover whether the revision helps reliably, compare both versions on the same cases under the same conditions.
The difficult part is keeping the experiment honest while you learn from failures. If every disappointing case changes the prompt, the scoring rule, or the test set, the final score becomes hard to interpret.
Separate practice from the final check
Imagine building a tool that extracts the requested seat quantity from a short message. Its output must be a whole number or unknown. When a message changes an earlier quantity, the latest requested quantity wins. If no quantity is stated, the tool must abstain.
Write the expected answers before comparing prompts. Include number words, revised quantities, missing quantities, and irrelevant questions. Anthropic's evaluation guidance recommends task-specific cases, edge cases, and exact-match grading where answers are unambiguous. Claude evaluation guidance
Suppose you collect 60 cases. You might use 40 for development and reserve 20 as a holdout. These numbers illustrate the separation; they are not a reliable sample-size recommendation. Keep paraphrases of the same original message together so a near-copy does not appear in both groups.
Use development cases to rewrite the prompt. Once you choose two versions, freeze them before opening the holdout results. Repeatedly tuning against a test set leaks information into the choice being evaluated, a problem documented in model selection. The same separation is useful when choosing prompts. Scikit-learn evaluation guidance
Change one thing you can name
Version A might say: “Return the seat quantity requested in the message as an integer, or unknown if it is missing. Return nothing else.”
Version B adds: “Interpret number words and use the latest explicitly requested quantity when a message revises an earlier one.”
Your hypothesis is specific: the added instruction should improve number-word and revision cases. Keep the model version, other instructions, generation settings, available context, output limit, and grading logic unchanged. Run each case in a fresh conversation so earlier examples do not provide accidental hints.
Record case ID, expected answer, prompt version, raw output, and pass/fail. Decide acceptable normalization in advance. For this task, trimming surrounding whitespace is reasonable; silently extracting a digit from an otherwise invalid paragraph changes the output requirement.
Read changes case by case
The following outputs are invented solely to demonstrate scoring. They are not live-model results or evidence about any provider.
| Case | Message | Expected | A | B |
|---|---|---|---|---|
| 1 | Reserve 4 seats. | 4 | 4 | 4 |
| 2 | Please book two seats. | 2 | unknown | 2 |
| 3 | Move my booking to Friday. | unknown | unknown | 1 |
| 4 | We booked 3 seats; now make it 5. | 5 | 3 | 5 |
| 5 | What does a seat cost? | unknown | unknown | unknown |
| 6 | Book a pair of seats. | 2 | unknown | unknown |
A passes three of six cases. B passes four. The paired comparison shows two fixes, one regression, two shared successes, and one shared failure.
That detail matters. B handles cases 2 and 4 better but invents a quantity in case 3. If your decision rule forbids newly invented quantities on abstention cases, B fails that rule despite its higher total. Decide such a rule before seeing which prompt benefits from it.
Six examples teach the calculation; they cannot establish broad reliability. For a real evaluation, include enough representative cases to support the decision's stakes. If outputs vary across runs, predeclare the same repeat count for both versions and retain all results. Three runs of one case still cover only one kind of input.
Transfer exercise
On 20 untouched cases, A passes 16 and B passes 17. B fixes three cases A missed but breaks two that A passed. One broken case invents a missing quantity. Your predeclared rule requires a higher pass count and zero new invented quantities.
Should B replace A? What should happen if you revise B after inspecting these failures?
Answer
B fails the decision rule: its net gain of one comes with a prohibited regression. Keep the paired error list so the failure remains visible.
After you revise B using those cases, that set becomes development evidence for the revision. Preserve it as a regression suite, but obtain a fresh untouched set for a new final comparison. A trustworthy experiment records both what improved and what the improvement cost.