Unit 10.02: Comparing two runs of the same input

Two runs of the same input that behave differently are informative only if you recorded everything that could have differed.

Diff the config, then the output

With retrieval identical and one configuration field changed, attribution is immediate.

The code diffs two runs and identifies what changed.

RUN_A = {"prompt_version": "answer-v3", "model": "m-2026-06", "k": 4,
         "retrieved": ["c1", "c7"], "answer": "7 days [c1]"}
RUN_B = {"prompt_version": "answer-v4", "model": "m-2026-06", "k": 4,
         "retrieved": ["c1", "c7"], "answer": "about a week [c1]"}

differences = {k: (RUN_A[k], RUN_B[k]) for k in RUN_A if RUN_A[k] != RUN_B[k]}
print("same input, two runs. differences:")
for field, (a, b) in differences.items():
    print(f"   {field:16} {a!r}  ->  {b!r}")

changed_config = [f for f in differences if f in {"prompt_version", "model", "k"}]
print(f"\nconfiguration that changed: {changed_config}")
print(f"retrieval identical: {RUN_A['retrieved'] == RUN_B['retrieved']}")
print("so the answer difference is attributable to the prompt version")

# Comparing two runs is only useful if every input including versions is
# recorded. With retrieval identical and one config field changed, the
# attribution is immediate.

Retrieval was identical, prompt_version differs, so the answer difference is attributable to the prompt. That took one comparison rather than an afternoon of hypotheses.

It only works because both runs recorded their versions. Without prompt_version the two runs look identical in their inputs and different in their outputs, which reads as model non-determinism - and sends you looking for a problem that is not there.

The mistake this prevents

The mistake is recording the model version and not the prompt version. The model changes a few times a year and the prompt changes weekly, so the field you omitted is the one that explains most differences.

Takeaway

Record every input including prompt, policy and chain versions. A difference you cannot attribute reads as model non-determinism, which is the wrong thing to go looking for.

Mark complete