Unit 06.03: Reproducing a run exactly

Reproducing a run means knowing everything that went into it, and the things people forget to record are the ones that change most often.

A fingerprint over every input, including versions

Hash the inputs plus the policy version, the prompt version, the model and the flow version. Two runs with the same fingerprint should behave the same way.

The code fingerprints a support run.

import hashlib
import json

run_inputs = {
    "ticket_id": "T-8841",
    "ticket_text": "I was charged twice on 14 June",
    "policy_version": "support-policies-v4",
    "prompt_version": "reply-draft-v3",
    "model": "some-model-2026-06",
    "flow_version": "support-flow-1.2.0",
}
fingerprint = hashlib.sha256(
    json.dumps(run_inputs, sort_keys=True).encode()).hexdigest()[:16]

print(json.dumps(run_inputs, indent=2))
print(f"\nfingerprint: {fingerprint}")

print("""
Two runs with the same fingerprint should behave the same way. When they do
not, the difference is inside the model step -- which narrows the search
enormously.

The versions are the part people leave out. A run recorded without its policy
version and prompt version cannot be reproduced after either one changes, and
both change often.
""")

When two runs share a fingerprint and behave differently, the difference is inside the model step - which narrows the search enormously and is worth establishing before anything else.

The version fields are the ones left out. A run recorded without its policy version and prompt version cannot be reproduced once either changes, and both change often - usually by someone who did not think of it as a change to the system's behaviour.

The mistake this prevents

The mistake is recording the inputs and not the versions. Six weeks later the ticket text is still there, the prompt has been edited twice, and you cannot tell whether the old behaviour came from the input or from the prompt that no longer exists.

Takeaway

Fingerprint every input including prompt, policy, model and flow versions. Same fingerprint with different behaviour localises the problem to the model step immediately.

Mark complete