Unit 10.03: Rolling back when the gate fails

Rollback needs three things ready before you need them, and the third is the one people miss.

Previous version, previous config, fast path

The old build still deployable, the configuration that went with it, and a way to do it without a full release cycle.

The code evaluates a failing deploy and decides.

DEPLOY = {
    "version": "2026-07-28.3",
    "previous": "2026-07-26.1",
    "changed": ["prompt answer-v3 -> answer-v4", "k 6 -> 4"],
    "gate_results": {"correctness": 0.91, "grounding": 0.88, "newly_broken": 2},
    "bars": {"correctness": 0.90, "grounding": 0.95, "newly_broken": 0},
}
failures = [k for k, v in DEPLOY["gate_results"].items()
            if (v < DEPLOY["bars"][k] if k != "newly_broken"
                else v > DEPLOY["bars"][k])]
print(f"deploy {DEPLOY['version']}")
for change in DEPLOY["changed"]:
    print(f"   changed: {change}")
print(f"\nfailed gates: {failures}")
print(f"action: roll back to {DEPLOY['previous']}")

print("""
Rollback needs three things ready before you need them: the previous version
still deployable, the config that went with it (prompt AND policy version), and
a way to do it without a full release cycle.

Prompts and retrieval settings are configuration, so they must roll back with
the code or you get the old binary and the new prompt.
""")

Prompts and retrieval settings are configuration, and they must roll back with the code. Roll back the binary alone and you get the old application running the new prompt - a combination that was never tested and is often worse than either version.

The failing gates here are grounding and newly-broken-cases. Correctness passed, which is exactly the situation the extra gates exist for: one metric looked fine and the release should not ship.

The mistake this prevents

A staged rollout is what makes rollback rarely necessary. Send the new version a small share of real traffic first and compare its metrics against the rest: real traffic surfaces distribution problems an eval set cannot, because your set has the questions you thought of. A regression then affects a small population and is visible in a comparison rather than in a complaint.

The mistake is treating prompt changes as content rather than code - edited in a dashboard, deployed instantly, and outside the rollback path entirely. A prompt change is a behaviour change and belongs under the same version control and the same rollback as everything else.

Takeaway

Keep the previous version and its configuration deployable together, with a fast path. Prompts and retrieval settings are code and must roll back with it.

Mark complete