Rehearse a change and its recovery path

Your local report is green. The next question is not merely how to expose everyone to the new version. It is how to learn about a change while preserving a way to stop and recover. A release plan needs conditions, observations, an owner, and a known recovery target before it needs a button.

This module uses a pure state-machine simulation. It changes no traffic, account, infrastructure, or deployment. The words shadow, canary, and expand name fictional stages in a paper exercise. A genuine canary involves actual controlled exposure; this replay is not one.

Start with prerequisites, not optimism

Open plan in operations.json. It names a fictional owner, a baseline artifact, an offline_pass boolean, recovery readiness, and three stages. The boolean is deliberately supplied teaching input, not a computed authorization. It lets us study the stage logic separately. The capstone will reconcile that assumption against all the evidence, including semantic failures and blocked repeated comparisons.

A real plan would identify exactly which change is proposed, its dependency compatibility, its authorized exposure scope, and what “rollback” can restore. A pointer to an old prompt is insufficient if a new database format or irreversible external action cannot be undone. Our router has no real side effects; the simulator's recovery is just a named target in a returned dictionary.

If either the offline prerequisite or rollback_ready is false, the simulator returns HOLD before evaluating a stage. That is a useful boundary: preparation cannot be retroactively invented after the failure has already expanded.

Give each stage fresh evidence

The fictional sequence is shadow, then canary, then expand. The simulator requires exactly that order and a fresh observation window for each stage. Reusing w1 three times would make the state advance without learning anything new. It also requires windows to advance in the supplied chronological order.

In a real design, shadow processing would need to suppress external side effects and protect any duplicated data; it is not automatically harmless because its response is hidden. A canary would expose an authorized subset and compare its own observations with an appropriate control. Expansion would happen only after the prior stage's defined review. None of those actions occurs here.

Google's canarying chapter describes comparing canary and control populations and the risks of attributing time-related changes to a release. This is why our paper exercise uses matched candidate and control windows rather than comparing a quiet weekend baseline with a busy weekday candidate. The example still cannot establish causal effects: both windows and their counts are invented.

Inspect the stop conditions

Run from project:

python3 -B opslab.py rollout

The shadow and canary stages continue through the fictional w1 and w2 observations. At expand, w3 triggers RECOVER. It contains a critical route failure, fails operational limits, and has eight bad candidate requests against two bad control requests. Because both denominators are 100, the control-relative excess is six requests, exceeding the toy allowance of three.

Equal counts are a specific assumption of this fixture. With unequal exposure volumes, subtracting counts would be misleading; compare appropriately defined rates and consider uncertainty. Likewise, if the candidate and control share a broken dependency, a small relative difference does not prove health. An absolute good-fraction floor remains necessary.

The simulator returns immediately on a stop condition. It does not process later expansion after RECOVER. If evidence is healthy but there are fewer than 100 observations, or the fictional reviewed flag is absent/false, it returns HOLD. A known critical failure takes priority over waiting for enough requests: low sample count is not a reason to ignore an observed unacceptable event.

Understand the three kinds of outcome

HOLD means a prerequisite or adequate reviewed evidence is missing. RECOVER means a stop condition was observed in this simulation, and the response names the known baseline artifact. COMPLETE_SIMULATION means all supplied stage conditions passed. Even that final state carries live_authorization: false.

Recovery itself must be checked. A real recovery plan would confirm that the target is available and compatible, scope the action, verify its result, and determine whether previously completed actions need separate remediation. No state-machine result can grant permission to change a real service. Here the exit artifact is a reviewable plan, not an instruction executed against Aabha.

Practice

1. Predict the three stage events before running the command. Explain why w3 requires recovery even if someone is pleased with the earlier windows.
2. In a copied plan, set rollback_ready to False; then try setting the first step's reviewed to False with recovery ready. Explain the difference between the two holds.
3. Copy the plan and reuse w1 for the second stage. Run the function and explain why this is invalid evidence rather than a successful fast rollout.
4. Write a recovery checklist naming an artifact, owner, compatibility check, verification step, and one possible side effect that an old prompt alone cannot undo. Keep it hypothetical.

Your exit artifact is a stage decision log and a paper recovery plan. Explained solutions.