Recover without duplicating work

Time plan: 30 minutes teaching, 35 minutes example, 50 minutes practice and 20 minutes checks. Keep a duplicate ledger and failure matrix. Prerequisite: the route and review design.

What you will be able to do

Distinguish a replay from a revision, limit retries, handle an uncertain write outcome and explain why a successful-looking rollback does not undo every external effect.

Expect repeated events

A sender may deliver the same event twice after losing an acknowledgment. A polling configuration can replay older records. An operator may click Run again. A reliable workflow must consider repeated inputs instead of assuming that each run is new business work.

Idempotency means that repeating an operation has the same intended effect as doing it once. For our review queue, the intended effect is one queue record for one request revision. Use a key such as request_id + revision + operation, for example R101|1|queue_review. Do not add a fresh run ID or timestamp to this key; that makes every retry look new. Keep workflow and source versions in the audit record rather than automatically treating a software update as permission to repeat an old side effect.

The ledger holds the key, a comparison of the material input, current state and destination record reference. Compare as_of, requester_type, item, duration_days, request_kind and message after documented harmless normalisation; exclude event_id and delivery/run timestamps. Preserve the original values. For the small manual lab, record states STARTED, RECORDED and NEEDS_RECONCILIATION. A repeated identical RECORDED key produces DUPLICATE_SKIPPED. The same key with different content is an identity conflict requiring review. A new revision is a new work item, linked to the earlier one.

Watch the ledger across three deliveries

Read these snapshots as a small role-play. The intake operator brings a delivery; the queue operator checks the stable key; the desk reviewer sees only confirmed queue items. This is a serial, manual rehearsal.

Snapshot 1. E004 brings R104 revision 1 with missing duration. Before: the ledger has no R104|1|queue_review entry. The validated business result is NEEDS_INFO. The queue operator writes one question draft Q104-1, confirms it exists and records key R104|1|queue_review, material input “14 October; staff; speaker; duration missing; new; original message,” state RECORDED, destination Q104-1. Q104-1's review state is PENDING.

Snapshot 2. A new fictional delivery E104-repeat brings exactly the same R104 revision 1 details. Before: that key is RECORDED → Q104-1. The queue operator compares the material input and finds no change. New queue writes: zero. After: the same key still points to Q104-1; the audit log adds the new event ID and disposition DUPLICATE_SKIPPED. This extra delivery is a teaching example, not an additional record in Pack A.

Snapshot 3. E011 brings R104 revision 2, now requesting two days. Before: only the revision-1 operation is recorded. The revision-2 key is different. The operator checks the new details, records Q104-2 with DRAFT_READY and PENDING, then records R104|2|queue_review as RECORDED → Q104-2. Q104-1 remains in history and is linked as superseded by revision 2. There are two justified historical queue items for two different request revisions, not two items for the same revision.

Try the partly filled row in the practice cards: R210|1|queue_review exists as STARTED; an identical delivery arrives. Fill disposition ______, new writes ______ and evidence needed ______. Then handle the independent R211 card. Read the material fields, not just the key, before deciding whether it is a replay.

A check is not always a lock

Two workers can both check a ledger, see no existing row and then both create an item. Checking before writing helps in a single-operator rehearsal; it does not prove concurrent safety. Production needs a verified unique destination key, an atomic claim or another proven control. Atomic means the critical claim succeeds as one indivisible operation, so competing workers cannot both win.

The beginner course uses one operator and one item at a time. State this limitation in the runbook. Do not promise “exactly once” delivery from a simple check-then-write design. If your optional tool provides keyed record creation, test its duplicate behaviour and error response in a safe synthetic store before relying on it.

Classify failures before retrying

A temporary read failure may be safe to retry: the source is unavailable, and nothing has been changed. A missing permission will usually not be fixed by repetition. A malformed input needs correction. A timeout after a write is different again: the destination may already have completed the action even though your workflow did not receive confirmation.

Set a bounded recovery policy. For this teaching design, retry a temporary read at most twice after the first attempt, with increasing waits chosen by the operator or platform, while respecting the service's instructions. Record attempt count. If it still fails, hold the item and tell the owner what is known. This is an example policy, not a claim that every vendor uses those exact delays or supports the same error handling on every plan.

Never blindly repeat an uncertain write. First query or inspect the destination using the stable operation key. If the effect exists, record its reference and reconcile the run. If you can establish that it did not occur and the key/control makes a retry safe, permit at most one operator-controlled retry under this teaching policy. The operator must first confirm no effect and a suitable duplicate control; otherwise keep it on hold. This write-retry bound is separate from the read-retry bound. If you cannot establish the outcome, leave NEEDS_RECONCILIATION and escalate. “No success message” is not evidence of “no effect.”

Worked recovery

E010 asks for a two-day speaker and produces a valid draft for R109 revision 1. The workflow sets ledger key R109|1|queue_review to STARTED, then writes review record Q109. The response is lost, so the workflow sees a timeout. It cannot tell whether Q109 exists.

The safe sequence is to mark NEEDS_RECONCILIATION, prevent an automatic second write, inspect the queue for the key, find Q109, check that its content and revision match and update the ledger to RECORDED with Q109. A replay is now DUPLICATE_SKIPPED. One timeout led to one review item, not two.

A dangerous sequence would create Q110 immediately on retry and tell the reviewer both were new. Another dangerous sequence marks the run successful without checking the queue. Recovery means establishing the outcome, not merely hiding the error.

Watch what the operator knows after a timeout

Operator: “I started the Q109 write using R109|1|queue_review.” Ledger snapshot: state STARTED; destination not yet confirmed.

Screen: “Timed out.” Pause here. That sentence tells us confirmation did not reach the operator. It does not tell us whether the write happened. The operator changes the ledger to NEEDS_RECONCILIATION, keeps the same key and records “outcome unknown.” The destination field stays unconfirmed; no second write is attempted.

Operator: “Show the queue item for this exact operation key.” The supplied failure card now reveals Q109 with matching R109 revision 1 content. After inspection, the ledger becomes RECORDED → Q109. The audit note says the acknowledgment was lost and the destination was checked. The queue item's review state is still PENDING.

Now hide the last card. If the queue cannot be inspected, the ledger must remain NEEDS_RECONCILIATION. You can report the blocker without claiming either success or failure. In your practice trace, write the question you need the destination check to answer before opening the recovery evidence card.

The concurrency problem has a similar visible sequence. Operator A checks an empty key. Before A writes, operator B checks the same empty key. Both see “absent,” so both may attempt a write. The gap between checking and writing is the problem. A single-operator rehearsal avoids that race; a production design needs a verified uniqueness or atomic-claim control. The diagram alone cannot supply one.

Retrying the model is not the same as retrying a request

An invalid model response may be repairable with one controlled instruction that repeats the output contract. Keep the original response and the repaired response. Validate the repair again. Do not allow endless “try again until it approves” loops. If the evidence is missing, a new model answer cannot create it. A factual conflict or unsupported request needs review rather than repeated generation.

Similarly, a platform rollback cannot be assumed to unsend a message or undo every connected-app action. Design the review boundary before any irreversible effect. This course uses no outbound sender, so you can learn recovery without creating those consequences.

Read the recovery diagram

The visual 04-safe-recovery.svg follows a write from STARTED to either confirmed RECORDED or NEEDS_RECONCILIATION. Only a destination check can resolve an unknown outcome. Duplicate detection returns the existing record; it does not create a new one. The explanation applies to paper logs as well as a no-code implementation.

Practice

1. Process E001 then E006 in order. Fill the ledger key, state, payload comparison and disposition. Explain why different event IDs do not justify two queue records.
2. Process E004 then E011. Show both revisions and their link. Explain why revision 2 should not be suppressed as a duplicate.
3. Use Failure F-A to write a three-attempt trace: first attempt, first retry and a hypothetical second retry. Stop once the read succeeds. Record what you would do if all three failed.
4. Apply Failure F-B to E010. Write every state transition and the evidence needed before deciding to retry.
5. Classify Failure F-C, F-D and F-F. For each, name the responsible human, safe next action and what must not happen automatically.
6. If you use a tool, inspect its official error-handling settings. Do not assume incomplete-run storage or retries are enabled. Record current settings without changing security or permissions.

Expected artifact: duplicate ledger, recovery trace, failure matrix and an explicit single-operator limitation or verified stronger control.

Check your understanding

M6 Q1. Why does a timeout after a queue write require a different response from a timeout before reading a policy?

M6 Q2. Why does including the current timestamp in the duplicate key defeat replay protection?

M6 Q3. Two workers use the same check-then-write sequence. What can still go wrong, and how is the beginner lab scoped to avoid claiming more than it proves?

Before you continue

You should be able to say “the outcome is unknown” and keep work safely on hold. Next you will measure the system's behaviour instead of relying on a polished demonstration.