Recover and replay deliberately

The synthetic catalogue returns a fact, then the process fails before saving it. On restart the last saved phase is IN_FLIGHT. The desk cannot tell from that checkpoint whether a reply was received. Replaying the whole job from its first line might hide this uncertainty and reset its budget.

We will preserve what is known and charge uncertainty conservatively. The exercise uses a read-only fixture, so deliberate re-reading can be safe. That does not authorize the same strategy for a message, purchase or record update.

Reserve before doing work

Engine.plan records a pending operation ID and increments tool_calls before any invocation. Despite its name, that counter means reserved attempts in the advanced engine. An abandoned reservation still counts. The counter is an upper bound on permitted actual invocations, not proof that every reservation ran.

invoke accepts only RESERVED. It checkpoints IN_FLIGHT before calling the synthetic tool. After a known result, it records evidence or a classified failure. Repeated invocation of an IN_FLIGHT operation is rejected. This prevents a caller from guessing that a pending operation never ran.

Read the checkpoint timeline. The important gap lies between the tool returning and the result checkpoint committing. A snapshot transaction cannot make an unrelated remote action part of the same atomic operation. The lab's SQL update is atomic for one local row; no remote action is involved.

If the diagram extends beyond the screen, scroll inside it or read the text version below.

Read the diagram as text

• A result can be uncertain even after an invocation began
• Aabha AI Academy · original course visual · synthetic workflow only
• Reserve
• Save attempt ID
• and charge
• Mark in flight
• Save before
• invocation
• Read returns
• Synthetic read
• has happened
• Crash gap
• Result not yet
• checkpointed
• Recover
• Stop or abandon
• read explicitly
• An abandoned reservation stays charged. A new read needs another reservation.
• Export/import simulates state transfer. RAM SQLite plus an in-process JSON string is not durable recovery.

Make the failure reproducible

Start a run, call plan, then call invoke(..., crash_after_read=True). The synthetic tool performs one read and the method raises SimulatedCrash before recording its reply. Inspect the store: phase IN_FLIGHT, one reservation, no accepted fact.

This is fault injection: we deliberately interrupt at a known point. A real process did not unexpectedly crash. The test makes a particular recovery path reproducible; it does not exhaust all disk, operating-system or distributed failure modes.

The default recover response is STOPPED/uncertain_tool_outcome. An explicit abandon_read=True permits a narrower alternative: mark that old read reservation abandoned and return to ACTIVE. The next lookup needs a new reservation and consumes more budget. The original reserved attempt is never subtracted.

Transfer state without inventing durability

Checkpoints.export creates a JSON string containing a schema name and saved snapshots. restore constructs a new in-memory SQLite store from that string. In advanced_demo.py, the first store closes and a second engine resumes the transferred state.

That is a restart simulation. The JSON string remains in the same Python process; nothing is automatically saved to disk, encrypted, authenticated or backed up. Only import snapshots produced by this course program. The restore helper is not a hardened parser for arbitrary uploaded files.

The point is semantic continuity. The new engine sees the same request identity, source version, policy, phase, revision and consumed allowance. A real persistent system would need an appropriate storage lifecycle, migration policy, access control, integrity checks and restore testing. Those deployment mechanisms are outside this course lab.

Work through a recovery decision

After the injected crash, export and restore the snapshot. The new synthetic tool object has performed zero calls, but the snapshot still carries one reservation. Abandon the old read deliberately, reserve another, invoke it and draft. The final paused state has two reservations. Across the original and new tool objects, two actual synthetic reads occurred.

If you instead reset tool_calls to zero during restore, the restarted process could repeatedly spend its full allowance. If you accept the uncheckpointed reply from memory as a fact after pretending the process restarted, you have used evidence that the restored state did not contain. Both shortcuts make the demonstration look smoother by removing the failure you were meant to handle.

For a real write, abandoning and repeating might duplicate an action. AWS's idempotency discussion explains why caller identity and atomic handling of effects matter. The course does not implement an external idempotency service. A checkpoint row alone cannot guarantee exactly-once messages or payments.

Replay identity is not restart identity

Engine.start with the same input, source and policy returns its existing checkpoint without restarting work. Changed material under the same request ID raises a conflict. Restore should preserve this behavior. The run's revision says which snapshot is current; the pending operation ID identifies one charged attempt inside that run.

These identifiers solve different questions. Reusing the run ID does not mean every internal tool call has the same identity. Giving a duplicate delivery a new run ID discards the deduplication contract. Naming and testing those semantics is more useful than claiming a generic “retry-safe system.”

Practice

1. Reproduce the injected crash and export/restore sequence. Record original actual calls, restored actual calls and saved reservations separately. Demonstrate that calling invoke again while IN_FLIGHT is blocked.
2. Repeat with Policy(max_tool_calls=1). Abandon the uncertain read, then try to continue. Explain why the next proposal stops even though the restored tool object has made no calls.
3. Compare default recovery with deliberate read abandonment. Write a decision note naming what is known, what remains uncertain and why re-reading this fixture is permitted.
4. Consider a future send_announcement write whose connection fails after dispatch. Propose two pieces of evidence needed before retrying. Do not implement or send anything.

Check yourself: what transaction does the SQLite context actually cover? Is an exported string a durable checkpoint? Why must uncertain reservations remain charged?

Keep a failure table and a tested trace. Recovery should make uncertainty visible and preserve the contract, even when that means stopping instead of finishing the comparison.