Unit 10.03: Recovering a run instead of restarting it

When a long run fails near the end, restarting it repeats everything that already worked - including the expensive parts and any external calls.

Edit the state at the failure point and continue

Recovery means fixing the state where it went wrong and resuming from there. Everything upstream stays done.

The code pauses a run, has an operator correct the state, and resumes - counting how many times the fetch actually ran.

from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import InMemorySaver

CALLS = []


class State(TypedDict):
    fetched: list
    processed: str


def fetch(state: State):
    CALLS.append("fetch")
    return {"fetched": ["a", "b", "c"]}


def process(state: State):
    return {"processed": f"processed {len(state['fetched'])} items"}


g = StateGraph(State)
g.add_node("fetch", fetch)
g.add_node("process", process)
g.add_edge(START, "fetch")
g.add_edge("fetch", "process")
g.add_edge("process", END)
app = g.compile(checkpointer=InMemorySaver(), interrupt_before=["process"])

config = {"configurable": {"thread_id": "recover-1"}}
app.invoke({"fetched": [], "processed": ""}, config)   # stops before process
print("paused with:", app.get_state(config).values["fetched"])

app.update_state(config, {"fetched": ["a", "b", "c", "d"]})   # operator fixes it
print("resumed    :", app.invoke(None, config)["processed"])
print(f"\nfetch ran {len(CALLS)} time(s) -- recovery did not repeat it")

# Restarting re-runs fetch, re-pays for it, and may get different data.
# Recovery edits the state at the point of failure and continues.

fetch ran once. The operator added a missing item and the run continued into process with the corrected list.

A restart would have re-run fetch - paying for it again, and possibly getting different data, since sources change between runs. That last point is the one people miss: a restart is not a repeat, because the world moved underneath it.

The mistake this prevents

The mistake is treating restart as the standard remedy because it is the one button available. Building the recovery path - inspect, edit, resume - costs little once checkpointing exists, and it is the difference between an incident that costs a minute and one that costs a full re-run.

Takeaway

Recover by editing the state at the failure point and resuming. A restart repeats completed work, re-pays for it, and may produce different data because the sources have changed.

Mark complete