Unit 08.04: What a hung workflow should look like

Without a per-node time budget, a hung run and a slow run are the same observation, and nobody investigates either until a user complains.

Elapsed against an expected budget

Each node gets an expected duration. A run exceeding it on a node is not necessarily broken, but it is a fact worth surfacing rather than a spinner that keeps turning.

The code checks three in-flight runs against their budgets.

from datetime import datetime, timedelta

now = datetime(2026, 7, 29, 10, 30)
RUNS = [
    {"id": "r1", "node": "research", "started": now - timedelta(seconds=8),  "budget": 60},
    {"id": "r2", "node": "draft",    "started": now - timedelta(seconds=95), "budget": 60},
    {"id": "r3", "node": "review",   "started": now - timedelta(minutes=40), "budget": 300},
]

print(f"{'run':4} {'node':10} {'elapsed':>9} {'budget':>7} status")
for run in RUNS:
    elapsed = (now - run["started"]).total_seconds()
    over = elapsed > run["budget"]
    status = "OVER BUDGET -- surface it" if over else "running"
    print(f"{run['id']:4} {run['node']:10} {elapsed:>8.0f}s {run['budget']:>6}s {status}")

print("""
A per-node time budget turns "it is still going" into a fact. Without one, a
hung run and a slow run look identical, and nobody investigates until a user
complains.

What the user should see when the budget is exceeded: the node it is stuck on,
how long it has been there, and a way to cancel. Not a spinner that has been
turning for forty minutes.
""")

r2 is at 95 seconds against a 60-second budget and r3 at 40 minutes against five. Both are over; neither would be visible without the budget column, because both are simply "still running."

What the user should see when a budget is exceeded is specific: which node it is stuck on, how long it has been there, and a way to cancel. That is three fields you already have, and it converts an unexplained wait into something a person can act on.

The mistake this prevents

The mistake is setting one global timeout for the whole run. It has to be generous enough for the slowest legitimate case, which makes it useless for detecting a node that hangs early. Budget per node, where the expected durations actually differ.

Takeaway

Give each node a time budget and surface runs that exceed it, showing the node, the elapsed time and a cancel option. A single global timeout is too coarse to detect anything useful.

Mark complete