Unit 11.00: Reconstructing what happened

The timeline usually contains a finding more actionable than the bug itself.

Detection lag and metric lag

Deploy, first report, metric movement, diagnosis, rollback, recovery.

The code lays out a fifty-minute incident.

import json

timeline = [
    {"t": "10:02", "event": "deploy 2026-07-28.3 (prompt answer-v4, k 6->4)"},
    {"t": "10:14", "event": "first user report: 'it says 30 days'"},
    {"t": "10:31", "event": "grounding metric drops 0.96 -> 0.84 on the dashboard"},
    {"t": "10:40", "event": "trace r-8841 shows retrieval returned 2 chunks, not 6"},
    {"t": "10:52", "event": "rollback to 2026-07-26.1"},
    {"t": "11:05", "event": "grounding recovers to 0.96"},
]
for row in timeline:
    print(f"{row['t']}  {row['event']}")

print(f"""
detection lag : 12 minutes (deploy to first report)
metric lag    : 29 minutes (deploy to the dashboard showing it)
total impact  : 50 minutes

The metric lagged the user by 17 minutes. That is its own finding, and usually
a more actionable one than the bug -- the users noticed before the monitoring
did.
""")

The metric lagged the user by seventeen minutes. A user noticed at 10:14 and the dashboard showed it at 10:31 - so for seventeen minutes the monitoring said everything was fine while it was not.

That is its own finding, and usually a more valuable one than the root cause. Fixing the bug prevents this incident; fixing the metric lag shortens every future one.

The mistake this prevents

The mistake is writing the timeline as a narrative of the fix. The interesting numbers are the gaps: deploy to detection, detection to diagnosis, diagnosis to recovery. Each is a different thing to improve and they have different owners.

Takeaway

Build the timeline and measure the gaps. Detection lag and metric lag are findings in their own right, and improving them shortens every future incident.

Mark complete