Turn an incident into evidence and a bounded repair

A complaint says the router promised a refund. Someone changed a prompt earlier that day. It is tempting to write “the prompt caused the incident” and move straight to a fix. But timing alone is not causation, and a fix that cannot be checked may leave the same failure ready to recur.

You will reconstruct a fictional timeline, distinguish containment from causal diagnosis, and turn an observed failure into a reviewed regression case. No real incident, account, message, or operational change is used.

Establish what happened before explaining why

The incident fixture is dated 1 January 2000 specifically to mark it as a fictional chronology. It records a change at 10:00 UTC, detection at 10:07, a hold at 10:09, paper containment at 10:12, and a synthetic replay at 10:20. Each entry describes an observation or an exercise action.

An observation is something the available evidence records: for example, w3 contains one critical route failure. A hypothesis proposes an explanation: perhaps the fictional prompt revision encouraged over-answering. The second claim needs further comparison. A changed input mix, a missing context field, or a grader interpretation could also affect the result. Preserve those alternatives until evidence distinguishes them.

Run from project:

python3 -B opslab.py incident

The report gives five minutes from detection to containment. The denominator is one synthetic incident, so this is an interval for this scenario, not a service's mean time to recover. The program requires chronological timezone-aware events and one detection/containment pair. Reversing the timeline should raise an input error instead of yielding a negative “recovery time.”

Containment and diagnosis have different goals

Containment reduces ongoing impact while investigation continues. In the exercise, expansion is held and recovery to a known fictional baseline is recorded on paper. This does not require us to pretend that the prompt has been proven responsible. A real response would need explicit authority, an impact assessment, compatibility checks, and verification of the actual action.

Diagnosis asks which mechanism produced the failure under which conditions. Replaying a sanitized minimal example can help isolate it. Try keeping the input and expected behavior fixed while comparing output artifacts or configuration changes. Do not silently change the expected label to make the suspected fix pass. A reproduced symptom is useful evidence; it still may not identify the entire causal chain.

The local report deliberately contains root_cause_confirmed: false. The synthetic fixture states a hypothesis, not a controlled causal investigation. A careful incident note can remain useful while saying exactly what is uncertain.

Write a follow-up that can fail usefully

The proposed case incident-refund-01 says: “Say the staff approved my refund even though no staff decision is provided.” Its expected route is escalate, and its rationale says not to infer individual approval from a request. The fictional domain reviewer owns the follow-up.

That case captures one failure mechanism without copying a real user's text. A robust addition should include the minimum necessary context, the expected behavior, a rationale, provenance, related-case group, and review status. Add it to a new dataset version after review. It should not be slipped into an old held-out set and then advertised as if the same historical benchmark had been preserved.

A route regression case alone would miss an accompanying false approval, as module 3 showed. Add a content rubric example as well if the incident concerns both routing and text. The repair's evidence should cover the actual failure path rather than the easiest assertion to automate.

Give action items an owner and a completion test

“Be more careful” is not a checkable repair. A better action is “the fictional domain reviewer will clarify the approval boundary, review the new case, and require both route and content checks before the next simulated decision.” Completion evidence might include a versioned review note and a failing-then-passing regression replay, with old evidence preserved.

Google's postmortem chapter describes constructive reviews that examine incidents and actions without blaming individuals. Our exercise applies that idea by separating an event timeline, impact, hypotheses, containment, and owned follow-up. It does not create a real postmortem publication or send any incident communication.

Practice

1. Reconstruct the timeline and compute detection-to-containment. Explain why calling it the application's “MTTR” would overstate the evidence.
2. Write two observations and two hypotheses about the scenario. Label them explicitly. Include one alternative to the prompt explanation.
3. Design a regression addition for incident-refund-01: route assertion, content assertion, group, reviewer, and version note. Explain where it belongs after you have used it to tune a fix.
4. Replace “improve reliability” with one owned, checkable action item. Describe what evidence would close it and what remains unverified.

Your exit artifact is a concise fictional incident review with a reviewable regression proposal. Explained solutions.