Skip to course content
Free LLMOps course

LLMOps for Reliable AI Applications

Module 11 Activity

Scenario

Something broke in production. This activity turns it into a permanent improvement rather than a fixed bug.

What you build

A timeline with measured gaps, a root-cause test, a blameless review with owned actions, and a scheduled check that the fix held.

Steps

  1. Build the incident timeline and measure the gaps: deploy to detection, detection to diagnosis, diagnosis to recovery.
  2. Report the metric lag - how long between users noticing and monitoring showing it - and treat it as a finding.
  3. Write the test for the ROOT CAUSE, not the symptom. Confirm it would have failed in CI before the change shipped.
  4. Run a blameless review asking what made the wrong action look right. End with owned, dated actions.
  5. Set a review date four weeks out and check the metric on it.
  6. Add the resulting cases to the eval set and report what fraction of your set now comes from real failures.

Evidence to hand in

Review checklist