Module 12 Activity

Scenario

The capstone: a reliability report for a real LLM application, ending in a verdict and the gates that make it repeatable.

What you build

A complete reliability report - quality bar, eval results, operational measurements, documented failure modes, a verdict, and proposed gates.

Steps

1. Choose the application and state the consequence of an error. Define the bar naming the set, the judge, a zero-tolerance metric and operational bounds.
2. Run the eval set and report each dimension against its own bar. Let the weakest decide.
3. Measure latency at p50, p95 and p99, cost per successful request, and the rates - stating which are by design.
4. Document known failure modes with a tested column, and populate the untested rows honestly.
5. Reach a verdict. If it does not ship, say so.
6. Propose the gate set that turns this measurement into a standing check, and list next steps.

Evidence to hand in

• The application, the consequence of error, and the four-part bar.
• Eval results per dimension against per-dimension bars.
• Latency percentiles, cost per successful request, and the rates.
• The failure-mode table including a tested column.
• The verdict.
• The proposed gates and next steps.

Review checklist

• The bar names the judge and includes a zero-tolerance metric.
• Each dimension is compared against its own bar, not an average.
• Latency is reported at three percentiles.
• Rates that are by design are labelled as such.
• Untested failure modes are listed rather than omitted.
• The verdict follows from the bars set beforehand, and the gates are specific enough to run.

Mark complete