Module 01 Activity

Scenario

You have been asked whether an existing LLM feature is reliable enough to expand. Nobody can currently answer, because nobody has defined what reliable would mean.

What you build

A written reliability definition for a real system: an end-to-end target, derived stage targets, and a quality bar someone could run tomorrow.

Steps

1. Run the same question through the system five times and record the outputs. Judge them on substance and note what differs.
2. Map every stage from request to response and estimate a success rate for each. Multiply them and compare against what anyone assumed.
3. Set the end-to-end target first, then derive the per-stage targets from it.
4. Rank the failure modes by what they cost the user, not by whether your metrics can see them. Say which ones are currently invisible.
5. Write a quality bar naming the set, the dimension, the threshold and the judge. Check that all four are present.
6. List the failures you are already accepting, each with a mitigation and a residual risk.

Evidence to hand in

• Five runs with a substance judgement on each.
• The stage table with the multiplied end-to-end figure.
• The derived per-stage targets.
• The failure ranking, marking which are invisible to current metrics.
• The quality bar with all four components.
• The accepted-failures table.

Review checklist

• The definition is a rate over a set, not 'produces output X'.
• The end-to-end figure is multiplied, not averaged.
• Per-stage targets are derived from the end-to-end target, not set independently.
• At least one currently-invisible failure mode is named.
• Every accepted failure has both a mitigation and a residual risk.

Mark complete