Module 12 Activity

Scenario

The capstone: a traceable RAG assistant with an offline test suite and a quality report that states its limits.

What you build

A complete assistant - scoped, grounded, instrumented, tested offline - and a report a reviewer can challenge.

Steps

1. Write the scope: users, sources with staleness budgets, refusals, a latency budget, and observable success.
2. Build the chain with stable ids, scored retrieval, a measured threshold and a refusal branch in code.
3. Emit one trace per request with versions, what was discarded below threshold, generation cost, and the latency budget comparison.
4. Write the offline suite, forcing the refusal branch with a threshold parameter rather than a chosen query.
5. Report retrieval, citation and refusal separately, with latency percentiles against the budget.
6. List known limits and what you have not tested. Give the report to someone and record their questions.

Evidence to hand in

• The scope with its four checks.
• The chain with its refusal branch and threshold provenance.
• One complete trace.
• The offline suite and evidence it needs no key.
• The three-way report with latency against budget.
• Known limits, untested areas, and the reader's questions.

Review checklist

• The latency budget follows from who uses it and when.
• The refusal is a return value, not a prompt instruction.
• The trace records what was discarded below threshold.
• The refusal test forces the branch with a parameter, not a lucky query.
• The report states the share of requests that miss the latency budget.
• not_yet_tested is populated and specific.

Mark complete