Module 11 Activity

Scenario

The suite should run offline, cover the paths a real model will not produce on demand, and fail only for reasons worth acting on.

What you build

An offline test suite with injected fakes, structural assertions, and separate scoring for retrieval, citation and refusal.

Steps

1. Refactor chain construction into functions that take the model as a parameter.
2. Use scripted fake responses to drive the retry path and the refusal path deliberately.
3. Replace every exact-match assertion with structural ones, including a hedging check.
4. Build an eval set scoring retrieval, citation and refusal separately, with unanswerable cases included.
5. Audit the suite for brittle and flaky tests. Delete or move them, and say where the timing assertions went.
6. Confirm the whole suite runs with no key, no network and no cost.

Evidence to hand in

• Chain-building functions taking the model as a parameter.
• Tests driving the retry and refusal paths via scripted responses.
• The structural assertions replacing exact matches.
• The three-column eval results.
• The brittle/flaky audit with what moved where.
• Evidence the suite runs offline.

Review checklist

• No chain is built at import time with a real model.
• The retry and refusal paths are both driven deliberately.
• No assertion depends on exact model wording.
• The eval set includes unanswerable cases.
• Timing assertions live in monitoring, not the suite.

Mark complete