Module 04 Activity

Scenario

The eval set is slow and costs money to run. This activity builds the fast suite that catches most regressions without a model.

What you build

An offline suite covering prompt rendering, schema contracts and every tool refusal branch, running in under a minute.

Steps

1. Test prompt rendering separately from the model, with hostile inputs including braces in user content and an empty context.
2. Replace every exact-match assertion with structural ones, including a hedging check.
3. Test every tool directly, covering every refusal branch. Require the approver as a parameter where an action is gated.
4. Record a golden payload and a type contract, and run them in CI.
5. Split the suite into offline and paid, and report the count and wall clock of each.
6. Confirm the offline suite runs in under a minute on every commit.

Evidence to hand in

• Prompt rendering tests including the braces case.
• The structural assertions replacing exact matches.
• Tool tests covering every refusal branch.
• The golden payload and contract check.
• The offline/paid split with counts and timings.

Review checklist

• The braces-in-user-content case is tested.
• No assertion depends on exact model wording.
• Every tool refusal branch has a direct test.
• approved_by is a required parameter on any gated tool.
• The offline suite runs in under a minute.

Mark complete