Unit 11.00: Scoping the assistant and its refusals

The capstone starts where the course did: with a written scope. Everything downstream - chunk size, k, threshold, eval composition - follows from it.

Who asks, from what, and what is never answered

A scope needs four things: who the users are, what the answers come from, what the assistant never answers, and what success looks like in a sentence someone outside the team would accept.

Below is a scope for a support-agent assistant.

{
  "users": "support agents, during a live conversation",
  "answers_from": [
    "support policy documents",
    "product FAQ"
  ],
  "never_answers": [
    "questions about a specific customer's account",
    "anything requiring a judgement about a complaint",
    "questions whose answer is not in the retrieved documents"
  ],
  "latency_budget_ms": 800,
  "refusal_string": "The available documents do not cover that.",
  "success": "agent resolves the query without opening the policy document themselves"
}

users says "support agents, during a live conversation," and that phrase alone sets the latency budget in the next field. An assistant consulted between calls could take five seconds; one used mid-conversation cannot.

success is the field that keeps the project honest: the agent resolves the query without opening the policy document themselves. It is observable, it is not a model metric, and it can be false while every eval score looks excellent.

The mistake this prevents

The mistake is writing the scope after the prototype. A scope written afterwards describes what you built; every constraint in it will be one the system already satisfies, and it will rule nothing out. Its value comes entirely from being written while options are still open.

Takeaway

Write the scope first, including what the assistant must never answer and what success means in observable terms. Every technical decision later in this capstone is downstream of it.

Mark complete