Worked Example: A Policy Assistant
Map the separate roles of sign-in, retrieval, a language model, rules, human review, sending, and logs.
Meet Policy Guide
Let us study a made-up workplace service called Policy Guide.
An employee asks:
Can I claim the cost of a taxi home after an evening client meeting?
The service follows these steps:
- It checks that the employee is signed in.
- It searches approved travel policies.
- It picks passages that may answer the question.
- It gives the question and passages to a language model.
- The model writes a draft answer.
- A rule sends questions about exceptions to a policy officer.
- The officer sees the question, passages, and draft.
- The reply is sent only after the officer approves or changes it.
Calling this only an AI chatbot would hide most of the system.
Look at each part
Sign-in
The sign-in check shows which account is asking the question. It can also control which information the person may see.
This is not part of the language model.
Search and retrieval
The search part finds passages in the policy collection.
It may miss a useful passage. It may also return the wrong one.
Language model
The model uses the question and passages to write a draft.
The draft may be correct. It may also miss a condition or add a claim that is not in the policy.
Routing rule
A fixed rule sends exception questions to an officer.
The model did not need to learn this rule. A person wrote it into the process.
Human review
The officer can compare the draft with the policy. The officer can edit, reject, or approve it.
This review is useful only if the officer has enough time, information, power, and a real way to stop a wrong reply.
Sending and logs
The app sends the approved reply. It may also save a record of the review.
These are more parts of the full system.
Is it an agent?
The service above follows a set process and waits for approval. Using many parts does not automatically make it an agent.
Now imagine a later version. It can choose which policy tools to use, ask follow-up questions, create a claim form, and continue until the case is complete.
That version looks more like an agent. It works towards a goal and takes actions over several steps.
We would still need to ask:
- What can it do?
- What needs approval?
- What data can it read?
- When does it stop?
- Who is responsible for the result?
Apply TRACE
Task: help answer a policy question.
Result: policy passages, a draft, and an approved reply.
Actual method: sign-in, retrieval, a language model, a fixed rule, officer review, sending, and logs.
Controls: approved policies, routing of exceptions, and approval before sending.
Evidence gap: we do not know how the service was tested or how strong the review process is.
Best description: a system that combines retrieval, generative AI, rules, and human approval. The facts do not yet prove that it is an agent.
Which description is better?
A. Policy Guide uses advanced AI to understand policy.
B. Policy Guide finds passages in approved policies, uses a language model to write a draft, and requires officer approval for exceptions.
Answer: B. It tells us what the parts do. It does not claim that the system understands policy like a person.
Remember
- A product may contain search, models, rules, apps, logs, and people.
- Each part can fail in a different way.
- Many steps do not always mean that a system is an agent.
- Human review must be real, not just a label.
You are now ready for the main practice activity.