Tools as narrow contracts

A colleague writes a helper called museum_action(text). It can look up dates, edit records and send announcements. The demonstration is short: pass the model's requested action into that helper. The problem appears when a note says “send the correction to everyone.” Nothing at the interface reveals whether this is evidence, a proposed action or permission.

We will expose less power and require more precise inputs. The purpose is not to make the tool description longer. It is to make the difference between permitted and unsupported behavior executable.

Start from one observable operation

Our registry in engine.py contains lookup_catalogue, a synthetic read, and draft, a local controller action. It does not contain a shell tool, arbitrary URL fetch, record update or message operation. The planner proposes a Proposal(name, exhibit_id). It cannot supply a new implementation under a familiar name.

For a lookup, the exhibit argument must exactly match the request card. An EX-001 job cannot quietly look up EX-002. This is an authorization boundary specific to the exercise. A future research workflow might permit a set of related exhibits, but that would need an explicit target-scope rule. Syntactically valid arguments are not automatically in scope.

For a draft, no exhibit argument is accepted. The controller already owns the validated request and fact. Accepting an extra target would create ambiguity about which one controls. A narrow contract can eliminate needless interpretation rather than asking a model to resolve it.

Put validation on both sides

Read the tool-contract visual. A proposal first passes name and target checks. A permitted lookup reserves budget before invocation. Its reply then passes output checks before becoming evidence. The source field is not decoration: it associates the year with a particular fixture version and exhibit.

If the diagram extends beyond the screen, scroll inside it or read the text version below.

Read the diagram as text

• A plausible reply is still rejected when it answers the wrong question
• Aabha AI Academy · original course visual · synthetic workflow only
• Proposal
• Known operation
• only
• Target gate
• Requested exhibit
• only
• Reservation
• Charge attempt
• before call
• Reply gate
• Exact fields, ID
• and version
• Evidence
• Accepted fact
• or safe stop
• EX-002 facts cannot support an EX-001 audit, even when both entries are valid.
• Static two-operation allowlist: synthetic catalogue read and local draft. No generic command executor.

The successful reply has exactly five fields: outcome, exhibit ID, year, source version and source ID. The year must be an integer within the course range, the exhibit must match, and the source reference must match the pinned fixture. A result with an unexpected field is rejected. The contract is intentionally rigid for a small exercise; a real versioned interface would need a documented compatibility policy for new fields.

The synthetic catalogue is an authoritative fixture within the exercise. Validating a source ID's shape does not prove a real source trustworthy, and no cryptographic signature is implemented. If a real service lies consistently about its identity, these local shape checks would not discover that alone.

Work through a plausible wrong answer

An EX-001 request receives year=1986 with exhibit_id=EX-002. Both values are well formed and refer to a real fixture entry. A naive pipeline might compare 1927 with 1986, produce MISMATCH and look successful. Our controller rejects the reply before drafting because the tool answered a different question.

Run test_wrong_tool_output_never_becomes_evidence. SyntheticTool(wrong_output=True) deliberately supplies the wrong exhibit. The result stops with invalid_tool_output, and fact remains absent. That absence matters: later planning sees has_fact=False, so rejected output cannot accidentally become accepted evidence.

Now propose send_email. The name is absent from the registry. Engine.plan stops with unsupported_proposal before the tool's call count rises. A natural-language justification or an apparently urgent note does not add an operation to the registry.

Classify failure by what is known

The tool can return a known temporary failure or an absent exhibit. A temporary read failure may justify another reserved attempt. An absent exhibit means this fixture contains no answer; repetition does not create evidence. A Python exception of an unclassified kind stops without retry.

Avoid treating every exception as “temporary.” That hides errors in arguments, permissions, implementation and data. Avoid logging arbitrary exception strings into learner-facing traces too: a production tool might include sensitive details. The project records a bounded reason code instead. For a real service, protected diagnostics and user-facing explanations would be separate design work.

Anthropic's tool-interface discussion emphasizes clear operation definitions and testing how tools are used. Our lesson makes that concrete with an allowlisted operation and independent output validation; it does not rely on a model obeying descriptive prose.

Practice

1. Write the contract for a future lookup_label_observation read. Specify the allowed exhibit scope, required output evidence and two failure outcomes. Keep it separate from catalogue lookup so the two sources are not confused.
2. Add a tool double whose success reply has the right exhibit but source version CAT-v2 during a CAT-v1 run. Confirm that no draft is produced. Explain why silently accepting “newer” information would violate the current execution contract.
3. Add a test for an extra reply field named instruction. Confirm rejection. Then explain whether dropping that field would, by itself, establish safety when a future model reads natural-language source text.
4. Propose a draft action carrying an exhibit argument after collecting a valid fact. Predict the reason code and verify it. Do not weaken the contract to accommodate the redundant argument.

Check yourself: how does a target-scope check differ from a regular expression? Which component owns budget charging? Why is a well-formed source reference insufficient proof of historical truth?

Keep a contract table and at least one negative test. A useful negative test shows that a plausible but unsupported answer cannot cross the evidence boundary. Testing only a happy-path reply does not demonstrate that property.