Evaluate whole trajectories

Two implementations show the same clean draft in a demonstration. One used one lookup and paused for review. The other made several calls, ignored a rejected source and happened to finish correctly. Looking only at the final paragraph hides those differences.

We will evaluate the path as well as the outcome. The supplied runner compares the fixed baseline with an intentionally poor deterministic planner. It does not estimate live model quality or real learner outcomes.

Write the expected result before looking at the run

Open evaluation_cases.json. Eight authored cases cover a match, mismatch, unknown exhibit, one temporary failure, repeated failures, instruction-like inert note, wrong tool output and Boolean year. Each case states expected phase, reason and, where appropriate, finding.

These are regression cases developed with this project. They are not an independently held-out benchmark. A genuine development/evaluation split would reserve unseen cases before tuning, preserve their provenance and avoid fixing expectations merely to match the implementation. The current set makes a useful local correctness check without claiming that stronger experimental design.

Run PYTHONDONTWRITEBYTECODE=1 python3 project/evaluate.py --planner baseline. The report contains case-level outcomes and a summary. Inspect the records when a total surprises you; a single aggregate cannot explain whether a failure came from parsing, evidence, routing or budget.

Keep the denominator honest

The baseline matches all eight authored expectations. Four cases produce review drafts. The other four correctly stop or reject input. Reporting “four successes out of four drafts” would omit every stopping case and conceal whether stops were appropriate.

For this regression question, outcome correctness is matching the expected result across all eight cases. Draft yield is four of eight. Those are different measures. A correct safe stop can improve correctness while decreasing draft yield. Increasing draft yield by accepting invalid facts would make the system worse.

The baseline makes nine actual synthetic calls in total, so the mean across all eight cases is 9/8 = 1.125. This is not token usage, financial cost or real-service latency. The invalid-year case contributes zero calls and stays in the denominator. If you also report cost among drafting cases, name that conditional population separately.

Examine a deliberately bad candidate

Run the same command with --planner repeat. The always-lookup double still handles unknown exhibits, repeated temporary failures, wrong facts and invalid input as expected. It fails the four cases that should reach review because it never proposes a draft. Its result is four correct outcomes out of eight and zero review drafts.

Both planners pass the runner's narrow budget and authority gates: actual calls stay within saved reservations and limits, and no pre-review evaluation run enters approval or simulation. That does not make them equally useful. Safety invariants and task performance should be reported together rather than collapsed into one score.

Read the evaluation visual. It separates case contract, observed trajectory, outcome score, mandatory gates and comparison. A more mature evaluation would add source accuracy, reviewer load, latency distribution and appropriate model behavior tests, with clearly defined units and populations.

If the diagram extends beyond the screen, scroll inside it or read the text version below.

Read the diagram as text

• A good final draft cannot excuse a broken trajectory
• Aabha AI Academy · original course visual · synthetic workflow only
• Case contract
• Expected outcome
• and source
• Run evidence
• Phase, steps, calls
• stop reason
• Score outcome
• Correct result /
• all cases
• Check gates
• Authority and
• budget invariants
• Compare
• Baseline versus
• fixed bad planner
• Eight authored regression cases: baseline 8/8; always-lookup double 4/8. Not a held-out model benchmark.
• Safe stops stay in the denominator. No token, dollar, real learner or production quality claim is supported.

Make failure analysis actionable

For each wrong record, ask where the first unsupported decision appeared. The repeat planner's first lookup is reasonable; the repeated lookup after accepted evidence reveals the useful diagnostic. A final stop reason alone says the budget ended, but the trajectory shows why it ended.

Do not repair a bad score by quietly changing expected REVIEW to STOPPED. An expectation should change when the service contract changes, with a reason and a versioned evaluation set. Otherwise the test stops checking the intended job.

OpenAI's evaluation guidance supports defining task-specific criteria and evaluating systematically. Our eight deterministic cases are a local application of that principle, not evidence of population-level accuracy or a replacement for model and user evaluation.

Practice

1. Recalculate baseline correctness, draft yield and mean calls from the records. Name the denominator and unit for each. Explain why the Boolean-input case remains included.
2. Compare the repeat planner case by case. Identify the four failures and write one causal explanation shared by them. Explain why all_gates_pass can still be true.
3. Add a new independent case with observed year 2100 for EX-002. State expectations before running. Recalculate the denominator and explain whether this is a new development case or a genuinely held-out evaluation.
4. Design, without running a model, three future evaluation cases for a natural-language observation extractor: ambiguity, missing evidence and instruction-like content. Specify what counts as success and what must stop.

Check yourself: can a stopped run be correct? Does 8/8 on this set imply 100% real-world accuracy? Why should actual calls and reserved attempts be separate fields?

Keep the full small result table and a short failure analysis. Report what the system did on this case set, with its limits, instead of turning a regression total into an unsupported reliability promise.