Unit 01.03: Reading a vendor demo critically

A demo is a recording of a good day. Six questions turn it into something you can evaluate.

What the happy path does not show

Failure handling, loop protection, approvals, cost, input selection, and out-of-scope behaviour.

The table below lists all six with the usual honest answer.

ask the demo                                   answer
what happens when the tool call fails?         usually: the demo has no failure
what stops it if it loops?                     usually: nothing shown
who approved the action we just saw?           usually: nobody
what did that cost in tokens?                  usually: not displayed
was that input chosen or typical?              usually: chosen
what does it do with a question outside scope? usually: not demonstrated

6 questions. A demo that answers all six is a system;
one that answers none is a recording of a good day.

None of these is a hostile question. They are the properties that decide whether the system survives real inputs, and they are absent from a demo precisely because a demo is optimised to show capability rather than robustness.

"Was that input chosen or typical?" is the one to ask first. Everything else is downstream of it - a demo run on a curated input tells you the system can handle that input, which was never in doubt.

The mistake this prevents

The mistake is evaluating a demo on what it accomplished. Accomplishment is the easy part to arrange. Ask instead what it does when a tool call fails, and watch whether the answer is a designed behaviour or a shrug.

Takeaway

Evaluate demos on failure handling, stopping, approvals, cost and out-of-scope behaviour - none of which a demo shows. Start with whether the input was chosen.

Mark complete