Unit 09.03: Misuse that is not adversarial, just wrong

Most of the harm comes from reasonable people using a helpful tool slightly outside what it was built for.

Five uses nobody intended and nobody would call an attack

Deciding rather than looking up, pasting whole contracts, treating citations as authority, expecting live numbers, and sending answers unedited.

The code lists all five with why each is a problem.

MISUSES = [
    ("using it to decide a refund rather than to look one up",
     "it was scoped to answer, not to decide"),
    ("pasting a whole contract and asking 'is this ok?'",
     "out of scope, and it will answer anyway"),
    ("treating a cited answer as legal advice",
     "citation is provenance, not authority"),
    ("relying on it for a number that changes daily",
     "the corpus is a document set, not a database"),
    ("copying its answer into a customer email unedited",
     "no review step was designed for that path"),
]
print(f"{'what people do':52} why it is a problem")
for misuse, why in MISUSES:
    print(f"{misuse:52} {why}")

print("""
None of these is an attack. Every one is a reasonable person using a helpful
tool slightly outside what it was built for, and every one produces a bad
outcome nobody intended.

Design for it: state the scope in the interface, not only in a document, and
make the out-of-scope refusals specific enough to redirect rather than just
decline.
""")

The first is the most common and the most consequential. A system scoped to *look up* the refund policy gets used to *decide* a refund, because the output looks like a decision and the person has one to make.

None of these is prevented by a better model or a stricter prompt. They are prevented by the interface being honest about scope, and by out-of-scope refusals that redirect - "an agent can check the account system" rather than a flat decline.

The mistake this prevents

The mistake is stating scope only in a design document. The people misusing the system never read it. Scope has to appear where the system is used, and the refusals have to say where to go instead.

Takeaway

Design for reasonable misuse, not just attacks. State the scope in the interface and make out-of-scope refusals redirect rather than merely decline.

Mark complete