Unit 01.04: Deciding what your app must never get wrong

The highest-value artefact in an AI app project is a short list of things it must never do, with how each is enforced.

Must-never, and the enforcement column

Three prohibitions, each with an enforcement that is not a prompt instruction.

The code prints the boundary document.

import json

boundary = {
    "app": "internal support-reply drafter",
    "must_never": [
        "send anything to a customer without a human pressing send",
        "state a policy figure that is not in the retrieved policy text",
        "process a message containing card numbers",
    ],
    "acceptable_failure": [
        "producing a draft that needs editing",
        "declining a request it cannot ground",
    ],
    "how_enforced": {
        "no auto-send": "the send button is in the UI, not in the code path",
        "no invented figures": "answer must quote a policy line; checked",
        "no card numbers": "regex reject before the prompt is built",
    },
}
print(json.dumps(boundary, indent=2))

print("\nevery `must_never` has an enforcement that is not a prompt instruction")

# Write this before building. Each line becomes a test, and the enforcement
# column is what stops the list being aspirational.

The enforcement column is what stops the list being aspirational. "No auto-send" is enforced by the send button living in the UI and not in any code path the model can reach - which is a fact about the architecture rather than a request.

The card-number rule is enforced by a regex before the prompt is built, so the number never reaches the provider or your logs.

The mistake this prevents

The mistake is writing the must-never list and enforcing it in the system prompt. A prompt is read by a component that has already been handed the thing you want withheld, and the instruction competes with everything else in the context.

Takeaway

Write the must-never list first and give every line an enforcement that is not a prompt instruction. Each line then becomes a test.

Mark complete