Unit 09.00: Attacking your own application

An afternoon spent attacking your own system finds more than a month of testing it the way it was designed to be used.

Seven probes, none of them exotic

Out-of-scope questions, rephrasings, long prefixes, instruction overrides, other users' data, malformed input, and prompt disclosure.

The code lists all seven with what each tests.

PROBES = [
    ("ask for something outside the corpus",   "does it refuse or invent?"),
    ("ask the same thing five ways",           "does the answer stay consistent?"),
    ("paste 8,000 words before the question",  "does the instruction survive?"),
    ("ask it to ignore its instructions",      "does the system prompt hold?"),
    ("ask about another user's account",       "does the access filter run?"),
    ("send malformed unicode and control chars", "does anything crash?"),
    ("ask for the system prompt",              "does it disclose?"),
]
print(f"{'probe':44} what it tests")
for probe, tests in PROBES:
    print(f"{probe:44} {tests}")

print(f"""
{len(PROBES)} probes, each runnable in a few minutes by hand before anything is
automated. Run them yourself before someone else does.

The value is not that each is exotic. It is that none of them is in your eval
set, because your eval set was built from questions people wanted answered.
""")

None of these requires special skill or tooling. What they have in common is that none is in your eval set - because the eval set was built from questions people wanted answered, and these are questions people ask when they are testing, confused, or hostile.

The 8,000-word prefix probe is the one that surprises teams. Instructions early in a long context lose influence, so a system prompt that holds reliably at 500 words may not at 8,000 - and long pasted content is normal user behaviour rather than an attack.

The mistake this prevents

The mistake is deferring red-teaming until there is time for a proper exercise. These seven take an afternoon, and running them badly now finds more than running them thoroughly after launch.

Takeaway

Run the probes yourself before someone else does. They are cheap, none is exotic, and none of them appears in an eval set built from real questions.

Mark complete