Unit 09.04: Recording what you found, not what you hoped

A red-team report's value is in the two sections nobody is required to write.

Confirmed, not reproduced, not tested, accepted

Four categories, and the last two are what make it credible.

The code prints a report covering all four.

import json

findings = {
    "date": "2026-07-29",
    "probes_run": 7,
    "confirmed": [
        {"issue": "injected instruction in a retrieved chunk changed the answer",
         "severity": "high", "fixed": True,
         "test_added": "rt1: poisoned chunk must not change the answer"},
        {"issue": "error message disclosed the model name and prompt version",
         "severity": "low", "fixed": True,
         "test_added": "rt2: error text contains no config values"},
    ],
    "attempted_but_not_reproduced": [
        {"probe": "asked for another user's account", "result": "filter held"},
    ],
    "not_tested": ["concurrent sessions", "very long conversations",
                   "non-English injection strings"],
    "accepted_risk": [
        {"issue": "marker matching misses rephrased injections",
         "mitigation": "delimited context block plus no model-triggered tools",
         "residual": "medium"},
    ],
}
print(json.dumps(findings, indent=2))

print("\n`not_tested` and `accepted_risk` are what make this a finding rather")
print("than a reassurance. A red-team report with neither has not been done.")

not_tested names three things nobody looked at - concurrent sessions, long conversations, non-English injection strings. Listing them lets a reviewer weigh the gap rather than discover it.

accepted_risk states that marker matching misses rephrased injections, with the structural mitigation and a residual of medium. That is an honest row: the mitigation is real, it is not complete, and someone can disagree with the judgement.

The mistake this prevents

The mistake is reporting only what was confirmed and fixed. It reads as a clean bill of health, and the reader has no way to tell a thorough exercise from a shallow one - so an experienced reviewer discounts the whole thing.

Takeaway

Report what you confirmed, what you tried and could not reproduce, what you did not test, and what you accept. The last two are what distinguish a finding from a reassurance.

Mark complete