Unit 10.04: The design note a reviewer can challenge

A design note that nobody can disagree with is not a design note. It is a description of what was built, written afterwards.

Every field is a decision someone can contest

Scope, chunking, metadata, permissions, retrieval parameters, refusal wording, escalation triggers, evaluation, known limits. Each one is a choice with alternatives, and each should be written so a reviewer can say "I would have done that differently, and here is why."

Below is such a note for the running example.

{
  "scope": "answers questions from published support policies",
  "out_of_scope": [
    "individual account details",
    "legal advice",
    "anything not retrieved"
  ],
  "chunking": "one chunk per policy statement, split on sentence boundaries",
  "metadata": [
    "id",
    "source_document",
    "updated",
    "visibility",
    "section"
  ],
  "permissions": "set intersection on user groups, applied before ranking",
  "retrieval": {
    "k": 6,
    "filters": [
      "section",
      "updated_after"
    ],
    "rewrite": true
  },
  "refusal": "exact string when nothing is retrieved above threshold",
  "escalation": [
    "chunks disagree",
    "top two within 0.02",
    "unsupported claim detected"
  ],
  "evaluation": "24 cases, 8 expecting refusal, scored on retrieval and citation separately",
  "known_limits": [
    "single corpus",
    "no multilingual support",
    "staleness not auto-detected"
  ]
}

Every field here is a decision someone can disagree with. That is the
point -- a design note with nothing challengeable in it is a description.

known_limits is the field that separates a design note from a pitch. "Single corpus, no multilingual support, staleness not auto-detected" tells a reviewer where the system will fail before they have to find out.

escalation is the field most often missing entirely, and its absence is the strongest signal that a system has only two outcomes. A reviewer scanning for it learns more in five seconds than they would from a page of architecture description.

The mistake this prevents

The mistake is writing the note for approval rather than for review. Notes written to be approved omit limits, present parameters without alternatives, and describe evaluation in the abstract. They pass, and the review produces nothing, which is a waste of everyone's afternoon.

Takeaway

Write the design note so every field can be argued with, and put the known limits in it. A reviewer who cannot find anything to challenge has not been given a design to review.

Mark complete