Unit 03.02: Labelling without arguing about it later

A rubric written before labelling produces labels two people agree on. One written afterwards produces an argument.

Properties present or absent

Each label is defined by which properties the answer has, not by a judgement of quality.

The code applies a four-label rubric to three samples.

RUBRIC = {
    "correct": "states the 7-day window AND the individual-plans scope",
    "incomplete": "states the window but omits the scope",
    "wrong": "states any other window, or contradicts the policy",
    "refused": "returns the exact refusal string",
}
for label, definition in RUBRIC.items():
    print(f"{label:12} {definition}")

SAMPLES = [
    ("Refunds within 7 days, individual plans only.", "correct"),
    ("Refunds within 7 days.", "incomplete"),
    ("Refunds within 30 days.", "wrong"),
]
print("\ntwo labellers applying the rubric:")
agree = 0
for text, expected in SAMPLES:
    labeller_a = expected
    labeller_b = expected
    agree += labeller_a == labeller_b
    print(f"  {text:46} A={labeller_a:11} B={labeller_b}")
print(f"agreement: {agree}/{len(SAMPLES)}")

# Write the rubric before labelling, in terms of properties present or absent.
# "Is this a good answer?" produces disagreement you cannot resolve six weeks
# later; "does it state the scope?" produces a yes or a no.

"States the 7-day window AND the individual-plans scope" is checkable by two people independently. "Is this a good answer?" is not, and the disagreement it produces cannot be resolved six weeks later because neither labeller can reconstruct what they were thinking.

The incomplete label is worth having as a category of its own. Collapsed into wrong it hides a distinct failure - the model had the right information and dropped a qualifier - which has a different fix from getting the answer wrong.

The mistake this prevents

Version the set alongside the rubric. Comparing this month's score against last month's is meaningless if cases were added, removed or relabelled in between - and they will have been, because the previous unit's whole point is that the set should grow. Tag each run with the set version, and treat a score comparison across versions as invalid unless you re-run the old version too.

The mistake is labelling first and writing the rubric to describe what you did. The rubric then encodes the labeller's implicit standards, which the next labeller does not share, and inter-rater agreement collapses on the second batch.

Takeaway

Write the rubric in terms of properties before labelling anything, and keep incomplete distinct from wrong. Measure agreement between two labellers on a sample.

Mark complete