Unit 02.01: Correctness when there is no single answer

Correctness has to be judged on facts present or absent, because the wording will never be the same twice.

Grade the facts, not the string

Decide which facts the answer must contain, then check for each independently.

The code scores four answers on two required facts.

EXPECTED = {"window_days": 7, "applies_to": "individual plans"}

ANSWERS = [
    "Refunds are allowed within 7 days, for individual plans.",
    "You have a week to request a refund on an individual plan.",
    "Refunds are allowed within 7 days.",
    "Refunds are allowed within 30 days.",
]


def score(answer):
    """Check the facts, not the wording."""
    has_window = "7 day" in answer.lower() or "a week" in answer.lower()
    has_scope = "individual" in answer.lower()
    return {"window": has_window, "scope": has_scope}


for answer in ANSWERS:
    result = score(answer)
    verdict = "correct" if all(result.values()) else "incomplete/wrong"
    print(f"{verdict:17} {result}  {answer}")

# The second answer says "a week" and is correct. The third drops the scope
# qualifier and is incomplete rather than wrong. Grading on facts rather than
# strings is what makes both judgements possible.

The second answer says "a week" rather than "7 days" and is correct. The third states the window and drops the scope qualifier - incomplete rather than wrong, which is a distinction worth keeping because the two have different fixes.

This also makes grading reproducible. Two people applying "does it state the window?" agree; two people applying "is this a good answer?" argue, and neither can reconstruct their reasoning six weeks later.

The mistake this prevents

The mistake is grading against a reference answer by similarity. A paraphrase scores lower than a fluent wrong answer that happens to reuse the reference's vocabulary, which is exactly backwards.

Takeaway

Define the facts an answer must contain and check each independently. It handles paraphrase, distinguishes incomplete from wrong, and produces labels two people can agree on.

Mark complete