Unit 02.03: Task outputs you can actually check

The best way to write expected_output is to write the checking function first and then describe what it accepts.

The check comes before the contract

A check is a function from output to a list of problems. Writing it first forces the contract to be specific, because a vague contract produces a check you cannot write.

The code checks three candidate outputs against a three-field schema.

import json


def check_output(text):
    """The check you write BEFORE the task runs."""
    problems = []
    try:
        parsed = json.loads(text)
    except json.JSONDecodeError:
        return ["not valid JSON"]
    if parsed.get("decision") not in {"eligible", "not_eligible", "unknown"}:
        problems.append("decision not in the allowed set")
    if not isinstance(parsed.get("days_elapsed"), int):
        problems.append("days_elapsed missing or not an integer")
    if not parsed.get("policy_line"):
        problems.append("no policy line quoted")
    return problems


for candidate in [
    '{"decision": "not_eligible", "days_elapsed": 9, "policy_line": "within 7 days"}',
    '{"decision": "probably not", "days_elapsed": 9, "policy_line": "within 7 days"}',
    'The refund is not eligible because 9 days have passed.',
]:
    problems = check_output(candidate)
    print(f"{'PASS' if not problems else 'FAIL'} {candidate[:52]:54} {problems}")

# Write this function first, then write `expected_output` to describe what it
# accepts. A contract you cannot execute is a preference.

The second candidate is the instructive one. "probably not" is valid JSON, has all three keys, and fails because the decision is not in the allowed set - which is a check you only have because the allowed set exists as data.

The third is plain English and perfectly reasonable prose. It fails at the first hurdle, which is the right outcome: a downstream task expecting structured input cannot consume a sentence, however true.

The mistake this prevents

The mistake is writing the contract as a description and hoping to check it by eye later. Eye-checking does not scale past the first fifty runs, and by then the drift you are looking for is small enough that you will not see it.

Takeaway

Write the checking function first, then write expected_output to describe what it accepts. A contract you cannot execute is a preference.

Mark complete