Unit 11.03: A golden file that catches a format break

A golden file plus a type contract catches a format break at the commit that caused it.

Recorded response, declared types

Three payloads against a contract.

The code checks each.

import json

GOLDEN = {"category": "billing", "urgency": "high", "account_id": "ACC-1187"}
TYPES = {"category": str, "urgency": str, "account_id": str}


def contract(payload):
    problems = []
    for field, expected in TYPES.items():
        if field not in payload:
            problems.append(f"missing {field}")
        elif not isinstance(payload[field], expected):
            problems.append(f"{field} is {type(payload[field]).__name__}")
    return problems


for candidate in [GOLDEN,
                  {**GOLDEN, "urgency": 3},
                  {k: v for k, v in GOLDEN.items() if k != "account_id"}]:
    problems = contract(candidate)
    print(f"{'PASS' if not problems else 'FAIL'} {json.dumps(candidate)[:52]:54} {problems}")

# Run this in CI against a recorded response. It costs nothing, needs no key,
# and catches a format break at the commit that introduced it rather than when
# a page fails to render.

The middle candidate has an integer where a string belongs. It is valid JSON with every field present, and it breaks the first piece of code that treats urgency as text - somewhere unrelated to the prompt change that caused it.

Running this in CI against a recorded response costs nothing and needs no key.

The mistake this prevents

The mistake is updating the golden file whenever the check fails. Sometimes that is right - the schema genuinely changed. Done reflexively, the test becomes a record of current behaviour and can never fail.

Takeaway

Keep a recorded response and a type contract in CI. Update the golden only when the change was deliberate.

Mark complete