Unit 04.03: Catching a format break in CI

A contract check against a recorded response catches format breaks the moment they are introduced.

Golden payload, type contract

Record one known-good response and assert that every required field is present with the right type.

The code checks three payloads against the contract.

import json

GOLDEN = {"decision": "not_eligible", "days_elapsed": 9,
          "policy_line": "within 7 days of purchase"}
REQUIRED_TYPES = {"decision": str, "days_elapsed": int, "policy_line": str}


def contract_check(payload):
    problems = []
    for field, expected in REQUIRED_TYPES.items():
        if field not in payload:
            problems.append(f"missing {field}")
        elif not isinstance(payload[field], expected):
            problems.append(f"{field} is {type(payload[field]).__name__}, "
                            f"expected {expected.__name__}")
    return problems


CANDIDATES = [GOLDEN,
              {**GOLDEN, "days_elapsed": "9"},
              {k: v for k, v in GOLDEN.items() if k != "policy_line"}]
for payload in CANDIDATES:
    problems = contract_check(payload)
    print(f"{'PASS' if not problems else 'FAIL'} {json.dumps(payload)[:56]:58} "
          f"{problems}")

# This runs in CI against a recorded response, so it costs nothing and catches
# the break the moment a prompt or schema change introduces it -- rather than
# when a page fails to render in production.

The second candidate has "9" where an integer belongs. It is valid JSON, has every field, and will break the first piece of arithmetic downstream - and the error will surface somewhere unrelated to the prompt change that caused it.

Running this in CI against a recorded response costs nothing and catches the break at the commit that introduced it, rather than when a page fails to render in production.

The mistake this prevents

The mistake is updating the golden payload whenever the check fails. Sometimes that is right - the schema genuinely changed. Done reflexively it turns the test into a record of current behaviour, which is a test that can never fail.

Takeaway

Keep a golden payload and a type contract, and run them in CI. Update the golden only when the schema change was deliberate.

Mark complete