Unit 12.04: The model card and the limits you accept

The deliverable is a model card whose prohibitions are longer than its permissions.

Approved, not approved, accepted limits, untested

Intended use, per-slice results, what it is not approved for, and what has not been tested.

The code prints the card.

import json

card = {
    "model": "carton-damage-v3",
    "intended_use": "triage before human inspection, day shift, site A only",
    "measured": {"overall_recall": 0.62, "threshold": 0.35,
                 "chosen_because": "a miss costs 45x a false alarm"},
    "per_slice_recall": {"day shift": 0.66, "night shift": 0.41,
                         "phone A": 0.64, "phone B": 0.29},
    "not_approved_for": ["night shift", "phone model B", "any second site",
                         "identifying staff", "productivity scoring"],
    "accepted_limits": [
        {"limit": "label agreement measured at 67% before re-labelling",
         "mitigation": "edge-case rules written and re-audited",
         "residual": "medium"},
        {"limit": "'wet' class has 5 examples",
         "mitigation": "reported, not claimed as measured", "residual": "high"},
    ],
    "not_tested": ["wet weather", "night shift after new lighting was installed"],
    "review": "re-measure per slice in 3 months or on any camera change",
}
print(json.dumps(card, indent=2))

print("\n`not_approved_for` is longer than `intended_use`, and that is correct")

not_approved_for is longer than intended_use, and that is the correct shape for a model measured honestly. Two slices, one device, one site and two prohibited purposes.

The 'wet' class limit is marked residual high, because five examples support no claim at all. Saying so is stronger than quietly including it in an overall figure.

The mistake this prevents

The mistake is writing the card at the end as documentation. Its fields are decisions - which slices are approved, what the review trigger is - and they should have been made while there was still time to collect more data.

Takeaway

A model card measured honestly has more prohibitions than permissions. Mark unmeasurable classes as unmeasured, and state the review trigger.

Mark complete