Unit 11.03: The model card a reviewer can challenge

A model card's value is in the fields nobody is required to fill in.

Baseline, per-group, gaps, prohibitions

What it does, what it beats, how it varies, where it is weak, and what it must not be used for.

The code prints one.

import json

card = {
    "model": "carton-damage-v3",
    "task": "binary: visible damage present in a carton photograph",
    "trained_on": "warehouse-damage-2026, 1,040 images, one site, daylight",
    "baseline": "majority class 94.0% accuracy, 0% damage recall",
    "measured": {"accuracy": 0.91, "damage_recall": 0.62,
                 "threshold": 0.35, "chosen_because": "a miss costs 45x a false alarm"},
    "per_group": {"day shift": 0.93, "night shift": 0.79,
                  "phone A": 0.92, "phone B": 0.74},
    "known_gaps": ["no wet-weather images", "one site", "night shift under-sampled",
                   "label agreement measured at 67%"],
    "must_not_be_used_for": ["identifying staff", "productivity scoring",
                             "any second site without re-validation"],
    "owner": "warehouse operations",
}
print(json.dumps(card, indent=2))

print("\n`baseline` and `known_gaps` are the two fields a reviewer reads first")

baseline is the field a reviewer reads first - a 91% accuracy against a 94% majority baseline is a very different claim from 91% against 50%.

known_gaps includes the 67% label agreement. Publishing that is uncomfortable and it is the number that explains the model's ceiling, so omitting it means every reader draws the wrong conclusion about what more training would achieve.

The mistake this prevents

The mistake is writing the card to present the model well. A card with no gaps and no baseline reads to an experienced reviewer as an evaluation that was not done, and they discount everything else in it.

Takeaway

Put the baseline, the per-group results and the known gaps in the model card. The uncomfortable numbers are the ones that make the rest credible.

Mark complete