Unit 07.03: Looking at the mistakes the model was most sure about

Sorting mistakes by confidence puts the informative ones first.

Confident errors mean something was learned wrong

The five errors furthest from the decision boundary.

The code sorts and inspects them for a shared property.

import numpy as np

rng = np.random.default_rng(1)
IMAGES = [{"id": f"img-{i:03d}", "truth": int(i % 20 == 0),
           "score": float(rng.random()), "lighting": "dim" if i % 3 == 0 else "bright"}
          for i in range(60)]
for img in IMAGES:
    img["pred"] = int(img["score"] >= 0.5)

wrong = [i for i in IMAGES if i["pred"] != i["truth"]]
confident = sorted(wrong, key=lambda i: -abs(i["score"] - 0.5))[:5]

print("the five mistakes the model was most sure about:")
for img in confident:
    kind = "false alarm" if img["pred"] == 1 else "MISS"
    print(f"  {img['id']}  score {img['score']:.2f}  {kind:11} lighting={img['lighting']}")

dim_share = sum(1 for i in confident if i["lighting"] == "dim") / len(confident)
print(f"\n{dim_share:.0%} of the confident mistakes are dim images")

# Sorting errors by confidence puts the informative ones first. A confident
# mistake means the model learned something wrong; an uncertain one near the
# threshold is just a hard case.

Four of the five confident mistakes are dim images. That is a pattern, and it is actionable - collect more dim images, or refuse when the brightness is out of range.

An error just below the threshold is a hard case and tells you little. An error at 0.95 confidence in the wrong direction means the model learned a rule that is wrong, which is a different problem.

The mistake this prevents

The mistake is reviewing errors in whatever order they come out. Random errors are dominated by borderline cases, which are the least informative kind - the model was genuinely uncertain and happened to fall the wrong way.

Takeaway

Sort errors by confidence and look for what the confident ones share. A confident mistake means a learned rule is wrong; a borderline one means the case was hard.

Mark complete