Unit 08.03: Precision and recall as the threshold moves

Precision and recall move in opposite directions as the confidence threshold changes, always.

There is no setting that maximises both

The same detections evaluated at four confidence thresholds.

The code reports precision and recall at each.

DETECTIONS = [(0.95, True), (0.91, True), (0.88, False), (0.84, True),
              (0.79, False), (0.71, True), (0.66, False), (0.52, True)]
TOTAL_TRUTH = 8

print(f"{'threshold':>10} {'kept':>5} {'TP':>4} {'FP':>4} {'prec':>6} {'rec':>6}")
for t in (0.9, 0.8, 0.7, 0.5):
    kept = [d for d in DETECTIONS if d[0] >= t]
    tp = sum(1 for _, correct in kept if correct)
    fp = len(kept) - tp
    print(f"{t:>10.1f} {len(kept):>5} {tp:>4} {fp:>4} "
          f"{tp / len(kept):>6.2f} {tp / TOTAL_TRUTH:>6.2f}")

print("\nprecision falls and recall rises as the threshold drops -- always")

# There is no threshold that maximises both. Which end you want is decided by
# the cost of a miss against the cost of a false alarm, exactly as in Module 7.

Lowering the threshold keeps more detections: recall rises, precision falls. Raising it does the reverse. This is mechanical and holds for every detector ever built.

Which end you want is the same question as Module 7's threshold choice - the cost of a miss against the cost of a false alarm - applied to boxes instead of labels.

The mistake this prevents

The mistake is reporting a single precision and recall pair without the threshold that produced them. They are a point on a curve, and the point was chosen.

Takeaway

Precision and recall trade against each other at every threshold. Report the threshold, and choose it from the cost of each error type.

Mark complete