Unit 07.02: Choosing a threshold from the cost of each error

0.5 is a default, not an answer.

The cheapest threshold, computed

Sweeping the threshold and pricing the resulting errors.

The code finds the minimum-cost operating point.

import numpy as np

rng = np.random.default_rng(0)
truth = np.array([0] * 940 + [1] * 60)
scores = np.clip(rng.normal(np.where(truth == 1, 0.65, 0.35), 0.18), 0, 1)
COST_FP, COST_FN = 2.0, 90.0

print(f"{'threshold':>10} {'FP':>5} {'FN':>5} {'cost':>9}")
best = None
for t in (0.3, 0.4, 0.5, 0.6, 0.7):
    pred = (scores >= t).astype(int)
    fp = int(((pred == 1) & (truth == 0)).sum())
    fn = int(((pred == 0) & (truth == 1)).sum())
    cost = fp * COST_FP + fn * COST_FN
    if best is None or cost < best[1]:
        best = (t, cost)
    print(f"{t:>10.1f} {fp:>5} {fn:>5} {cost:>9.0f}")

print(f"\ncheapest threshold: {best[0]} at ${best[1]:.0f}")
print("0.5 is a default, not an answer")

# The threshold falls out of the cost of each error. When a miss costs 45x a
# false alarm, the cheapest operating point is well below 0.5 -- and nothing
# about the model changed.

The cheapest threshold sits well below 0.5, because a miss costs 45 times a false alarm. Nothing about the model changed - only where the decision boundary was placed.

This is often the single largest improvement available on a trained model, and it costs one sweep over a validation set.

The mistake this prevents

The mistake is treating 0.5 as neutral. It is the point of equal probability, which is only the right decision boundary when the two errors cost the same - a situation that essentially never occurs.

Takeaway

Choose the threshold by minimising the cost of the errors it produces. It is usually the cheapest available improvement and 0.5 is rarely correct.

Mark complete