Unit 07.01: The confusion matrix and which cell costs most

The confusion matrix shows both error types, and they are not interchangeable.

Four cells, two costs

True and false, positive and negative - plus what each error costs in this specific application.

The code prints the matrix and prices both error types.

import numpy as np
from sklearn.metrics import confusion_matrix

truth = np.array([0] * 940 + [1] * 60)
pred = truth.copy()
pred[:20] = 1
pred[940:970] = 0

tn, fp, fn, tp = confusion_matrix(truth, pred).ravel()
print(f"{'':16}{'predicted ok':>14}{'predicted damaged':>20}")
print(f"{'actually ok':16}{tn:>14}{fp:>20}")
print(f"{'actually damaged':16}{fn:>14}{tp:>20}")

COSTS = {"fp": 2.0, "fn": 90.0}
print(f"\nfalse positives: {fp} x ${COSTS['fp']:.0f} inspection = ${fp * COSTS['fp']:.0f}")
print(f"false negatives: {fn} x ${COSTS['fn']:.0f} customer claim = ${fn * COSTS['fn']:.0f}")
print(f"the {fn} misses cost {fn * COSTS['fn'] / (fp * COSTS['fp']):.0f}x what the "
      f"{fp} false alarms do")

# The two error types are not interchangeable and the matrix shows both. Which
# cell costs more is a business fact, not a modelling one, and it decides
# everything in the next unit.

The 30 misses cost 45 times what the 20 false alarms do. That ratio is a business fact rather than a modelling one, and it decides the threshold in the next unit.

Without the costs, the matrix is four numbers with no way to compare them. With the costs, it is a single figure you can minimise.

The mistake this prevents

The mistake is treating the two error types as symmetric because both are errors. Almost no real application has symmetric costs, and assuming symmetry silently optimises for the wrong thing.

Takeaway

Read the confusion matrix with the cost of each error type attached. The ratio between them is a business fact and it determines the operating point.

Mark complete