Unit 04.03: Loss choice for numeric, binary, and multiclass targets

The loss defines what "wrong" means. Choosing one that does not match your target type produces a model that optimises the wrong thing, usually without an error.

Matching the loss to the target

MSE squares the error, so one large mistake outweighs several small ones — good when big errors genuinely matter more, bad when your data has outliers. L1 treats them proportionally.

BCEWithLogitsLoss takes raw logits for binary targets. CrossEntropyLoss takes raw scores and class *indices* — not one-hot vectors.

import torch
from torch import nn

# Numeric target -> MSE. Penalises large errors quadratically.
pred, true = torch.tensor([[2.0], [3.0]]), torch.tensor([[2.5], [5.0]])
print("MSE :", round(nn.MSELoss()(pred, true).item(), 4))      # (0.25 + 4)/2
print("MAE :", round(nn.L1Loss()(pred, true).item(), 4))       # (0.5 + 2)/2

# Binary target -> BCEWithLogitsLoss, fed RAW logits.
logits = torch.tensor([[2.0], [-1.0]])
labels = torch.tensor([[1.0], [0.0]])
print("BCEWithLogits:", round(nn.BCEWithLogitsLoss()(logits, labels).item(), 4))

# Applying sigmoid yourself and then BCELoss gives the same number but is
# less numerically stable at the extremes.
print("sigmoid + BCE:", round(nn.BCELoss()(torch.sigmoid(logits), labels).item(), 4))

# Multiclass -> CrossEntropyLoss. Targets are class INDICES, not one-hot.
scores = torch.tensor([[2.0, 0.5, 0.1], [0.2, 3.0, 0.4]])
targets = torch.tensor([0, 1])
print("CrossEntropy :", round(nn.CrossEntropyLoss()(scores, targets).item(), 4))

# The classic error: passing one-hot targets to CrossEntropyLoss.
try:
    nn.CrossEntropyLoss()(scores, torch.tensor([[1, 0, 0], [0, 1, 0]]))
except Exception as exc:
    print("one-hot targets ->", type(exc).__name__)

Compare the MSE and L1 numbers: the same two errors (0.5 and 2.0) give 2.125 under MSE and 1.25 under L1, because squaring magnifies the larger one.

The final block shows what happens when you pass one-hot targets to CrossEntropyLoss — it fails, which is the good case. The silent failures are worse.

The mistake this prevents

Passing probabilities to BCEWithLogitsLoss. It applies a sigmoid internally, so your already-squashed values get squashed again. Training still runs; the model just learns badly.

Takeaway

Numeric to MSE or L1, binary to BCEWithLogits, multiclass to CrossEntropy with class indices.

Mark complete