Unit 11.02: Saliency and attribution as clues, not proof

Saliency shows what the output was locally sensitive to. That is genuinely useful and much weaker than it is usually presented as being.

A gradient at one point

Take the gradient of the output with respect to the input. Large magnitude means a small change in that feature moves the prediction a lot — at this specific input.

Here only features 0 and 1 carry signal, so the ranking is checkable:

import torch
from torch import nn

torch.manual_seed(0)
# Only features 0 and 1 matter; 2-5 are noise.
X = torch.randn(400, 6)
y = (3 * X[:, 0] - 2 * X[:, 1]).unsqueeze(1)

model = nn.Sequential(nn.Linear(6, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(500):
    opt.zero_grad()
    nn.MSELoss()(model(X), y).backward()
    opt.step()

# Saliency: how much does the output move when each input moves?
sample = X[:1].clone().requires_grad_(True)
model(sample).backward()
saliency = sample.grad.abs().squeeze(0)

print("feature  saliency")
for i, v in enumerate(saliency.tolist()):
    marker = "  <-真 signal" if i < 2 else ""
    print(f"{i:>7}  {v:>8.3f}{marker}")

print("\nranking:", saliency.argsort(descending=True).tolist())
print("""
Saliency is a local, first-order approximation for ONE input. It shows what the
model responded to here, not what it uses in general and not what causes the
outcome. Treat a high score as a lead worth investigating, never as evidence
that a feature matters.
""")

The two real features rank highest, which confirms the method works when the answer is known.

But read the caveats carefully. This is a first-order approximation, at a single point, for a single example. It describes the model's local sensitivity — not what the model uses in general, and certainly not what causes the outcome in the world.

The mistake this prevents

Presenting a saliency map as an explanation of a decision. It is a lead worth investigating. Treating it as evidence of causation is a claim the method cannot support.

Takeaway

Saliency generates hypotheses. Confirm them by intervening on the feature and measuring what changes.

Mark complete