Unit 04.00: Linear layers and activation functions

A linear layer is a matrix multiply plus a bias. Stack two with nothing between them and you still have one linear layer — you can prove it in a few lines.

Why the activation is the whole point

nn.Linear(4, 3) stores a weight of shape (out, in) — note the order, it catches people out — and a bias of shape (out,).

The interesting part is what happens when you stack them. Two linear layers back to back compose into a single matrix multiplication, so the pair has exactly the expressive power of one layer. The code below constructs that equivalent layer explicitly and checks:

import torch
from torch import nn

layer = nn.Linear(4, 3)
print("weight shape:", tuple(layer.weight.shape))   # (out, in) -- not (in, out)
print("bias shape  :", tuple(layer.bias.shape))
print("parameters  :", 4 * 3 + 3)

# Stacking linear layers with no activation collapses to ONE linear layer.
torch.manual_seed(0)
x = torch.randn(5, 4)
two_linear = nn.Sequential(nn.Linear(4, 8), nn.Linear(8, 3))
equivalent = nn.Linear(4, 3)
with torch.no_grad():
    equivalent.weight.copy_(two_linear[1].weight @ two_linear[0].weight)
    equivalent.bias.copy_(two_linear[1](two_linear[0].bias))
print("stacked == single linear:", torch.allclose(two_linear(x), equivalent(x), atol=1e-5))

# The activation is what makes depth mean anything.
for name, fn in [("ReLU", nn.ReLU()), ("Tanh", nn.Tanh()), ("Sigmoid", nn.Sigmoid())]:
    out = fn(torch.tensor([-2.0, 0.0, 2.0]))
    print(f"{name:8}: {[round(v, 3) for v in out.tolist()]}")

The stacked pair and the single layer produce identical output. All those extra parameters bought nothing.

Insert a non-linear activation between them and that collapse becomes impossible. ReLU zeroes negatives, Tanh squashes to (-1, 1), Sigmoid to (0, 1) — each bends the space so the next layer sees something a single matrix could not have produced.

The mistake this prevents

Building a deep stack of nn.Linear layers with no activations and expecting depth to help. The model has the capacity of one linear layer, trains more slowly, and is harder to debug.

Takeaway

Depth without non-linearity is width you paid for and cannot use.

Mark complete