Unit 06.00: What a convolution actually does to a patch

A convolution asks one question of every patch in the image.

One kernel, one question

The same patch convolved with a vertical-edge kernel and a horizontal-edge kernel.

The code shows the two responses.

import numpy as np
from scipy import signal

patch = np.array([
    [10, 10, 10, 200, 200],
    [10, 10, 10, 200, 200],
    [10, 10, 10, 200, 200],
], dtype=float)

vertical_edge = np.array([[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]], dtype=float)
horizontal_edge = vertical_edge.T

for name, kernel in [("vertical edge", vertical_edge),
                     ("horizontal edge", horizontal_edge)]:
    response = signal.convolve2d(patch, kernel, mode="valid")
    print(f"{name:16} max response {response.max():>7.0f}")

print("\nthe same patch, two kernels, completely different answers")
print("a conv layer learns the kernels; you choose how many")

# One kernel is one question asked of every patch: "how much does this look
# like a vertical edge?" A layer with 64 kernels asks 64 questions, and the
# next layer asks questions about the answers.

The vertical-edge kernel responds strongly; the horizontal one responds barely at all. Same patch, different question, completely different answer.

A convolutional layer with 64 kernels asks 64 such questions at every position, and the next layer asks questions about those answers. That composition is the whole architecture.

The mistake this prevents

The mistake is thinking of a CNN as a black box that looks at images. The first layer is a bank of small pattern detectors, and they are inspectable - visualising them is a genuine diagnostic when a model is not learning.

Takeaway

A convolution is a question asked of every patch. A layer is a bank of such questions, and depth is questions about answers.

Mark complete