Convolutions, Receptive Fields, and Why Stacked 3×3 Kernels Work

Convolutional networks are built from one operation repeated with different kernels, strides and dilations, and most design decisions about them come down to a single number: how many input pixels can influence one output value. This post implements the operation in NumPy, works out the receptive-field arithmetic, runs a Sobel edge detector on a synthetic image, and then does the parameter count that made VGG-style architectures abandon large kernels.

The operation

What deep-learning libraries call a convolution is, strictly, a cross-correlation: the kernel is slid over the input without being flipped. For a single-channel image $x$ and a $k \times k$ kernel $w$ ,

y [i, j] = \sum_{u = 0}^{k - 1} \sum_{v = 0}^{k - 1} w [u, v] x [i \cdot s + u, j \cdot s + v],

with stride $s$ and whatever zero-padding was applied to $x$ beforehand. The output size along each axis is $⌊ (n + 2 p - k) / s ⌋ + 1$ for input size $n$ and padding $p$ . A loop over the $k^{2}$ kernel entries, rather than over output pixels, keeps the NumPy version short and fast:

import numpy as np

def conv2d(x, k, stride=1, padding=0):
    """Cross-correlation of a 2-D array with a 2-D kernel (what CNNs call 'convolution')."""
    if padding:
        x = np.pad(x, padding)
    H, W = x.shape
    kh, kw = k.shape
    oh, ow = (H - kh) // stride + 1, (W - kw) // stride + 1
    out = np.zeros((oh, ow))
    for i in range(kh):                      # k*k shifted, strided slices, each weighted
        for j in range(kw):
            out += k[i, j] * x[i:i + stride * oh:stride, j:j + stride * ow:stride]
    return out

For a $128 \times 128$ input and a $3 \times 3$ kernel this gives a $128 \times 128$ output with padding=1, $126 \times 126$ without, and $64 \times 64$ with stride=2, padding=1, matching the formula.

TIP

Because the kernel is not flipped, a learned CNN kernel is "the pattern the unit responds to", read directly. If you ever port a hand-designed filter from signal-processing literature, which assumes true convolution, flip it in both axes first; for symmetric kernels such as Gaussians the distinction does not matter, for Sobel-type derivative kernels it flips the sign.

Edge detection on a synthetic image

A $128 \times 128$ synthetic image with a bright square, a mid-grey disc, a thin diagonal line, a horizontal brightness ramp and a little noise is enough to see what derivative kernels do. The Sobel kernels approximate the horizontal and vertical derivatives with a bit of smoothing across the other axis:

G_{x} = (\begin{matrix} - 1 & 0 & 1 \\ - 2 & 0 & 2 \\ - 1 & 0 & 1 \end{matrix}), G_{y} = G_{x}^{⊤} .

sx = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], float)
gx = conv2d(img, sx, padding=1)
gy = conv2d(img, sx.T, padding=1)
mag = np.hypot(gx, gy)

Synthetic image and its Sobel responses — Left to right: the synthetic input, the horizontal-derivative response (vertical edges light up with a sign that encodes the direction of the step), the vertical-derivative response, and the gradient magnitude. The gentle ramp in the background produces a weak constant response in Gx and nothing in Gy.

Two details carry over to learned networks. First, the sign of the response matters: a bright-to-dark step and a dark-to-bright step give opposite signs, which is why networks follow convolutions with a non-linearity rather than taking the magnitude. Second, the ramp shows that a derivative kernel responds to gradients, not to objects; "edges" are a property of the kernel's shape, not of the scene.

Receptive-field arithmetic

The receptive field of a unit is the set of input pixels that can change its value. For a stack of layers with kernel sizes $k_{l}$ , strides $s_{l}$ and dilations $d_{l}$ , the receptive-field side length $r_{L}$ after $L$ layers obeys

r_{L} = r_{L - 1} + (k_{L} - 1) d_{L} \prod_{i = 1}^{L - 1} s_{i}, r_{0} = 1 .

Where the recurrence comes from

Think of the output of layer $L - 1$ as a grid whose cells are spaced $j_{L - 1} = \prod_{i < L} s_{i}$ input pixels apart (the "jump"). A kernel of size $k_{L}$ with dilation $d_{L}$ touches $k_{L}$ such cells spread over $(k_{L} - 1) d_{L}$ steps of that grid, so it extends the receptive field by $(k_{L} - 1) d_{L} j_{L - 1}$ input pixels. Padding shifts where the receptive field is centred but not its size; stride at layer $L$ affects only later layers, through the jump.

With stride 1 and no dilation the recurrence is linear: $r_{L} = 1 + L (k - 1)$ . Stacking $3 \times 3$ kernels grows the receptive field by two pixels per layer, a $5 \times 5$ by four, a $7 \times 7$ by six. Striding or dilating changes the picture completely, because every later layer inherits the multiplied jump.

def receptive_field(kernels, strides, dilations=None):
    dilations = dilations or [1] * len(kernels)
    r, jump, out = 1, 1, [1]
    for k, s, d in zip(kernels, strides, dilations):
        r += (k - 1) * d * jump
        jump *= s
        out.append(r)
    return out

Receptive field size versus depth for several kernel configurations — Receptive-field side length after each of eight layers (log scale). Plain 3×3 stacks grow linearly; one stride-2 layer at the start doubles the growth rate of everything after it; exponentially increasing dilation reaches 511 pixels in eight layers with no loss of resolution.

The dilated configuration ( $d_{l} = 2^{l}$ ) is the trick behind WaveNet-style and segmentation architectures: a receptive field of 511 in eight layers while the output stays at full resolution, whereas plain $3 \times 3$ layers reach only 17.

Why two 3×3 layers instead of one 5×5

Two stacked $3 \times 3$ layers see exactly the same $5 \times 5$ window as one $5 \times 5$ layer. The difference is the cost and the non-linearity count. With $C$ input and $C$ output channels and no biases:

Receptive field	Option	Weights	C=64	C=256	Non-linearities
5 × 5	one 5 × 5	$25 C^{2}$	102,400	1,638,400	1
5 × 5	two 3 × 3	$18 C^{2}$	73,728	1,179,648	2
7 × 7	one 7 × 7	$49 C^{2}$	200,704	3,211,264	1
7 × 7	three 3 × 3	$27 C^{2}$	110,592	1,769,472	3

A one-dimensional cross-section of the two options. Both outputs depend on the same five input pixels; the stacked version uses 28 % fewer weights for the same receptive field and inserts an extra non-linearity.

The stacked option has fewer parameters and more non-linearities, so it is both cheaper and more expressive; the price is that the composed linear map is constrained (a product of two $3 \times 3$ filters is a $5 \times 5$ filter of a particular low-rank form, which is why the non-linearity in between matters). The same argument, carried to its conclusion, is the design rule of VGG: use only $3 \times 3$ kernels and get large receptive fields from depth.

The effective receptive field is smaller than the arithmetic says

The recurrence counts every pixel that can influence the output. How much each pixel actually influences it is a different question. Repeatedly applying a $3 \times 3$ averaging kernel to an impulse shows the answer: the influence profile is a repeated convolution of box filters, which converges to a Gaussian, so pixels near the edge of the theoretical receptive field contribute almost nothing.

Normalized sensitivity of the centre output to each input offset for stacks of 1, 2, 4 and 8 box filters. The nominal receptive field of the 8-layer stack is 17 pixels wide, but the outer pixels have a few percent of the centre's weight.

Luo et al. measured this on trained networks and found the same Gaussian-like profile, with an effective radius growing like $\sqrt{L}$ rather than $L$ . Practically, this means that a network whose nominal receptive field just covers the object is probably not seeing all of it, which is one of the arguments for strided or dilated layers, and later for attention.

Summary

conv2d is a sum of $k^{2}$ shifted, strided, weighted slices; that view also explains the output-size formula.
Receptive fields grow additively with kernel size and multiplicatively with stride and dilation.
Two $3 \times 3$ layers replace one $5 \times 5$ with $18 C^{2}$ instead of $25 C^{2}$ weights and one more non-linearity; three replace a $7 \times 7$ with $27 C^{2}$ instead of $49 C^{2}$ .
The effective receptive field is Gaussian-shaped and considerably smaller than the nominal one.