Residual Network
A residual network is a deep architecture built from residual blocks. It was introduced for vision models, but the residual pattern is now a general design principle across CNNs, Transformers, GNNs, and diffusion backbones.
The core block is:
where is usually a small stack of convolutions, normalizations, and activations.
For a sequence of residual blocks:
the representation after layers is:
This additive path makes the network learn refinements around an identity signal rather than replacing the whole representation at every layer.
Why It Matters
- Makes very deep networks easier to optimize.
- Lets layers learn refinements around an identity path.
- Connects directly to residual connections in Transformers and other models.
Block Variants
Basic image residual block:
Bottleneck block:
The bottleneck reduces channel width before the expensive spatial convolution, then restores it.
When shape changes, the skip path uses a projection:
where may be a convolution, linear projection, or pooling/projection pair.
Residual Scaling
Some very deep networks scale the residual branch:
with or learned gating. This controls hidden-state magnitude and improves stability in deep stacks.
Where It Appears
- CNN backbones for image and grid inputs.
- U-Net encoder and decoder blocks.
- Transformer blocks through Residual connection.
- Graph and geometric models that repeatedly refine node or coordinate states.
Checks
- Does the residual branch preserve shape, or does it need a projection?
- Are normalization and activation placed before or after the residual addition?
- Is the residual path scaled, gated, or stochastic?
- Is the skip path identity, projection, downsampling, or cross-stage connection?
- Are residual blocks being compared at equal depth, width, and compute?