Jaemin's Research Notes

Normalization Placement

2 min read

Normalization Placement

Normalization placement describes where normalization is applied relative to residual branches. The placement can strongly affect gradient flow, depth scaling, and training stability.

For a residual block with sublayer :

Pre-norm keeps a direct identity path from output to input, while post-norm normalizes the residual sum.

Transformer Block

A pre-norm Transformer block is often written as:

The normalization sits before attention and feed-forward sublayers.

Gradient Path

In a pre-norm block:

the residual stream keeps an unnormalized identity path. In a post-norm block:

the gradient passes through normalization after the residual sum. This can affect depth scaling because normalization changes both the forward activation and the backward Jacobian.

Common Patterns

PatternFormTypical Reading
pre-normstable for deep sequence models
post-normoutput is normalized after each block
sandwich normextra stabilization, extra compute
parallel blockattention and FFN share the same residual input
norm-free or scaledresidual scaling without explicit normrelies on initialization and scale control

Comparison Boundary

When comparing architectures, normalization placement is not a minor implementation detail. It changes:

  • trainable depth;
  • learning-rate tolerance;
  • activation scale;
  • compatibility with residual scaling;
  • whether the baseline has the same stability budget.

If a paper changes both architecture family and norm placement, do not attribute all gains to the family name.

Tradeoffs

  • Pre-norm usually improves optimization stability for deep Transformers.
  • Post-norm can make layer outputs more uniformly normalized but can be harder to optimize at depth.
  • Sandwich or hybrid norms add normalization both before and after sublayers.
  • Residual scaling, initialization, and learning-rate schedule interact with placement.

Checks

  • Is the block pre-norm, post-norm, sandwich-norm, or another variant?
  • Is the normalization LayerNorm, RMSNorm, BatchNorm, or GroupNorm?
  • Are residual branches scaled or gated?
  • Is the reported architecture comparable to the baseline after changing norm placement?
  • Does instability appear as depth increases?
  • Is the output before the task head normalized, or only the internal residual stream?
  • Are inference-time normalization statistics identical to training-time behavior?

Graph View

Backlinks