Feed-Forward Network

A feed-forward network maps each position independently through dense layers and nonlinearities. In Transformers, the feed-forward network is the token-wise MLP between attention blocks.

For token matrix $X\in {\mathbb{R}}^{T\times d}$:

$$H=\sigma (X{W}_1+b_1)$$ $$Y=H{W}_2+b_2$$

where $W_1\in {\mathbb{R}}^{d\times d_{\mathrm{ff}}}$, $W_2\in {\mathbb{R}}^{d_{\mathrm{ff}}\times d}$, $d_{\mathrm{ff}}$ is the hidden expansion width, $\sigma$ is an activation function, and $Y\in {\mathbb{R}}^{T\times d}$ preserves the residual dimension.

Transformer FFN

A pre-norm Transformer usually applies:

$$X_{\mathrm{out}}=X+\mathrm{FFN}(\mathrm{LN}(X))$$

The FFN does not mix tokens by itself. Token mixing comes from attention, recurrence, convolution, or message passing. The FFN changes the representation at each token after information has been mixed.

Parameter and Compute Scale

For hidden width $d$ and expansion width $d_{\mathrm{ff}}$, a two-layer FFN has approximately:

$$2d{d}_{\mathrm{ff}}+d_{\mathrm{ff}}+d$$

parameters including biases. If $d_{\mathrm{ff}}=4d$, the FFN parameter count is roughly:

$$8d^2$$

per block. This is often a large fraction of Transformer parameters and compute.

For a sequence length $T$, token-wise FFN cost scales as:

$$O(Td{d}_{\mathrm{ff}})$$

unlike attention’s dense token-mixing cost, which often includes an $O(T^{2}d)$ term.

Gated FFN

Many modern models use a gated variant:

$$H=\sigma (X{W}_g+b_g)\odot (X{W}_u+b_u)$$ $$Y=H{W}_o+b_o$$

Here $\odot$ is elementwise multiplication. The gate controls which hidden features pass through, linking FFNs to gating and mixture-style computation.

Common gated variants include GLU-style, GEGLU-style, and SwiGLU-style FFNs. The paper note GLU Variants Improve Transformer is the canonical reference in this wiki for reading those variants as Transformer architecture blocks:

$$\mathrm{SwiGLU}(x)=\mathrm{SiLU}(x{W}_g)\odot x{W}_u$$

The gate changes both capacity and parameter count, so comparisons should state the expansion ratio and whether the total parameter budget is matched.

Role in Different Architectures

Architecture	FFN Role	What Mixes Positions
Transformer	token-wise channel mixing after attention	self-attention
MLP-Mixer style model	channel mixing and sometimes token mixing through separate MLPs	token-mixing MLP
Graph network	node-wise update after aggregation	message passing
CNN/ViT hybrid	channel mixing after spatial mixing	convolution or attention
MoE model	sparse expert FFNs	router selects experts

The phrase “MLP block” can mean channel-only mixing or token mixing depending on the architecture. State the axis.

Canonical Papers

Paper	Why It Matters
MLP-Mixer	makes token-axis and channel-axis MLP mixing explicit in a vision backbone
GLU Variants Improve Transformer	canonical note for gated Transformer FFN variants

Checks

• Does the block mix tokens, or only transform each token independently?
• What is the expansion ratio $d_{\mathrm{ff}}/d$?
• Is the activation ReLU, GELU, SiLU, GLU, or another gated form?
• Does the output dimension match the residual connection?
• Is dropout, normalization, or routing applied inside the FFN?
• Is parameter count matched when comparing gated, MoE, or wider FFNs?
• Which axis is mixed: channel, token, node, spatial location, or expert route?

Related

• Linear layer
• Activation function
• Gating
• Transformer
• Mixture of experts
• MLP-Mixer
• GLU Variants Improve Transformer