Gaussian Error Linear Units

GELU replaces hard ReLU gating with a smooth probability-weighted activation, becoming a default nonlinearity in BERT-style and many later Transformer feed-forward blocks.

Metadata

Field	Value
Paper	Gaussian Error Linear Units (GELUs)
Authors	Dan Hendrycks, Kevin Gimpel
Year	2016
Venue	arXiv preprint
arXiv	1606.08415
Code	hendrycks/GELUs
Status	seed note started

One-Line Takeaway

GELU is the activation:

$$\mathrm{GELU}(x)=x\Phi (x),$$

where $\Phi$ is the standard Gaussian CDF, giving a smooth input-dependent gate rather than a hard threshold at zero.

Question

ReLU applies a hard gate:

$$\mathrm{ReLU}(x)=x{1}_{x>0}.$$

This is cheap and effective, but the gate is discontinuous in derivative at zero and fully suppresses all negative inputs.

GELU asks:

Can an activation weight an input by how likely it is to be positive under a Gaussian uncertainty model?

Formula

The GELU activation is:

$$\mathrm{GELU}(x)=x\Phi (x),$$

where:

$$\Phi (x)=\frac{1}{2}\left[1+\mathrm{erf}\left(\frac{x}{\sqrt{2}}\right)\right].$$

So:

$$\mathrm{GELU}(x)=\frac{x}{2}\left[1+\mathrm{erf}\left(\frac{x}{\sqrt{2}}\right)\right].$$

A common approximation is:

$$\mathrm{GELU}(x)\approx 0.5x\left(1+\tanh \left[\sqrt{\frac{2}{\pi }}(x+0.044715x^3)\right]\right).$$

Architecture Contract

Component	Role
input $x$	pre-activation value from linear layer
$\Phi (x)$	smooth probability-like gate
product $x\Phi (x)$	gates value by input magnitude and sign
approximation	faster implementation option
feed-forward placement	activation inside MLP/FFN block

In a Transformer feed-forward block:

$$\mathrm{FFN}(x)=W_2\sigma (W_{1}x+b_1)+b_2,$$

GELU sets:

$$\sigma =\mathrm{GELU}.$$

This is why GELU matters as an architecture component: it changes the nonlinearity in the token-wise channel mixer.

ReLU vs GELU

Axis	ReLU	GELU
Formula	$\max (0,x)$	$x\Phi (x)$
Gate	hard sign gate	smooth probability-weighted gate
Negative values	all zeroed	small negatives can pass
Derivative	kink at zero	smooth
Common use	CNNs and many MLPs	BERT-style Transformers and modern FFNs

GELU is not a new macro-architecture. It is a small block-level choice that became durable because activation behavior compounds over many layers.

Derivative

Let $\varphi (x)$ be the standard Gaussian density:

$$\varphi (x)=\frac{1}{\sqrt{2\pi }}\exp (-x^2/2).$$

Then:

$$\frac{d}{dx}\mathrm{GELU}(x)=\Phi (x)+x\varphi (x).$$

This derivative is smooth. For large positive $x$:

$$\Phi (x)\approx 1\Rightarrow \mathrm{GELU}(x)\approx x.$$

For large negative $x$:

$$\Phi (x)\approx 0\Rightarrow \mathrm{GELU}(x)\approx 0.$$

Near zero, GELU behaves like a soft gate instead of an abrupt switch.

Why It Matters

GELU became important because many Transformer blocks are built from:

$$\text{attention}+\text{token-wise FFN}+\text{normalization}+\text{residual paths}.$$

The FFN is often where much of the parameter count lives:

$$d_{\text{model}}\to d_{\text{ff}}\to d_{\text{model}}.$$

So the activation inside the FFN is not cosmetic. It affects channel mixing, gradient flow, sparsity, saturation, and training stability.

What To Watch

• GELU is usually an activation choice, not a standalone architecture claim.
• Exact and approximate GELU can differ slightly; implementations should be recorded.
• Performance gains are recipe-dependent and can interact with normalization, initialization, optimizer, and depth.
• In modern LLMs, GELU is often replaced by gated activations such as SwiGLU; compare with GLU Variants Improve Transformer.
• If a paper claims an architecture improvement, isolate whether the gain came from activation, width, normalization, or training recipe.

Related

• Activation function
• Feed-forward network
• Transformer
• Gating
• BERT
• GLU Variants Improve Transformer
• Attention Is All You Need