Activation Function
An activation function adds nonlinearity between linear transformations. Without nonlinear activations, stacked linear layers collapse into a single linear layer.
For an MLP layer:
Here is the activation function.
If every layer were linear:
so depth would not create a new nonlinear function class. Activations make the model piecewise, smooth, gated, or saturating depending on the choice.
Common Activations
ReLU:
GELU:
where is the standard normal CDF. GELU is common in Transformer feed-forward blocks.
SiLU, also called swish, is:
where is the sigmoid function. SiLU and gated variants are common in modern feed-forward networks.
Tanh maps values into :
Sigmoid maps values into :
They are useful for gates and bounded outputs, but can saturate when is large.
Gradient Behavior
The derivative determines how gradients pass through the layer. For ReLU:
For sigmoid:
When is close to 0 or 1, the derivative is small. This is one source of vanishing gradients in deep or recurrent models.
Gated Activations
Many modern blocks use a gate:
or a smooth variant:
Gating lets the network modulate information flow, connecting activations to Gating and feed-forward networks.
Choosing an Activation
- ReLU: simple, sparse, cheap, but can create dead units.
- GELU: smooth, common in Transformer blocks.
- SiLU/SwiGLU: smooth gated behavior, common in modern large models.
- Tanh/sigmoid: useful for gates or bounded outputs, but saturation must be watched.
- Output activation: should match the target semantics and loss.
Checks
- Check whether the activation is ReLU, GELU, SiLU, tanh, sigmoid, or gated.
- Watch saturation for sigmoid/tanh.
- Check whether the activation is hidden-layer nonlinearity, gate, or output transform.
- Confirm whether the loss expects logits or probabilities.
- In generative models, distinguish hidden activations from output distributions.
- In scientific models, ensure output activation matches the target range.