Gating

Gating lets a model modulate information flow with learned multiplicative controls. It appears in RNNs, GLUs, state-space models, mixture-of-experts routers, and some attention variants.

A simple gate is:

$$g=\sigma (x{W}_g+b_g)$$ $$y=g\odot f(x)$$

where $g$ controls how much of $f(x)$ passes through.

Gated residual updates often have the form:

$$h_{t+1}=g_t\odot {\tilde{h}}_{t+1}+(1-g_t)\odot h_t$$

where $g_t$ controls whether the state changes or is preserved. In GLU-style feed-forward blocks such as GLU Variants Improve Transformer:

$$\mathrm{GLU}(x)=(x{W}_a)\odot \sigma (x{W}_b)$$

The multiplicative path gives the model a data-dependent way to suppress, pass, or route features.

Gate Types

Gate type	Formula sketch	Appears in
feature gate	$y=g(x)\odot f(x)$	GLU, gated MLP
state update gate	$h^{\prime }=g\odot \tilde{h}+(1-g)\odot h$	GRU, LSTM-like memory
routing gate	$p(e\mid x)=\mathrm{softmax}(W_{r}x)$	mixture-of-experts
selective scan gate	input-dependent transition or mixing	state-space models
attention gate	content-dependent weighted aggregation	attention variants, Mega

The shared idea is multiplicative control. The difference is what is being controlled: feature magnitude, memory update, expert choice, or sequence dynamics.

Gradient Behavior

For sigmoid gates:

$${\sigma }^{\prime }(a)=\sigma (a)(1-\sigma (a))$$

When $a$ is very positive or negative, ${\sigma }^{\prime }(a)$ becomes small. Saturated gates can preserve or block information, but they also reduce gradient signal through the gate.

Regime	Behavior	Risk
$g\approx 0$	suppress path	useful feature blocked
$g\approx 1$	pass path	gate stops modulating
$g\approx 0.5$	high sensitivity	unstable if scale is poorly normalized
sparse routing	few paths active	load imbalance or expert collapse

Gating vs Attention

Gating modulates a representation, while attention chooses or mixes information from a set of positions.

$$\text{gate}:y_i=g_i\odot x_i$$ $$\text{attention}:y_i=\sum _j{\alpha }_{ij}{v}_j$$

Both can be content-dependent, but they answer different questions:

Mechanism	Question
gating	should this feature/state/path pass?
attention	which other positions should this position read?
routing	which expert/path should process this token?

Failure Modes

• Saturation: gates stay near 0 or 1 and reduce gradient signal.
• Routing collapse: a small set of experts or paths receives most tokens.
• Spurious interpretability: gate value is treated as explanation without intervention or ablation.
• Instability: multiplicative interactions amplify activation scale when normalization is weak.
• Confused explanation: a large gate value is not automatically causal evidence.

Where It Appears

• LSTM and GRU update gates in RNNs.
• Selective updates in Mamba and other state-space models.
• Gated linear units in feed-forward networks.
• Gated attention blocks in Mega, where moving-average memory is combined with gated content mixing.
• Routing weights in Mixture of experts.

Checks

• Is the gate scalar, channel-wise, token-wise, or expert-wise?
• Does the gate saturate near 0 or 1?
• Is gating used for memory, routing, feature modulation, or output control?
• Does the gate create an interpretation claim that needs verification?
• Are gate logits normalized or regularized enough to avoid collapse?
• Is the gate measured by distribution, ablation, or only visual inspection?

Related

• RNN
• GRU
• LSTM
• Feed-forward network
• State-space model
• Attention
• Mega
• Mixture of experts
• Normalization