Recurrent Neural Networks

Recurrent neural networks process sequences through a carried hidden state. They are an older sequence-modeling family and a useful reference point for state-space models.

The basic recurrence is:

$$h_t=\varphi (W_x{x}_t+W_h{h}_{t-1}+b)$$

The hidden state $h_t$ is the compressed memory of the prefix up to time $t$.

For sequence-to-sequence outputs, the recurrence is often paired with an output projection:

$$y_t=W_y{h}_t+b_y$$

For sequence-level prediction, the model may use $h_T$, a pooled sequence of hidden states, or an attention/readout layer over $\{h_t{\}}_{t=1}^T$.

Key Ideas

• RNNs update a hidden state one step at a time, so order is built into the computation.
• The hidden state is a bottleneck that must carry useful past information forward.
• Gated variants such as LSTM and GRU add learned mechanisms for retaining and forgetting information.
• Bidirectional RNNs use past and future context for representation tasks, but not for strict causal generation.
• RNNs are useful comparisons for state-space models because both scan sequences through state updates.

Sequence Contract

Field	Question
Direction	causal, bidirectional, or encoder-only?
State size	how much memory can $h_t$ carry?
Output	token-level, sequence-level, pooled, or autoregressive?
Masking	how are padding and variable lengths handled?
Truncation	full BPTT or truncated backpropagation through time?
Streaming	can inference process one token at a time?

For long sequences, the hidden state is both the memory and the bottleneck:

$$h_t=F_{\theta }(x_t,h_{t-1})$$

so information from $x_1$ must survive repeated updates to affect $y_T$.

Complexity

For sequence length $L$ and hidden width $d$, recurrent computation is usually:

$$O(L{d}^2)$$

for dense recurrent matrices, with sequential dependence over $t$. This makes RNNs efficient for streaming but harder to parallelize across time than Transformers.

Transformer-Era Recurrent Models

Modern recurrent language models try to keep the streaming advantage while recovering Transformer-era training scale:

$$\text{parallel training}+\text{recurrent inference state}.$$

RWKV and RetNet are canonical paper notes here. They can be formulated in parallel-friendly modes for training and recurrent modes for inference:

$$s_t=F_{\theta }(s_{t-1},x_t),y_t=G_{\theta }(s_t,x_t).$$

RetNet makes this explicit with retention:

$$S_t=\gamma S_{t-1}+k_t^{\top }{v}_t,y_t=q_t{S}_t.$$

The practical tradeoff is:

Benefit	Risk
constant-size recurrent state at inference	compressed state may bottleneck retrieval-heavy tasks
no growing KV cache	less direct pairwise token access than attention
streaming-friendly generation	training and implementation details are architecture-specific

Paper Reading Boundary

Claim	What To Check
long-context modeling	truncation length and gradient stability
streaming inference	latency, state carry-over, reset policy
compact baseline	parameter count and hidden size
protein sequence modeling	homolog split, sequence identity, length handling
time-series or trajectory	sampling rate, missing data, and episode boundary

Practical Checks

• Check whether the task needs causal, bidirectional, or streaming inference.
• Watch sequence length, truncation, masking, and padding behavior.
• Inspect how the final representation is chosen: last state, pooled states, attention over states, or token-level outputs.
• Compare with Transformers when long-range interactions or parallel training are central.
• Is hidden state reset per example, carried across chunks, or initialized from context?
• Are sequence-level metrics separated from token-level metrics?

Related

• State-space models
• LSTM
• GRU
• Transformer
• Attention
• RWKV
• RetNet
• Learning methods