Transformer

Transformers use attention to model relationships across tokens. They are common in language models, protein sequence models, molecular sequence models, and agent systems.

A pre-norm Transformer block with multi-head self-attention is:

$$\tilde{X}=\mathrm{LN}(X)$$ $$Q_h=\tilde{X}{W}_Q^{(h)},K_h=\tilde{X}{W}_K^{(h)},V_h=\tilde{X}{W}_V^{(h)}$$ $${\mathrm{head}}_h=\mathrm{softmax}\left(\frac{Q_h{K}_h^{\top }}{\sqrt{d_k}}+M\right)V_h$$ $$\mathrm{MHA}(\tilde{X})=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_H)W_O$$ $$X^{\prime }=X+\mathrm{MHA}(\tilde{X})$$ $$\mathrm{FFN}(Z)=\sigma (Z{W}_1+b_1)W_2+b_2$$ $$X_{\mathrm{out}}=X^{\prime }+\mathrm{FFN}(\mathrm{LN}(X^{\prime }))$$

Here $X$ is the token matrix, $M$ is a padding or causal mask, $H$ is the number of heads, and $\sigma$ is the feed-forward nonlinearity. Decoder-only Transformers use a causal $M$; encoder-style models usually use bidirectional attention with only padding masks. The block combines tokenization, embeddings, linear layers, softmax, normalization, residual connections, and a token-wise feed-forward network.

Shape Contract

For a batch of token embeddings:

$$X\in {\mathbb{R}}^{B\times T\times d_{\mathrm{model}}}$$

each head uses projections:

$$W_Q^{(h)},W_K^{(h)},W_V^{(h)}\in {\mathbb{R}}^{d_{\mathrm{model}}\times d_k}$$

and produces:

$${\mathrm{head}}_h\in {\mathbb{R}}^{B\times T_q\times d_v}$$

The attention score tensor has shape:

$$S_h=\frac{Q_h{K}_h^{\top }}{\sqrt{d_k}}\in {\mathbb{R}}^{B\times T_q\times T_k}$$

For self-attention, $T_q=T_k=T$. For cross-attention, query tokens and key/value tokens may come from different sources.

Mask Semantics

The mask $M$ defines what information is available:

Mask	Allows	Typical use
padding mask	ignore nonexistent tokens	variable-length batches
causal mask	attend only to previous tokens	autoregressive decoding
bidirectional mask	attend to all valid tokens	representation learning
block/local mask	attend to restricted windows	long-context efficiency
cross mask	connect query tokens to external memory	retrieval or encoder-decoder models

Mask choice is part of the model claim. A representation model, a generator, and a retrieved-context agent can all use Transformers while having different information boundaries.

Complexity

Full self-attention scales quadratically in sequence length:

$${\mathrm{cost}}_{\mathrm{attn}}=O(BH{T}^2{d}_k)$$

and attention logits require:

$$O(BH{T}^2)$$

memory before optimization. This is why long-context systems use caching, local attention, sparse attention, state-space models, retrieval, or chunking.

Transformer vs Recurrent Inference

Decoder-only Transformers usually avoid recomputing the full prefix by keeping a KV cache:

$$\text{KV cache size}\propto L\cdot n_{\mathrm{layers}}\cdot d.$$

Recurrent or retention-based alternatives such as RWKV and RetNet aim for a compact per-layer state:

$$\text{recurrent state size}\propto n_{\mathrm{layers}}\cdot d.$$

RetNet frames this as retention state:

$$S_t=\gamma S_{t-1}+k_t^{\top }{v}_t.$$

This does not make recurrence automatically better. It changes the bottleneck: attention keeps explicit past keys and values, while recurrence or retention compresses the prefix into state.

Model Family	Inference Memory	Main Risk
decoder-only Transformer	grows with context through KV cache	long-context memory cost
recurrent language model	fixed recurrent state	compressed-state bottleneck
retention model	decayed key-value state	retention horizon and normalization details
SSM/selective SSM	scan state or structured cache	state design and retrieval behavior
hybrid Transformer-SSM	fewer attention layers need KV cache	allocation and implementation details matter

Hybrid models such as Jamba keep some attention layers for explicit token-token retrieval while using Mamba-style layers to reduce long-context cache pressure:

$$L_{\mathrm{attn}}<L_{\mathrm{total}}\Rightarrow M_{\mathrm{KV}}=O(L_{\mathrm{attn}}Td).$$

This is not a free replacement for attention. It is an architecture allocation choice.

Key Ideas

• Self-attention lets each token mix information from other tokens according to learned relevance.
• Feed-forward networks, normalization, residual connections, and positional encodings are part of the core pattern.
• Encoder-only Transformer, Decoder-only Transformer, and encoder-decoder variants serve different training and inference workflows.
• Causal masking supports autoregressive generation; bidirectional attention supports representation learning.
• Transformers can process text, protein sequences, molecular strings, retrieved context, and multimodal tokens when inputs are tokenized.

Practical Checks

• Identify the variant: encoder-only, decoder-only, or encoder-decoder.
• Check context length, attention pattern, positional encoding, and masking.
• Track what each token represents: wordpiece, residue, atom token, graph node, image patch, tool call, or retrieved chunk.
• For scientific tasks, separate model architecture from data split, leakage, calibration, and downstream evaluation.
• Check whether reported context length is training length, inference length, or extrapolated length.
• Check whether memory optimization changes exact attention, approximation, or only implementation.

Related

• Attention
• Cross-attention
• Feed-forward network
• Normalization placement
• Weight initialization
• Tokenization
• Mixture of Experts
• State-space model
• RetNet
• Jamba
• Protein
• Self-supervised learning
• Coding agents