Decoder-Only Transformer

A decoder-only Transformer uses causal self-attention and predicts the next token autoregressively. It is the standard architecture pattern for many large language models and sequence generators.

Canonical paper notes in this wiki include GPT-2, GPT-3, and LLaMA.

The causal mask is:

$$M_{ij}=\{\begin{array}{ll}0,& j\le i\\ -\infty ,& j>i\end{array}$$

The sequence likelihood factorizes as:

$$p_{\theta }(x_{1:T})=\prod _{t=1}^T{p}_{\theta }(x_t\mid x_{<t})$$

At layer $\ell$, causal self-attention computes:

$$Q=H^{(\ell )}{W}_Q,K=H^{(\ell )}{W}_K,V=H^{(\ell )}{W}_V$$ $$\mathrm{Attn}(H^{(\ell )})=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}+M\right)V$$

where $M$ prevents a token from attending to future tokens. During inference, previously computed keys and values can be cached:

$$K_{\le t}=[K_{<t};K_t],V_{\le t}=[V_{<t};V_t]$$

This makes incremental decoding avoid recomputing the whole prefix, but memory grows with context length, layers, heads, and batch size.

Training and Inference

• Training usually uses teacher forcing: every position predicts the next token in parallel under the causal mask.
• Inference samples or decodes one step at a time, often with KV cache.
• Long-context behavior depends on positional encoding, attention implementation, and memory budget.
• Valid output may require constrained decoding for molecules, tool calls, code, or structured actions.

Uses

• Text generation.
• Autoregressive protein or molecular sequence generation.
• Tool-call and agent action generation.
• Next-token pretraining.

Checks

• Does the task require generation or only representation?
• Is context truncated or cached during inference?
• Are generated tokens constrained to valid syntax, molecules, or actions?
• Is evaluation separated into likelihood, sample quality, and task utility?

Related

• Transformer
• Autoregressive model
• Attention
• Positional encoding
• GPU memory
• Tool use