LLaMA

The paper made the modern efficient open-weight decoder-only language model recipe concrete: train smaller dense Transformers longer, on public data, with a stable pre-normalized architecture.

Metadata

Field	Value
Paper	LLaMA: Open and Efficient Foundation Language Models
Authors	Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
Year	2023
Venue	arXiv preprint
arXiv	2302.13971
Meta page	Meta AI publication
Status	full note started

One-Line Takeaway

LLaMA is important less because it invented a single new block and more because it established a practical decoder-only foundation model recipe: public-data pretraining, compute-conscious model sizes, pre-normalization, RMSNorm, SwiGLU feed-forward layers, rotary position embeddings, and careful evaluation across many tasks.

Question

Large language model progress before LLaMA was dominated by very large closed models. The architectural and systems question was:

How strong can a dense decoder-only Transformer become if trained on more tokens, with an efficient architecture recipe, without relying on private training data?

The practical question:

$$\text{smaller model}+\text{more training tokens}+\text{stable Transformer recipe}\Rightarrow \text{strong foundation model}$$

The paper should be read as an architecture-and-training recipe paper, not only as a model release.

Main Claim

The narrowed claim:

$$\text{decoder-only Transformer}+\text{public-data trillion-token training}+\text{compute-efficient scaling}\Rightarrow \text{competitive foundation models}$$

The paper reports a family of dense language models from 7B to 65B parameters. The most important architectural message is not that the Transformer changed completely. It is that a specific set of now-standard decoder-only choices can support strong language models under a public-data constraint.

Architecture Contract

Item	Contract
Input	token sequence
Output	next-token probability distribution
Backbone	decoder-only Transformer
Attention pattern	causal self-attention
Normalization	pre-normalization with RMSNorm
Position handling	rotary positional embeddings
Feed-forward block	SwiGLU-style gated MLP
Training objective	autoregressive next-token prediction
Main scaling route	more tokens for smaller models, public data mixture

The core interface is the same as GPT-2 and GPT-3:

$$p_{\theta }(x)=\prod _{t=1}^T{p}_{\theta }(x_t\mid x_{<t}).$$

The training loss is:

$${\mathcal{L}}_{\text{LM}}=-\sum _{t=1}^T\log p_{\theta }(x_t\mid x_{<t}).$$

The architectural difference is in the modernized block choices and the scaling recipe.

Decoder-Only Transformer

LLaMA uses a causal Transformer stack. For a sequence of token embeddings:

$$X_0\in {\mathbb{R}}^{T\times d},$$

each layer maps:

$$X_{\ell }\to X_{\ell +1}.$$

A simplified pre-normalized decoder block is:

$$U_{\ell }=X_{\ell }+{\mathrm{Attn}}_{\ell }(\mathrm{Norm}(X_{\ell })),$$ $$X_{\ell +1}=U_{\ell }+{\mathrm{FFN}}_{\ell }(\mathrm{Norm}(U_{\ell })).$$

This is different from a post-norm form:

$$X_{\ell +1}=\mathrm{Norm}(X_{\ell }+F(X_{\ell })).$$

Pre-normalization improves optimization stability in deep Transformer stacks because each sublayer sees normalized inputs before the residual addition.

Causal Self-Attention

For hidden states:

$$H\in {\mathbb{R}}^{T\times d},$$

attention forms:

$$Q=H{W}_Q,K=H{W}_K,V=H{W}_V.$$

For a single head:

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_h}}+M\right)V,$$

where:

• $d_h$ is the head dimension;
• $M_{ij}=0$ if token $i$ may attend to token $j$;
• $M_{ij}=-\infty$ if $j>i$ under causal masking.

The mask enforces:

$$p(x_t\mid x_{\le T})\to p(x_t\mid x_{<t}).$$

This is the same basic next-token contract as GPT-style models, but LLaMA’s block choices became a reference point for later open-weight models.

RMSNorm

RMSNorm is one of the small but important block choices in the LLaMA recipe. LayerNorm normalizes by subtracting the mean and dividing by the standard deviation:

$$\mathrm{LayerNorm}(x)=\gamma \odot \frac{x-\mu (x)}{\sqrt{{\sigma }^2(x)+ϵ}}+\beta .$$

RMSNorm removes mean-centering and normalizes by root mean square:

$$\mathrm{RMSNorm}(x)=g\odot \frac{x}{\mathrm{RMS}(x)+ϵ},$$

where:

$$\mathrm{RMS}(x)=\sqrt{\frac{1}{d}\sum _{i=1}^d{x}_i^2}.$$

Why this matters architecturally:

Choice	Effect
no mean subtraction	simpler normalization path
scale normalization	controls activation magnitude
pre-norm placement	stabilizes deep residual stacks
fewer operations than full LayerNorm	useful for large-scale training and inference

The point is not that RMSNorm alone explains LLaMA performance. It is one part of a stable decoder-only block recipe.

Rotary Positional Embeddings

Transformers need some way to encode token position. Absolute position embeddings add a learned or fixed vector:

$$h_t\leftarrow h_t+p_t.$$

Rotary positional embeddings instead rotate query and key vectors as a function of position before attention:

$$q_t^{\prime }=R_t{q}_t,k_s^{\prime }=R_s{k}_s,$$

where $R_t$ is a block-diagonal rotation matrix parameterized by token position $t$.

The attention score becomes:

$$(q_t^{\prime }{)}^{\top }{k}_s^{\prime }=(R_t{q}_t{)}^{\top }(R_s{k}_s)=q_t^{\top }{R}_t^{\top }{R}_s{k}_s.$$

Because:

$$R_t^{\top }{R}_s=R_{s-t},$$

the dot product can express relative displacement between positions.

Reading contract:

Positional method	What is injected
learned absolute embedding	a position vector added to hidden state
sinusoidal absolute embedding	fixed position basis added to hidden state
rotary embedding	position-dependent rotation applied to queries and keys

For LLaMA-style models, RoPE became a default because it fits causal attention naturally and preserves the dot-product attention interface.

SwiGLU Feed-Forward Network

GLU Variants Improve Transformer is the paper-level reference for the gated FFN family behind SwiGLU. The standard Transformer feed-forward block is:

$$\mathrm{FFN}(x)=W_2\varphi (W_{1}x),$$

where $\varphi$ is often ReLU or GELU.

A gated feed-forward block uses one projection to produce content and another to produce a gate:

$$\mathrm{GLU}(x)=(x{W}_a)\odot \sigma (x{W}_b).$$

SwiGLU uses the Swish activation:

$$\mathrm{Swish}(z)=z\sigma (z),$$

and a common form is:

$$\mathrm{SwiGLU}(x)=(x{W}_a)\odot \mathrm{Swish}(x{W}_b).$$

The output projection maps back to model dimension:

$${\mathrm{FFN}}_{\text{SwiGLU}}(x)=W_o\left[(x{W}_a)\odot \mathrm{Swish}(x{W}_b)\right].$$

Architectural reading:

FFN style	Inductive effect
ReLU/GELU MLP	feature expansion and nonlinear mixing
GLU family	feature expansion with multiplicative gating
SwiGLU	smoother gate used in many modern LLM blocks

The gated MLP is a token-wise computation. It does not mix sequence positions; attention does that. It changes how each token representation transforms after contextual mixing.

Tokenization and Vocabulary Interface

LLaMA uses subword tokenization. The exact tokenizer is not the architecture contribution, but tokenization defines the model’s input/output alphabet:

$$\text{text}\to (x_1,\dots ,x_T),x_t\in \{1,\dots ,V\}.$$

The model outputs logits:

$$z_t\in {\mathbb{R}}^V.$$

Next-token probabilities are:

$$p_{\theta }(x_{t+1}=v\mid x_{\le t})=\frac{\exp z_{t,v}}{{\sum }_{u=1}^V\exp z_{t,u}}.$$

Tokenization matters because:

• vocabulary size affects embedding and output projection size;
• token boundaries change sequence length;
• long words, code, and multilingual text can become different tokenization regimes;
• token-level loss is not always comparable across tokenizers.

Public Data Constraint

The LLaMA paper emphasizes training on publicly available datasets. For architecture reading, this matters because it changes what the result means.

The claim is not:

$$\text{secret dataset}\Rightarrow \text{strong model}.$$

The claim is closer to:

$$\text{known architecture recipe}+\text{public data mixture}+\text{large token budget}\Rightarrow \text{strong open research model}.$$

This made the paper useful as a reference point for researchers who could not inspect closed-model training data.

Scaling Reading

LLaMA follows the post-Chinchilla intuition that many large models were under-trained relative to parameter count. The practical architecture lesson:

$$\text{more parameters only}\ne \text{best compute allocation}.$$

A simplified compute view:

$$C\propto N_{\text{params}}\cdot N_{\text{tokens}},$$

where:

• $C$ is approximate training compute;
• $N_{\text{params}}$ is parameter count;
• $N_{\text{tokens}}$ is number of training tokens.

For a fixed compute budget, one can trade:

$$\text{larger model, fewer tokens}\text{vs.}\text{smaller model, more tokens}.$$

The paper helped popularize the second direction for open foundation models.

Model Family

The paper presents several sizes rather than a single architecture instance.

Family Aspect	Reading
7B	deployable research-scale dense LM
13B	stronger small model; important comparison point against much larger models
33B	mid-scale foundation model
65B	large open research model of the original family

The same architectural recipe across sizes makes the paper useful for comparing scaling behavior without changing model family.

What Changed Relative to GPT-2

Axis	GPT-2	LLaMA
backbone	decoder-only Transformer	decoder-only Transformer
objective	next-token prediction	next-token prediction
position	learned absolute positions	rotary positional embeddings
normalization	LayerNorm-style Transformer recipe	pre-norm RMSNorm recipe
FFN	standard Transformer MLP	SwiGLU gated MLP
central claim	zero-shot behavior from web-scale LM	efficient open foundation models from public data
model release role	early GPT-style public reference	central open-weight research reference

This comparison is important. LLaMA did not replace the decoder-only Transformer paradigm. It made a newer implementation recipe the practical baseline for open LLM work.

What Changed Relative to BERT and T5

Model	Architecture Contract	Main Use
BERT	encoder-only masked language model	representation learning and fine-tuning
T5	encoder-decoder text-to-text model	supervised and multitask seq2seq transfer
LLaMA	decoder-only causal language model	generative continuation, prompting, instruction tuning base

LLaMA is not a general replacement for encoder-only or encoder-decoder architectures. It is the canonical path for open autoregressive foundation models.

Evidence Reading

The paper evaluates language understanding, reasoning, commonsense, reading comprehension, and mathematical benchmarks. For a paper note, the exact benchmark scores are less important than what the evidence is trying to support:

Evidence Type	What It Supports	What It Does Not Prove
comparison to larger models	smaller public-data models can be competitive	architecture alone caused the gain
many benchmark categories	broad transfer behavior	robust real-world reliability
model family scaling	same recipe scales across sizes	optimality of every hyperparameter
public-data training	strong models need not depend entirely on private corpora	all data provenance issues are solved

The key reading discipline:

$$\text{benchmark score}\ne \text{isolated architecture proof}.$$

The benchmark result is an outcome of architecture, data, compute, tokenizer, training recipe, and evaluation protocol.

Implementation Notes

When implementing a LLaMA-style block, separate the concerns:

Component	Implementation Question
RMSNorm	is normalization before attention and before FFN?
RoPE	are rotations applied to query and key, not value?
causal mask	does token $t$ avoid attending to future tokens?
SwiGLU	are hidden dimensions adjusted to keep parameter count comparable?
KV cache	does inference reuse past keys and values?
tokenizer	are input IDs compatible with the checkpoint vocabulary?

Autoregressive inference without caching recomputes attention keys and values for previous tokens. With KV cache:

$$K_{\le t}=[K_{\le t-1};K_t],V_{\le t}=[V_{\le t-1};V_t].$$

The next step attends over cached history:

$$\mathrm{Attn}(q_t,K_{\le t},V_{\le t}).$$

This is an inference-systems concern, but it is inseparable from decoder-only architecture in deployment.

Failure Modes

Failure Mode	Why It Matters
treating LLaMA as a new Transformer type	misses that it is mainly a modern decoder-only recipe
attributing gains only to RMSNorm/RoPE/SwiGLU	ignores data and compute allocation
comparing token-level loss across tokenizers naively	tokenization changes the unit of prediction
ignoring public-data mixture	weakens interpretation of what the paper demonstrates
assuming benchmark breadth means reliability	benchmark coverage is not the same as robust behavior
mixing base model and instruction-tuned model claims	architecture paper concerns the base pretraining recipe

Common Misreadings

”LLaMA invented decoder-only Transformers.”

No. GPT-style decoder-only Transformers came earlier. LLaMA popularized a strong open foundation-model recipe.

”The architecture explains the whole performance jump.”

No. Performance depends on architecture, data, token budget, optimization, and evaluation choices.

”Public data means the model is fully transparent.”

No. Public-data training is more inspectable than private-data training, but exact preprocessing, filtering, deduplication, and data governance still matter.

”A smaller model beating a larger model means parameters do not matter.”

No. It means parameter count alone is a bad comparison axis when token budget and training recipe differ.

Why It Matters

LLaMA became a reference architecture because it made several choices converge:

• decoder-only causal language modeling remained the dominant generative interface;
• pre-normalized RMSNorm blocks became common in open LLMs;
• RoPE became a standard positional method for causal attention;
• gated MLPs became a standard FFN choice;
• public-data foundation models became credible research infrastructure;
• open-weight models became a practical base for fine-tuning, alignment, retrieval, agents, and domain adaptation.

For this wiki, LLaMA is the bridge between:

$$\text{GPT-2 as decoder-only LM proof}\to \text{modern open foundation model recipe}.$$

Later-Paper Checklist

When reading later LLaMA-family or open LLM papers, check:

• Is the model still dense decoder-only, or does it use MoE?
• Does it keep RMSNorm, RoPE, and gated FFN?
• Does it change context length?
• Does it use grouped-query or multi-query attention?
• What data mixture changed?
• How much of the gain comes from tokenizer changes?
• Are results for base models, instruction-tuned models, or RLHF/DPO models?
• Are comparisons compute-matched or only parameter-matched?
• Are benchmarks contaminated by training data?

Limitations

The paper is not a clean architecture ablation. It does not isolate each block choice under a fully controlled compute/data setup. It is better read as a strong model-family recipe and evidence that open, public-data foundation models can be competitive.

The note should therefore avoid claims like:

$$\text{RMSNorm}\Rightarrow \text{LLaMA performance}.$$

The more defensible claim is:

$$\text{modern decoder-only recipe}+\text{large public-data pretraining}+\text{compute-conscious scaling}\Rightarrow \text{strong open foundation model family}.$$

Connections

• Decoder-only Transformer
• Transformer
• Attention
• Normalization
• Normalization placement
• Feed-forward network
• Gating
• Positional encoding
• Tokenization
• Computational complexity
• Autoregressive model
• Scaling claim contract
• GPT-2
• BERT
• T5
• Mamba
• Switch Transformer
• Architecture papers