Weight Decay

Weight decay penalizes large parameters during optimization. It is commonly used as a form of regularization in neural network training.

Classical $L_2$ regularization adds a penalty to the objective:

$$J(\theta )=\hat{R}(\theta )+\frac{\lambda }{2}\Vert \theta {\Vert }_2^2$$

The gradient becomes:

$${\nabla }_{\theta }J(\theta )={\nabla }_{\theta }\hat{R}(\theta )+\lambda \theta$$

For plain SGD, this gives:

$${\theta }_{t+1}={\theta }_t-\eta \left(g_t+\lambda {\theta }_t\right)$$

$\eta$ is the learning rate, $g_t$ is the data gradient, and $\lambda$ is the decay coefficient.

Decoupled Weight Decay

Adaptive optimizers often use decoupled weight decay:

$${\theta }_{t+1}=(1-\eta \lambda ){\theta }_t-\eta u_t$$

$u_t$ is the optimizer update direction. This separates parameter shrinkage from adaptive gradient normalization.

Coupled vs Decoupled

The same coefficient $\lambda$ can mean different things depending on the optimizer.

Setup	Formula Shape	Interpretation
$L_2$ objective penalty	$\hat{R}(\theta )+\frac{\lambda }{2}\Vert \theta {\Vert }_2^2$	changes the optimized objective
SGD with $L_2$	${\theta }_{t+1}=(1-\eta \lambda ){\theta }_t-\eta g_t$	equivalent to multiplicative shrinkage
Adam with $L_2$	Adam receives $g_t+\lambda {\theta }_t$	decay is scaled by adaptive denominator
AdamW	${\theta }_{t+1}=(1-\eta \lambda ){\theta }_t-\eta u_t$	direct decoupled shrinkage

For plain SGD, coupled $L_2$ and weight decay are closely related. For adaptive optimizers, they should be treated as different training recipes.

Effective Decay Over Training

With decoupled decay, a parameter is repeatedly multiplied by:

$$1-{\eta }_t\lambda$$

Ignoring gradient updates, after $T$ optimizer steps:

$${\theta }_T\approx \left[\prod _{t=1}^T(1-{\eta }_t\lambda )\right]{\theta }_0$$

So effective decay depends on learning-rate schedule and optimizer step count, not only on $\lambda$.

Parameter Group Contract

Weight decay should be documented as a parameter-group rule:

Group	Typical Choice	Check
matrix weights	decay	linear, convolution, attention projections
bias terms	no decay	otherwise shifts may be over-regularized
normalization parameters	no decay	scale/shift parameters control activation statistics
embeddings	depends	can dominate parameter count; tune separately
task heads	depends	small heads may need different decay

The public note should state the rule, not private run identifiers or internal file paths.

Interaction With Generalization

Weight decay can reduce overfitting, but it is not proof of better generalization:

$${\lambda }^{\star }=\arg \underset{\lambda }{\min }{M}_{\mathrm{val}}({\theta }_{\lambda })$$

The selected value depends on validation metric, split, schedule, batch size, optimizer, augmentation, and training length. If those change, the best $\lambda$ may change too.

Checks

• Is weight decay coupled to the loss or decoupled in the optimizer?
• Are bias, normalization, and embedding parameters excluded when appropriate?
• Is the decay coefficient reported separately from learning rate?
• Does stronger decay improve validation performance or only reduce training loss?
• Is weight decay being confused with dropout or early stopping?
• Is the effective decay affected by schedule length or number of optimizer steps?
• Are parameter groups logged in the run record?

Related

• Regularization
• Optimizer
• Adam
• AdamW
• Gradient descent
• Empirical risk minimization
• Learning rate schedule
• Model selection
• OOD generalization