Optimizer

An optimizer turns gradients into parameter updates. Different optimizers use momentum, adaptive moments, weight decay, or second-order information to change the update rule.

For an objective $\mathcal{L}(\theta )$, the optimizer receives a gradient estimate:

$$g_t=\widehat{{\nabla }_{\theta }\mathcal{L}}({\theta }_t)$$

and produces an update:

$$\Delta {\theta }_t=U(g_t,o_t,h_t)$$ $${\theta }_{t+1}={\theta }_t+\Delta {\theta }_t$$

where $o_t$ is optimizer state and $h_t$ denotes hyperparameters such as learning rate, momentum, and weight decay.

Stochastic gradient descent is:

$${\theta }_{t+1}={\theta }_t-\eta g_t$$

Momentum keeps a velocity:

$$v_{t+1}=\beta v_t+g_t$$ $${\theta }_{t+1}={\theta }_t-\eta v_{t+1}$$

Adam-style optimizers track moving averages of gradients and squared gradients:

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$$ $$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$$

With bias correction:

$${\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t},{\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}$$

and the Adam update is:

$${\theta }_{t+1}={\theta }_t-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$$

AdamW separates adaptive optimization from weight decay:

$${\theta }_{t+1}=(1-\eta \lambda ){\theta }_t-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$$

where $\lambda$ is the decoupled weight decay coefficient.

Update Boundary

The optimizer usually steps after gradients are accumulated, synchronized, optionally clipped, and optionally unscaled for mixed precision:

$$g_t=\mathrm{Reduce}\left(\sum _{a=1}^A{\nabla }_{\theta }\frac{{\mathcal{L}}_{B_a}}{A}\right)$$ $$g_t^{\prime }=\mathrm{Clip}(g_t)$$ $$({\theta }_{t+1},o_{t+1})=\mathrm{Step}({\theta }_t,o_t,g_t^{\prime },{\eta }_t)$$

This boundary should match training step accounting, scheduler updates, logging, and checkpointing.

Checks

• Is weight decay applied to the intended parameter groups?
• Are bias, normalization, embedding, and matrix parameters treated appropriately?
• Is the optimizer state saved in checkpoints?
• Is the optimizer step counted separately from micro-steps?
• Are gradient clipping and mixed-precision unscaling applied before the update?
• Does the optimizer choice matter after controlling learning rate and schedule?

Related

• Gradient descent
• Adam
• AdamW
• Learning rate schedule
• Weight decay
• Training step accounting
• Gradient accumulation
• Gradient clipping
• Checkpointing
• GPU memory