Gradient Descent

Gradient descent updates parameters in the direction that locally decreases a loss. It is the basic optimization idea behind most neural network training.

The update is:

$${\theta }_{t+1}={\theta }_t-{\eta }_t{\nabla }_{\theta }\mathcal{L}({\theta }_t)$$

where ${\theta }_t$ are parameters, ${\eta }_t$ is the learning rate, and ${\nabla }_{\theta }\mathcal{L}$ is the gradient.

More generally, training updates an objective $J(\theta )$:

$$J(\theta )=\hat{R}(\theta )+\lambda \Omega (\theta )$$

and the update is:

$${\theta }_{t+1}={\theta }_t-{\eta }_t{\nabla }_{\theta }J({\theta }_t)$$

where $\hat{R}$ is empirical risk, $\Omega$ is a regularizer, and $\lambda$ controls regularization strength.

The local geometry of the loss is controlled by Jacobian and Hessian information: gradients give first-order direction, while Hessian curvature affects stability and step size.

This geometry is often described as the Loss landscape. Gradient descent only uses first-order local information, so the same update rule can behave differently in flat, sharp, noisy, or badly scaled regions.

Direction and Step Size

For a small perturbation $\Delta \theta$, first-order approximation gives:

$$J(\theta +\Delta \theta )\approx J(\theta )+{\nabla }_{\theta }J(\theta {)}^{\top }\Delta \theta$$

Choosing $\Delta \theta =-\eta {\nabla }_{\theta }J(\theta )$ gives:

$$J(\theta +\Delta \theta )\approx J(\theta )-\eta {\Vert {\nabla }_{\theta }J(\theta )\Vert }_2^2$$

This explains why the negative gradient is locally downhill. It does not guarantee global improvement when the step is too large, gradients are noisy, or the objective is non-convex.

With mini-batches, the gradient is estimated from a batch $B$:

$$g_t=\frac{1}{|B|}\sum _{i\in B}{\nabla }_{\theta }\mathcal{L}(f_{\theta }(x_i),y_i)$$

Then the practical update is:

$${\theta }_{t+1}={\theta }_t-{\eta }_t{g}_t$$

possibly after gradient clipping, mixed-precision scaling, distributed averaging, or optimizer-specific transformations.

Failure Modes

• Overshooting: ${\eta }_t$ is too large, so loss oscillates or diverges.
• Slow progress: ${\eta }_t$ is too small or curvature is badly scaled.
• Bad gradient estimate: masking, padding, sampling, or distributed averaging changes $g_t$.
• Train-only improvement: training loss decreases while validation loss or target metric does not.
• Resume drift: optimizer, scheduler, scaler, or random sampler state is not restored.

Checks

• Is the gradient computed from the intended loss and valid examples only?
• Is the learning rate schedule stable?
• Are gradients clipped, accumulated, synchronized, or scaled?
• Does lower training loss improve validation performance?
• Is the x-axis counted in optimizer steps, samples, tokens, or epochs?
• Is the same update boundary used across compared runs?
• Are optimizer and scheduler states restored when resuming?

Related

• Empirical risk minimization
• Backpropagation
• Automatic differentiation
• Loss landscape
• Jacobian and Hessian
• Optimizer
• Learning rate schedule
• Gradient clipping
• Training step accounting
• Training loop