Second-Order Optimization

Second-order optimization uses curvature information, not only gradients, to choose parameter updates. In machine learning, this usually means using the Hessian or an approximation to it.

For a scalar loss $\mathcal{L}(\theta )$:

$$g_t={\nabla }_{\theta }\mathcal{L}({\theta }_t),H_t={\nabla }_{\theta }^2\mathcal{L}({\theta }_t)$$

Newton’s method uses:

$${\theta }_{t+1}={\theta }_t-H_t^{-1}{g}_t$$

This rescales the gradient by local curvature.

Local Quadratic View

Near ${\theta }_t$, a second-order approximation is:

$$\mathcal{L}({\theta }_t+\delta )\approx \mathcal{L}({\theta }_t)+g_t^{\top }\delta +\frac{1}{2}{\delta }^{\top }{H}_t\delta$$

Minimizing this quadratic gives the Newton step when $H_t$ is invertible and well-conditioned.

Why Deep Learning Rarely Uses Full Hessians

Issue	Meaning
Size	$H_t$ is $d\times d$ for $d$ parameters
Memory	full Hessian storage is usually impossible for large models
Cost	exact inversion is expensive
Indefiniteness	nonconvex losses can have negative curvature
Noise	mini-batch Hessians can be unstable

Practical methods often use Hessian-vector products, diagonal approximations, low-rank approximations, quasi-Newton methods, natural-gradient approximations, or adaptive first-order optimizers.

Curvature Diagnostics

Curvature still matters even when training uses AdamW or SGD:

Signal	Interpretation
Large Hessian eigenvalues	sharp directions, learning-rate sensitivity
Ill-conditioning	some directions need much smaller steps than others
Negative curvature	saddle or nonconvex region
Large gradient norm with unstable loss	step may be too large for local curvature
Flat validation response	hyperparameter may not affect the selected metric strongly

Boundaries

Second-order information can support different claims:

Use	Claim
Optimization method	curvature directly changes updates
Diagnostic	curvature explains instability or sharpness
Uncertainty approximation	local curvature approximates posterior shape
Architecture analysis	curvature compares trainability or sensitivity

Do not treat a curvature plot as proof of generalization. It needs a metric, split, and uncertainty boundary.

Checks

• Is the method using a full Hessian, Hessian-vector products, diagonal approximation, or a proxy?
• Is curvature computed on train, validation, or test data?
• Is the Hessian of loss, log likelihood, energy, or another objective?
• Are reported eigenvalues comparable across model scales and losses?
• Does the second-order claim affect optimization, uncertainty, or only interpretation?

Related

• Jacobian and Hessian
• Optimization
• Loss landscape
• Gradient descent
• Optimizer
• Training stability
• Eigenvalue and eigenvector
• Bayesian inference