Gradient Checking

Gradient checking compares an automatically computed gradient with a numerical finite-difference estimate. It is a debugging tool for custom losses, custom layers, coordinate transforms, masking logic, and scientific model code.

For a scalar objective $L(\theta )$, the central finite-difference estimate for parameter ${\theta }_j$ is:

$$\frac{\partial L}{\partial {\theta }_j}\approx \frac{L(\theta +ϵe_j)-L(\theta -ϵe_j)}{2ϵ}$$

where $e_j$ is a unit vector and $ϵ$ is a small perturbation.

Relative Error

A common comparison is:

$$\mathrm{relerr}=\frac{\Vert g_{\mathrm{auto}}-g_{\mathrm{num}}\Vert }{\max (\Vert g_{\mathrm{auto}}\Vert ,\Vert g_{\mathrm{num}}\Vert ,{ϵ}_{\mathrm{safe}})}$$

The threshold depends on dtype, function smoothness, reduction scale, and numerical conditioning. The point is not to prove all training is correct; it is to catch broken derivative paths.

When to Use

• A custom differentiable operation is added.
• A loss has masks, weights, or reductions.
• A coordinate transform affects geometry-sensitive outputs.
• A constraint, projection, or reparameterization affects the optimized variable.
• A model uses manual gradient manipulation.
• Training diverges before basic gradients have been validated.

Pitfalls

• Finite differences are unreliable around discontinuities, argmax, hard thresholds, indexing, and random branches.
• Float32 can be too noisy; use double precision when possible.
• Stochastic layers should be disabled or seeded.
• Regularization terms and loss reductions must match exactly.
• Checking every parameter is expensive; sample representative elements.
• Projected or constrained updates may be nondifferentiable at boundaries.

Constrained Variables

For a reparameterized constrained variable:

$$x=\varphi (z)$$

the gradient check should usually compare derivatives with respect to $z$, not only $x$:

$$\frac{\partial L(\varphi (z))}{\partial z}$$

For projected updates:

$${\Pi }_{\mathcal{C}}(x)$$

finite differences can fail near the boundary of $\mathcal{C}$. In that case, check interior points, smooth penalty versions, or task-level invariants.

Checks

• Is the model in deterministic mode?
• Is the same scalar loss used for both automatic and numerical gradients?
• Is the perturbation $ϵ$ appropriate for the dtype and scale?
• Are masks and constraints applied identically in both evaluations?
• Are gradients checked before running a long experiment?
• If variables are constrained, is the check performed in the right parameterization?

Related

• Automatic differentiation
• Backpropagation
• Constrained optimization
• Training stability
• Experiment lifecycle
• Minimum viable experiment