Loss Function

A loss function converts prediction error into a scalar training signal. It defines what the optimizer tries to reduce, so it must match the task, target semantics, and evaluation metric.

Empirical risk minimization averages a loss over examples:

$$\hat{R}(\theta )=\frac{1}{n}\sum _{i=1}^n\mathcal{L}(f_{\theta }(x_i),y_i)$$

Mean squared error is common for regression:

$${\mathcal{L}}_{\mathrm{MSE}}(\hat{y},y)=\Vert \hat{y}-y{\Vert }_2^2$$

See Mean squared error for the full regression loss and likelihood interpretation.

Cross-entropy is common for categorical prediction:

$${\mathcal{L}}_{\mathrm{CE}}(p,y)=-\sum _{k=1}^K{y}_k\log p_k$$

Here $p_k$ is the predicted probability for class $k$, and $y_k$ is the target distribution or one-hot label.

See Cross-entropy loss and Negative log-likelihood for the probabilistic view.

Batch Reduction

Implementation details matter because the optimizer receives one scalar. For per-example losses ${\ell }_i$:

$${\mathcal{L}}_{\mathrm{mean}}=\frac{1}{|B|}\sum _{i\in B}{\ell }_i$$ $${\mathcal{L}}_{\mathrm{sum}}=\sum _{i\in B}{\ell }_i$$

Changing mean to sum changes gradient scale:

$${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{sum}}=|B|{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{mean}}$$

so the effective learning rate changes unless the update rule compensates.

For weighted data:

$${\mathcal{L}}_B=\frac{{\sum }_{i\in B}{w}_i{\ell }_i}{{\sum }_{i\in B}{w}_i}$$

where $w_i$ may encode class imbalance, sampling correction, confidence, or task weighting.

Training Loss vs Reported Metric

The optimized loss and reported metric can differ:

$${\theta }^{\star }=\arg \underset{\theta }{\min }{\mathcal{L}}_{\mathrm{train}}(\theta )\nRightarrow \arg \underset{\theta }{\max }{M}_{\mathrm{valid}}(\theta )$$

This is normal when the loss is a smooth surrogate for a discrete or domain-specific metric. The important question is whether lower validation loss and better task metric move together.

Loss Decomposition

For paper notes, write the training objective as:

$$\mathcal{J}(\theta )={\mathbb{E}}_{u\sim q_{\mathrm{train}}}\left[w(u)\ell (f_{\theta }(r(u)),y(u))\right]+\lambda \Omega (\theta )$$

where $u$ is the sampled training unit, $r(u)$ is the representation, $y(u)$ is the target, $w(u)$ is an optional weight, and $\Omega$ is regularization. This makes hidden assumptions visible: sampling distribution, label semantics, representation, weighting, and regularization.

Choosing A Loss

Target / Claim	Common Loss	Check
calibrated class probability	cross-entropy, NLL	evaluate NLL, Brier score, calibration, not only accuracy
hard classification	cross-entropy, focal loss, hinge loss	threshold and class prevalence must match deployment
numeric regression	MSE, MAE, Huber, Gaussian NLL	unit, scale, censoring, and outlier policy define meaning
ranking or retrieval	pairwise, listwise, contrastive	sampled negatives must match candidate corpus
sequence generation	token NLL, sequence loss, RL-style objective	teacher forcing may not match free-running generation
molecule or structure generation	denoising, score, velocity, validity-filtered objective	loss may not imply validity, novelty, or task utility
pose or coordinate prediction	coordinate loss, distance loss, equivariant loss	atom mapping, symmetry, and frame must be explicit

Reduction Boundary

For variable-length examples, the denominator matters:

$$\mathcal{L}=\frac{{\sum }_i{\sum }_{j\in {\mathcal{V}}_i}{\ell }_{ij}}{{\sum }_i|{\mathcal{V}}_i|}$$

where ${\mathcal{V}}_i$ is the valid token, residue, atom, edge, pixel, or candidate set for example $i$. Averaging per token and averaging per example can optimize different populations.

Common Loss Families

• Cross-entropy loss for categorical labels and next-token prediction.
• Mean squared error for regression and reconstruction.
• Negative log-likelihood for explicit probabilistic models.
• Pairwise or listwise losses for Ranking.
• Contrastive losses for Contrastive learning.

Checks

• Does the loss match the evaluation metric?
• Are targets continuous, categorical, ordinal, structured, or pairwise?
• Is the loss reduced by mean, sum, token count, valid element count, or task-specific weights?
• Does gradient accumulation divide the loss at the right boundary?
• Does imbalance require weighting, sampling, or calibration?
• Is label noise large enough that a robust loss matters?
• Is the sampled unit an example, token, residue, atom, edge, pair, pose, or trajectory?
• Does the loss denominator match the population implied by the reported metric?
• Is the loss compatible with censored, missing, weak, or noisy labels?

Related

• Expectation
• Maximum likelihood
• Entropy and KL divergence
• Label semantics
• Objective-metric alignment
• Optimization
• Training loop
• Training step accounting
• Evaluation
• Supervised learning