Empirical Risk Minimization

Empirical risk minimization is the basic training principle of choosing parameters that minimize average loss on observed data.

Given a dataset:

$$\mathcal{D}=\{(x_i,y_i){\}}_{i=1}^n$$

the empirical risk is:

$$\hat{R}(\theta )=\frac{1}{n}\sum _{i=1}^n\mathcal{L}(f_{\theta }(x_i),y_i)$$

Training chooses:

$$\hat{\theta }=\arg \underset{\theta }{\min }\hat{R}(\theta )$$

$x_i$ is an input, $y_i$ is a target, $f_{\theta }$ is the model, and $\mathcal{L}$ is the loss.

Why It Matters

Many learning methods are variations on this template. The difference is usually in the data distribution, loss, regularizer, sampling process, or optimization method.

ERM is an approximation to population risk:

$$R(\theta )={\mathbb{E}}_{(x,y)\sim p_{\mathrm{data}}}\left[\mathcal{L}(f_{\theta }(x),y)\right]$$

The training set replaces the unknown data distribution with an empirical distribution:

$${\hat{p}}_{\mathcal{D}}(x,y)=\frac{1}{n}\sum _{i=1}^n{\delta }_{(x_i,y_i)}(x,y)$$

so $\hat{R}(\theta )$ is only as meaningful as the dataset, labels, preprocessing, and split policy behind it.

With a regularizer:

$$\hat{\theta }=\arg \underset{\theta }{\min }\left[\frac{1}{n}\sum _{i=1}^n\mathcal{L}(f_{\theta }(x_i),y_i)+\lambda \Omega (\theta )\right]$$

$\Omega (\theta )$ penalizes model complexity or parameter size, and $\lambda$ controls the penalty strength.

Weighted Risk

Real training objectives often use weights, masks, or sampling rules:

$${\hat{R}}_w(\theta )=\frac{{\sum }_{i=1}^n{w}_i\mathcal{L}(f_{\theta }(x_i),y_i)}{{\sum }_{i=1}^n{w}_i}$$

where $w_i$ may encode class weights, label confidence, missing-label masks, token masks, or task balancing. This changes the target empirical distribution. A model trained on reweighted risk should not be evaluated as if it optimized the unweighted population.

Protocol Boundary

ERM is not only a formula. It includes decisions that must be fixed before final evaluation:

$$\text{training protocol}=({\mathcal{D}}_{\mathrm{train}},\text{preprocessing},\mathcal{L},\Omega ,\text{sampler},\text{optimizer},\text{selection rule})$$

Changing any component after seeing validation or test behavior changes the learning procedure. This connects ERM to model selection and leakage.

Checks

• What examples define the empirical distribution?
• Does the loss match the task and target semantics?
• Is the training objective the same quantity as the evaluation metric?
• Are weights, masks, or sampling probabilities changing the risk?
• Is regularization explicit or hidden in the optimizer?
• Which parts of preprocessing are fit only on training data?
• Is the final test set untouched by objective, sampler, or hyperparameter choices?
• Does the claimed deployment distribution match the empirical distribution used for training?

Related

• Loss function
• Regularization
• Optimization
• Stochastic gradient
• Model selection
• Generalization
• Preprocessing contract
• Sampling strategy
• Leakage
• test split