Expectation

Expectation is an average under a probability distribution. Many objectives and evaluation metrics are expectations.

For a discrete random variable:

E_{x \sim p} [f (x)] = x \sum p (x) f (x)

For a continuous random variable:

E_{x \sim p} [f (x)] = \int p (x) f (x) d x

Empirical averages approximate expectations from finite data:

\frac{1}{n} i = 1 \sum n f (x_{i}) \approx E_{x \sim p_{data}} [f (x)]

Expected risk is the population version of a supervised loss:

R (f) = E_{(x, y) \sim p_{data}} [L (f (x), y)]

Empirical risk replaces the unknown distribution with a dataset:

\hat{R} (f) = \frac{1}{n} i = 1 \sum n L (f (x_{i}), y_{i})

The gap between $R$ and $\hat{R}$ is why sampling, splits, confidence intervals, and distribution shift matter.

Linearity

Expectation is linear:

E [a X + bY] = a E [X] + b E [Y]

This holds even when $X$ and $Y$ are not independent. Many loss decompositions and metric averages use this property.

Conditional expectation averages under a conditional distribution:

E [Y ∣ X = x] = \int y p (y ∣ x) d y

For regression with squared error, the optimal prediction is the conditional mean:

f^{\*} (x) = E [Y ∣ X = x]

This explains why different losses imply different target summaries.

Sampling or class weighting changes the expectation:

E_{q} [w (X, Y) L (f (X), Y)]

If $q$ is the training sampler and deployment follows $p$ , the metric should make clear which expectation is being estimated.

Training objectives average losses over data.
Evaluation estimates risk over a held-out distribution.
Sampling changes which expectation the model optimizes.
Monte Carlo estimates approximate intractable integrals.
Reinforcement learning, generative modeling, and evaluation all differ mainly in which distribution the expectation is taken under.

Which distribution is the expectation under?
Is the estimate biased by sampling, filtering, or missing data?
Is the average over examples, tokens, classes, tasks, seeds, or folds?
Does the reported metric include uncertainty or variance?
Is the empirical average weighted by the sampling process?
Are macro, micro, per-class, per-token, and per-example averages being mixed?
Does the loss target a mean, median, mode, quantile, ranking, or calibrated probability?
Is the expectation over data, model samples, seeds, folds, or annotators?