Probability Distribution

A probability distribution assigns probabilities to possible outcomes. In machine learning, data, labels, latent variables, model outputs, noise, and deployment environments are often described as distributions.

For a discrete variable:

$$\sum _{x}p(x)=1,p(x)\ge 0$$

For a continuous variable:

$$\int p(x)dx=1,p(x)\ge 0$$

Joint, Marginal, and Conditional

Joint distribution:

$$p(x,y)$$

Marginal distribution:

$$p(x)=\sum _{y}p(x,y)$$

Conditional distribution:

$$p(y\mid x)=\frac{p(x,y)}{p(x)}$$

The product rule factorizes a joint distribution:

$$p(x,y)=p(y\mid x)p(x)=p(x\mid y)p(y)$$

For a sequence $x_{1:T}$, the chain rule is:

$$p(x_{1:T})=\prod _{t=1}^{T}p(x_t\mid x_{<t})$$

This is the probability factorization behind autoregressive modeling.

Independence

Two variables are independent when:

$$p(x,y)=p(x)p(y)$$

Conditional independence is:

$$p(x,y\mid z)=p(x\mid z)p(y\mid z)$$

Many modeling assumptions are independence assumptions: IID datasets, factorized likelihoods, naive Bayes models, token factorization choices, and some loss decompositions.

Distribution Over Structured Objects

Distributions can be defined over vectors, sequences, graphs, coordinates, or sets:

$$p(y\mid x),y\in \mathcal{Y}$$

where $\mathcal{Y}$ may be a class set, sequence vocabulary, molecule graph space, coordinate space, or action space. The output space determines whether probabilities are easy to normalize or require approximation.

Model vs Data Distribution

Machine learning often compares the data distribution and a model distribution:

$$p_{\mathrm{data}}(x,y)\text{and}{p}_{\theta }(y\mid x)\text{ or }{p}_{\theta }(x)$$

Evaluation asks whether performance under a held-out or deployment distribution supports the intended claim.

Shift

Distribution shift can be written as:

$$p_{\mathrm{train}}(x,y)\ne p_{\mathrm{test}}(x,y)\ne p_{\mathrm{deploy}}(x,y)$$

The shift may be in $p(x)$, $p(y\mid x)$, label availability, metadata, source mixture, or task definition.

Checks

• What random variable does the distribution describe?
• Is the distribution over inputs, labels, latents, outputs, or noise?
• Is the model estimating $p(x)$, $p(y\mid x)$, $p(x,y)$, or a score?
• Is the variable discrete, continuous, structured, or mixed?
• Does the training distribution match the deployment distribution?
• Is the factorization assumption explicit?
• Are independence assumptions valid for the example unit and split unit?
• Is the distribution normalized, implicit, energy-based, or only sampled from?
• Which distribution is used for training, validation, test, and deployment?

Related

• Random variable
• Normal distribution
• Expectation
• Maximum likelihood
• Covariance and correlation
• Bayes rule
• Density estimation
• Data distribution
• Task output space
• Generative models
• OOD generalization