Covariance and Correlation

Covariance measures how two random variables vary together. Correlation normalizes covariance so the scale is easier to compare.

For random variables $X$ and $Y$:

$$\mathrm{Cov}(X,Y)=\mathbb{E}\left[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])\right]$$

The correlation coefficient is:

$${\rho }_{X,Y}=\frac{\mathrm{Cov}(X,Y)}{\sqrt{\mathrm{Var}(X)}\sqrt{\mathrm{Var}(Y)}}$$

where:

$$\mathrm{Var}(X)=\mathrm{Cov}(X,X)=\mathbb{E}\left[(X-\mathbb{E}[X]{)}^2\right]$$

The empirical covariance from paired samples $(x_i,y_i)$ is:

$$\widehat{\mathrm{Cov}}(X,Y)=\frac{1}{n-1}\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})$$

where:

$$\bar{x}=\frac{1}{n}\sum _{i=1}^n{x}_i,\bar{y}=\frac{1}{n}\sum _{i=1}^n{y}_i$$

Vector Case

For a random vector $x\in {\mathbb{R}}^d$ with mean $\mu =\mathbb{E}[x]$, the covariance matrix is:

$$\Sigma =\mathbb{E}\left[(x-\mu )(x-\mu {)}^{\top }\right]$$

The diagonal entries are variances. Off-diagonal entries describe pairwise covariance.

The sample covariance matrix is:

$$\hat{\Sigma }=\frac{1}{n-1}\sum _{i=1}^n(x_i-\bar{x})(x_i-\bar{x}{)}^{\top }$$

For centered data matrix $X_c\in {\mathbb{R}}^{n\times d}$:

$$\hat{\Sigma }=\frac{1}{n-1}{X}_c^{\top }{X}_c$$

This connects covariance directly to eigenvalue decomposition and singular value decomposition.

Correlation Matrix

If $D$ is the diagonal matrix of feature standard deviations, the correlation matrix is:

$$R=D^{-1}\Sigma D^{-1}$$

Correlation is useful when feature scales are arbitrary, but it can hide absolute variance differences that matter for optimization or measurement noise.

Whitening

Whitening transforms features so their covariance is close to identity:

$$z={\Sigma }^{-1/2}(x-\mu )$$

Then:

$$\mathrm{Cov}(z)\approx I$$

In practice, ${\Sigma }^{-1/2}$ may be regularized:

$$z=(\Sigma +ϵI{)}^{-1/2}(x-\mu )$$

Whitening can help analysis, but it can also amplify noise in low-variance directions.

Representation Analysis

Embedding covariance is often used to detect collapse or redundancy. If all embeddings become similar, the covariance spectrum becomes low-rank:

$${\lambda }_1\gg {\lambda }_2,\dots ,{\lambda }_d$$

where ${\lambda }_j$ are eigenvalues of $\hat{\Sigma }$. This can indicate that a representation has lost useful degrees of freedom.

Why It Matters

• Dimensionality reduction often studies covariance structure.
• Feature correlation can make linear models unstable.
• Evaluation results can be misleading when samples are not independent.
• Representation analysis often compares embedding covariance or collapse.
• Covariance assumptions appear in Gaussian models, uncertainty estimates, and whitening.

Watch For

• Correlation is not causation.
• Zero covariance does not always mean independence.
• Strong correlation can be an artifact of leakage, duplicates, or confounders.
• Covariance depends on scale; correlation removes scale but not bias.
• Pairwise correlation can miss nonlinear dependence.
• Dataset shift can change covariance even when means look stable.
• High correlation between features can destabilize coefficient interpretation.
• Correlation computed over pooled data can hide subgroup-specific behavior.

Related

• Random variable
• Expectation
• Eigenvalue and eigenvector
• Singular value decomposition
• Normal distribution
• Dimensionality reduction
• Linear model
• Representation learning
• Leakage