Metric

A metric is the number used to summarize model behavior under a specific evaluation protocol. A good metric matches the decision that will be made from the model, not just the training loss.

For a test set ${\mathcal{D}}_{\mathrm{test}}=\{(x_j,y_j){\}}_{j=1}^m$, a generic metric is:

$$\hat{M}(f)=\frac{1}{m}\sum _{j=1}^{m}m(f(x_j),y_j)$$

where $m(\cdot )$ is the per-example score.

Classification accuracy is:

$$\mathrm{Acc}=\frac{1}{m}\sum _{j=1}^{m}1\left[\arg \underset{k}{\max }{p}_{\theta }(k\mid x_j)=y_j\right]$$

Mean absolute error is:

$$\mathrm{MAE}=\frac{1}{m}\sum _{j=1}^m|{\hat{y}}_j-y_j|$$

Metric as an Estimator

A reported metric is an estimator of behavior under a target distribution:

$$M(f)={\mathbb{E}}_{(x,y)\sim p_{\mathrm{target}}}\left[m(f(x),y)\right]$$

The finite test-set estimate is:

$$\hat{M}(f)=\frac{1}{m}\sum _{j=1}^{m}m(f(x_j),y_j)$$

This is why split design and confidence intervals matter. A metric can be precisely computed on the wrong test set and still support the wrong claim.

Primary and Diagnostic Metrics

The primary metric decides model selection or headline comparison:

$$f^{\text{*}}=\arg \underset{f\in \mathcal{F}}{\max }{M}_{\mathrm{primary}}(f;{\mathcal{D}}_{\mathrm{val}})$$

Diagnostic metrics explain failure modes:

$$\{M_{\mathrm{diagnostic},k}{\}}_{k=1}^K$$

Examples include calibration, subgroup error, invalid-output rate, uncertainty coverage, latency, cost, or robustness slices. Diagnostics should not silently become selection criteria after seeing results.

Aggregation

Metric aggregation should match the example unit. Per-row averaging can overweight duplicated molecules, long documents, large protein families, or repeated prompts. Grouped aggregation can be written as:

$${\hat{M}}_{\mathrm{group}}=\frac{1}{G}\sum _{g=1}^G\left[\frac{1}{|D_g|}\sum _{(x,y)\in D_g}m(f(x),y)\right]$$

where $D_g$ is a group such as a target, scaffold, query, protein family, dataset source, or user.

Checks

• Does the metric match the scientific or product decision?
• Has the metric been selected using Metric selection rather than habit?
• Is higher or lower better, and is the scale interpretable?
• Does class imbalance require AUROC, AUPRC, balanced accuracy, or calibration?
• Are hard-decision, ranking, probability, and calibration metrics being mixed?
• Are confidence intervals or multiple seeds needed?
• Is the metric valid under the chosen split?
• Is the metric averaged per row, per group, per query, per target, or per task?
• Are invalid, abstained, timed-out, or failed outputs counted rather than silently dropped?
• Is the primary metric fixed before model selection?

Related

• Evaluation
• Metric selection
• Evaluation protocol
• Classification metrics
• Regression metrics
• Ranking metrics
• Probability metrics
• Generation evaluation
• Failure mode taxonomy
• Confidence interval
• test split
• Calibration
• Loss function