Multiple Comparisons

Multiple comparisons occur when many models, prompts, thresholds, seeds, datasets, metrics, or ablations are tried and only the best-looking result is emphasized. The more comparisons are made, the easier it is to find a false positive.

If each comparison has false-positive probability $\alpha$ and $K$ independent comparisons are tried, the chance of at least one false positive is:

$$P(\text{at least one false positive})=1-(1-\alpha {)}^K$$

The independence assumption is often imperfect, but the direction is the same: more unreported attempts make a selected result less trustworthy.

Where It Appears

• Hyperparameter sweeps where only the best trial is reported.
• Many random seeds with the best seed selected.
• Prompt engineering against a public benchmark.
• Many ablations, metrics, or datasets with selective reporting.
• Threshold tuning on the final test set.
• Leaderboard submissions used as iterative feedback.

Controls

• Predefine the primary metric and selection rule.
• Separate validation from final test evaluation.
• Report the search space, number of trials, failed trials, and seed policy.
• Use held-out confirmation for the final selected model.
• Adjust interpretation when many comparisons were attempted.
• Treat exploratory ablations as hypothesis generation unless confirmed.

Comparison Ledger

For public claims, record enough context to distinguish search from evidence:

Field	Meaning
candidate set	models, prompts, thresholds, seeds, datasets, or metrics tried
selection rule	how the winning variant was chosen
validation budget	how much feedback was used before final testing
final test boundary	which data was untouched until selection finished
negative results	failed or worse variants that affect interpretation
claim type	exploratory, confirmatory, reproduced, or deployment decision

Bonferroni Bound

A conservative threshold divides the target false-positive rate by the number of tests:

$${\alpha }_{\mathrm{per}-\mathrm{test}}=\frac{{\alpha }_{\mathrm{family}}}{K}$$

$K$ is the number of planned comparisons. This is simple but can be too conservative when tests are correlated; the important habit is to disclose and separate exploration from confirmation.

For false discovery rate control, the Benjamini-Hochberg rule sorts $p$-values and finds the largest $i$ such that:

$$p_{(i)}\le \frac{i}{K}q$$

where $q$ is the target false discovery rate. This is useful when many discoveries are expected, but it does not replace a clean final-test boundary for a single headline claim.

Checks

• How many variants, prompts, thresholds, seeds, or metrics were tried?
• Was the final test set used once, after model selection?
• Are negative and failed runs visible?
• Is the claim exploratory or confirmatory?
• Would the result still look meaningful under a stricter threshold or held-out confirmation?
• Does the paper distinguish model selection evidence from final claim evidence?

Related

• Statistical significance
• Effect size
• Test-set contamination
• Benchmark claim contract
• Hyperparameter tuning
• Ablation study
• Threat to validity