Benchmark Claim Contract

Benchmark claim contract maps a reported score to the narrow claim it can support. Use it when a paper says a method is better, more general, more useful, or more robust.

$$\text{claim support}=g(\mathcal{D},\mathcal{S},\mathcal{T},\mathcal{M},b,u)$$

where $\mathcal{D}$ is the dataset, $\mathcal{S}$ is the split, $\mathcal{T}$ is the task, $\mathcal{M}$ is the metric set, $b$ is the baseline, and $u$ is uncertainty or variance.

Contract Fields

Field	Must State	Why It Matters
Dataset	name, version, source, filtering, preprocessing	defines the population being measured
Example unit	one molecule, protein, complex, image, prompt, trajectory, or generated sample	prevents mismatched denominators
Split unit	random row, scaffold, protein family, time, source, complex pair, or task instance	defines the generalization boundary
Split axis	single-axis or multi-axis holdout, with grouped-unit counts	prevents broad generalization claims from a weak split
Task	input, output, validity rule, allowed context	separates classification, ranking, generation, and coordinate claims
Metric	primary metric, diagnostics, aggregation, invalid-output policy	states what success means
Selection rule	validation metric, threshold choice, checkpoint choice, prompt choice, filtering rule	separates model selection from final evidence
Baseline	simple baseline, previous method, oracle upper bound, or ablation	makes the score interpretable
Uncertainty	seed variance, bootstrap interval, paired comparison, subgroup variance	prevents overreading small differences
Allowed information	pretraining data, templates, retrieval corpus, tools, prompts, featurizers	detects contamination and hidden assistance
Compute budget	data scale, model size, training steps, hardware, precision, inference budget	separates quality claims from resource changes

Score to Claim Map

Reported Score	Conservative Claim
Higher IID test score	better on this dataset distribution under this protocol
Higher scaffold split score	better chemical-series generalization if scaffold grouping is valid
Higher protein-family split score	better target-family generalization if homolog leakage is controlled
Higher multi-axis split score	better joint-axis transfer only for the named held-out axes
Higher pose success	better pose placement under the same receptor, ligand, and symmetry policy
Higher docking enrichment	better early ranking under this active/decoy construction
Higher generation validity	fewer invalid samples under this sampler and filtering policy
Higher benchmark average	better aggregate score, not necessarily better on every subgroup
Better leaderboard rank	better under leaderboard rules, not necessarily better deployment utility

Claim Formula

A paper’s benchmark table usually reports an estimate:

$$\hat{\Delta }={\mathrm{Agg}}_{i\in {\mathcal{D}}_{test}}\left[m(f_{{\theta }^{\text{*}}}(x_i),y_i)-m(f_{{\varphi }^{\text{*}}}(x_i),y_i)\right]$$

where $f_{{\theta }^{\text{*}}}$ is the proposed model, $f_{{\varphi }^{\text{*}}}$ is the baseline, $m$ is the per-example metric, and $\mathrm{Agg}$ is the aggregation rule.

The claim is weak when ${\theta }^{\text{*}}$, ${\varphi }^{\text{*}}$, thresholds, prompts, filtering, or preprocessing are chosen after looking at the final test set.

Molecular Modeling Additions

Area	Extra Check
Molecule property	standardization, scaffold split, assay/source split, activity cliffs
Protein model	sequence identity, family split, residue indexing, structure source
Protein-ligand complex	ligand scaffold, protein family, pair split, pocket definition, template leakage
Docking and pose	atom mapping, symmetry correction, receptor preparation, ligand protonation, pose validity
Virtual screening	active/decoy provenance, early enrichment, duplicate handling, target leakage
Conformer model	conformer source, ensemble size, energy ranking, downstream representation shift

Stop Before Claiming

• The dataset version or filtering policy is missing.
• The split does not match the claimed deployment shift.
• The held-out axis is unnamed or weaker than the claim.
• The baseline uses different data, examples, or allowed information.
• The proposed method uses a larger compute, data, model-selection, or inference budget without stating it.
• The primary metric was chosen after seeing many alternatives.
• Invalid generations or failed predictions are removed without a denominator.
• Only aggregate performance is shown while subgroup failures are plausible.
• The result is within seed variance or confidence intervals.
• The table reports a best seed or best checkpoint without the selection rule.
• The paper’s evidence supports a benchmark-specific claim, but the draft says the method is generally superior.

Audit Evidence

Claim Risk	Evidence Needed
leakage	duplicate, near-duplicate, scaffold, homolog, source, and preprocessing audit
contamination	pretraining/retrieval/template/prompt corpus cutoff or exclusion policy
weak split	group counts and label distribution per split
filtered denominator	attempted, failed, repaired, filtered, and scored counts
unstable improvement	paired comparison, confidence interval, or seed variance

Related

• Benchmark intake
• Dataset card
• Dataset split contract
• Evaluation protocol
• Test-set contamination
• Claim-evidence boundary
• Seed variance
• Result table reading
• Metric selection
• Baseline
• Paired comparison
• Confidence interval
• Scaling claim contract
• Molecular modeling data and evaluation