Implementation Readiness

Implementation readiness asks whether a paper has enough public detail to justify spending time on a reimplementation. It sits between Artifact availability and Reproduction plan.

A paper is implementation-ready only for a scoped claim:

$$\mathrm{ready}(p,c)=\mathrm{artifacts}(p)\wedge \mathrm{spec}(c)\wedge \mathrm{feasible}(c)\wedge \mathrm{verifiable}(c)$$

where $p$ is the paper and $c$ is the target claim.

Readiness Levels

Level	Meaning
metadata-only	The paper can be cited or tracked, but not implemented yet
concept-ready	The idea can update concept notes, but implementation details are incomplete
baseline-ready	A simplified baseline or diagnostic can be implemented
claim-ready	One named claim has enough artifacts, data, metric, and scope to test
reproduction-ready	The original result can plausibly be rerun or closely reimplemented

Required Fields

• Target claim: one claim, not the whole paper.
• Input/output contract: data, preprocessing, representation, target, and validity rule.
• Method specification: architecture, objective, optimizer, schedule, and inference procedure.
• Evaluation contract: benchmark, split, metric, aggregation, baseline, and failure criteria.
• Artifact status: code, data, split, config, weights, logs, predictions, and environment.
• Compute estimate: public hardware class, memory, runtime, and fallback plan.
• Risk: missing details, dependency drift, ambiguous preprocessing, or unavailable data.

Readiness Matrix

Field	Ready When	Common Blocker
object	input/output units are explicit	row, molecule, protein, pose, or trajectory is ambiguous
data	public source and filtering are reconstructable	private processed data only
split	train/validation/test rule or files are available	paper reports only aggregate score
objective	loss, reward, score, energy, or constraint is specified	auxiliary weights or masks missing
implementation	architecture and inference path are concrete	hidden preprocessing or custom kernels
evaluation	metric script or exact formula is available	invalid samples or failed cases omitted
compute	feasible minimum run is defined	original scale is too expensive

Readiness should be assigned per claim, not per paper.

Decision Rule

Do not start a full reproduction when a smaller check can falsify the useful part:

$$\text{start}⟺\text{claim-ready}\wedge \text{compute-feasible}\wedge \text{failure-informative}$$

A failed run is useful when it can distinguish paper ambiguity, implementation error, data mismatch, compute limitation, or a false claim.

Minimum Viable Reproduction

Before a full reproduction, define the smallest check:

$$R_{\min }=\arg \underset{R}{\min }\mathrm{cost}(R)\text{s.t.}R\text{ can falsify claim }c$$

Examples:

Claim	Minimum check
better metric	rerun metric on released predictions
better split generalization	audit split files and rerun baseline
better sampler	evaluate released samples before retraining
better docking pose	verify pose RMSD/validity on released poses
better agent workflow	replay a small public task subset with logs

Checks

• Is the implementation target one explicit claim?
• Are all missing artifacts marked to verify, not found, or not applicable?
• Can a minimum viable experiment test the claim before a full reproduction?
• Is the benchmark public and compatible with the claimed split?
• Are expected metrics and tolerances defined before running?
• Would a failure produce a useful Reproduction result note?
• Is the first run a metric check, baseline rerun, small-scale run, or full training run?
• Can failure be attributed to data, code, config, compute, or paper ambiguity?

Related

• Artifact availability
• Reproducibility checklist
• Reproduction plan
• Reproduction result
• Minimum viable experiment
• Benchmark claim contract
• Training run
• Run artifact