Issta reproducibility

ISSTA Reproducibility

Use this before submission and again before the artifact deadline. Verifiability and transparency are named ISSTA evaluation criteria, so reproducibility is scored, not optional. Reopen the current call and artifact instructions to confirm what the cycle requires.

Evidence map

• Map each empirical claim — a detection rate, a coverage gain, a bug count, a speedup — to a verifiable location: a table, the artifact, or a logged run that a reader could regenerate.
• Pin the substrate: benchmark version (e.g. the Defects4J revision), subject-program commit SHAs, the extraction date of any mined corpus, and the exact toolchain versions. "The latest version" is not reproducible.
• Fix and report the stochastic knobs: random seeds, timeout budgets, iteration or generation counts, and hardware, because a fuzzing or search result at one budget says little about another.
• Disclose non-determinism honestly. Where results vary between runs, report the run count and the observed spread rather than a single golden run, and say which tables are means over runs.
• Give an availability statement: what is released, under what license, and where, or an honest reason for withholding (proprietary subjects, license terms) with enough detail for in-principle reproduction.
• Keep the artifact and the paper in lockstep; a table the artifact cannot regenerate is a verifiability failure the criteria will catch.

Claim-to-evidence audit table

Claim type	Minimum reproducibility evidence	Common failure caught
Detection / bug-finding rate	Labelled subjects + ground-truth labels archived	Rate reported against an unshared or hand-picked subject set
Coverage or analysis-precision gain	Subject SHAs, tool config, and the measurement script	Baseline run under a different configuration than the tool
Fuzzing throughput / bugs found	Seed corpus, time budget, run count, hardware	A single lucky campaign presented as typical
Speedup over a baseline	Same machine, same subjects, wall-clock protocol	Speedup measured on incomparable inputs

Marking a stochastic result as if it were deterministic — no seed, no run count — is a recognizable ISSTA red flag, because reviewers know these techniques do not produce the same number twice.

Vignette: a fuzzing evaluation

Consider a paper claiming a new mutation strategy finds more bugs. Its reproducibility spine: the seed corpus and target binaries pinned by hash; the CPU-time budget per campaign and the number of repeated campaigns; the deduplication method for counting distinct bugs; a statement of which bugs are previously known versus new; and a script that turns the raw campaign logs into the paper's bug-count table — plus one honest sentence about variance across campaigns.

Degrees of reproducibility

Turnkey     one command regenerates each table from logged runs (aim for the smoke path)
Scripted    scripts exist but need documented manual steps or large external data
Descriptive prose detailed enough that a competent reader could rebuild the pipeline

For ISSTA, aim for at least a turnkey smoke path plus scripted full runs; long fuzzing or symbolic-execution campaigns may stay scripted if the budget and variance are documented. Stating the achieved level honestly beats promising turnkey behaviour that fails on a clean machine.

Output format

[Claim inventory] <claim -> evidence location>
[Substrate pinned] benchmark version / subject SHAs / toolchain: yes/no
[Stochastic disclosure] seeds / budgets / run count / variance: complete/partial/missing
[Availability] released / partially released / withheld-with-reason
[Paper fixes] <must appear in the body>
[Artifact fixes] <package or script additions>