Issta artifact evaluation

ISSTA Artifact Evaluation

Use this to package evidence for ISSTA's artifact-evaluation track. ISSTA has a genuinely strong artifact culture: a runnable tool and a shared benchmark are treated as normal, and the badges carry weight. Reopen the current artifact call before packaging — the badge set and the archival requirement are cycle-specific.

The ACM badge targets

Badge	What it certifies	What you must ship
Artifacts Available	The artifact is publicly, permanently retrievable	A DOI-issuing deposit (Zenodo) — a GitHub link alone does not qualify
Artifacts Evaluated — Functional	The artifact runs and does what the paper says	A working entry point, dependencies, and a documented expected output
Artifacts Evaluated — Reusable	Others can inspect, adapt, and build on it	Clean structure, real documentation, and configurability beyond the paper's runs
Results Reproduced	The paper's key results regenerate from the artifact	Scripts that produce the paper's tables/figures within stated tolerance

Available is about archival; Functional and Reusable are about engineering quality; Results Reproduced is about matching the paper. They are earned independently — decide which you are going for before you package.

Artifact plan

• Decide the badge set, then package to its bar. A Reusable badge needs documentation and structure a Functional-only artifact can skip.
• Ship the tool with a single documented entry point and a container or environment file, so an evaluator reaches a result without reconstructing your machine.
• Pin subjects and benchmark versions: the exact Defects4J revision, subject-program commit SHAs, and any fuzzing seed corpora, archived rather than referenced by name.
• Provide a results-regeneration path: a script that takes logged runs to the paper's tables, so "Results Reproduced" is a command, not an argument.
• Anonymize the review-time copy — repository owners, commit authors, container labels — because ISSTA artifact review is double-anonymous alongside the paper.
• After acceptance, make the Zenodo deposit public and citable, and record its DOI for the camera-ready badge display.

What ISSTA evaluators try first

• The README's quick-start, then the single command that produces one headline number. If that fails on a clean machine, no amount of internal quality is visible.
• A small, fast subset that finishes in minutes, before any full multi-day fuzzing or symbolic-execution campaign. Ship a "smoke" configuration explicitly.
• The mapping from a paper claim to the artifact output that supports it; an artifact whose outputs cannot be tied back to Table N reads as unverifiable.

Handling long-running and non-deterministic tools

Testing and analysis artifacts often run for hours and vary between runs. Package for that reality:

artifact/
  README.md            # quick-start, smoke config, full config, expected outputs, runtime
  Dockerfile           # pinned toolchain and dependencies
  subjects/            # pinned subject SHAs or a fetch script that pins them
  run_smoke.sh         # minutes: reproduces one representative row
  run_full.sh          # hours/days: reproduces all tables
  results/             # logged raw outputs from the authors' runs
  scripts/tables.py    # regenerates paper tables from results/

State the run count and expected variance for non-deterministic results, and have the regeneration script accept the evaluator's fresh runs as well as the shipped logs, so a partial reproduction still lands on the paper's numbers within tolerance.

Calibration anchors

• Evaluators are time-boxed; assume they run the smoke config and skim the full one. Design so the smoke path alone justifies Functional.
• Badge names, the Zenodo requirement, and any single-blind vs. double-blind detail vary by cycle; verify against the current artifact call rather than a past year.

Output format

[Badge target] Available / Functional / Reusable / Results Reproduced
[Entry point] <command + smoke runtime>
[Pinned subjects] <benchmark version / SHAs archived?>
[Reproduction level] turnkey / scripted / descriptive / weak
[Anonymity risks] <owners/authors/labels/paths>
[Fixes before upload] <ordered list>