Pldi artifact evaluation

PLDI Artifact Evaluation

PLDI runs artifact evaluation after paper acceptance, deliberately separating paper review from artifact review, and it accepts "any artifact authors wish to submit, broadly defined" — compilers, proofs, benchmark suites, measurement infrastructure (PLDI 2026 research-artifacts track, read 2026-07-08; AE co-chairs for 2026 were Raphaël Monat and Qirun Zhang). Badges appear on the published PACMPL article, so the artifact is part of the paper's permanent record.

The badge ladder

Badge	2026-cycle criterion	What actually earns it
Functional	Artifact supports the paper's claims and works as documented	An evaluator reproduces your headline flow from the README alone
Reusable	Only awarded to Functional artifacts judged especially well packaged, documented, and designed for future research	Extension points documented; someone could swap in a new benchmark or pass
Available	Automatic once the complete artifact is archived publicly in an archival location	A Zenodo snapshot with a DOI — a lab GitHub link does not qualify

Two structural kindnesses in the 2026 rules: artifacts are requested only after acceptance, and there is no artifact camera-ready deadline because Zenodo accepts new versions at any time. Use that slack for polish, not procrastination — the evaluation itself still runs on the AE committee's timetable.

Package for a hostile machine

Your evaluator has a laptop, a deadline, and no institutional access to your cluster. Design for that:

artifact/
  README.md            # claims covered, est. runtimes, kick-the-tires steps
  LICENSE
  Dockerfile           # pinned base image, pinned compiler versions
  claims.md            # paper claim -> script -> expected output, one row each
  scripts/
    smoke.sh           # <10 min: builds, runs one benchmark, checks one number
    reproduce_all.sh   # full run with per-experiment time estimates
  benchmarks/          # provenance + versions of every program measured
  results/expected/    # our outputs, for diffing

Rules of thumb that decide badges:

• A ten-minute smoke test is the highest-leverage file in the archive. Most negative AE experiences begin with a build that fails in minute one.
• Pin everything: compiler versions, benchmark-suite revisions, flags. "Latest LLVM" is a different artifact every month.
• Map claims to scripts explicitly. Evaluators check the paper's tables against your outputs; make the correspondence a table, not a scavenger hunt.
• State hardware sensitivity honestly. If speedups need AVX-512 or 64 GB of RAM, say so up front and provide a reduced-scale mode that still shows the trend.
• For proof artifacts, pin the proof assistant version and make make check verify the exact theorem names cited in the paper.

Cross-check with the paper

The evaluation protocol inside the artifact must match what pldi-experiments put in the paper: same warmup discipline, same repetition counts, same statistics. An artifact that reruns 3 iterations when the paper reports 30-run confidence intervals invites exactly the doubt AE exists to dispel.

待核实 each cycle

Kick-the-tires/rebuttal phases, badge wording, submission format (VM vs container), and the AE timetable are reset per edition — reread the current research-artifacts track page before packaging.

Output format

[Badge target] Available / +Functional / +Reusable
[Smoke test] exists? runtime? passes in a fresh container?
[Claim map] paper table/claim -> script -> expected output (n rows)
[Pinning] compilers / benchmarks / base image / proof assistant
[Archive] Zenodo DOI minted? version matches camera-ready?