Oopsla artifact evaluation

OOPSLA Artifact Evaluation

SPLASH runs a unified artifact-evaluation track for its research papers; the 2026 edition used a two-phase process — an opening kick-the-tires pass where evaluators check the artifact builds and starts at all, then the full evaluation — awarding Functional (documented, complete enough to exercise), Reusable (Functional plus organization and documentation that support reuse by others), and Available (a snapshot deposited on Zenodo with a DOI), with Available strongly encouraged for every passing artifact absent licensing or privacy constraints. Per-cycle dates and submission mechanics live on the current track page (待核实 each cycle).

Design for the kick-the-tires failure mode

Most artifacts that miss a badge die in the first hour of a stranger's time: a build that assumes your machine. The discipline is to treat the evaluator as a hostile fresh VM.

Golden-path contract (put this at the top of README.md):
  1. Requirements: OS/arch, RAM, disk, expected wall-clock time
  2. One command to build (container image or pinned toolchain)
  3. One command for a 10-minute smoke result mapped to a named claim
  4. One command per paper table/figure, each with expected output range
  5. What may legitimately differ on other hardware, and by how much

Dry-run the contract yourself on a machine that has never seen the project — laptop-hosted paths, private registries, and license-gated dependencies are the classic silent breakers.

Badge strategy

Badge	What actually earns it	Cheap mistakes that forfeit it
Functional	Inventory + docs sufficient to exercise the artifact; results consistent with the paper	Missing inputs; undocumented flags; claims that need hardware you didn't disclose
Reusable	Clear structure, extension points, explained internals — a stranger could build on it	A working but opaque tarball; hard-coded paths; no guidance beyond replication
Available	Zenodo (or equivalent archival) deposit with DOI	GitHub-only "archive" (mutable, not archival); DOI minted after the camera-ready statement was frozen

Reusable is where OOPSLA-style artifacts differentiate: language implementations, calculi mechanizations, and corpus studies are exactly the artifacts other groups extend, so structure the repository as a tool, not as a paper appendix.

Claim-to-artifact mapping

Evaluators read the paper's claims against what the artifact demonstrates. Build the mapping explicitly and reuse it in three places: the artifact README, the evaluation submission form, and the paper's Data-Availability Statement (oopsla-submission requires the statement; oopsla-camera-ready updates it with the final DOI).

• Every quantitative table/figure → a script that regenerates it.
• Every qualitative claim ("scales to", "handles all of") → a named test or corpus directory that witnesses it.
• Every claim the artifact cannot support (proprietary benchmark, cluster scale) → an explicit exclusion with justification, declared up front rather than discovered by the evaluator.

Scope and timing notes

• Artifact evaluation is tied to acceptance rounds; papers accepted in Round 1 and Round 2 flow through the process in different windows — read the track page for the round you were accepted in.
• Zenodo deposits are versioned: mint the DOI early, then publish updated versions as evaluation feedback lands; the DOI in the published article should resolve to the final version.
• Anonymity is over at this stage, but the paper's review-time supplement and the artifact must not contradict each other (oopsla-supplementary).

Output format

[Phase readiness] kick-the-tires dry-run: pass / failures listed
[Badge targets] Functional / +Reusable / +Available, with gaps per badge
[Claim map] <n claims mapped / m unmapped — list unmapped>
[Deposit] Zenodo DOI status + version plan
[Statement sync] Data-Availability Statement consistent: yes/no