Ppopp artifact evaluation

PPoPP Artifact Evaluation

Use this for the artifact track. PPoPP runs a post-acceptance artifact evaluation with its own deadline (PPoPP 2026: artifact submission 17 November 2025, notification 5 January 2026), separate from the camera-ready and shared in culture with co-located CGO. Two things to internalize: PPoPP has a distinctive badge policy, and the hard part of a parallel artifact is that the results must reproduce on someone else's hardware, where core counts and topology differ from yours.

PPoPP's badge policy (verify the current set each cycle)

Badge	Colour	What it certifies	How it is awarded
Artifacts Available	green	The artifact is publicly, permanently retrievable	By the publisher, from a deposited-artifact link — no formal audit
Artifacts Evaluated — Functional	lighter red	The artifact runs, is documented, consistent, complete, exercisable	AE committee runs it
Artifacts Evaluated — Reusable	darker red	Functional plus quality that exceeds minimal functionality; carefully documented and structured for reuse	AE committee runs it
Results Reproduced	darker blue	The paper's main results were obtained by the evaluators (within tolerance)	AE committee re-runs and matches

Two PPoPP-specific facts to get right:

• The committee awards one of Functional / Reusable (Reusable is the higher bar) plus possibly Results Reproduced.
• PPoPP does not award the lighter-blue "Results Replicated" badge — do not target or claim it. Available is not evaluated; it is publisher-granted from a link, so it is the cheapest badge to secure and worth doing regardless.

What makes a parallel artifact hard

An evaluator does not have your machine. A speedup that depended on your 96-core dual-socket node will not reproduce on a reviewer's 16-core laptop, and "Results Reproduced" requires agreement within a stated tolerance. Design for portability of the conclusion, not the absolute numbers:

[Hardware doc]  state the exact machine you used (CPU/GPU model, sockets, cores, NUMA, memory,
                interconnect) and the minimum config on which the trend still holds
[Scaled runs]   provide a small/fast configuration whose *shape* (linear region, saturation)
                matches the paper, so an evaluator can reproduce the claim on modest hardware
[Tolerance]     state explicitly what "reproduced" means for your numbers (e.g. within 10%, or
                "monotone speedup to the machine's core count")
[Determinism]   fix seeds; pin threads; document warm-up; make runs repeatable
[GPU]           document driver/CUDA/ROCm versions and the GPU class needed; provide a CPU
                fallback path where possible

What evaluators open first

Claim type	First thing inspected	Common failure caught
A concurrent structure	README + a one-command throughput run	Only builds on the authors' toolchain; hardcoded core count
A runtime/scheduler	The script that reproduces a scaling figure	Numbers in the PDF that no script regenerates
A GPU technique	Build + a small kernel run	Requires a specific GPU with no fallback; missing driver versions
A parallel algorithm	Build + a scaled input run	Data missing; topology assumptions undocumented

Assume a bounded time budget on a clean machine; make the first ten minutes succeed.

Packaging plan

[Container]   a Dockerfile or pinned environment (spack/conda/lockfile); avoid manual 40-step builds
[README]      one-screen orientation: what it is, how to build, how to run the demo, how to
              reproduce each figure, expected runtime, and the hardware each run needs
[Mapping]     an explicit table: paper claim/figure -> script -> expected result (+ tolerance)
[Scaling kit] scripts that sweep thread/core count and emit the paper's curve on the eval machine
[Provenance]  exact hardware used, compiler/flags, driver versions, seeds, pinning policy
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository (Zenodo/figshare/Software Heritage) for Available

Anonymized review artifact vs. badge artifact

• At paper submission: the artifact (if attached for reviewers) is anonymized — no owner strings, cluster/account paths, lab names, or identifying links.
• At the AE track (post-acceptance): the de-anonymized, licensed, DOI-archived version is what evaluators badge and the camera-ready cites.

Calibration

• The AE deadline is after acceptance and independent of the camera-ready; do not conflate them.
• Confirm each cycle whether AE is run jointly with CGO, the exact badge set and colours, and whether Available is auto-granted by the publisher (as verified) or handled by the committee.

Output format

[Target badges] Available (publisher) / Functional or Reusable / Results Reproduced  (NOT Replicated)
[Artifact role] anonymized review artifact / public badge artifact
[Portability] scaled config whose trend matches the paper on modest hardware? tolerance stated?
[Ten-minute test] build + demo succeeds on a clean machine? yes/no
[Claim mapping] each figure -> script -> expected result (+ tolerance) present? yes/no
[Fixes before upload] <ordered list>