Atc artifact evaluation

ATC Artifact Evaluation

Use this for the artifact track. ATC follows the shared USENIX/SIGOPS artifact-evaluation scheme: an Artifact Evaluation Committee (AEC) runs a separate, post-acceptance process through HotCRP and awards up to three badges — Artifacts Available, Artifacts Functional, and Results Reproduced. Two things to internalize: badges are earned by evaluators actually using your package, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived). ATC's artifact culture is strong and high-participation — for USENIX ATC '25, most evaluated artifacts earned Functional and a large majority earned Reproduced.

The three badges (verify the current set and names)

Badge	What it certifies	What earns it
Artifacts Available	The artifact is permanently, publicly retrievable	Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage)
Artifacts Functional	It is documented, complete, and exercisable — it runs and does what the paper says	Clean-machine install, a demo, scripts + data to run the experiments
Results Reproduced	An evaluator reproduced the paper's main results	A turnkey path from the artifact to the headline figures/tables

Available is low-cost, high-value (archive the package). Functional and Reproduced require the evaluator's own run to succeed, so the failure mode is always "did not run on their machine," never "the idea was weak."

What AEC evaluators open first

Claim type	First thing inspected	Common failure caught
A built system / mechanism	The README and one install/run command	Undocumented deps; only-works-on-authors'-hardware
A performance result	The scripts that turn a run into the paper's figures	Numbers in the PDF no script reproduces
A result needing specific hardware (RDMA/NVMe/NIC)	The documented host requirements + a small-scale mode	Requires unavailable hardware; no fallback
A trace-driven study	The traces + the processing scripts	Query shipped, data missing; provenance unpinned

Assume the evaluator has a bounded time budget on a clean machine and may lack your exact hardware. Design the first ten minutes to succeed and provide a reduced-scale path.

Packaging plan

[Container]   a Dockerfile or pinned environment; document any host/hardware the container cannot abstract
[README]      one-screen orientation: what it is, install, run the demo, reproduce each claim, expected runtime/outputs
[Mapping]     an explicit table: paper claim -> script -> expected figure/table -> tolerance
[Scale]       a full-scale path AND a small-scale mode for evaluators without the hardware
[Data]        the traces/datasets themselves (or durable documented access), not just the query
[Provenance]  kernel/OS versions, hardware models, commit SHAs, workload seeds, run counts
[License]     an OSI-approved license so the artifact can be shared and reused
[Archive]     deposit in a DOI-issuing repository for the Available badge

Anonymized review artifact vs. badge artifact

• At submission (review): anonymized for the paper's double-blind reviewers — no owner strings, cluster hostnames, lab/product names, or identity-revealing links, and no live repo that discloses authors (see atc-reproducibility).
• After acceptance (badge): replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version the AEC badges and the camera-ready cites.

Worked vignette: a storage-cache policy + measurement

A paper contributes a flash-cache admission policy and a testbed evaluation. To target Functional and Reproduced: ship a container with the modified server pre-built; a run_demo.sh that exercises the policy on a bundled trace sample in a minute; a reproduce/ directory whose scripts regenerate each figure from logged runs; a claim-to-script mapping with tolerances in the README; the production-derived trace sample (with pinned provenance) plus documentation of the full trace's access; a documented note that the full-scale p99 result needs the 12-node testbed, with a single-node small-scale mode; and an MIT/Apache license. State honestly which results are full-scale-only.

Calibration

• The AEC deadline is after acceptance and independent of the camera-ready — do not conflate them (plan both in atc-workflow).
• Badge names, the exact set offered, and whether evaluation is single- or double-anonymous can vary by cycle and moved in the SIGOPS transition — confirm on the current call for artifacts.

Output format

[Target badges] Available / Functional / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <system/data/scripts/provenance/license/archive>
[Ten-minute test] install + demo succeed on a clean machine? yes/no
[Reproduce] claim -> script -> expected figure present? small-scale mode for missing hardware? yes/no
[Fixes before upload] <ordered list>