Fast artifact evaluation

FAST Artifact Evaluation

Use this for the artifact track. FAST follows the USENIX artifact-evaluation model, and evaluation is a separate, post-acceptance process with its own timeline and an Artifact Evaluation Committee (AEC). Two things to internalize: badges are earned by evaluators actually using your package, and the review-time artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived, with an artifact appendix).

The three USENIX badges (verify the current set and names)

Badge	What it certifies	What earns it
Artifacts Available	The artifact is permanently, publicly retrievable	Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage)
Artifacts Functional	The artifact works and is useful for the paper's outcomes	A documented build/run, a demo, and expected outputs on the AEC's setup
Results Reproduced	An evaluator reproduced the paper's main results	A path from the artifact to the headline numbers

Available is a low-cost, high-value badge (archive the package). Functional and Reproduced require the evaluator's own run to succeed — and for a storage paper that run may need specific hardware, which shapes everything below.

The storage-artifact problem: hardware dependence

A storage result often depends on a particular SSD/NVM device, firmware, or a large trace. Design the package so an AEC can get as far as possible without your exact drive:

[Tier the claims]  which results are hardware-agnostic (analysis, plots from logs) vs. device-bound
                   (measured WA, tail latency)? Make the agnostic ones turnkey.
[Ship the logs]    include the raw device-counter logs and the scripts that turn them into figures,
                   so "Reproduced" is achievable from data even when the device is unavailable
[Document devices] state exact models + firmware; where feasible, offer a smaller commodity-SSD path
                   the AEC can run to see the trend, and say the factor is device-specific
[Bound the runtime] long endurance/aging runs: provide a short demo + the full slow protocol, and
                   label which results need the full run

What a storage AEC opens first

Claim type	First thing inspected	Common failure caught
A storage system/tool	The README and one build+run command	Undocumented deps; only-builds-on-authors'-kernel
A measured storage cost (WA/latency)	The logs + the script from logs to the figure	Numbers in the PDF no script regenerates
A trace-driven study	The replay tool + the archived trace	Trace named, not shipped; replay settings missing
A crash-consistency claim	The fault-injection / replay harness	Only the FS code, no way to reproduce the test
A reliability field study	The (de-identified) dataset + analysis scripts	Aggregates only; nothing an evaluator can re-run

Assume an evaluator gives your package a bounded time budget. Design for the first ten minutes (build

• a small demo + a figure-from-logs) to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment; avoid "install these 40 things by hand"
[README]      one screen: what it is, build, run the demo, reproduce each claim, expected runtime/outputs
[Mapping]     an explicit table: paper claim -> script -> expected result (mark device-bound ones)
[Data]        the archived traces and raw device-counter logs, not just names
[Provenance]  device models + firmware, host/kernel, mkfs/mount options, seeds
[License]     an OSI-approved license so the artifact can be evaluated toward Functional/Reproduced
[Archive]     deposit in a DOI-issuing repository for the Available badge
[Appendix]    write the artifact appendix; add the earned badges to the camera-ready appendix after AE

Anonymized review artifact vs. badge artifact

• At submission: anonymized for the paper's reviewers — no owner strings, hostnames, cluster paths, serials in device dumps, lab names, or identity-revealing trace hosts.
• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; write the artifact appendix; this is the version the AEC badges and the camera-ready cites, with badges shown on the first page.

Worked vignette: packaging a KV-store + measurement study

A paper contributes an SSD-conscious KV store and a write-amplification study. To target Reproduced: ship a Docker image with the store pre-built; a run_demo.sh that runs a short YCSB load on a commodity SSD (or a loopback file) in minutes; a reproduce/ directory whose scripts regenerate each figure from the shipped device-counter logs so the headline numbers reproduce even without the exact datacenter SSD; a claim-to-script table marking which results are device-bound; the archived trace with replay scripts; and an MIT/Apache license. State honestly which numbers are the specific drive's and which are the trend.

Calibration

• The artifact timeline is after acceptance and independent of the camera-ready deadline; do not conflate them.
• Badge names, the exact set offered, and whether AE is single- or double-anonymous vary by cycle (待核实 for the precise FAST '27 AEC anonymity model) — confirm on the current Call for Artifacts.

Output format

[Target badges] Available / Functional / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <system/traces/device-logs/scripts/provenance/license>
[Hardware tiering] <which claims turnkey-from-logs vs. device-bound; demo path present?>
[Ten-minute test] does build + demo + a figure-from-logs succeed on a clean machine? yes/no
[Fixes before upload] <ordered list>