Sigmetrics artifact evaluation

SIGMETRICS Artifact Evaluation

Use this for artifact/reproducibility packaging. SIGMETRICS sits in the ACM ecosystem and, where an artifact track runs, follows the ACM Artifact Review and Badging scheme. Two things to internalize: badges are earned by evaluators actually using your package, and — distinctively for SIGMETRICS — the package usually has to let an evaluator regenerate the figures from a seeded simulation and see them match the analytic prediction, not only run a tool. Confirm on the current cycle whether a formal artifact track exists and its timing (待核实).

The ACM badges (verify the current set and names)

Badge	What it certifies	What earns it
Artifacts Available	The artifact is permanently, publicly retrievable	Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage)
Artifacts Evaluated - Functional	The artifact runs and does what the paper says	A clean-machine install, a demo run of the simulator, documented expected outputs
Artifacts Evaluated - Reusable	Others can build on it	The Functional bar plus careful docs, structure, and licensing
Results Reproduced	An evaluator reproduced the paper's key results	A turnkey path from the artifact to the headline figures/numbers (and the analytic overlay)

Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced require the evaluator's own run to succeed, so the failure mode is "did not run / figures did not match on their machine," never "the theorem was weak."

What performance-evaluation evaluators open first

Claim type	First thing inspected	Common failure caught
An analytic bound + simulation	The script that regenerates the analysis-vs-simulation figure	Figure hard-coded; simulator does not actually produce the plotted curve
A measurement study	The scripts that turn the trace into the paper's tables	Numbers in the PDF that no script reproduces; trace missing
A scheduling/queueing policy	The seeded simulator and its steady-state handling	Non-deterministic runs; no seeds; warm-up not handled
A learning-for-systems result	Code plotting empirical regret against the proven bound	Requires unavailable data; guarantee not empirically checked

Assume an evaluator gives your package a bounded time budget on a clean machine. Design for the first ten minutes to succeed: a small demo that regenerates one headline figure quickly.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what the model is, how to install, how to run the simulator
              demo, how to regenerate each figure, expected runtime and outputs
[Mapping]     an explicit table: paper claim/figure -> script -> expected result (incl. the
              analytic overlay it should match)
[Simulator]   seeded, steady-state-aware; a fast demo config and the full (slow) config
[Data]        the processed trace/dataset (or documented access), not just the collection query
[Proofs]      the derivation appendix, so the analytic side is checkable alongside the code
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository for the Available badge

Anonymized review artifact vs. badge artifact

• At submission: the artifact is anonymized for the paper's reviewers — no owner strings, cluster paths, group names, or identity-revealing trace provenance (Operational Systems Track excepted).
• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version the badges attach to and the POMACS camera-ready cites.

Worked vignette: packaging a scheduling-policy paper

A paper contributes a scheduling theorem and a trace-driven evaluation. To target Reusable and Reproduced: ship a Docker image with the simulator pre-built; a run_demo.sh that regenerates the analysis-vs-simulation figure on a small config in under a minute; a reproduce/ directory whose scripts regenerate each table and the trace-driven comparison from logged runs; a claim-to-figure-to-script mapping in the README; the processed trace with pinned provenance; the proof appendix; and an MIT/Apache license. State honestly which figures are turnkey and which need the full (slow) simulation sweep.

Calibration

• Whether an artifact/reproducibility track runs a given cycle, its badges, timing, and whether it is mandatory or optional — all cycle-volatile; confirm on the current call (待核实).
• For POMACS, badges are typically pursued around/after acceptance; do not conflate the artifact process with the paper's shepherding.

Output format

[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <model/simulator/trace/proofs/provenance/license>
[Ten-minute test] does install + demo regenerate one headline figure on a clean machine? yes/no
[Claim mapping] <claim/figure -> script -> expected result (matches analysis?) present? yes/no>
[Fixes before upload] <ordered list>