Fse artifact evaluation

FSE Artifact Evaluation

Use this for the artifact track. FSE follows the ACM Artifact Review and Badging scheme, and the artifact evaluation is a separate, post-acceptance process with its own deadline. Two things to internalize: badges are earned by evaluators actually using your package, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived).

The ACM badges (verify the current set and names)

Badge	What it certifies	What earns it
Artifacts Available	The artifact is permanently, publicly retrievable	Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage)
Artifacts Evaluated - Functional	The artifact runs and does what the paper says	A clean-machine install, a demo, and documented expected outputs
Artifacts Evaluated - Reusable	Others can build on it	The Functional bar plus careful docs, structure, and licensing
Results Reproduced	An evaluator reproduced the paper's key results	A turnkey path from the artifact to the headline numbers

Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced require the evaluator's own run to succeed, so the failure mode is always "did not run on their machine," never "the idea was weak."

What SIGSOFT evaluators open first

Claim type	First thing inspected	Common failure caught
A tool/technique	The README and one install/run command	Undocumented dependencies; only-works-on-authors'-laptop
An empirical study	The scripts that turn data into the paper's tables	Numbers in the PDF that no script reproduces
A mined dataset	The extraction scripts + the extracted data	Query shipped, data missing; provenance unpinned
An LLM-based result	Cached prompts/outputs + model IDs	Requires live API keys; not reproducible

Assume an evaluator gives your package a bounded time budget on a clean machine. Design for the first ten minutes to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what it is, how to install, how to run the demo, how to
              reproduce each claim, expected runtime and outputs
[Mapping]     an explicit table: paper claim -> script -> expected result
[Data]        the extracted dataset itself (or documented access), not just the query
[Provenance]  repo SHAs, extraction dates, model IDs/dates, seeds
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository for the Available badge

Anonymized review artifact vs. badge artifact

• At submission: the artifact is anonymized for the paper's reviewers — no owner strings, cluster paths, lab names, or identity-revealing links, and no live repository that discloses authors.
• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version the artifact evaluators badge and the camera-ready cites.

Worked vignette: packaging a detection tool + study

A paper contributes a defect-detection tool and an empirical evaluation. To target Reusable and Reproduced: ship a Docker image with the tool pre-built; a run_demo.sh that detects on a small bundled project in under a minute; a reproduce/ directory whose scripts regenerate each table from logged results; a claim-to-script mapping table in the README; the extracted evaluation dataset with pinned SHAs; and an MIT/Apache license. State honestly which results are turnkey and which need the full (slow) dataset run.

Calibration

• The artifact deadline is after acceptance and independent of the camera-ready; do not conflate them.
• Badge names, the exact set offered, and whether evaluation is single- or double-anonymous vary by cycle — confirm on the current artifact-track call.

Output format

[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <tool/data/scripts/provenance/license>
[Ten-minute test] does install + demo succeed on a clean machine? yes/no
[Claim mapping] <claim -> script -> expected result present? yes/no>
[Fixes before upload] <ordered list>