Ndss artifact evaluation

NDSS Artifact Evaluation

NDSS added artifact evaluation to the symposium in recent editions (running on the community secartifacts platform, verified for 2024-2026), and it is opt-in after conditional acceptance — a separate track, not a submission requirement. Papers that pass carry badges on the first page and add a 2-page artifact appendix (Call for Artifacts, checked 2026-07-08). Decide early whether you are engineering for it, because the packaging choices are cheap during the project and expensive afterward.

The three badges

Badge	Claim to the committee	What you must ship
Available	The artifact is publicly, permanently retrievable	Deposit in an archival repository with a DOI (Zenodo, figshare); a bare GitHub URL is not "available"
Functional	It does what the paper says, is usable and documented	Build/run instructions that work on a clean machine; the components matching the paper's claims
Reproduced	The paper's results can be regenerated by others	One documented path per headline result, with expected output and tolerances

Badges are independent — you can earn Available and Functional without Reproduced when the world (live targets, proprietary data) genuinely prevents re-execution. Claim only what you can defend; an evaluator who cannot reproduce a "Reproduced" claim writes that in the report.

From claims ledger to artifact

The claims.yml ledger built in ndss-reproducibility is the artifact's spine. Convert it into an evaluator-facing package:

artifact/
  README.md          # what this is, badges sought, hardware/time needs, safety notes
  INSTALL.md         # clean-machine setup; pinned deps / container image
  claims/            # one entry per headline result:
    fig4/  run.sh    # → expected_output/, runtime + resource note
    tab2/  run.sh
  data/              # snapshots or synthetic substitutes (never raw victim traffic)
  src/               # the code, with the dangerous parts gated (below)
  ETHICS.md          # disclosure status, gating rationale, usage boundaries

Give evaluators the shortest true path to each result. A "kick-the-tires" phase means the first five minutes decide the evaluator's mood — make the smoke test one command.

Packaging dangerous material responsibly

NDSS artifacts routinely include working exploits, attack tooling, or malware-adjacent code. This is allowed; unbounded release is not.

• Gate the weaponized parts. Ship the analysis and measurement pipeline openly; put live exploit modules behind documented access conditions or a defanged demonstration mode consistent with your Ethics Considerations section.
• Align with the disclosure clock. If a vulnerability is under a coordinated-disclosure embargo, the artifact's release timing must respect it — state the date the full module becomes available.
• Sandbox by default. Anything that touches a network should target a supplied local testbed, not the Internet, out of the box. Document the blast radius.
• No live victim data. Replace with prefix-preserved or synthetic traces carrying the statistics the result needs (see ndss-reproducibility).

When re-execution needs your lab

Hardware-bound results (SDRs, specific switches, IoT devices) cannot be reproduced by a committee at their desks. Bridge the gap: provide recorded traces plus a replay harness so the analysis is reproducible even when the capture is not, and say plainly in the README which results are trace-replayable and which need the physical setup. This commonly yields Functional + Reproduced-on-recorded-data rather than a failed full Reproduced.

The 2-page artifact appendix

Written after evaluation for the camera-ready. It states the badges earned, the artifact's scope, access instructions (the DOI), hardware/software requirements, and the mapping from paper claims to artifact components. Keep it factual and navigational — it is a map for future readers of the open-access paper, not a second results section.

Timing

The opt-in comes shortly after (conditional) acceptance and runs on its own schedule (2027 dates 待核实 — confirm on the Call for Artifacts). It overlaps the camera-ready window (Jan 6, 2027), so a team that starts packaging at acceptance, not at the AE deadline, avoids a two-front crunch.

Output format

[Badges targeted] Available / Functional / Reproduced — with feasibility per badge
[Package status] README / INSTALL / per-claim runners / data substitutes — done or gaps
[Danger handling] gated modules, sandbox default, disclosure-aligned release date
[Hardware gap] results needing physical setup + trace-replay bridge
[Appendix] 2-page artifact appendix drafted? claim→component map complete?