Conext artifact evaluation

CoNEXT Artifact Evaluation

Turn the accepted paper's package into optional ACM reproducibility badges, administered by CoNEXT's reproducibility committee. This is a distinct process from the paper review, with its own timing: the decisive move — opting in before the paper submission deadline — happens long before you have an accept. Treat badging as a track you enter at submission and complete after acceptance.

The badge timeline (opt-in is the gate)

[Before submission deadline]  OPT IN for ACM badging  <-- miss this and you are ineligible this cycle
[Within ~1 week of accept]    send a ONE-PAGE artifact description + pointers to the committee
[Post-accept evaluation]      committee reviews the artifact against the ACM badge criteria
[Camera-ready]                approved badges appear on the PACMNET article (see conext-camera-ready)

The ACM badges (what each requires)

CoNEXT uses the ACM Artifact Review and Badging scheme. In practice the ladder is:

Badge	What it certifies	What you must provide
Artifacts Available	The artifact is publicly archived	A DOI-issuing archive (Zenodo/figshare/Software Heritage) with an open license
Artifacts Evaluated — Functional	It is documented, complete, and runs	Clear docs, a runnable package, and a "getting started" path that works
Artifacts Evaluated — Reusable	It is functional and others can build on it	Reusability docs, clean structure, parameterization, and a license that permits reuse
Results Reproduced	Key results were regenerated by evaluators	A scripted path that regenerates the paper's central figures/tables from the inputs

Confirm the exact badge set and names offered by the current CoNEXT reproducibility committee — the scheme is stable but wording is cycle-volatile (待核实).

The one-page artifact description

The committee starts from your one-pager, so make it do work:

• Inventory: code, traces, configs, testbed scripts — with a pointer to each.
• Requirements: the hardware/software an evaluator needs. Networking artifacts often need specific hardware (a particular switch ASIC, NIC, or SDR); if evaluators need access to hardware you must provide, flag it prominently and describe how you will give access.
• Claims supported: which paper results the artifact reproduces, and which it cannot (be honest about hardware-bound results).
• Getting started: the shortest path to a first successful run.

Networking-specific evaluation hurdles

• Hardware dependence. A result that needs a specific switch or NIC is hard for a remote evaluator; provide a testbed reservation, a remote-access path, or a faithful emulation for the parts that can run in software, and clearly mark what is hardware-bound.
• Trace size and privacy. Large or sensitive traces may not be fully releasable; ship a representative or synthetic subset plus the pipeline, and document the restriction.
• Environment drift. Pin firmware/OS/kernel versions and provide images; "works on our testbed" is not reproducible.
• Live dependencies. Cache anything that would otherwise require a live API/network call so the evaluator's run is deterministic.

Reusability that survives an evaluator

• A README that assumes nothing about your lab's setup; every path and dependency explicit.
• A single command (or a short script) for the "getting started" run, and a documented full-reproduction path for the central results.
• An open license present and consistent across the archive.
• A mapping from paper claim → artifact step → expected output, so an evaluator can check "Results Reproduced" without guessing.

Pre-evaluation checklist

[Opt-in]        confirmed before the submission deadline? yes/no
[Archive]       DOI-issuing archive + open license? yes/no
[One-pager]     inventory + requirements + hardware-access plan + claims map? yes/no
[Functional]    fresh-environment run succeeds from the docs alone? yes/no
[Reusable]      structure + docs let a third party adapt it? yes/no
[Reproduced]    central figures/tables regenerate by script? yes/no
[Hardware]      hardware-bound results flagged; access path provided? yes/no

Output format

[Badge target] Available / Functional / Reusable / Reproduced (which are in scope)
[Opt-in status] done before submission? yes/no
[Artifact readiness] <per-badge gaps>
[Hardware plan] <how remote evaluators reach any required hardware>
[One-pager] <drafted / sent to committee>