Sensys artifact evaluation

SenSys Artifact Evaluation

SenSys awards three independent ACM badges through an Artifact Evaluation Committee: Artifacts Available, Artifacts Evaluated — Functional, and Results Reproduced. They are independent — you may pursue one, two, or all three — and awarded badges are printed on the paper and recorded in the ACM DL. The hard part at SenSys is that your evidence is physical: an evaluator usually does not have your motes, your harvester, or your deployment, so the artifact must be built to be judged without them.

Choose the badges deliberately

Badge	Bar	Hardest part at SenSys
Artifacts Available	Artifact deposited in a permanent public archive with a DOI	Deciding what firmware/traces you can legally and safely release
Artifacts Evaluated — Functional	The artifact runs and does what the paper says	Giving an evaluator without your hardware a way to reach "it runs"
Results Reproduced	Key results re-obtained by the evaluator	Reproducing hardware-measured energy/latency numbers off your testbed

Available is the cheapest and worth claiming almost always. Functional and Reproduced are where the hardware-optional path earns its keep.

Build the hardware-optional path first

Most AEC members will not have your testbed. Give them a graded way in:

Tier 0 — Available:   archived repo (DOI), firmware sources, traces, README.
Tier 1 — Bench/replay: recorded sensor + power traces the analysis re-runs on any laptop,
                       reproducing the paper's figures from stored data.
Tier 2 — Emulation:   a QEMU/renode-class emulator or a single dev board that reaches
                       "it runs" without the full deployment.
Tier 3 — Full HW:     scripts + BOM for an evaluator who does have the platform.

A trace-replay path is the single most valuable thing you can ship: it lets an evaluator reproduce your figures from your recorded energy and sensor data even if they cannot re-run the deployment. Document exactly which figures/tables the replay reproduces and which need hardware.

Document the provenance the AEC cannot infer

artifact/
  README.md            # claims → which script/trace reproduces each figure/table
  firmware/            # sources + toolchain version + build flags
  traces/              # recorded power + sensor data behind each figure
  analysis/            # scripts that turn traces into the paper's plots
  hardware/            # BOM, board revision, wiring, instrument model + settings
  ENERGY.md            # instrument, sampling rate, integration boundaries
  GROUND_TRUTH.md      # how reference labels were obtained and their error
  LICENSE

The ENERGY.md and GROUND_TRUTH.md files are SenSys-specific: an evaluator reproducing a number needs the measurement method, not just the code (see sensys-reproducibility).

Smoke-run before you submit

Prove the package installs and runs from clean before an evaluator ever sees it:

# Package hygiene (manifests, seeds, scripts) — see resources/code/README.md
python3 ../../../shared-resources/ml-conference-methods/code/check_repro_package.py ./artifact
# Then the SenSys-specific smoke: does the trace-replay reproduce a headline figure?
cd artifact/analysis && ./reproduce_fig4.sh   # should regenerate Fig. 4 from traces/, no hardware

If the replay does not regenerate a figure on a clean machine, no badge claim is safe yet.

Work with the AEC's iteration

The committee may ask for revisions and iterate with you. Respond fast and concretely: a missing dependency or an unclear step is a quick fix, and the AEC is trying to award the badge, not deny it. Keep the anonymity rules of the round if evaluation overlaps the review window.

Output format

[Badges]   which of the three you are pursuing, and why each is reachable
[HW-path]  the graded path (Available/Replay/Emulation/Full-HW) — which tiers exist
[Provenance] ENERGY.md + GROUND_TRUTH.md + firmware toolchain present? pass/gap
[Smoke]    does trace-replay reproduce a headline figure on a clean machine? Y/N
[Open]     the gap most likely to block Functional or Reproduced