Mobicom artifact evaluation

MobiCom Artifact Evaluation

MobiCom accepted papers may earn the three ACM badges — Artifacts Available, Artifacts Evaluated — Functional, and Results Reproduced — from a separate artifact-evaluation committee (AEC). The badges are independent: you may pursue one, some, or all. The engineering problem specific to MobiCom is that the evaluation depends on a radio testbed the AEC does not have, so the packaging must let an evaluator get as far as possible without one.

Choose badges as a claim of what you can prove

Badge	What it asserts	What the AEC needs
Artifacts Available	the artifact is publicly and permanently retrievable	a stable public archive (DOI-backed)
Artifacts Evaluated — Functional	the artifact runs and does what the paper describes	a working build + smoke run on reachable hardware
Results Reproduced	the paper's key results regenerate	a path from artifact to the headline numbers

Pick badges honestly. Available is almost always worth claiming and is the lowest-friction. Functional requires a path that runs without your specific radios. Results Reproduced is hardest for a wireless paper because the channel is not portable — claim it only where a trace-replay or downscaled path can regenerate the numbers (mobicom-reproducibility). The exact 2027 badge deadlines and process are 待核实 — read the current artifact call.

The hardware-optional path is the core deliverable

Most AEC members will not have your SDRs, tags, or testbed. Design the package so a reviewer without hardware still reaches Functional:

artifact/
  README            # claims, badge targets, hardware needed vs optional, time budget
  env/              # container or pinned deps; no radio required to build
  smoke/            # runs in ~minutes on a laptop, no radio; proves the pipeline
  replay/           # recorded captures + decode/analysis -> regenerates key figures
  hardware/         # optional: scripts + exact radio/firmware to reproduce on a testbed
  data/             # traces that can legally ship (or a loader + instructions)
  expected/         # expected outputs AND expected variance ranges

• The smoke run is the functionality proof: a short, laptop-only command that exercises the pipeline end-to-end on bundled data and prints a recognizable result.
• The replay path feeds recorded captures through the same processing that produced the paper's figures, so results regenerate without the radio.
• The hardware path is documented for the rare evaluator (or future reader) with a matching testbed, with exact radio model, firmware, and channel setup.

Make availability real

• Deposit in a permanent, DOI-backed archive, not a personal URL that rots.
• Confirm what can legally ship: firmware and proprietary radio stacks often cannot; user-identifiable captures must be stripped or aggregated; site data may be bound by an agreement (mobicom-reproducibility).
• If part of the artifact cannot be released, say so explicitly and ship the largest functional subset rather than a broken whole.

Document variance honestly

An over-the-air result differs run to run. Ship expected-variance files alongside expected outputs so a rerun that lands within a stated margin reads as a reproduction, not a failure. An AEC member who sees a number move and has no variance guidance will log it as a non-reproduction.

Pre-submission checklist

• Badge targets chosen to match what the package can actually prove.
• Smoke run works on a clean laptop with no radio, in a stated time budget.
• Replay path regenerates the key figures from bundled captures.
• Hardware path documents exact radio/firmware/channel for testbed reproduction.
• Available artifact is in a permanent DOI-backed archive.
• Legal/ethical shipping decision made for firmware, traces, and site data.
• Expected outputs and expected variance both provided.

Output format

[Badges] targeted vs justified by the package
[Smoke] laptop-only run present + time budget
[Replay] key figures regenerate without hardware? y/n
[Hardware path] radio/firmware/channel documented? y/n
[Availability] archive + DOI + legal-shipping decision
[Gaps] what blocks each targeted badge