Percom artifact evaluation

PerCom Artifact Evaluation

Use this for reproducibility packaging. First, a cycle caveat: unlike SIGSOFT venues, PerCom has not historically run a mandatory formal artifact-evaluation track with a fixed badge set, and whether a given edition offers a reproducibility/badging track (e.g., IEEE Open Research Objects / Results Reproduced) is 待核实 — confirm on the current call. Regardless of whether a badge is offered, a well-packaged, de-identified sensing dataset and reproducible pipeline is a scored strength in the double-blind review and a lasting community contribution.

What "reproducible" means for a ubicomp sensing paper

Two deliverables, kept distinct:

• The anonymized review package (at submission): dataset link and code scrubbed of owner, testbed, and lab identity, for the double-blind reviewers.
• The public deposit (after acceptance): a de-identified dataset and code in a DOI-issuing archive under an open license — the version others cite and reuse, and the version any badge program evaluates.

What a ubicomp evaluator opens first

Claim type	First thing inspected	Common failure caught
An activity/context recognizer	The script that regenerates cross-subject (LOSO) results	Only a pooled-accuracy script; no leave-one-subject-out path
A sensing dataset	The data itself + a datasheet (subjects, sensors, labels)	Link present, data missing; no de-identification described
A deployed system	A demo on bundled sample data	Only-runs-on-authors'-testbed; hardware not documented
A model result	Trained weights + inference on sample input	Requires the full raw dataset or private compute to run

Assume an evaluator gives your package a bounded time budget on a clean machine with no access to your sensors or subjects. Design for the first ten minutes — a demo on bundled, de-identified sample data — to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what it is, install, run the demo, reproduce each claim,
              expected runtime and outputs
[Datasheet]   a dataset datasheet: subjects (count, relevant demographics), sensors (device,
              firmware, sampling rate, placement), labels + protocol, and known biases
[Mapping]     an explicit table: paper claim -> script -> expected result (with the LOSO split)
[Data]        the de-identified dataset itself (or documented restricted-access), not just a query
[Ethics]      IRB/consent status and the de-identification performed before release
[License]     an open, DOI-issuing deposit (IEEE DataPort, Zenodo) so others can reuse and cite

De-identification is the ubicomp-specific bar

Human-subjects sensing data leaks identity in ways code does not: raw audio/video, GPS traces, timestamps that pinpoint a home, and even accelerometer gait can re-identify. Before any release:

• Remove or transform direct identifiers and re-identifying traces; document exactly what you did.
• Confirm your consent and IRB approval permit public release of the de-identified data — some approvals do not, and then the honest move is restricted access with a documented request path.
• Never ship a dataset that a subject did not consent to have published.

Worked vignette: a wearable-HAR dataset + recognizer

To make a HAR paper reproducible: ship a Docker image with the recognizer pre-built; a run_demo.sh that classifies on a small bundled, de-identified sample in under a minute; a reproduce/ directory whose scripts regenerate the leave-one-subject-out F1 table (not just a pooled number) from logged features; a datasheet listing subjects, sensor placement, and sampling rate; the de-identified dataset with a documented consent/IRB basis; and an open license with a DOI. State honestly which results are turnkey and which need the full (slow) training run.

Calibration

• Whether a badge/reproducibility track runs, and which badges, is 待核实 per cycle — confirm on the current call rather than assuming a SIGSOFT-style scheme.
• The DOI-issuing deposit and datasheet are portable value even when no badge is offered.
• Anonymize the review package; de-identify (and confirm consent for) the public dataset — these are different obligations.

Output format

[Track status] formal reproducibility/badge track this cycle? yes/no/待核实
[Artifact role] anonymized review package / public de-identified deposit
[Contents] <recognizer/dataset/datasheet/scripts/ethics/license>
[Ten-minute test] does install + demo on bundled sample data succeed on a clean machine? yes/no
[Cross-subject reproduction] does a script regenerate the LOSO result? yes/no
[De-identification] documented + consent/IRB permits release? yes/no
[Fixes before deposit] <ordered list>