Uai artifact evaluation

UAI Artifact Evaluation

UAI posted no separate artifact-evaluation track or badge system for the 2026 cycle (existence of one in later cycles: 待核实). What it did post is sharper than a badge: code and data release is strongly encouraged, a 50 MB anonymous ZIP rides with the submission, and reviewers judge whether claims are "backed up convincingly." The artifact's job is to convert a skeptical probabilistic-ML reviewer's spot-check into confirmation.

Design for the five-minute skeptic

Because reviewers are explicitly not required to consult supplementary material, assume whoever opens the ZIP gives it five minutes. Optimize for that reader:

• A top-level README that states, in its first ten lines, what claim each script reproduces and how long it takes.
• One command per headline result, each with a small-scale mode that finishes on a laptop: a reviewer who reproduces Figure 2 at n=500 in ninety seconds will extend you trust for the n=50,000 version.
• Pinned dependencies and one canonical entry point. A Makefile or single driver script beats a directory of loose notebooks.
• Expected outputs stored beside the scripts, so "did it work?" needs no judgment.

What inference artifacts must expose

Probabilistic-modeling papers fail reproduction in venue-specific ways. Package accordingly:

Contribution type	Must be in the artifact	Spot-check the reviewer will try
MCMC / SMC sampler	Seeds, chain configs, convergence diagnostics code	Re-run short chains; compare R-hat and ESS to reported
Variational method	ELBO logging, initialization scheme, optimizer settings	Reproduce the ELBO trace shape, not just the endpoint
Causal discovery	Graph-generation scripts, SHD/SID evaluation code	Regenerate synthetic graphs; verify metrics on one seed
Calibration / conformal	Score functions, split definitions, coverage computation	Recompute empirical coverage at one α
Probabilistic programming / PGM inference	Model source, query set, exact-baseline harness	Run the exact baseline on the smallest instance
Theory with simulations	Every constant used to instantiate the bound	Check simulation matches theorem conditions

Anonymity inside the archive

The double-blind requirement covers all supplementary material explicitly. Archive leaks are quieter than PDF leaks, so build from a clean export:

# Build an anonymous artifact from a clean tree, never from the working repo
git archive --format=tar HEAD | tar -x -C /tmp/uai-artifact
cd /tmp/uai-artifact
grep -rniE 'university|\.edu|author|thanks|grant' --include='*.py' --include='*.md' . | head
find . -name '*.ipynb' -exec grep -l '"authors"' {} \;      # notebook metadata
rm -rf .git .github; find . -name '.DS_Store' -delete
zip -r ../supplement.zip . && du -h ../supplement.zip        # must be ≤ 50 MB (2026 cap)

Watch for: license headers with names, pyproject.toml author fields, conda environment exports embedding usernames, wandb/MLflow run URLs tied to an account, dataset paths containing /home/<name>/, and model checkpoints whose training config JSON names a cluster.

README skeleton for the archive

The archive's README is the artifact's abstract. A shape that fits the five-minute reader:

# Supplementary code for submission #<OpenReview number>

## Claim → command map (small-scale modes; full-scale flags noted)
| Paper claim | Command | Runtime (laptop) |
|---|---|---|
| Fig. 2 (coverage vs n) | `make fig2-small`  | ~2 min |
| Table 1 (SHD, 10 seeds) | `make table1-small` | ~4 min |
| Thm. 3 simulation | `make thm3-check` | ~1 min |

## Environment
- `pip install -r requirements.txt` (pinned versions; Python 3.11)
- No GPU required for small-scale modes.

## Expected outputs
- `expected/` holds reference CSVs; each command prints PASS/FAIL against them.

## Full-scale reproduction
- Flags, hardware used, and total runtime per experiment in `FULL_RUNS.md`.

Everything above stays anonymous by construction — no names, no lab conventions in paths, no acknowledgement of infrastructure that identifies an institution.

Data that cannot ship

• Under the 50 MB cap, ship generators, not corpora: the script that produces the synthetic SCMs or the subsampled benchmark, plus checksums for the full data.
• For licensed or private datasets, include the loader, the preprocessing pipeline, the schema, and summary statistics, and state access terms in the README.
• Never bypass a data-use agreement to help reviewers; explain the constraint instead. A stated limitation is a clarity point, a violated license is misconduct under the AUAI code of conduct.

Licensing notes

• The review-time ZIP needs no license drama, but the post-acceptance release does: choose deliberately between permissive (MIT/BSD/Apache-2.0) and copyleft, and check employer policy before the camera-ready deadline forces a rushed default.
• Third-party code vendored into the archive keeps its own license; verify compatibility now, since reviewers occasionally notice a GPL file inside an "MIT" archive and read it as carelessness.
• Data licensing is separate from code licensing; a permissive code license does not launder a restricted dataset.

After acceptance

Convert the anonymous ZIP into a public artifact worth citing: a tagged repository, an archival DOI where your institution supports one, a license chosen deliberately, and the README rewritten from "reviewer instructions" to "user instructions." Link it from the camera-ready — post-acceptance is when the CFP's release encouragement costs you nothing and earns citations.

Output format

[Artifact scope] supplement ZIP / public release / both
[Five-minute path] <command a reviewer runs first, and its runtime>
[Claim coverage] <headline results reproducible / total>
[Anonymity scan] clean / leaks: <files>
[Size] <ZIP size vs current cap>
[Data strategy] shipped / generated / loader-plus-terms