Vis artifact evaluation

VIS Artifact Evaluation

Use this for the reproducibility track. IEEE VIS does not use ACM-style artifact badges; its recognition is the Graphics Replicability Stamp Initiative (GRSI) — issued to TVCG papers as the TVCG Replicability Stamp — plus the conference's Open Practices program. Two things to internalize: the stamp is earned by an independent volunteer actually reproducing your results from a public archive, and the review artifact (anonymized, for your paper's reviewers) is not the same deliverable as the stamp artifact (de-anonymized, permanently archived).

What GRSI certifies (verify the current process)

Mechanism	What it certifies	What earns it
TVCG Replicability Stamp (GRSI)	An independent volunteer reproduced the paper's results from your code/data	A public repository + a single documented build/run path that regenerates the key figures/results
Open Practices disclosures	Transparent reporting of open code, data, preprints, preregistration	Filling the camera-ready Open Practices form honestly and posting the materials

Unlike a graded badge ladder, the stamp is binary: an evaluator either reproduces your results or does not. The failure mode is therefore always "it did not build/run on their machine," never "the idea was weak" — design for a stranger's clean environment.

What a GRSI volunteer opens first

Claim type	First thing reproduced	Common failure caught
A visualization technique/algorithm	The build + a script that regenerates a key figure	Undocumented deps; only-builds-on-authors'-GPU
A system/tool	The install and a demo on bundled sample data	Requires a private server, API key, or paid license
A perceptual/empirical study	The analysis scripts that turn raw responses into the paper's stats	Numbers in the PDF no script reproduces; raw data missing
A rendering result	The pipeline + reference images with a comparison	Non-deterministic output with no tolerance/seed documented

Assume the evaluator gives your package a bounded time budget on a clean machine. The first build and the first regenerated figure must succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile, exact toolchain
              versions); avoid "install these 40 things by hand" and undocumented GPU/driver needs
[README]      one-screen orientation: what it is, how to build, how to run the demo, how to
              regenerate each figure/result, expected runtime and outputs
[Mapping]     an explicit table: paper figure/result -> script -> expected output
[Data]        the actual dataset or stimuli (or documented access), not just a pointer
[Determinism] seeds, tolerances, and reference images for anything stochastic or GPU-dependent
[License]     an OSI-approved license so others can reuse the visualization code
[Archive]     deposit in a DOI-issuing repository (OSF, Zenodo, Software Heritage) for permanence

Anonymized review artifact vs. stamp artifact

• At submission (if double-blind): the supplemental code/data/video is anonymized for the paper's reviewers — no owner strings, lab names, institutional URLs, or identity-revealing demo links, and no live repository that discloses authors.
• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version a GRSI volunteer reproduces and the camera-ready cites.

Worked vignette: stamping a technique + system paper

A paper contributes a new graph-layout technique and an interactive system. To target the stamp: ship a Docker image with the layout code pre-built; a run_demo.sh that lays out a small bundled graph and writes the teaser figure in under a minute; a reproduce/ directory whose scripts regenerate each quantitative figure from logged benchmark data; a figure-to-script mapping in the README; the benchmark graphs themselves with provenance; and an MIT/BSD license. State honestly which figures are turnkey and which need the full (slow) benchmark run.

Calibration

• The GRSI review is independent of and after camera-ready; do not conflate them, and do not block the paper on the stamp result.
• The exact GRSI submission process, the Open Practices requirements, and whether any element is mandatory vary by cycle — confirm on the current Open Practices page and the GRSI site.

Output format

[Target recognition] TVCG Replicability Stamp / Open Practices disclosures
[Artifact role] anonymized review artifact / public stamp artifact
[Contents] <code/data/stimuli/determinism aids/license>
[Clean-machine test] does build + demo + one regenerated figure succeed? yes/no
[Figure mapping] <figure/result -> script -> expected output present? yes/no>
[Fixes before archiving] <ordered list>