Naacl artifact evaluation

NAACL Artifact Evaluation

NAACL has no separate artifact-badging track; artifacts are judged inside the ARR review itself, through the supplement upload and section B of the Responsible NLP checklist ("scientific artifacts"). That placement matters: your artifact documentation is not an optional extra but a set of sworn answers reviewers cross-examine against the PDF.

What the artifact must let a reviewer do

• Trace provenance. Where did every corpus come from, under what license, and does your use match the terms and the creators' intent?
• Inspect the instrument. Prompts, annotation guidelines, interface screenshots, and pay rates are artifacts too — a human-evaluation claim without its instrument is unverifiable.
• Rerun the cheap parts. Scoring scripts and metric code should execute from the archive alone; nobody will retrain your model, but everybody can re-score your outputs if you include them.
• Audit the data card. Language varieties, dialect coverage, speaker demographics where relevant, known gaps, and intended use.

Language-data documentation ladder

Data situation	Minimum documentation for a NAACL reviewer	Extra step
Standard public benchmark	Version, split, license, citation	Note any known contamination reports
Web-scraped text	Collection dates, filtering rules, deduplication, license basis	PII handling statement
New annotated corpus	Guidelines, annotator recruitment and pay, agreement scores	Release the guidelines verbatim in the supplement
Dialectal / code-switched data	Variety labels and how they were assigned	Native-speaker validation description
Indigenous or community-owned language data	Consent and partnership terms, community approval for release	Verify whether public release is permitted at all

The last row is a NAACL signature concern. Work on languages of the Americas increasingly follows community-controlled data norms: some corpora may be used but not redistributed, some require named attribution (which conflicts with anonymous review — use a placeholder and restore at camera-ready), and some communities set conditions on derived models. "We release everything" is not automatically the ethical high ground here; the checklist rewards accuracy about constraints, not maximal openness.

Anonymous packaging that survives inspection

artifact.zip
├── README.md          # one-screen orientation: what, how, how long
├── data/
│   ├── data_card.md   # provenance, license, varieties, gaps
│   └── samples/       # enough rows to judge quality, not the corpus
├── prompts/           # exact strings, all variants tried
├── eval/
│   ├── score.py       # runs on outputs/ with no network access
│   └── outputs/       # raw model outputs backing the main tables
└── annotation/
    └── guidelines.pdf # the instrument, scrubbed of institution marks

Scrub before zipping: repository history, notebook execution metadata, absolute paths with usernames, license headers naming the lab, and any consent form carrying institutional letterhead (replace with a redacted copy; note that the original exists).

Vignette: a Quechua-Spanish parallel corpus package

A submission introduces a 40k-pair Quechua-Spanish parallel corpus built with two community organizations, plus MT baselines. The packaging calls that follow from this skill:

• The corpus itself does not ship in the review archive — the partnership terms permit research use but defer public release to a community decision. The data card states this, and section B of the checklist answers "no, with reason" for artifact release.
• What does ship: 200 sample pairs cleared for review purposes, the collection protocol, annotator recruitment and payment description, the cleaning scripts, and the full MT evaluation pipeline with outputs.
• The consent-form template ships with the letterhead redacted and a note that the original is held by the partner organizations.
• The README's first paragraph tells the reviewer exactly which claims the archive can and cannot let them verify — pre-empting the "authors refuse to release data" misreading with a governance explanation instead.

The result is an artifact that scores as honest and inspectable rather than incomplete, which is the realistic best outcome for community-governed data.

Cycle-volatile mechanics

Archive size caps, accepted formats, and whether supplements upload as one file or several are OpenReview-form details that shift between ARR cycles; read the live submission form before building the final zip, and never reverse-engineer the limits from a previous cycle's folklore.

Post-acceptance conversion

At camera-ready, the anonymous bundle becomes the public record: move it to a persistent host with a DOI or a tagged release, apply the real license, restore attribution the community partnership requires, and update the checklist-facing statements in the paper if the release scope changed between review and publication.

Output format

[Artifact inventory] <data / code / prompts / guidelines / outputs>
[Checklist B alignment] <each B answer -> where the artifact proves it>
[Provenance gaps] <unlicensed, undocumented, or unclear-consent items>
[Community constraints] <redistribution / attribution / approval terms>
[Anonymity sweep] clean / issues found
[Release plan] <anonymous now -> public form at camera-ready>