Eacl artifact evaluation

EACL Artifact Evaluation

Use this to turn a paper's evidence into artifacts that survive review and become a public release. EACL runs through ACL Rolling Review, so the artifact lives two lives: an anonymized supplement attached at ARR submission, and a public release after commitment acceptance. Both are audited against the Responsible NLP checklist. Reopen the current checklist before packaging.

The two lives of an EACL artifact

Stage	Form	Must be	Owner
ARR submission	Anonymized .zip/.tgz supplement	Fully de-identified, self-contained	Authors
Commitment acceptance	Public repo + Anthology link	Licensed, versioned, reproducible	Authors

Do not conflate them: the review supplement must contain no author-identifying strings, while the public release must contain exactly the identifying and licensing information the supplement omitted.

What belongs in an EACL artifact

• Code to reproduce the headline tables, with a top-level entry point.
• Data: the dataset or a loader plus a documented path to it; if redistribution is restricted, document access precisely rather than implying release.
• Prompts and decoding settings verbatim for any LLM-based result — these are part of the method, not an afterthought.
• Model outputs retained so scores can be re-computed without re-running expensive models.
• Annotation materials: guidelines, interface, pay information, and inter-annotator agreement.

Anonymized-supplement checklist

[ ] No author names in paths, file headers, LICENSE, or notebook metadata
[ ] Git history stripped or repo re-initialized
[ ] No personal hosting URLs (Drive/Dropbox) that identify authors
[ ] Prompts + decoding params included verbatim
[ ] Model outputs included for re-scoring
[ ] A README that reproduces at least one reported table
[ ] Smoke-checked (see resources/code/README.md)

Run the shared smoke checker before upload:

python3 ../../../shared-resources/ml-conference-methods/code/check_repro_package.py /path/to/anonymous-supplement

Licensing and documentation for the public release

• Choose a license appropriate to code (e.g. permissive) and data (respecting upstream dataset terms); the paper text should state it.
• Document intended use and known limitations of any released dataset — required by the checklist and expected by the European community's data-governance norms.
• Version the release with a tag that matches the camera-ready, so the Anthology PDF and the repo cannot drift.

Multilingual and lower-resource specifics

• If the artifact covers lower-resourced languages, document provenance and speaker/annotator context carefully; thin documentation of a low-resource dataset is a common EACL reviewer concern.
• Keep language codes and scripts explicit (ISO codes, script variants) so the artifact is usable by others working on those languages.

Output format

[Artifact stage] Anonymized supplement / Public release
[Contents] <code/data/prompts/outputs/annotation coverage>
[Anonymization] <pass/fail with specific leaks>
[Reproduces] <which reported table the README regenerates>
[Licensing + docs] <license, dataset terms, intended-use note>
[Gaps] <what a reviewer could still not reproduce>