Eacl reproducibility

EACL Reproducibility

Use this to audit an EACL paper against the Responsible NLP checklist, which ARR files at submission and which the action editor and reviewers read alongside the PDF. The checklist is not paperwork: misleading answers are desk-rejection grounds, and inconsistencies between the checklist and the paper are what careful EACL reviewers hunt for. Reopen the current checklist at aclrollingreview.org/responsibleNLPresearch before auditing.

Treat the checklist as a claims audit

Every "yes" in the checklist implies a location in the paper. Walk the paper claim by claim and confirm each is backed:

Claim type	Must disclose	Common EACL failure
Model results	Hyperparameters, tuning, model size	"Default settings" with no numbers
Compute	Hardware, run time, total budget	Silent on cost of large runs
LLM prompts	Verbatim prompts + decoding params	Paraphrased or omitted prompts
Data	Source, license, splits, preprocessing	Undocumented or "on request" data
Metrics	Variance over seeds / significance test	Single-run deltas reported as wins
Multilingual	Languages, resource levels, per-language results	Aggregate hides where it fails

Contamination stance (LLM era)

• State explicitly whether evaluation data could have leaked into training — for closed LLMs this is often unknowable, and the honest move is to say so and bound the risk rather than claim clean evaluation. The "Leak, Cheat, Repeat" exemplar in ../../resources/exemplars/library.md is the reference discipline.
• Where feasible, run an overlap or decontamination check and report it.

Variance and significance floor

Reporting rule of thumb:
  - >= 3-5 seeds for any headline comparison
  - report mean +/- CI or std, not a single run
  - a significance test when two systems are "close"
  - never claim a win on an unreplicated single-run delta

Multilingual coverage honesty

• If the paper claims a cross-lingual or multilingual result, the checklist audit must confirm the languages are named, the resource levels are stated, and per-language results exist somewhere — an aggregate average is not evidence for every language.
• For lower-resourced languages, confirm dataset provenance and annotation context are documented; this is a recurring EACL reviewer expectation.

Checklist-to-paper consistency sweep

[ ] Every checklist "yes" maps to a section/appendix number
[ ] Hyperparameters + search space stated
[ ] Compute budget stated for expensive runs
[ ] Prompts + decoding params verbatim (if LLMs used)
[ ] Data source, license, splits, preprocessing documented
[ ] Variance/significance reported for headline claims
[ ] Contamination risk addressed honestly
[ ] Per-language results present for multilingual claims
[ ] AI-assistance use disclosed truthfully

Output format

[Reproducibility risk] Low / Medium / High
[Checklist-paper mismatches] <specific "yes" answers not backed in text>
[Disclosure gaps] <hyperparameters/compute/prompts/data>
[Evidence floor] <seeds/variance/significance findings>
[Contamination] <stance + any check run>
[Fix order] <what to add before the cycle deadline>